I’ve shipped software for 35 years, and I have never trusted a dashboard I didn’t build myself. I’ve been on both sides of one — writing the code that reports its own health, and sitting in the meeting where somebody reads that report out loud. Every organization I’ve worked in has had a status meeting where every indicator was green and the thing under discussion had not run in weeks. The tell is always the same. Somebody asks a second question, and the room goes quiet.
So I stopped planning for the loud failure. The software you depend on will not fail loudly — it will report that it is fine.
The setting that read Enabled for months
Everything below is my own work. I’m not describing a client’s mess.
A cloud security feature in my own environment read “Enabled” for months. It was not enabled in any sense that mattered. The plan behind it was the free tier, and the part that actually watches things had never been turned on — zero dollars a month, none of the protection I thought I was paying for, and a status page that said Enabled the whole time. The dashboard wasn’t wrong, exactly. It was answering a different question than the one I thought I was asking.
Nobody was lying. Nobody was checking.
That incident produced the one rule I’ve since written into an internal charter word for word: verify function, not flags. A setting that reads Enabled is a claim. Prove the control works — fresh audit data with a recent timestamp, a probe that comes back refused, a score that exists — or report it as unverified and say why. Unverified is an honest answer. Green is not.
The deploy test that looks for a failure
My deployment pipeline finishes by calling the API with no credentials and asserting that it gets back a 401, the server refusing to answer. Not a 200. A rejection.
That looks backwards until you name the failure you’re actually afraid of. A deploy that reports success while leaving the application down is the one that hurts, because the pipeline is the thing you were trusting. A refusal proves the host started and that something is checking credentials before it answers anything. A green checkmark on the deploy job proves a file copy finished.
Most pipelines end with the words “deployment succeeded.” That is a claim about the deployer, not about the application.
What the 200 was actually telling me
Last week I shipped a small add-in — a button inside Outlook that files an email as a task. Before deploying it, I checked the two web addresses the thing depends on. Both came back 200, the code that means “here it is.” Neither file existed.
The host was configured to answer any unknown address with the main page, so every URL I could type returned a byte-identical copy of a completely different application. A file that did not exist looked exactly like a file that did.
The same shape of bug was waiting one layer down. The application keeps an offline cache, and that cache had a fallback rule of its own pointing at the main app’s entry page — so once installed, it answered a request for the add-in by rendering the wrong application inside the Outlook pane.
Check the body, not the status code. If you’re verifying that something shipped, the test is whether the response contains something only the new build has — a string, a version number, a header. A fallback page can satisfy anything weaker than that.
The check I nearly hid from myself
My repository requires a build-and-test check to pass before anything merges. The efficient-looking move is to run that check only when files under the application folder change. Don’t.
A required check that never runs does not fail. It reads as permanently pending, and pending waits forever. The first pull request touching only documentation would have sat there with a greyed-out merge button and nothing red anywhere on the screen to explain it.
So the trigger carries no path filter at all — no rule about which files have to change for the check to fire. It builds and tests on every pull request, including the trivial ones. Two wasted minutes per pull request beats an afternoon of wondering why a merge won’t go through when nothing has failed.
That’s an instance of a rule I now apply to anything that runs on a schedule: “nothing happened” must be distinguishable from “nothing ran.” A skipped run is a result and it deserves a row of its own with a reason attached. Silence is the only result I don’t accept, because silence is also what a dead system produces.
An unset value looks like a bug in the feature
After one production deploy, two configuration values were missing. The result was two navigation links that worked perfectly and led to server errors.
Nothing failed fast. A missing setting doesn’t announce itself as a misconfiguration — it presents as a bug in the feature, which sends you hunting through code that was never wrong. What I’d do about it is unglamorous: make the application validate its own configuration at startup and refuse to start when something required is absent. A service that won’t boot is a five-minute problem. A service that boots and serves an error on one screen is a two-day problem. I sent that invoice to myself.
The honest part: this is open in my own project
If I stopped here it would be a piece about lessons learned, and I’ve been doing this too long to write one of those.
The project where I formalized “verify function, not flags” has a telemetry connection string configured in production — the address monitoring data is supposed to be sent to — and no monitoring service behind it. It looks instrumented. It is not instrumented. That was the state at the start of this month, and I have not fixed it.
The reason I know is that I wrote the runbook, and the runbook says so in plain language: there is no monitoring service, so a human watching is the monitoring, and therefore deploys happen at the start of a work block rather than the end of one.
That is the same failure, recurring inside the project created to prevent it. The connection string is still there. The gap is still open. What changed is that the document stopped claiming to be monitored and named the actual control, which is a person with an hour of attention left. That’s a weak control. It is a real one, and it is written down where the next person will read it before they push at 4:45 on a Friday.
What a review actually tests
You have a system running right now that somebody built fast, and it works. You know which one. I’d bet it has a status page nobody opens, a backup job nobody has restored from, and a monitoring integration somebody configured in a hurry and never revisited.
None of that is negligence, and it isn’t a vendor problem either. It’s the ordinary consequence of nobody having a reason to look. A control that gets checked when somebody thinks of it is not a control. The checking has to be somebody’s standing job with a cadence attached, and the output has to be allowed to say unverified without anybody treating it as an accusation.
The AI tools made it easy to build something that runs. They did nothing to make it easier to tell whether it is running. That gap is wider now than at any point in my career, and it is entirely a review problem — you read the configuration against the behavior instead of asking the system how it feels.
Which green lights are load-bearing, and which are decoration.
I have never worked on a system where every green light was earned. Not one, including the ones I built.
The question isn’t whether your dashboard says the system is healthy.
It’s who last proved it.
Datos reviews systems that got built fast — architecture, data model, security, hosting, and every control that reports working without being asked. Read about the fixed-fee production assessment, or start a conversation if something you built has quietly become load-bearing.