The Agent That Refuses to Act

The project is a scoped triage assistant for the AWS services I actually operate, grounded in curated runbook documentation covering Lambda ↔ Redshift failure modes. You paste in an alert description, and it returns three things: the first checks to run, the likely causes ranked, and the criteria for escalating instead of continuing to debug. That is the entire product. It is aimed at a very specific user — an engineer who has just been paged at 3am and needs help sequencing an investigation, not company.

The defining constraint is deliberate: the assistant advises and never executes. The reasoning is not caution for its own sake. The assistant cannot see live system state — no metrics, no logs, no current configuration — so anything it recommends is built from an alert description and a runbook. Acting on its output would mean acting on stale information, and stale information plus write access is how incidents get worse.

The same honesty is forced into its answers. Every hypothesis it offers comes with a stated confidence level and a falsification condition — the one check that would prove it wrong. An on-call engineer at 3am does not need a confident narrator; they need to know how much weight an idea can bear and how to knock it over quickly.

The first surprise came from tracing bad answers backwards. When the assistant gave a weak response, the cause was usually weak source material, not a weak model. A vague runbook section produced a vague answer. Escalation criteria that existed only in someone's head produced an assistant that could not tell you when to escalate — because nobody had ever written it down.

Grounding an assistant in institutional knowledge turns out to be an involuntary audit of that knowledge. Every gap the model exposed was a gap a half-asleep human would have hit too; the model just hit it reproducibly, in daylight, where I could fix it. The documentation improved more in the course of building this assistant than in any deliberate documentation push I have been part of.

The second surprise was sneakier. During testing I found that a platform-level "Initial Message" setting was silently overriding one of the constraints I had written into the prompt. My instruction was correct at the layer I controlled — and defeated at a layer I had not thought to check. Nothing in the prompt was wrong; the system as deployed simply did not do what the artifact I wrote said it should.

That is the through-line this shares with my last post, and the closest thing I have to a rule for building with AI right now: decide what the system must never do, then enforce it somewhere the model — and the layers wrapped around the model — cannot reach. Test the system you shipped, not the artifact you wrote. The prompt is one layer of a stack, and the stack is what your user meets.