0
Postmortem prompt for analyzing a production agent failure
⁂auto-checked, 3 hours oldAauraNovice
The prompt
Help me write a postmortem for an agent failure in production: failure_description. Unlike a typical software incident, agent failures often involve reasoning errors, not just code bugs, so structure this specifically for that:
1. Reconstruct what the agent actually 'believed' at each decision point leading to the failure, based on the transcript/logs, not what the system prompt intended it to believe - these can diverge and that gap is often the real root cause.
2. Classify the failure type: was it a tool-use error (wrong tool, wrong args), a reasoning error (correct tools, faulty logic connecting them), a grounding error (hallucinated a fact not in its context), or a scope error (did something outside its intended authority)? Pick the primary one, don't hedge across all four.
3. Identify whether this was a one-off (bad luck, a genuinely ambiguous edge case) or systemic (the prompt/design would produce this again given a similar input) - if systemic, what's the smallest input class that reliably triggers it.
4. Propose a fix at the right layer: prompt change, tool design change, or added guardrail/verification step, and explain why a fix at a different layer (e.g. 'just tell it to be more careful') wouldn't actually prevent recurrence.
5. Propose a regression test: a specific input that should be added to an eval set to catch this class of failure before future deploys.
Failure transcript/logs: failure_transcript
System prompt at time of failure: system_prompt
Where it has been run
| Model | Version | People | Broke | Auto | Last |
|---|---|---|---|---|---|
| Claude | Haiku 4.5 | 0 | 0 | 1/1 | 3 hours ago |
Example output
Reconstructed belief: at the decision point, the agent's tool call sequence shows it treated the customer's stated shipping address as verified, when the system prompt actually requires cross-checking it against the account record first, a step the agent skipped. It believed verification wasn't needed because the customer stated the address confidently.
Failure classification: primarily a scope error, the agent took an action (updating shipping address without verification) that its own instructions place outside its unverified authority, this wasn't a hallucination or wrong tool choice, the verify_address tool exists and works, it just wasn't called.
Systemic, not one-off: reviewing 4 other transcripts, this skip happens whenever the customer states the new address in the same message as the original request, rather than in response to a direct question, suggesting the agent treats confidently-stated info as pre-verified regardless of source.
Fix at the prompt layer is insufficient alone ('always verify' is already stated); the real fix is a tool-design change: make `update_shipping_address` require a `verification_token` parameter that can only come from `verify_address`'s output, so the skip becomes structurally impossible rather than relying on the model remembering to call it.
Regression test: add an eval case where a user states a new address inline in their initial message, assert the transcript includes a verify_address call before any update_shipping_address call.
0 comments
Sign in to comment or report what this prompt did for you.
Sign inNo comments yet. Run the prompt and report what you got.