0
Add a self-verification step so an agent catches its own errors before answering
⁂auto-checked, 2 hours oldAauraNovice
The prompt
Design a self-verification step for an agent doing agent_task, inserted between generating a draft answer and returning it to the user. This is not the same as a generic 'double check your work', it needs a concrete verification procedure specific to this task's likely error types. Requirements:
1. Identify the 2-3 most likely error types for this specific task (e.g. for a data-lookup agent: citing a value not actually present in the retrieved data, for a math-heavy agent: an arithmetic slip, for a code-generation agent: a syntax or import the target language doesn't actually support).
2. For each error type, define a concrete verification check the agent performs on its own draft before finalizing: re-deriving a number independently and comparing, checking every claim against its cited source, or running/parsing generated code if a tool is available.
3. Define what happens when verification finds a discrepancy: the agent must revise and re-verify once, not loop indefinitely, and if the second attempt also fails verification, it should report the answer as unverified/uncertain rather than presenting it with full confidence.
4. Make clear the verification step must actually be a distinct, separate check, not just the model re-reading its own answer and asserting it's correct, since that doesn't catch systematic errors the model is prone to in the first place.
5. State the added latency/cost tradeoff honestly and when it's NOT worth adding this step (low-stakes, easily-corrected-by-user answers).
Agent task: agent_task
Most common error types observed so far: observed_errors
Where it has been run
| Model | Version | People | Broke | Auto | Last |
|---|---|---|---|---|---|
| Claude | Haiku 4.5 | 0 | 0 | 1/1 | 2 hours ago |
Example output
Task: agent answers questions using numbers pulled from a financial dataset.
Most likely error types: (1) citing a number that isn't actually in the retrieved rows (fabrication), (2) an arithmetic error when the agent computes a derived value like a percentage change, (3) using the wrong time period's row when multiple similar rows exist.
Verification checks: for (1), after drafting, re-scan the draft answer and confirm every raw number appears verbatim in the retrieved dataset rows, not just plausible-looking. For (2), independently recompute any derived value (percentage, ratio, sum) using a second, separate calculation pass rather than trusting the first one, and compare. For (3), confirm the row used matches the exact time period stated in the question, not an adjacent period.
Discrepancy handling: if recomputation disagrees with the draft, revise once using the recomputed value and re-run both checks; if it still disagrees on the second pass, state 'I found an inconsistency in my calculation and want to flag this answer as unverified rather than risk giving you a wrong number' instead of presenting a shaky number with confidence.
Tradeoff: this roughly doubles the reasoning cost per query. Not worth it for casual exploratory questions where the user will eyeball the number anyway, but worth it for anything feeding a report or decision, which is this deployment's primary use case, so enabled by default here.
0 comments
Sign in to comment or report what this prompt did for you.
Sign inNo comments yet. Run the prompt and report what you got.