0
Design an evaluation rubric that catches an agent's real failure modes
⁂auto-checked, 3 days oldAauraNovice
The prompt
I'm building an agent that does agent_task_description. Design an evaluation rubric to score its outputs, but ground it in the actual failure modes this kind of agent tends to have, not generic 'helpfulness/harmlessness' scores. Structure:
1. List 5-8 specific failure modes this agent is likely to exhibit given its task (e.g. for a data-extraction agent: hallucinating a field that isn't in the source, silently skipping ambiguous cases instead of flagging them, inconsistent formatting across similar inputs).
2. For each failure mode, write a rubric criterion that's binary or low-cardinality (pass/fail or 1-3 scale), not a vague 1-10 'quality' score that different graders would interpret differently.
3. For each criterion, give one example output that would fail it and one that would pass, concretely, using realistic (not toy) input.
4. Note which criteria could be graded automatically (regex, schema validation, exact match) versus which genuinely require an LLM-judge or human, and don't claim a criterion is automatable if it actually requires judgment.
5. Weight the criteria: which failures are disqualifying (any instance fails the whole output) versus which are minor deductions.
Agent task: agent_task_description
Example inputs/outputs if you have them: examples
Where it has been run
| Model | Version | People | Broke | Auto | Last |
|---|---|---|---|---|---|
| Claude | Haiku 4.5 | 0 | 0 | 1/1 | 3 days ago |
Example output
Failure mode 1: hallucinated field. Criterion (disqualifying, binary): every value in the output JSON must trace to a literal substring in the source document; automatable via substring match for exact-copy fields, but requires LLM-judge for computed fields like 'total' since those are derived, not literal.
Fail example: output contains `"vendor": "Acme Corp"` when the invoice actually says 'ACME Corporation Inc.' and the agent normalized/guessed rather than extracting verbatim from the allowed field.
Pass example: `"vendor": "ACME Corporation Inc."` matching the source exactly.
Failure mode 2: silent skip on ambiguity. Criterion (disqualifying): if a field is genuinely ambiguous in the source (e.g. two possible totals), the agent must flag it in an `_uncertain_fields` array rather than picking one silently. Requires LLM-judge to assess whether the source was actually ambiguous. Weighting: failure modes 1 and 2 are disqualifying since they produce confidently wrong data; formatting inconsistency (failure mode 5) is a minor deduction only.
0 comments
Sign in to comment or report what this prompt did for you.
Sign inNo comments yet. Run the prompt and report what you got.