0
Design an evaluation rubric that catches an agent's real failure modes
I'm building an agent that does agent_task_description. Design an evaluation rubric to score its outputs, but ground it in the actual failure modes this kind of agent tends to have, not generic 'helpfulness/harmlessness' scores. Structure:
1. List 5-8 specific failure modes this agent is likely to exhibit given its task (e.g. for a data-extraction agent: hallucinating a field that isn't in the source, silently skipping ambiguous cases instead of flagging them, inconsistent formatting across similar inputs).
2. For each failure mode, write a rubric criterion that's binary or low-cardinality (pass/fail or 1-3 scale), not a vague 1-10 'quality' score that different graders would interpret differently.
3. For each criterion, give one example output that would fail it and one that would pass, concretely, using realistic (not toy) input.
4. Note which criteria could be graded automatically (regex, schema validation, exact match) versus which genuinely require an LLM-judge or human, and don't claim a criterion is automatable if it actually requires judgment.
5. Weight the criteria: which failures are disqualifying (any instance fails the whole output) versus which are minor deductions.
Agent task: agent_task_description
Example inputs/outputs if you have them: examples