0
Write an eval rubric for a prompt before you scale it up
⁂auto-checked, 2 hours oldAauraNovice
The prompt
I'm about to scale up usage of this prompt: prompt for use_case. Before I do, help me write an evaluation rubric so I can measure quality systematically instead of spot-checking. Requirements:
1. Derive 4-6 scoring criteria directly from what the prompt itself demands (if the prompt requires a specific format, that's a criterion; if it requires groundedness, that's a criterion) rather than generic LLM-output quality metrics that don't reflect this specific prompt's actual requirements.
2. Make each criterion as objectively checkable as possible: state whether it's programmatically checkable (regex, schema, exact match), or genuinely requires an LLM-judge or human, and don't overstate how automatable a subjective criterion is.
3. For each criterion, write the LLM-judge instruction that would grade it, calibrated with one example of a passing and one failing output, so the judge prompt itself is reproducible and not vague.
4. Define a pass/fail threshold for the overall rubric (e.g. must pass all disqualifying criteria plus 80% of others) rather than leaving 'good enough' undefined.
5. Suggest how many sample outputs I'd need to eval before I can trust an aggregate pass rate, given the stakes of use_case.
Prompt: prompt
Use case and stakes: use_case
Where it has been run
| Model | Version | People | Broke | Auto | Last |
|---|---|---|---|---|---|
| Claude | Haiku 4.5 | 0 | 0 | 1/1 | 2 hours ago |
Example output
Criterion 1 (disqualifying, programmatically checkable): output must be valid JSON matching the given schema, checked via schema validation, not an LLM judge.
Criterion 2 (disqualifying, requires LLM-judge): every claim in the 'summary' field must be traceable to the source text provided, no added information. Judge instruction: 'Given the source text and the summary field, flag PASS only if every factual statement in the summary appears in or is directly inferable from the source; flag FAIL if any detail is added, exaggerated, or not present.' Passing example: summary states a fact present verbatim in source. Failing example: summary adds a number not present anywhere in source text.
Criterion 3 (minor, requires LLM-judge): tone matches the calm, neutral register requested, not alarmist language even when describing a problem.
Threshold: must pass both disqualifying criteria on every sample (any failure blocks that output from use), and score at least 80% pass rate on the tone criterion across the batch, since that one is a softer preference not a correctness issue.
Sample size: given this feeds customer-facing content (moderate stakes, not safety-critical), evaluate at least 30 diverse samples before trusting an aggregate pass rate, fewer than that risks missing a failure mode that only shows up on less common input shapes.
0 comments
Sign in to comment or report what this prompt did for you.
Sign inNo comments yet. Run the prompt and report what you got.