0
Systematically A/B two prompt variants instead of eyeballing outputs
⁂auto-checked, 23 hours oldAauraNovice
The prompt
I have two versions of a prompt for task_description and I want a rigorous comparison, not a vibes-based 'this one sounds better'. Given both prompt variants and n representative test inputs:
1. Before generating any outputs, state 2-4 concrete criteria you'll judge by, specific to this task (not generic 'quality'), and for each, how you'll detect it in an output (a presence/absence check, a count, a structural property).
2. Run both prompts against each test input (or reason about what each would produce if you can't execute code) and score each output against every criterion, showing your scoring, not just a final verdict.
3. Report per-criterion win/loss/tie counts across all inputs, not just an overall 'variant A is better' impression, since a prompt can win on one axis and lose on another.
4. Flag any input where the two variants produced meaningfully different outputs for a reason unrelated to your criteria (e.g. one variant happened to get a lucky/unlucky example), so I don't over-attribute that difference to the prompt wording itself.
5. Give a final recommendation with the specific tradeoff named, not just a winner declared.
Variant A: prompt_a
Variant B: prompt_b
Test inputs: test_inputs
Where it has been run
| Model | Version | People | Broke | Auto | Last |
|---|---|---|---|---|---|
| Claude | Haiku 4.5 | 0 | 0 | 1/1 | 23 hours ago |
Example output
Criteria: (1) does the output include a specific numeric estimate, not just qualitative language, detected by presence of a number; (2) is the response under 100 words, detected by word count; (3) does it acknowledge uncertainty when the input is genuinely ambiguous, detected by hedging language presence when input has 2+ interpretations.
Across 6 test inputs: Variant A included a numeric estimate in 6/6, Variant B in 2/6 (Variant B's phrasing 'give your best sense of' seems to invite qualitative answers). Word count: A averaged 140 words (over budget in 4/6), B averaged 85 (within budget in 6/6). Uncertainty acknowledgment: tied, both hedged appropriately on the 2 genuinely ambiguous inputs.
Flagged: on input 4, A produced a notably worse response, but this looks like it hit an unusual edge case (a negative number in the input) rather than a prompt-wording effect, both variants likely would have struggled here.
Recommendation: Variant B for length compliance, but graft Variant A's explicit 'give a specific number, not a range or qualitative estimate' clause onto B, since that's the one dimension B loses on and it's a targeted, isolated fix.
0 comments
Sign in to comment or report what this prompt did for you.
Sign inNo comments yet. Run the prompt and report what you got.