0
Compare how different models handle the same prompt and why they diverge
💀failed our automated checkAauraNovice
The prompt
I want to understand behavioral differences between models on the same prompt, not just which output I personally prefer. Given this prompt: prompt and outputs from models_list:
1. For each pair of models that diverge meaningfully, identify the specific instruction or ambiguity in the prompt that each model resolved differently, don't just describe the outputs as 'different in style'.
2. Distinguish divergence that's a genuine capability difference (one model got a factual/logical detail wrong the other got right) from divergence that's a preference/interpretation difference (both are defensible readings of an underspecified instruction).
3. For preference/interpretation divergences, propose the specific prompt change that would make the instruction unambiguous enough that both models would converge, and state your confidence that the fix would actually work versus just being a reasonable guess.
4. Note if any model appears to have ignored an explicit instruction entirely (not just interpreted it loosely) since that's a different, more concerning category than reasonable interpretive divergence.
5. Give a practical recommendation: if I need consistent behavior across these models, what's the minimum prompt change needed, versus what divergence I should just accept as inherent.
Prompt: prompt
Outputs by model: outputs_by_model
Where it has been run
| Model | Version | People | Broke | Auto | Last |
|---|---|---|---|---|---|
| Claude | Haiku 4.5 | 0 | 0 | 0/1 | 3 hours ago |
Example output
Model A and Model B diverge on how they handled 'list the top factors' with no number specified: Model A gave 3, Model B gave 7. This is a preference divergence (underspecified count), not a capability gap, both readings are defensible.
Fix: specify 'exactly 4 factors' explicitly; confidence this resolves it is high since both models clearly can count and follow explicit numeric constraints, the divergence stems purely from the prompt's ambiguity, not a model limitation.
Capability-difference divergence: on the calculation embedded in the task, Model C's arithmetic in step 3 is simply wrong (states 15% of 340 as 61 instead of 51), that's a factual error, not an interpretation choice, and worth flagging separately from the stylistic divergences above.
Ignored-instruction flag: Model B's output doesn't include the requested citation format at all, despite the prompt explicitly requiring it in a labeled 'Output format' section; the other two models complied. This looks like an instruction being dropped rather than reasonably reinterpreted, worth testing again since it may be inconsistent rather than a stable model behavior.
Recommendation: add the explicit count constraint (fixes the interpretation divergence), the arithmetic issue isn't prompt-fixable, it's a per-model reliability question, consider adding a verification step for numeric outputs regardless of which model you use.
0 comments
Sign in to comment or report what this prompt did for you.
Sign inNo comments yet. Run the prompt and report what you got.