APromptAura

#evaluation

Top prompts tagged evaluation.

0
p/metaAauraNovice4 days agohas example⁂auto-checked

Systematically A/B two prompt variants instead of eyeballing outputs

I have two versions of a prompt for task_description and I want a rigorous comparison, not a vibes-based 'this one sounds better'. Given both prompt variants and n representative test inputs: 1. Before generating any outputs, state 2-4 concrete criteria you'll judge by, specific to this task (not generic 'quality'), and for each, how you'll detect it in an output (a presence/absence check, a count, a structural property). 2. Run both prompts against each test input (or reason about what each would produce if you can't execute code) and score each output against every criterion, showing your scoring, not just a final verdict. 3. Report per-criterion win/loss/tie counts across all inputs, not just an overall 'variant A is better' impression, since a prompt can win on one axis and lose on another. 4. Flag any input where the two variants produced meaningfully different outputs for a reason unrelated to your criteria (e.g. one variant happened to get a lucky/unlucky example), so I don't over-attribute that difference to the prompt wording itself. 5. Give a final recommendation with the specific tradeoff named, not just a winner declared. Variant A: prompt_a Variant B: prompt_b Test inputs: test_inputs
ClaudeGPT#prompt-testing#evaluation#ab-testing
0
0
p/agentsAauraNovice4 days agohas example⁂auto-checked

Design an evaluation rubric that catches an agent's real failure modes

I'm building an agent that does agent_task_description. Design an evaluation rubric to score its outputs, but ground it in the actual failure modes this kind of agent tends to have, not generic 'helpfulness/harmlessness' scores. Structure: 1. List 5-8 specific failure modes this agent is likely to exhibit given its task (e.g. for a data-extraction agent: hallucinating a field that isn't in the source, silently skipping ambiguous cases instead of flagging them, inconsistent formatting across similar inputs). 2. For each failure mode, write a rubric criterion that's binary or low-cardinality (pass/fail or 1-3 scale), not a vague 1-10 'quality' score that different graders would interpret differently. 3. For each criterion, give one example output that would fail it and one that would pass, concretely, using realistic (not toy) input. 4. Note which criteria could be graded automatically (regex, schema validation, exact match) versus which genuinely require an LLM-judge or human, and don't claim a criterion is automatable if it actually requires judgment. 5. Weight the criteria: which failures are disqualifying (any instance fails the whole output) versus which are minor deductions. Agent task: agent_task_description Example inputs/outputs if you have them: examples
ClaudeGPT#evaluation#rubric#agent-testing
0