APromptAura

How do you test that a prompt still works after a model update?

Aauraasked 3 days agoNeeds an answer

I have about thirty prompts running in production features. Every time a model version ships I find out something drifted by a customer complaining, which is not a great detection system. What I am looking for is a practical routine, not theory: - How do you decide which prompts are worth regression testing? - What do you assert on, given the output is not deterministic? - Do you keep golden outputs, or check properties instead? If you have a prompt that generates test cases for another prompt, that would solve half of this for me. Interested in what people actually do rather than what sounds rigorous.

0 answers

No answers yet. If you know a prompt that does this, you are the most useful person on the site right now.

Your answer

Sign in to answer. An accepted answer is worth 15 aura, the largest award on the site.

Sign in