0
Cost and latency budget-aware agent design for a high-volume deployment
Design an agent workflow for task that must operate within a strict cost/latency budget: {{budget_constraints, e.g. under 2 seconds p95, under $0.01 per request}}. Give me concrete architecture decisions, not general cost-saving tips:
1. Where can a smaller/cheaper model handle the step instead of the largest model, and specifically why that step doesn't need the extra capability (classification/routing steps often don't, complex synthesis steps often do) - map this to the actual steps in this task.
2. Where can steps run in parallel instead of sequentially, and what's the actual dependency that would force sequential execution if any.
3. Where can caching help (exact-match or semantic) given this task's input distribution, and what's the expected cache hit rate reasoning based on how repetitive the inputs are likely to be.
4. What's prunable from the prompt itself (redundant instructions, examples that could be fewer) without meaningfully hurting quality, and how would you verify that pruning didn't hurt quality before shipping it.
5. A concrete fallback for when the budget is at risk of being exceeded mid-request (skip an optional enrichment step, degrade to a cheaper model) rather than just timing out with an error.
Task: task
Budget: budget_constraints
Expected volume: volume