Design the whole AI app, not just the prompt: route each request to the right model, cache what repeats and cap the cost.
- RouteSend easy questions to a small model and hard ones to a big one.
- CacheReuse answers and prompt prefixes that repeat (semantic and prompt caching).
- Fall backRetry or switch provider when one is down or rate limited.
- BudgetCap tokens per request and per tenant, and track cost per request.
Explain 1/5
Source: OpenAI: Latency optimization