Six Levers to Reduce LLM Inference Costs
Enterprise AI teams are increasingly facing the same issue: an AI use case works in a pilot, but its inference cost becomes difficult to justify once usage scales.
The solution is not simply to replace every premium model with a cheaper one – It is to design each request around the required quality, context, latency and unit economics. The six levers below are a practical starting point.
1. Route by complexity
Not every request needs a frontier model. Classification, extraction, tagging, routing and simple customer-service tasks can often run on smaller, less expensive models. Premium models should be reserved for complex reasoning, sensitive decisions or tasks where evaluation shows a clear quality advantage.
Use smaller, lower-cost models for routine work. Escalate only high-complexity, low-confidence or high-risk requests to premium models. This is commonly called model routing or model cascading.
Open-source tool: Brick routes simpler queries to smaller models while aiming to preserve quality on more demanding requests.
The key metric is not model cost alone, but cost per successful outcome. A routing mistake that creates retries, poor answers or manual rework is not a saving.
2. Compress prompts
Long system instructions, redundant examples and oversized RAG context increase input-token cost on every request. Prompt compression reduces the amount of information sent to the model while attempting to retain the instructions and evidence needed for a good answer.
Open-source tool: LLMLingua is a prompt-compression project from Microsoft Research. Its documentation describes compression of prompts and KV cache, with the goal of faster inference and lower token use.
Compression requires evaluation. If critical instructions, product facts or safety constraints are removed, lower token spend can quickly become a higher business cost.
3. Compress agent context
Long-running agents create a distinct problem: they repeatedly send conversation history, tool outputs, logs, file contents and intermediate results. Much of that context is duplicated or no longer relevant, but it is still paid for and consumes the model’s context window.
Open-source tool: Foveance is designed to structurally compress OpenAI-style message histories and agentic workflows, including tool-use conversations, through a library, proxy or wrapper approach.
This is different from generic summarisation. The objective is to remove redundancy while preserving the structure an agent needs to continue working reliably. For tool-heavy workflows, context compression can often matter as much as model selection.
4. Cache repeated work
Caching prevents the same—or semantically very similar—request from triggering a new model call. It is especially useful for recurring support questions, internal knowledge queries, standard analysis requests and stable shared context.
Open-source tool: GPTCache provides a semantic cache for LLM applications, storing and retrieving responses based on query similarity to reduce requests and tokens sent to the provider.
Caching and compression solve different problems. Caching reuses a previous result; compression reduces the tokens that reach the model when a new invocation is still necessary.
5. Enforce token budgets
Most LLM cost spikes do not come from one expensive request. They come from missing controls: verbose prompts, unlimited output lengths, oversized retrieval, recursive agent loops and repeated tool calls.
Set explicit budgets by use case: maximum input tokens, maximum output tokens, maximum tool calls, maximum retries and maximum cost per session. Make the system stop, downgrade or require approval when it exceeds the budget.
Open-source tool: Floe Guard is an open-source billing guardrail for agents, designed to stop runaway loops before they create unexpected spend.
A useful governance metric is cost per completed task, broken down by model, workflow, customer segment and application version.
6. Batch non-urgent workloads
Many AI tasks do not need a real-time response: document extraction, catalogue enrichment, classification, offline evaluation, report generation and back-office workflows can be queued and processed in batches.
Batching improves utilisation and can reduce infrastructure cost, particularly for self-hosted models. It is not suitable for every use case, but it should be the default question for any workflow without a real-time user requirement.
Open-source tool: vLLM is an open-source inference and serving engine that uses techniques such as continuous batching to improve throughput and GPU utilisation for self-hosted models.
For provider-hosted models, the equivalent is usually an asynchronous or discounted batch tier. The trade-off is simple: lower cost in exchange for less predictable turnaround time.
Build the stack
These levers work best together:
- Route the request to the right model.
- Send only the context that matters.
- Compress repeated agent context.
- Reuse answers through caching.
- Cap token, tool and retry budgets.
- Batch anything that does not require immediate output.
Savings will vary by workload. A customer-support assistant with repeated queries will benefit most from routing and caching; a coding agent may benefit more from context compression and strict tool-loop budgets; a high-volume enrichment workflow may gain most from batching and self-hosted serving.