AI API Cost Optimization - Caching, Model Matching, Batch Mode and Context Compression

The AI project works, tests pass, it goes to production - and after a month the API bill is several times the budget. Usually the model price isn't to blame; it's what you send with every call: a long system prompt, tool definitions, a growing conversation history and document fragments. Five techniques - caching, matching the model to the task, batch mode, context compression and limits - can cut the bill in half or more without hurting quality, and caching alone cuts the cost of repeated context by 90-97.5%.

How to cut your OpenAI, Claude and Gemini API bill without losing quality: one call broken down before and after (-68%), caching at three vendors, model matching, batch and Flex modes, history compression and a two-week plan.

In this post I show where costs come from, how caching works at OpenAI, Anthropic and Google, how to match models to tasks, when to use batch mode, how to compress conversation history and how to monitor spend. Using a single agent call as an example, I break down the cost before and after optimization.

Where the costs come from

Every call consists of input tokens (the system prompt, tool definitions, history, knowledge base context, the user's message) and output tokens (the response). Output is usually about 5 times more expensive per token, but in chat and agent applications input makes up most of the bill, because you send the same elements at every step.

A typical breakdown of input tokens in an agent application:

• System prompt - a few hundred to a few thousand tokens, the same on every call.
• Tool definitions - from 100 to a few hundred tokens per tool, always sent.
• Conversation history - grows with every turn.
• Knowledge base context - from a few hundred to a few thousand tokens.
• User message - usually the smallest part.

One call broken down - before and after

An example: a customer service agent on GPT-6.1 Sol ($2 per million input tokens, $10 per million output, $0.10 for cached reads). The tenth turn of a conversation:

At 100,000 such calls a month, that's the difference between about $3,280 and $1,040 - 68% less, without changing the model. Moving simple queries to a cheaper model lowers the bill even further.

Caching - the biggest and cheapest saving

Caching means you don't pay full price for the same start of a prompt on subsequent calls. It works at all three major vendors, but on different terms:

Writing to the cache can cost slightly more than regular input (1.25 times the price at OpenAI and Anthropic), so caching pays off after the first reuse.

The key rule: fixed elements first, variable ones last. The order in the prompt:

1. 1.System prompt - fixed for the whole application.
2. 2.Tool definitions - fixed.
3. 3.Context documents - fixed within a session.
4. 4.Conversation history - changes every turn.
5. 5.User message - always new.

A common mistake: putting the date, a session ID or the user's name into the system prompt. Every such change invalidates the cache. Move this data to the end, into the user message.

from openai import OpenAIfrom anthropic import AnthropicSYSTEM_PROMPT = "You are a customer service assistant for Acme. [...a fixed description of products, procedures and pricing...]"def chat_openai(history: list, message: str):    r = OpenAI().chat.completions.create(        model="gpt-6.1-sol",        messages=[{"role": "system", "content": SYSTEM_PROMPT}, *history, {"role": "user", "content": message}],    )    details = r.usage.prompt_tokens_details    return r.choices[0].message.content, {"input": r.usage.prompt_tokens, "cached": details.cached_tokens if details else 0}def chat_anthropic(history: list, message: str):    r = Anthropic().messages.create(        model="claude-sonnet-5-5",        max_tokens=1024,        system=[{"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}],        messages=[*history, {"role": "user", "content": message}],    )    return r.content[0].text, {"input": r.usage.input_tokens, "cached": r.usage.cache_read_input_tokens}

Log the number of cached tokens on every call and track the hit rate. If it drops, someone usually added a variable element to the start of the prompt.

The right model for the task

The cheapest models now cost 10-30 cents per million input tokens and the most capable ones $2-10. In a typical business application, most queries are simple tasks - classification, short answers from provided context, data extraction - that a cheap model handles just as well.

• The cheap tier (GPT-6 Luna, Gemini 3.1 Flash-Lite, Claude Haiku 4.5) - classification, routing, simple FAQ, schema-based data extraction.
• The mid tier (GPT-6.1 Sol, Claude Sonnet 5.5, Gemini 3.8 Flash) - document analysis, answers that affect a customer's decision, content generation.
• The most capable models (GPT-6 Astra, Claude Opus 5.5, Gemini 3.1 Pro) - only the hardest multi-step steps, where an error costs more than the call.

I describe a router in which a cheap model decides which tier is needed, plus a model selection test on your own data, in detail in the post on how to choose an LLM. Before you switch traffic to a cheaper model, run it in shadow mode for a week: the stronger model still gives the answer, and the cheaper one generates its own in the background for comparison.

Batch and Flex - half price for patience

Tasks that don't need an immediate result can go through batch mode (Batch API). OpenAI, Anthropic and Google offer a 50% discount there, and results usually come back within a day. It's a good fit for generating product descriptions, classifying a document archive, overnight reports or evaluations.

OpenAI also has Flex mode - individual requests at batch prices, but with longer response times and a risk of temporary unavailability. It suits background tasks that can wait and be retried.

Discounts stack: a batch job with a cached prompt costs a fraction of the standard price.

Context compression - stop the bill from growing

In agents and chats, history grows every turn. Three strategies, from the simplest:

• A sliding window. You send only the last N messages. No extra cost, but the model forgets the start of the conversation. Good when older context matters little.
• Summarization. Every few turns a cheap model summarizes the older part of the conversation into a few hundred tokens, while the latest turns stay intact. Keeps key facts with big savings.
• Selection by relevance. Only the earlier fragments closest to the current question go into context (an embeddings comparison). The most precise for very long and varied sessions.

from openai import OpenAI

def compress_history(history: list[dict], keep_last: int = 6, max_words: int = 200) -> list[dict]: if len(history) <= keep_last: return history old, recent = history[:-keep_last], history[-keep_last:] text = "\n".join(f"{m['role']}: {m['content']}" for m in old) r = OpenAI().chat.completions.create( model="gpt-6-luna", messages=[ {"role": "system", "content": f"Summarize the conversation in at most {max_words} words. Keep facts, decisions and open issues, skip greetings."}, {"role": "user", "content": text}, ], ) summary = r.choices[0].message.content return [{"role": "user", "content": f"[Summary of the earlier conversation] {summary}"}, *recent]

Put the summary after the fixed part of the prompt so it doesn't break caching. Check quality on a test set: compare answers with full and compressed history. I cover managing agent context in more depth in the post on context engineering.

Limits and monitoring

• A response length limit matched to the task, instead of the default.
• Only the tools needed at a given step - every definition costs tokens on every call.
• A shorter system prompt - remove repetition and instructions that don't change results.
• A budget per task and per user - protection against an agent loop that runs up a big bill overnight.
• Cost per endpoint - log the tokens and cost of every call (your own logs or tools such as Helicone or Langfuse) and set an alert when the cost of a single request exceeds a threshold.

A two-week plan

---

I run AI cost audits: I analyze call logs, show where the budget is leaking, and implement caching, model matching, context compression and monitoring. I do this as part of AI app development and AI consulting. Get in touch - I start with an analysis of your API logs.

Worth reading next:

• How to Choose an LLM in 2026 - OpenAI, Anthropic, Google or Open Weights (Prices, Testing, Routing)
• Context Engineering - How to Manage Model Context So Your AI Agent Doesn't Lose the Thread
• Monitoring AI in Production: Metrics, Tracing and Alerts for LLM Applications
• Ollama, vLLM and Self-Hosted LLM - How to Run a Local AI Model in Your Company