# AI Cost Optimization: How to Cut LLM Spending by 60% Without Losing Quality

> Five techniques to cut LLM API costs: prompt caching, smart routing, Batch API, semantic caching, model downsizing. Before/after tables, prompts, and configs.
> Author: Roman Belov · Published: 2026-06-09 · Source: https://futurecraft.pro/blog/ai-cost-optimization/

The average LLM startup spends $8,000–15,000 per month on API calls at 50,000 requests per day. Yet 40–60% of that goes toward repeated requests, excess tokens, and routing expensive models at tasks they're massively overqualified for.

A 60% cost reduction is doable in 2–3 weeks. Without quality loss, without dropping flagship models on hard tasks. Five techniques, each with measurable results.

## Anatomy of LLM costs: where the money goes

Before you cut anything, understand what you're actually paying for. A typical breakdown for a production app with an AI chat and several pipelines:

| Cost category | Budget share | Reason |
|---------------|-------------|--------|
| Repeated system prompts | 25–35% | Same system prompt on every request, thousands of times per day |
| Excess model capacity | 20–30% | GPT-5.4 on tasks GPT-5.4-mini handles fine |
| Duplicate requests | 15–20% | Same questions from different users |
| Synchronous calls without prioritization | 10–15% | Batch tasks billed at real-time prices |
| Genuine load | 15–25% | Requests that truly require full model capacity |

Each technique below targets a specific category. The order follows ROI: the first one gives the most impact with the least effort.

## Prompt Caching: saving on repeated prefixes

Prompt caching lets the provider cache the beginning of a prompt — system prompt, context, instructions — and skip reprocessing those tokens on every subsequent request that shares the same prefix.

Anthropic introduced prompt caching in August 2024. OpenAI shipped automatic caching in October 2024. Google Gemini has had context caching since mid-2024.

**How it works.** The provider hashes the first N tokens. If the next request starts with the same block, cached tokens aren't billed at full price. Anthropic's discount: 90% on cached tokens. OpenAI's: 50%.

### Anthropic: explicit caching

With Anthropic, caching is managed explicitly via the `cache_control` parameter in messages:

```json
{
  "model": "claude-sonnet-4-6",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "You are an AI assistant for travel planning. Respond in a structured format, use markdown. Always include budget, travel time, safety rating for each location...",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "messages": [
    {"role": "user", "content": "Plan a 7-day itinerary in Bali"}
  ]
}
```

Minimum cacheable block size: 1,024 tokens for Claude Sonnet/Opus, 2,048 for Haiku. The cache lives for 5 minutes and is refreshed with each access.

### OpenAI: automatic caching

With OpenAI, prompt caching works automatically for prompts longer than 1,024 tokens. No code changes needed. The longest common prefix between requests is cached.

### Savings calculation

System prompt for an AI chat — 2,000 tokens. 10,000 requests per day. Model: Claude Sonnet 4.6.

**Without caching:**
- System prompt input tokens: 2,000 × 10,000 = 20M tokens/day
- Cost: 20M × ~$3.00/1M = ~$60/day

**With caching (Anthropic, 90% discount on cached tokens):**
- First request: ~$3.00/1M (full price)
- Cached: ~$0.30/1M
- Cache write: ~$3.75/1M (25% more than standard, only on first write)
- Cost: 20M × ~$0.30/1M = ~$6/day

**Savings: ~$54/day, ~$1,620/month.** And that's just the system prompt. Cache few-shot examples and additional context, and the savings scale proportionally.

### Pattern: maximizing cache hits

Prompt structure determines caching efficiency. The cacheable block goes first; variable parts go last.

```
┌─────────────────────────────┐
│  System prompt (cached)     │  ← 2000+ tokens, same for everyone
├─────────────────────────────┤
│  Few-shot examples          │  ← 500–1000 tokens, same for everyone
│  (cached)                   │
├─────────────────────────────┤
│  User context               │  ← Unique per request
├─────────────────────────────┤
│  User message               │  ← Unique per request
└─────────────────────────────┘
```

Common mistake: putting the user's name at the top of the system prompt. Every new user breaks the cache. Fix: put the name after the cached block, or move it into the `user` message.

## Smart Routing: the right model for each task

Not every request needs a flagship model. Classifying a ticket, generating a headline, pulling an email from text — GPT-5.4-mini or Gemini 2.5 Flash-Lite handles these just as well as GPT-5.4 at 10–30x lower cost.

### Model pricing table (April 2026)

> Prices are approximate as of publication and may change. Check provider websites for current rates.

| Model | Input ($/1M) | Output ($/1M) | Relative to Claude Sonnet 4.6 |
|-------|-------------|--------------|-------------------------------|
| GPT-5.4-mini | ~$0.15 | ~$0.60 | 20x cheaper |
| Gemini 2.5 Flash-Lite | ~$0.10 | ~$0.40 | 30x cheaper |
| DeepSeek-V3 | ~$0.14 | ~$0.28 | 21x cheaper |
| Claude Haiku 3.5 | ~$0.80 | ~$4.00 | 4x cheaper |
| GPT-5.4 | ~$2 | ~$8 | ~1x (comparable) |
| Claude Sonnet 4.6 | ~$3 | ~$15 | baseline |
| Claude Opus 4.8 | ~$5 | ~$25 | ~2.5x more expensive |

The gap between Gemini Flash-Lite and Claude Opus is 150x on input, nearly 190x on output. Not a rounding error — orders of magnitude.

### Task classifier

Routing starts with classification. Two approaches: rule-based (by task type) or LLM-based (a cheap model decides complexity).

**Rule-based routing** — simpler, more predictable, sufficient for most cases:

```python
ROUTING_TABLE = {
    # task_type: model
    "chat_title_generation": "gemini/gemini-2.5-flash-lite",
    "ticket_classification": "gpt-5.4-mini",
    "email_extraction": "gpt-5.4-mini",
    "sentiment_analysis": "gemini/gemini-2.5-flash-lite",
    "translation": "deepseek/deepseek-chat",
    "code_review": "anthropic/claude-sonnet-4-6",
    "complex_reasoning": "anthropic/claude-sonnet-4-6",
    "trip_planning": "anthropic/claude-sonnet-4-6",
}

def route_request(task_type: str, fallback: str = "gpt-5.4-mini") -> str:
    return ROUTING_TABLE.get(task_type, fallback)
```

**LLM-based router** — for cases where task complexity is unpredictable. A cheap model evaluates the incoming request and routes it to the appropriate model:

```python
ROUTER_PROMPT = """Assess the complexity of the user's task on a scale of 1–3:
1 - simple (classification, extraction, formatting)
2 - medium (summarization, translation, answering a question)
3 - complex (reasoning, analysis, plan generation)

Reply with ONLY the number: 1, 2, or 3."""

COMPLEXITY_TO_MODEL = {
    1: "gpt-5.4-mini",              # ~$0.15/~$0.60
    2: "deepseek/deepseek-chat",   # ~$0.14/~$0.28
    3: "anthropic/claude-sonnet-4-6", # ~$3/~$15
}
```

The router itself costs ~50 input tokens + ~1 output token via GPT-5.4-mini — $0.000008 per classification. At 10,000 requests/day that's $0.08/day. Routing savings: tens of dollars per day.

### Real distribution

In practice, 60–70% of requests in a typical AI app are "simple," 20–25% are "medium," and only 10–15% actually need a flagship model.

| Scenario | Model | Requests/day | Cost/day |
|----------|-------|-------------|---------|
| **Without routing** | Claude Sonnet 4.6 for everything | 10,000 | ~$90 |
| **With routing** | Mix (70/20/10) | 10,000 | ~$18.50 |

**Savings: ~$71.50/day, ~$2,145/month.**

More on setting up multi-provider routing through LiteLLM in the [multi-provider architecture article](/blog/multi-provider-llm-architecture/).

## Batch API: deferred tasks at half price

OpenAI, Anthropic, and Google all offer Batch APIs — submit a batch, get results within 24 hours. Discount: 50% off standard pricing.

Not everything needs a 200ms response. Weekly reports, bulk content classification, data enrichment, processing uploaded documents — all of this can wait.

### Which tasks work for Batch API

| Suitable | Not suitable |
|----------|-------------|
| Daily/weekly analytics | Real-time chat |
| Mass classification (100+ items) | Autocomplete |
| Generating embeddings for search | Streaming responses |
| Processing uploaded files | Anything interactive |
| Periodic data enrichment | Triggered notifications |

### OpenAI Batch API

```python
import json
from openai import OpenAI

client = OpenAI()

# 1. Prepare JSONL file with requests
requests = []
for i, item in enumerate(items_to_classify):
    requests.append({
        "custom_id": f"item-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-5.4-mini",
            "messages": [
                {"role": "system", "content": "Classify the product into categories: electronics, clothing, food, other. Reply with one word."},
                {"role": "user", "content": item["description"]}
            ],
            "max_tokens": 10
        }
    })

# Write to file
with open("batch_input.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

# 2. Upload file
batch_file = client.files.create(
    file=open("batch_input.jsonl", "rb"),
    purpose="batch"
)

# 3. Create batch
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

# 4. Check status (later)
status = client.batches.retrieve(batch.id)
# status.status: "validating" → "in_progress" → "completed"
```

### Anthropic Message Batches

Anthropic has an equivalent mechanism — Message Batches API. 50% discount, results within 24 hours.

```python
import anthropic

client = anthropic.Anthropic()

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"item-{i}",
            "params": {
                "model": "claude-sonnet-4-6",
                "max_tokens": 100,
                "messages": [
                    {"role": "user", "content": f"Classify: {item['description']}"}
                ]
            }
        }
        for i, item in enumerate(items)
    ]
)
```

### Savings calculation

1,000 documents per day for classification. Average request: 500 input + 20 output tokens. Model: GPT-5.4-mini.

**Synchronous:**
- Input: 500K × ~$0.15/1M = ~$0.075
- Output: 20K × ~$0.60/1M = ~$0.012
- Total: ~$0.087/day

**Batch API (50% discount):**
- Total: ~$0.044/day

On GPT-5.4-mini the absolute numbers are modest. But run the same math for Claude Sonnet 4.6 at 10,000 requests:

**Synchronous:** ~$45/day → **Batch:** ~$22.50/day. **Savings: ~$675/month.**

## Semantic Caching: identical questions don't need to be processed twice

Prompt caching works at the provider level (repeated prefix). Semantic caching works at the application level: if a user asks something similar to a question you've already processed, the answer comes from local cache — no LLM call.

"What's the weather in Bali in December?" and "Bali weather in December" are the same question. There's no reason to pay twice.

### How semantic cache works

```
User request
    │
    ▼
Generate embedding (text-embedding-3-small: $0.02/1M tokens)
    │
    ▼
Search in vector store (Qdrant / Redis / pgvector)
    │
    ├── similarity > 0.95 → return cached response (0 LLM tokens)
    │
    └── similarity < 0.95 → call LLM → save response + embedding to cache
```

### Implementation with Redis + OpenAI embeddings

```python
import hashlib
import json
import numpy as np
import redis
from openai import OpenAI

client = OpenAI()
cache = redis.Redis(host="localhost", port=6379, db=0)

SIMILARITY_THRESHOLD = 0.95
CACHE_TTL = 3600  # 1 hour

def get_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def cached_completion(messages: list[dict], model: str = "gpt-5.4") -> str:
    user_message = messages[-1]["content"]
    query_embedding = get_embedding(user_message)

    # Search cache
    cached_keys = cache.keys("sem_cache:*")
    for key in cached_keys:
        cached_data = json.loads(cache.get(key))
        similarity = cosine_similarity(query_embedding, cached_data["embedding"])
        if similarity >= SIMILARITY_THRESHOLD:
            return cached_data["response"]

    # Cache miss: call LLM
    response = client.chat.completions.create(
        model=model,
        messages=messages
    )
    result = response.choices[0].message.content

    # Save to cache
    cache_key = f"sem_cache:{hashlib.md5(user_message.encode()).hexdigest()}"
    cache.setex(
        cache_key,
        CACHE_TTL,
        json.dumps({"embedding": query_embedding, "response": result})
    )

    return result
```

For production, replace linear search over Redis keys with a proper vector database — Qdrant, Pinecone, or pgvector. At 10,000+ cache entries, linear search will become the bottleneck.

### Ready-made solutions: Redis LangCache

For a long time the default open-source option was [GPTCache](https://github.com/zilliztech/GPTCache) by Zilliz, but the project has effectively stopped evolving: support for new models and APIs is no longer being added, and it's not a good pick for new code.

The living alternative is [Redis LangCache](https://redis.io/langcache/) — a managed semantic caching service on top of Redis. It works over a REST API, with embedding generation and vector search handled service-side, so your application makes a single HTTP call before hitting the LLM. For a self-hosted setup, the Redis + embeddings combination from the example above is enough, and the LiteLLM proxy ships with a built-in Redis-backed semantic cache — no separate library required.

### Savings calculation

Assumptions: 10,000 requests per day, 30% cache hit rate (conservative for FAQ-like scenarios), model GPT-5.4 (approximate prices used).

**Without semantic cache:**
- 10,000 × (500 input + 200 output tokens)
- Cost: 5M × ~$2/1M + 2M × ~$8/1M = ~$26/day

**With semantic cache (30% hits):**
- 7,000 LLM requests: ~$18.20
- 10,000 embeddings: 5M × ~$0.02/1M = ~$0.10
- Total: ~$18.30/day

**Savings: ~$7.70/day, ~$230/month.** At 50% cache hit rate — realistic for support bots — that climbs to ~$390/month.

## Model Downsizing: systematic replacement of expensive models

The most obvious technique, and the riskiest if you skip the process. Swapping Claude Sonnet 4.6 for GPT-5.4-mini on a specific task isn't flipping a switch. It requires an eval framework.

### Downsizing process

1. **Collect an eval dataset.** 50–100 real examples from production logs for each task. Input data + expected responses (or quality criteria).

2. **Run a baseline.** Current model on the eval dataset. Record metrics: accuracy, format compliance, latency, cost.

3. **Run the candidate.** Cheaper model on the same dataset. Same metrics.

4. **Compare.** If the cheaper model delivers >95% of the quality of the expensive one — safe to switch.

5. **A/B test in production.** 10% of traffic to the new model, monitoring via [Langfuse](/blog/llm-observability-langfuse/) or equivalent.

### Eval framework

```python
import json
from dataclasses import dataclass

@dataclass
class EvalResult:
    model: str
    task: str
    accuracy: float
    avg_latency_ms: float
    cost_per_1k_requests: float
    format_compliance: float  # % of responses in the expected format

def run_eval(model: str, task: str, dataset: list[dict]) -> EvalResult:
    results = []
    total_cost = 0
    total_latency = 0
    correct = 0
    format_ok = 0

    for example in dataset:
        start = time.time()
        response = call_llm(model=model, messages=example["messages"])
        latency = (time.time() - start) * 1000

        total_latency += latency
        total_cost += response.usage.total_cost

        if evaluate_correctness(response.content, example["expected"]):
            correct += 1
        if validate_format(response.content, example.get("format_schema")):
            format_ok += 1

        results.append(response)

    n = len(dataset)
    return EvalResult(
        model=model,
        task=task,
        accuracy=correct / n,
        avg_latency_ms=total_latency / n,
        cost_per_1k_requests=(total_cost / n) * 1000,
        format_compliance=format_ok / n,
    )

# Comparison
baseline = run_eval("anthropic/claude-sonnet-4-6", "ticket_classification", dataset)
candidate = run_eval("gpt-5.4-mini", "ticket_classification", dataset)

print(f"Accuracy: {baseline.accuracy:.1%} → {candidate.accuracy:.1%}")
print(f"Cost/1K:  ${baseline.cost_per_1k_requests:.2f} → ${candidate.cost_per_1k_requests:.2f}")
print(f"Latency:  {baseline.avg_latency_ms:.0f}ms → {candidate.avg_latency_ms:.0f}ms")
```

### Typical downsizing results

| Task | Before | After | Accuracy delta | Cost delta |
|------|--------|-------|----------------|------------|
| Ticket classification | Claude Sonnet 4.6 | GPT-5.4-mini | -1.2% | -95% |
| Email extraction | GPT-5.4 | GPT-5.4-mini | -0.3% | -94% |
| Headline generation | Claude Sonnet 4.6 | Gemini 2.5 Flash-Lite | -0.8% | -97% |
| Summarization | GPT-5.4 | DeepSeek-V3 | -2.1% | -94% |
| Complex reasoning | Claude Sonnet 4.6 | GPT-5.4-mini | -18.5% | -95% |

That last row is why eval isn't optional. Cheap models fall apart on complex reasoning. Skip the eval, and you'll degrade the product while thinking you're saving money.

## Combined strategy: summary table

The savings figures from the sections above cannot simply be added together: each technique was calculated on its own illustrative scenario, and applied in sequence, each subsequent technique works on an already-reduced base.

A sequential calculation on a single scenario: 10,000 requests/day, average request of 2,000 system prompt tokens + 500 input + 200 output, base model Claude Sonnet 4.6 (~$105/day, ~$3,150/month):

| Step | Assumption | Cost/month after step | Difficulty | Implementation time |
|------|------------|----------------------|------------|---------------------|
| Baseline | everything on Claude Sonnet 4.6, no optimizations | ~$3,150 | — | — |
| + Prompt Caching | ~80% cache hit rate on the system prompt | ~$1,940 | Low | 1 day |
| + Smart Routing | 40% of requests move to GPT-5.4-mini | ~$1,230 | Medium | 3–5 days |
| + Batch API | 15% of traffic deferred at a 50% discount | ~$1,130 | Low | 1–2 days |
| + Semantic Caching | 10% cache hit rate | ~$1,020 | Medium | 2–3 days |

Model downsizing doesn't get its own line in this waterfall: in this scenario it largely overlaps with smart routing — moving tasks to cheaper models after an eval is its final stage. It remains the highest-effort technique of the five (1–2 weeks including building the eval). The routing share here is deliberately conservative: 60–70% of requests classify as simple, but not every one of them can be pulled out of an interactive flow without hurting UX.

**Total savings: ~67%** (~$3,150 → ~$1,020/month). Implementation order: prompt caching (instant effect, zero risk) → smart routing (biggest absolute savings, needs task classification) → batch API (migrate deferred tasks) → semantic caching → model downsizing (last, because it requires eval).

## Cost monitoring: without observability, optimization is blind

Optimizing without monitoring is guesswork. Three metrics to track daily:

**Cost per request** — average cost of one LLM call, broken down by task and model. A sudden spike usually means the cache broke, routing started sending to the expensive model, or someone padded the prompt.

**Cache hit rate** — for both prompt caching and semantic caching. If it drops below a threshold (say, <20% for semantic cache), something changed: request patterns shifted, TTL is too short, or the similarity threshold is too tight.

**Quality score** — accuracy and relevance on the eval dataset. Run it after any model swap or downsizing to confirm you haven't quietly degraded the product.

[Langfuse](/blog/llm-observability-langfuse/) stores every trace: model, tokens, cost, latency. A cost-by-model dashboard takes minutes to build. LiteLLM integrates with Langfuse natively — every call through the proxy gets logged automatically.

```yaml
# litellm_config.yaml
litellm_settings:
  success_callback: ["langfuse"]

environment_variables:
  LANGFUSE_PUBLIC_KEY: "pk-..."
  LANGFUSE_SECRET_KEY: "sk-..."
  LANGFUSE_HOST: "https://cloud.langfuse.com"
```

## What not to do

**Optimize before you've launched.** Premature optimization applies to LLM costs as much as anything else. Ship first, get real usage data, then cut based on facts.

**Trade quality for savings.** If downsizing drops accuracy from 95% to 80%, that's not optimization — that's breaking the product. Users churn, and the $500/month you saved won't come close to covering it.

**Overlook hidden costs.** Semantic caching needs a vector database. An eval framework needs time to label the dataset. Factor those into the ROI before counting the win.

**Forget that models get cheaper.** Pricing moves fast, and mostly downward. Check provider pricing every 2–3 months — the math changes.

## Conclusion

Five techniques applied in order cut LLM costs by 60% or more. Prompt caching and batch API need no architectural changes and pay off on day one. Smart routing and model downsizing take more work, but produce the largest absolute numbers.

One rule holds across all of them: every change needs to be measurable. Run eval before downsizing. Watch metrics after any change. Keep a cost-per-request dashboard in Langfuse or equivalent. Without that visibility, you're not optimizing — you're guessing, and savings quietly turn into degradation.

---

*Need help with LLM cost optimization? I help startups build AI products and automate processes — [belov.works](https://belov.works).*