LLM Model Routing: Cut Your AI Inference Costs by 40–85% Without Sacrificing Quality

Most teams pick one model and run everything through it. That's the wrong default.

A simple classification task, a one-sentence summarization, a yes/no extraction — these do not need Claude Opus or GPT-4o. They need a fast, cheap model. Routing every request to your most capable model is like taking a taxi to the corner store: you'll get there, but you'll pay for it.

Model routing fixes this. You build a thin layer that looks at each request and decides which model to send it to. Done well, it cuts inference costs by 40–85% with less than 5% degradation in output quality.

This post covers how to build one.

The Core Idea

A model router sits between your application and the LLM APIs. It classifies each incoming request by complexity or type, then dispatches it to the appropriate model tier:

Request → Router → [simple?] → Llama 3.1 8B / Gemini Flash
                 → [complex?] → Claude Sonnet / GPT-4o
                 → [expert?]  → Claude Opus / GPT-4o-heavy

The router itself is lightweight — it adds single-digit milliseconds of latency. The payoff is that 60–80% of your requests (the easy ones) never touch your expensive model.

Two Routing Strategies

1. Rule-Based Routing

The simplest approach: use heuristics to classify requests before sending them.

def route_by_heuristic(prompt: str, system: str = "") -> str:
    """Return the model name to use for this request."""
    token_estimate = len(prompt.split()) * 1.3

    # Long context or complex reasoning → powerful model
    if token_estimate > 2000:
        return "claude-sonnet-5"
    
    # Keyword signals for complexity
    complex_signals = [
        "analyze", "compare", "synthesize", "evaluate",
        "write a detailed", "explain the tradeoffs", "step by step"
    ]
    if any(signal in prompt.lower() for signal in complex_signals):
        return "claude-sonnet-5"
    
    # Classification, extraction, short Q&A → cheap model
    simple_signals = [
        "summarize in one sentence", "extract", "classify",
        "yes or no", "true or false", "what is the", "translate"
    ]
    if any(signal in prompt.lower() for signal in simple_signals):
        return "gemini/gemini-2.0-flash"
    
    # Default: mid-tier
    return "gemini/gemini-2.0-flash"

This is predictable and zero-latency, but it relies on prompt patterns holding. It breaks when users phrase things unexpectedly.

2. Learned Routing with RouteLLM

RouteLLM (from the LMSys team behind Chatbot Arena) trains a lightweight classifier on human preference data. It predicts whether a "weak" or "strong" model will produce a satisfying answer for a given query.

pip install routellm[serve,eval]

from routellm.controller import Controller

# Initialize with a trained router
client = Controller(
    routers=["mf"],  # matrix-factorization router
    strong_model="claude-sonnet-5",
    weak_model="ollama_chat/llama3.1:8b",
)

response = client.chat.completions.create(
    model="router-mf-0.11593",  # threshold controls routing aggressiveness
    messages=[{"role": "user", "content": "What is the capital of France?"}]
)

print(response.choices[0].message.content)
# Routes to llama3.1:8b — simple factual query

The 0.11593 threshold is the key parameter. It controls what fraction of calls go to the strong model. Lower threshold = more calls go to weak model = cheaper but riskier. The RouteLLM paper shows that with the right threshold, you can send 40% of traffic to the weak model and lose less than 1% on MT-Bench scores.

Production Setup with LiteLLM Router

For most teams, LiteLLM is the practical production choice. It gives you a unified API across 100+ models, plus built-in routing, fallbacks, and load balancing.

pip install litellm

from litellm import Router

# Define your model pool
model_list = [
    {
        "model_name": "fast-tier",
        "litellm_params": {
            "model": "gemini/gemini-2.0-flash",
            "api_key": os.environ["GEMINI_API_KEY"],
        },
        "tpm": 1_000_000,  # tokens per minute limit
        "rpm": 1000,        # requests per minute limit
    },
    {
        "model_name": "standard-tier",
        "litellm_params": {
            "model": "claude-sonnet-5",
            "api_key": os.environ["ANTHROPIC_API_KEY"],
        },
        "tpm": 100_000,
        "rpm": 500,
    },
    {
        "model_name": "expert-tier",
        "litellm_params": {
            "model": "claude-opus-4-8",
            "api_key": os.environ["ANTHROPIC_API_KEY"],
        },
        "tpm": 50_000,
        "rpm": 100,
    },
]

router = Router(model_list=model_list)

Now wrap it with your routing logic:

import re

def classify_request(messages: list[dict]) -> str:
    """Return the tier name for this set of messages."""
    user_content = " ".join(
        m["content"] for m in messages if m["role"] == "user"
    )
    total_tokens = sum(len(m["content"].split()) for m in messages) * 1.3

    if total_tokens > 3000:
        return "expert-tier"

    # Code generation or multi-step reasoning
    if re.search(r"(write|implement|build|debug|refactor)\s.*(function|class|api|service)", user_content, re.I):
        return "standard-tier"

    # Simple extraction or lookup
    if total_tokens < 200 and not any(
        word in user_content.lower()
        for word in ["explain", "analyze", "compare", "write"]
    ):
        return "fast-tier"

    return "standard-tier"


async def routed_completion(messages: list[dict], **kwargs):
    tier = classify_request(messages)
    return await router.acompletion(model=tier, messages=messages, **kwargs)

Adding Fallbacks

The real production value of LiteLLM: automatic fallback when a model is rate-limited or down.

router = Router(
    model_list=model_list,
    fallbacks=[
        {"fast-tier": ["standard-tier"]},       # if flash is down, use sonnet
        {"standard-tier": ["expert-tier"]},      # if sonnet fails, escalate
    ],
    allowed_fails=2,          # mark model unhealthy after 2 failures
    cooldown_time=60,         # wait 60s before retrying unhealthy model
    retry_after=5,            # retry rate-limited requests after 5s
    num_retries=3,
)

This means your app keeps working even during provider outages. The router handles the escalation; your application code sees nothing.

Measuring What You're Saving

Track routing decisions so you can tune thresholds over time:

import time
from dataclasses import dataclass, field

@dataclass
class RoutingMetrics:
    calls_by_tier: dict = field(default_factory=lambda: {
        "fast-tier": 0, "standard-tier": 0, "expert-tier": 0
    })
    cost_by_tier: dict = field(default_factory=lambda: {
        "fast-tier": 0.0, "standard-tier": 0.0, "expert-tier": 0.0
    })

    # Approximate cost per 1M tokens (input + output blended), mid-2026
    COST_PER_MTok = {
        "fast-tier": 0.10,      # Gemini Flash
        "standard-tier": 3.00,  # Claude Sonnet
        "expert-tier": 15.00,   # Claude Opus
    }

    def record(self, tier: str, input_tokens: int, output_tokens: int):
        self.calls_by_tier[tier] += 1
        total_tokens = input_tokens + output_tokens
        self.cost_by_tier[tier] += (total_tokens / 1_000_000) * self.COST_PER_MTok[tier]

    def report(self):
        total_cost = sum(self.cost_by_tier.values())
        total_calls = sum(self.calls_by_tier.values())
        # Cost if everything went to expert-tier
        hypothetical_all_expert = sum(
            calls * 15.00 * 0.001  # rough average tokens
            for calls in self.calls_by_tier.values()
        )
        print(f"Total calls: {total_calls}")
        for tier, calls in self.calls_by_tier.items():
            pct = 100 * calls / max(total_calls, 1)
            print(f"  {tier}: {calls} calls ({pct:.1f}%) — ${self.cost_by_tier[tier]:.4f}")
        print(f"Total cost: ${total_cost:.4f}")

Routing Strategy Comparison

Strategy	Latency overhead	Accuracy	Best for
Rule-based (heuristics)	~0ms	Medium	Stable, predictable prompt patterns
RouteLLM (MF router)	~5–10ms	High	General chat, mixed workloads
LiteLLM auto-routing	~2–5ms	Medium-high	API gateway, multi-provider setup
Task-type classifier	~20–50ms	High	Domain-specific apps with labeled data
Token-count only	~0ms	Low	Quick wins, not production-grade

Common Pitfalls

Routing on input length alone. A 50-token prompt like "Explain the difference between TCP and UDP, considering NAT traversal, congestion windows, and QUIC's hybrid approach" is complex. Token count is a weak signal — combine it with keyword patterns or a classifier.

Skipping quality regression tests. When you shift 60% of traffic to a cheaper model, run your existing eval suite against both models on your real query distribution. If you don't have evals, start with 200 sampled production queries and manually check the weak model's outputs.

Not monitoring routing drift. If your users start asking harder questions over time, the routing thresholds that worked at launch will route too many hard queries to the cheap model. Set up weekly checks on your tier distribution.

Forgetting latency. Gemini Flash and Llama 3.1 8B are not just cheaper — they're often faster. For streaming responses, this is a quality-of-life improvement, not just a cost saving.

No fallback for the weak model. If Gemini Flash is down, make sure your router can fall back to the next tier, not fail open with an error. Users do not care that you were trying to save money.

When Not to Route

Routing adds complexity. If you're processing fewer than 50K requests per month, the engineering cost of a router may not pay off in dollar terms — just use a mid-tier model everywhere. Model routing makes most sense when:

• You're spending more than $200/month on inference
• Your workload has clearly separable easy/hard query types
• You have existing evals you can run to validate tier quality

Next Steps

1. Audit your last 500 requests. What fraction were simple extraction or factual lookups? That's your routing opportunity.
2. Set up LiteLLM with two tiers and a basic heuristic classifier. Track cost per tier for two weeks.
3. Run your evals against both models to confirm quality holds.
4. If you're processing high volume, evaluate RouteLLM's trained routers against your query distribution.

The goal is not to always use the cheapest model — it's to use the cheapest model that will give a good answer. That distinction is what routing makes precise.

---

Building AI products and want to go deeper on production AI engineering — inference optimization, evals, agent architecture? The Joinloop AI Engineering Cohort covers this with hands-on projects and expert mentors.