Code examples:

Model Selection: Cheap vs Capable Models

November 5, 2026 13 min read By Jaffar Kazi
Real-World AI Engineering Model Routing Cost Optimization
C# Python Azure

The Problem

Most teams ship an LLM feature by picking one model and calling it for every request. Early on this is the right call — there's one integration to build and one set of outputs to reason about. The problem shows up later, once volume grows: a listing-compliance classifier that checks every marketplace submission against platform policy and extracts the structured reason for a decision ends up calling the same top-tier model for an obviously-compliant single-item listing as it does for a borderline multi-item bundle with ambiguous, policy-adjacent language. The first case is, in plain terms, easy. The model barely has to think. The second case is genuinely hard — it needs judgment, not pattern matching.

Routing everything through the capable model isn't wrong, exactly. It's just wasteful in a specific, measurable way: a fixed cost per request regardless of how much reasoning that request actually needed. The instinct to "just use a cheaper model everywhere" is just as wrong in the other direction — it trades a known cost problem for an unknown correctness problem, because nobody has actually measured where the cheap model's judgment runs out. Both are guesses. What's missing is a way to find out, for a specific task, where the line between "cheap model handles this fine" and "cheap model gets this wrong" actually sits — and a way to act on that line in production without re-litigating the decision on every request.

The Shape of a Real Solution

The fix has two halves that have to stay separate. Offline, there's a fixed, labeled evaluation set run through every model under consideration, scored the same way every time, so "is the cheap model good enough" has an actual answer instead of an impression from a few manual spot-checks. Online, there's a routing policy that doesn't try to predict task difficulty in advance — it calls the cheap model first, checks the result against the schema and a confidence signal, and escalates to the capable model only when that check fails. The offline harness answers whether routing is worth doing at all and where to set the threshold; the online policy is what actually runs on every request.

Crucially, the escalation decision at request time never has access to ground truth — only the eval harness does. That asymmetry is the whole reason the production routing signal has to be something observable without a label: schema validity, a self-reported confidence score, or a deterministic rule, never "was this actually correct."

Architecture Overview

The offline and online paths share a model-client abstraction and a pricing table, but run on different triggers: the harness runs whenever the eval set, prompt, or model lineup changes; the routing policy runs on every live request.

Architecture diagram showing the offline evaluation harness feeding a confidence threshold into the online cheap-first escalation routing policy

What's notable here is the dotted line: the harness doesn't gate deploys by itself, it produces a number (the confidence threshold) that the online router consumes. Change the eval set or swap in a new model generation, and that number moves — the router's behavior changes without a code change.

Core Implementation

Three types carry the whole pattern: a typed eval task and result (so scoring is structured, not string-matched after the fact), a pricing table keyed by model tier, and the harness loop that runs every task through every model and scores it the same way.

EvalModels.cs / eval_models.py

public enum ModelTier { Cheap, Capable }

public record ModelPricing(decimal InputPerMillion, decimal OutputPerMillion);

// At the time of writing: Claude Haiku 5.5 (Cheap) and Claude Sonnet 5.5 (Capable).
public static class Pricing
{
    public static readonly Dictionary<ModelTier, ModelPricing> Table = new()
    {
        [ModelTier.Cheap]    = new ModelPricing(0.10m, 0.50m),   // per 1M tokens, prompts under 100K
        [ModelTier.Capable]  = new ModelPricing(2.00m, 10.00m),  // per 1M tokens
    };
}

public record EvalTask(
    string TaskId,
    string ListingText,
    string ComplexityTier,      // "clear-cut" or "borderline"
    string ExpectedDecision,    // "compliant" | "non_compliant"
    string? ExpectedReasonCode  // null when ExpectedDecision is "compliant"
);

public record ModelResponse(
    string Decision,
    string? ReasonCode,
    double SelfReportedConfidence,
    bool SchemaValid,
    int InputTokens,
    int OutputTokens,
    int LatencyMs
);

public record EvalResult(
    string TaskId,
    ModelTier Tier,
    bool SchemaValid,
    bool CorrectnessPass,
    double EstimatedCostUsd,
    int LatencyMs
);

from enum import Enum
from pydantic import BaseModel

class ModelTier(str, Enum):
    CHEAP = "cheap"
    CAPABLE = "capable"

class ModelPricing(BaseModel):
    input_per_million: float
    output_per_million: float

# At the time of writing: Claude Haiku 5.5 (Cheap) and Claude Sonnet 5.5 (Capable).
PRICING: dict[ModelTier, ModelPricing] = {
    ModelTier.CHEAP: ModelPricing(input_per_million=0.10, output_per_million=0.50),
    ModelTier.CAPABLE: ModelPricing(input_per_million=2.00, output_per_million=10.00),
}

class EvalTask(BaseModel):
    task_id: str
    listing_text: str
    complexity_tier: str          # "clear-cut" or "borderline"
    expected_decision: str        # "compliant" | "non_compliant"
    expected_reason_code: str | None = None

class ModelResponse(BaseModel):
    decision: str
    reason_code: str | None
    self_reported_confidence: float
    schema_valid: bool
    input_tokens: int
    output_tokens: int
    latency_ms: int

class EvalResult(BaseModel):
    task_id: str
    tier: ModelTier
    schema_valid: bool
    correctness_pass: bool
    estimated_cost_usd: float
    latency_ms: int

The harness itself just runs every task through every tier, scores the response, and prices it from the same table used in production — so the cost numbers in the eval report and the cost numbers in the production ledger mean the same thing.

EvaluationHarness.cs / evaluation_harness.py

public class EvaluationHarness
{
    private readonly Dictionary<ModelTier, IModelClient> _clients;

    public EvaluationHarness(Dictionary<ModelTier, IModelClient> clients) => _clients = clients;

    public async Task<List<EvalResult>> RunAsync(IEnumerable<EvalTask> tasks)
    {
        var results = new List<EvalResult>();
        foreach (var task in tasks)
        {
            foreach (var (tier, client) in _clients)
            {
                var response = await client.ClassifyAsync(task.ListingText);
                results.Add(Score(task, tier, response));
            }
        }
        return results;
    }

    private static EvalResult Score(EvalTask task, ModelTier tier, ModelResponse r)
    {
        bool correctnessPass = r.SchemaValid
            && r.Decision == task.ExpectedDecision
            && (task.ExpectedDecision != "non_compliant" || r.ReasonCode == task.ExpectedReasonCode);

        var price = Pricing.Table[tier];
        double cost = (r.InputTokens / 1_000_000.0 * (double)price.InputPerMillion)
                    + (r.OutputTokens / 1_000_000.0 * (double)price.OutputPerMillion);

        return new EvalResult(task.TaskId, tier, r.SchemaValid, correctnessPass, cost, r.LatencyMs);
    }
}

class EvaluationHarness:
    def __init__(self, clients: dict[ModelTier, "ModelClient"]):
        self._clients = clients

    async def run(self, tasks: list[EvalTask]) -> list[EvalResult]:
        results: list[EvalResult] = []
        for task in tasks:
            for tier, client in self._clients.items():
                response = await client.classify(task.listing_text)
                results.append(self._score(task, tier, response))
        return results

    @staticmethod
    def _score(task: EvalTask, tier: ModelTier, r: ModelResponse) -> EvalResult:
        correctness_pass = (
            r.schema_valid
            and r.decision == task.expected_decision
            and (task.expected_decision != "non_compliant" or r.reason_code == task.expected_reason_code)
        )
        price = PRICING[tier]
        cost = (r.input_tokens / 1_000_000 * price.input_per_million) + (
            r.output_tokens / 1_000_000 * price.output_per_million
        )
        return EvalResult(
            task_id=task.task_id, tier=tier, schema_valid=r.schema_valid,
            correctness_pass=correctness_pass, estimated_cost_usd=cost, latency_ms=r.latency_ms,
        )

Key Technical Challenge: Building a Fair Eval Set

Thirty to fifty tasks sounds small, and it is — small enough that how those tasks are chosen matters more than how many there are. A set built only from the last week's traffic will skew toward whatever was common that week, not toward the cases that actually separate the models. The eval set here is deliberately stratified into two tiers up front: "clear-cut" tasks (single-item listings, obvious policy violations or obvious compliance, the kind a keyword filter would mostly get right) and "borderline" tasks (bundles mixing compliant and non-compliant items, claims that depend on context, phrasing that's policy-adjacent without crossing the line). Each task is hand-labeled once with an expected decision and, for violations, an expected reason code — and that label is never touched again, because the eval set's value comes entirely from staying fixed while models and prompts change around it.

The failure mode worth naming: teams add new tasks to the eval set after a model fails them in production, which quietly makes the set easier to pass next time rather than more representative. New failure cases belong in the set, but only as a separate, dated addition — not a silent edit to the existing baseline.

Key Technical Challenge: Scoring Correctness When Answers Aren't Exact-Match

Schema validity is binary — the response either parses into the typed model or it doesn't. Correctness is not. For this task, decision and reason code are both enums, so exact-match scoring works, which is the easy case. A task where the model has to summarize or justify in free text needs a rubric instead: a second LLM call acting as a judge, or a human-reviewed sample used to spot-check the automated score. The mistake is applying exact-match scoring to a free-text field anyway, because it's simpler to implement — it produces a score that looks precise and measures something else entirely (surface phrasing, not correctness). Pick the scoring method per field, not per task: structured fields get exact match, judgment fields get a rubric, and the harness tags each `EvalResult` with which method produced it so nobody later treats both numbers as equally reliable.

Key Technical Challenge: Choosing a Production Routing Signal Without Ground Truth

The eval harness can score against the expected answer because it has one. The production router never does — by the time a real listing arrives, there's no label to check against, only the response itself. That leaves two usable signals: schema validity (did the output parse and does it reference a valid reason code) and a self-reported confidence score the model is explicitly prompted to return alongside its decision. Neither is a measurement of correctness; both are a measurement of how likely the response is to be correct, which is a meaningfully weaker thing, and the harness's job is to establish empirically how well that weaker signal predicts the stronger one it can actually check.

This is where the confidence threshold comes from: run the harness, take the cheap model's correctness-pass results, and find the self-reported-confidence cutoff below which correctness drops below the minimum acceptable rate. Set the router's escalation threshold there — not at a round number, not at a value picked by eye from a handful of examples.

ComplianceRoutingService.cs / compliance_routing_service.py

public class ComplianceRoutingService
{
    private readonly IModelClient _cheap;
    private readonly IModelClient _capable;
    private readonly ICostLedger _ledger;
    private const double ConfidenceThreshold = 0.82; // set from the harness report, not guessed

    public ComplianceRoutingService(IModelClient cheap, IModelClient capable, ICostLedger ledger)
    {
        _cheap = cheap; _capable = capable; _ledger = ledger;
    }

    public async Task<ModelResponse> ClassifyAsync(string listingText, string listingId)
    {
        var cheapResponse = await _cheap.ClassifyAsync(listingText);

        bool needsEscalation = !cheapResponse.SchemaValid
            || cheapResponse.SelfReportedConfidence < ConfidenceThreshold;

        if (!needsEscalation)
        {
            await _ledger.RecordAsync(listingId, ModelTier.Cheap, cheapResponse);
            return cheapResponse;
        }

        var capableResponse = await _capable.ClassifyAsync(listingText);
        await _ledger.RecordAsync(listingId, ModelTier.Capable, capableResponse);
        return capableResponse;
    }
}

CONFIDENCE_THRESHOLD = 0.82  # set from the harness report, not guessed

class ComplianceRoutingService:
    def __init__(self, cheap: "ModelClient", capable: "ModelClient", ledger: "CostLedger"):
        self._cheap = cheap
        self._capable = capable
        self._ledger = ledger

    async def classify(self, listing_text: str, listing_id: str) -> ModelResponse:
        cheap_response = await self._cheap.classify(listing_text)

        needs_escalation = (
            not cheap_response.schema_valid
            or cheap_response.self_reported_confidence < CONFIDENCE_THRESHOLD
        )

        if not needs_escalation:
            await self._ledger.record(listing_id, ModelTier.CHEAP, cheap_response)
            return cheap_response

        capable_response = await self._capable.classify(listing_text)
        await self._ledger.record(listing_id, ModelTier.CAPABLE, capable_response)
        return capable_response

The cheap call still happens on every escalated request. That's deliberate, not wasted spend — at Haiku 5.5 pricing the discarded cheap call costs a fraction of a cent, and running it first is what produces the confidence signal that triggers escalation in the first place. A separate pre-classifier that decides "easy or hard" before calling any model is a reasonable alternative, but it's a second model to maintain and its own source of misrouting error — don't add it until the simpler cheap-first pattern is proven insufficient.

What This Actually Costs

At the time of writing, Claude Haiku 5.5 runs $0.10 per million input tokens and $0.50 per million output tokens for prompts under 100K tokens; Claude Sonnet 5.5 runs $2.00 and $10.00 per million respectively — a 20x gap on both input and output. For a compliance check averaging roughly 600 input tokens and 150 output tokens, the capable-only cost is about $0.0027 per request; the cheap-only cost is about $0.000135 — roughly 20x cheaper per call, as the pricing ratio implies.

The number that actually matters isn't either of those — it's the blended cost once routing is in place, which depends entirely on the escalation rate the harness measures. If 80% of real traffic looks like the "clear-cut" eval tier and clears the confidence threshold on the cheap model, the blended cost per request lands close to the cheap price plus a 20% tax at the capable price — still a large reduction from calling the capable model on everything, but nowhere near the full 20x the sticker prices suggest. Reporting routing as "faster and dramatically cheaper" without the escalation rate attached is the single most common way this pattern gets oversold internally; the harness report should always carry that percentage alongside the cost figures, and it should be re-measured whenever the model lineup or the mix of incoming listings shifts meaningfully.

Observability

Every production decision logs which tier actually served the request, the self-reported confidence score, schema validity, latency, and estimated cost — the same fields the harness scores, so a production dashboard and an eval report can sit side by side without translation. What doesn't get logged is the listing text itself in that same record; store a hashed or truncated reference to the listing ID for correlation, and keep the raw text in whatever system already owns it under its own retention policy. A cost-and-routing ledger is infrastructure that outlives any one model generation — it shouldn't also become an unintended second copy of user-submitted content.

Testing

The LLM calls are the only non-deterministic part; everything else — scoring, the escalation decision, cost arithmetic — is ordinary code and gets tested as such, with both model clients mocked to return fixed responses. The test that matters most proves the escalation path actually reaches the capable model rather than silently returning the cheap model's low-confidence answer, via an assertion on the capable client's call count.

ComplianceRoutingServiceTests.cs / test_compliance_routing_service.py

[Fact]
public async Task ClearCutListing_StaysOnCheapModel_NoEscalation()
{
    var cheap = new FakeModelClient(new ModelResponse("compliant", null, 0.95, true, 500, 80, 220));
    var capable = new FakeModelClient(new ModelResponse("compliant", null, 0.99, true, 500, 80, 410));
    var service = new ComplianceRoutingService(cheap, capable, new FakeCostLedger());

    var result = await service.ClassifyAsync("Single-item listing, clean description.", "listing-1");

    Assert.Equal("compliant", result.Decision);
    Assert.Equal(0, capable.CallCount); // proves no escalation happened
}

[Fact]
public async Task BorderlineListing_LowConfidence_EscalatesToCapableModel()
{
    var cheap = new FakeModelClient(new ModelResponse("compliant", null, 0.55, true, 700, 90, 240));
    var capable = new FakeModelClient(new ModelResponse("non_compliant", "BUNDLED_PROHIBITED_ITEM", 0.97, true, 700, 110, 430));
    var service = new ComplianceRoutingService(cheap, capable, new FakeCostLedger());

    var result = await service.ClassifyAsync("Bundle listing mixing two categories.", "listing-2");

    Assert.Equal("non_compliant", result.Decision);
    Assert.Equal(1, capable.CallCount); // escalation called the capable model exactly once
    Assert.Equal(1, cheap.CallCount);   // and the cheap call still happened first
}

async def test_clear_cut_listing_stays_on_cheap_model_no_escalation():
    cheap = FakeModelClient(ModelResponse(
        decision="compliant", reason_code=None, self_reported_confidence=0.95,
        schema_valid=True, input_tokens=500, output_tokens=80, latency_ms=220))
    capable = FakeModelClient(ModelResponse(
        decision="compliant", reason_code=None, self_reported_confidence=0.99,
        schema_valid=True, input_tokens=500, output_tokens=80, latency_ms=410))
    service = ComplianceRoutingService(cheap, capable, FakeCostLedger())

    result = await service.classify("Single-item listing, clean description.", "listing-1")

    assert result.decision == "compliant"
    assert capable.call_count == 0  # proves no escalation happened

async def test_borderline_listing_low_confidence_escalates_to_capable_model():
    cheap = FakeModelClient(ModelResponse(
        decision="compliant", reason_code=None, self_reported_confidence=0.55,
        schema_valid=True, input_tokens=700, output_tokens=90, latency_ms=240))
    capable = FakeModelClient(ModelResponse(
        decision="non_compliant", reason_code="BUNDLED_PROHIBITED_ITEM", self_reported_confidence=0.97,
        schema_valid=True, input_tokens=700, output_tokens=110, latency_ms=430))
    service = ComplianceRoutingService(cheap, capable, FakeCostLedger())

    result = await service.classify("Bundle listing mixing two categories.", "listing-2")

    assert result.decision == "non_compliant"
    assert capable.call_count == 1  # escalation called the capable model exactly once
    assert cheap.call_count == 1    # and the cheap call still happened first

Technology Choices

C# with ASP.NET Core suits the production routing path well — the escalation decision sits in a request-scoped service, the cost ledger is a straightforward EF Core write, and the typed records (`EvalTask`, `ModelResponse`, `EvalResult`) catch a mismatched field at compile time rather than at the first bad production response. The offline harness is naturally a .NET worker service or console job triggered from CI whenever the eval set or prompts change.

Python with FastAPI and Pydantic is the stronger choice when the eval harness itself is the primary deliverable rather than the production service — notebook-style analysis of the `EvalResult` set, quick plotting of confidence against correctness to pick the threshold, and integration with whatever ML tooling already exists on the team lean Python. Both stacks share the same pricing table and the same scoring logic conceptually; which one owns the harness versus the router is a team-fit decision, not a correctness one.

Engineering Trade-offs

Routing pays off when three conditions hold together: meaningful volume (the saved cost per request has to multiply into something worth the added complexity), a genuine mix of easy and hard cases (if every request is borderline, there's nothing to route away from the capable model), and a task where "good enough" has a measurable, defensible threshold. The listing-compliance case fits all three — high daily volume, a real split between obvious and ambiguous listings, and a correctness bar that can be set and checked.

Skip it when any of those is missing. Low volume means the savings never add up to more than the maintenance cost of a second model integration, a pricing table, and a threshold that needs periodic re-tuning. A task where every case is already hard — fraud investigation write-ups, say — has no cheap tier worth building the harness for. And anywhere a wrong answer is disproportionately expensive to get wrong — safety-critical moderation decisions, financial approvals — the engineering answer is to keep the capable model as the default and treat cost savings as a non-goal, not to lower the confidence threshold until the savings look good on paper.

Key Takeaways

  • A fixed, stratified eval set is what turns "is the cheap model good enough" from an impression into a measured, reproducible answer — and it only stays useful if new failures are added as dated extensions, not silent edits to the baseline.
  • Score structured fields by exact match and judgment fields by rubric; tag every result with which method produced it so the two scores are never treated as equally precise.
  • The production router never has ground truth — only schema validity and self-reported confidence are available at request time, and the harness's job is to show how well those predict actual correctness.
  • Cheap-first-then-escalate beats a separate pre-classifier for most teams: it needs no second model to maintain, and the "wasted" cheap call is what generates the confidence signal driving escalation.
  • The blended production cost depends entirely on the real escalation rate, not the sticker-price ratio between models — always report both together.
  • Model selection isn't a one-time decision; it's a report (the harness output) feeding a tunable parameter (the confidence threshold) that moves every time the model lineup or the traffic mix changes.

Interview Questions

How would you decide whether a task can use a cheaper model?
Build a fixed, labeled eval set stratified across the task's real difficulty range, run it through every candidate model with identical prompts and output constraints, and score correctness, schema validity, latency, and cost per model. The decision comes from that report against a minimum acceptable quality bar, not from spot-checking a handful of examples by eye.

What makes an LLM benchmark trustworthy versus misleading?
It stays fixed once built, so it measures model/prompt changes rather than drifting alongside them; it's stratified by real difficulty rather than sampled from whatever traffic happened to arrive; and it scores each field with a method that matches what that field actually is — exact match for structured values, a rubric or judge model for free text — rather than one scoring approach applied uniformly for simplicity.

How do you reason about latency, quality, and cost together when picking a model?
None of the three can be optimized in isolation — a cheaper model that's also faster but fails the correctness bar isn't a trade-off, it's just wrong for the task. The useful framing is: fix the minimum acceptable quality first, then compare cost and latency only among models that clear it, and report cost as the blended production number, not the per-call sticker price.

How would you design a system that routes requests between a cheap and a capable model?
Call the cheap model first on every request, validate the response against the schema and a confidence signal set from the eval harness, and escalate to the capable model only when that check fails. Avoid a separate pre-classifier that guesses difficulty before calling any model — it adds a second model to maintain and its own misrouting error, and is rarely justified until the simpler pattern proves insufficient.

Why does reproducibility matter in model evaluation, and how do you preserve it?
Without a fixed eval set, "the cheap model got better" and "we made the eval set easier" are indistinguishable, which quietly erodes the quality bar over time. Preserve it by versioning the eval set, labeling tasks once and never editing existing labels, and adding newly discovered failure cases only as dated, separate additions rather than modifications to the original baseline.

Written by Jaffar Kazi, a software engineer in Sydney building AI-powered applications. Connect on LinkedIn.