The Problem
Nobody sets out to build an expensive LLM feature. It happens one reasonable decision at a time. The system prompt grows a paragraph every time someone finds an edge case. The full conversation history gets sent on every turn because that's the easiest way to keep the model "aware". The retrieval step returns ten passages because ten felt safer than three. The model replies in friendly prose that the backend then parses with a regex. Each choice is defensible. Together they produce a request that carries around 14,000 input tokens to answer a question that needed 1,500.
The running example here is an internal IT helpdesk assistant: an employee asks about VPN access, a laptop refresh, or a locked account, and the service answers using a handful of knowledge-base passages and the conversation so far, returning a short structured result. It works. It's also slow and far more expensive than it should be, and nobody can say exactly why, because the only number anyone looks at is the monthly invoice.
The naive fix is to start cutting: trim the prompt, drop some history, lower the retrieval count. That's a guess in the same way "just use a cheaper model" is a guess. Without measurement you can't tell which cut saved a lot and which one quietly broke answers for a whole category of questions. Token optimisation is an engineering exercise, not a clean-up job: measure, change one thing, measure again, and prove quality didn't move.
The Shape of a Real Solution
The approach is deliberately boring. Start with a baseline that's honestly inefficient, because you need something to measure against. Instrument it so every request reports input tokens by section (system prompt, history, retrieved passages, user message), output tokens, cost, and latency. Then apply one lever at a time, in order of likely payoff, and re-run the same fixed evaluation set after each change.
- Shorter prompts: remove redundant instructions and examples that no longer earn their tokens.
- Better context selection: cap retrieval at the passages that actually matter, within a token budget.
- Conversation summarisation: replace old turns with a rolling summary, keep recent turns verbatim.
- Structured, bounded output: a small schema and a
max_tokensceiling instead of free prose. - Caching: put the stable part of the prompt first so the provider can reuse it.
- Model selection: send easy requests to a cheaper model first (covered in depth in the previous article on cheap versus capable models).
Two rules hold the whole thing together. A change is only kept if the eval set says quality didn't regress, and the savings are reported per request, not as a percentage of a monthly bill that also contains everything else.
Architecture Overview
The request path is a context builder feeding a prompt assembler, with a meter and an eval guard sitting after the model call. The measurement loop is what turns this from a clever pipeline into something you can iterate on safely.
Notice the dotted line back from the report. Each optimisation is a switch that the report either keeps or reverts. Nothing in the pipeline is "always on" just because it sounded like a good idea.
Core Implementation
Before optimising anything, build the meter. It records tokens per prompt section, and computes cost from the pricing table, with cached input billed at its own rate. Per-section counts come from a local tokenizer (they're estimates, and tokenizers differ between model families). The totals that drive cost should always come from the usage figures the provider returns.
public record SectionTokens(int System, int History, int Retrieval, int User)
{
public int Total => System + History + Retrieval + User;
}
public record Usage(SectionTokens Sections, int Output, int CachedInput = 0)
{
public int Input => Sections.Total;
public int Total => Input + Output;
}
public record Pricing(decimal InputPerMillion, decimal OutputPerMillion, decimal CachedInputPerMillion);
public record MeterRecord(string Variant, Usage Usage, decimal CostUsd, long LatencyMs);
public class TokenMeter
{
private readonly List<MeterRecord> _records = new();
public MeterRecord Record(string variant, Usage usage, Pricing p, long latencyMs)
{
var uncached = usage.Input - usage.CachedInput;
var cost = (uncached * p.InputPerMillion
+ usage.CachedInput * p.CachedInputPerMillion
+ usage.Output * p.OutputPerMillion) / 1_000_000m;
var record = new MeterRecord(variant, usage, cost, latencyMs);
_records.Add(record);
return record;
}
public (double AvgInput, double AvgOutput, decimal AvgCost) Summary(string variant)
{
var rows = _records.Where(r => r.Variant == variant).ToList();
return (rows.Average(r => r.Usage.Input),
rows.Average(r => r.Usage.Output),
rows.Average(r => r.CostUsd));
}
}
from dataclasses import dataclass
from statistics import mean
@dataclass(frozen=True)
class SectionTokens:
system: int
history: int
retrieval: int
user: int
@property
def total(self) -> int:
return self.system + self.history + self.retrieval + self.user
@dataclass(frozen=True)
class Usage:
sections: SectionTokens
output: int
cached_input: int = 0
@property
def input(self) -> int:
return self.sections.total
@dataclass(frozen=True)
class Pricing:
input_per_million: float
output_per_million: float
cached_input_per_million: float
@dataclass(frozen=True)
class MeterRecord:
variant: str
usage: Usage
cost_usd: float
latency_ms: int
class TokenMeter:
def __init__(self):
self._records: list[MeterRecord] = []
def record(self, variant: str, usage: Usage, p: Pricing, latency_ms: int) -> MeterRecord:
uncached = usage.input - usage.cached_input
cost = (uncached * p.input_per_million
+ usage.cached_input * p.cached_input_per_million
+ usage.output * p.output_per_million) / 1_000_000
rec = MeterRecord(variant, usage, cost, latency_ms)
self._records.append(rec)
return rec
def summary(self, variant: str) -> dict:
rows = [r for r in self._records if r.variant == variant]
return {
"avg_input": mean(r.usage.input for r in rows),
"avg_output": mean(r.usage.output for r in rows),
"avg_cost": mean(r.cost_usd for r in rows),
}
Every optimisation then becomes a named configuration of the same pipeline. The baseline is deliberately wasteful, and each preset below changes exactly what its name says, so a before/after comparison isolates one variable.
public record PipelineOptions(
string SystemPromptVersion, // "verbose-v1" | "tight-v2"
int MaxPassages,
int PassageTokenBudget, // total tokens allowed for retrieval
bool SummariseHistory,
int RecentTurnsKept,
bool StructuredOutput,
int MaxOutputTokens)
{
// Deliberately inefficient: this is the thing we measure against.
public static readonly PipelineOptions Baseline =
new("verbose-v1", MaxPassages: 10, PassageTokenBudget: int.MaxValue,
SummariseHistory: false, RecentTurnsKept: int.MaxValue,
StructuredOutput: false, MaxOutputTokens: 1024);
public PipelineOptions WithTightPrompt() => this with { SystemPromptVersion = "tight-v2" };
public PipelineOptions WithRetrievalLimit() => this with { MaxPassages = 4, PassageTokenBudget = 1600 };
public PipelineOptions WithSummarisation() => this with { SummariseHistory = true, RecentTurnsKept = 2 };
public PipelineOptions WithStructured() => this with { StructuredOutput = true, MaxOutputTokens = 200 };
}
from dataclasses import dataclass, replace
@dataclass(frozen=True)
class PipelineOptions:
system_prompt_version: str # "verbose-v1" | "tight-v2"
max_passages: int
passage_token_budget: int # total tokens allowed for retrieval
summarise_history: bool
recent_turns_kept: int
structured_output: bool
max_output_tokens: int
def with_tight_prompt(self): return replace(self, system_prompt_version="tight-v2")
def with_retrieval_limit(self): return replace(self, max_passages=4, passage_token_budget=1600)
def with_summarisation(self): return replace(self, summarise_history=True, recent_turns_kept=2)
def with_structured(self): return replace(self, structured_output=True, max_output_tokens=200)
# Deliberately inefficient: this is the thing we measure against.
BASELINE = PipelineOptions(
system_prompt_version="verbose-v1", max_passages=10, passage_token_budget=10**9,
summarise_history=False, recent_turns_kept=10**9,
structured_output=False, max_output_tokens=1024,
)
Running the comparison is then a simple loop. For each preset, replay the same fixed set of helpdesk conversations, record one meter entry per request, and score each answer against its label. The report prints average input, output, cost, and eval score for every variant side by side. If a step saves 5,000 tokens but the score drops on the multi-turn conversations, you've learned the change is lossy for that category, and you fix or revert it before it gets anywhere near production.
Order matters too. Go after the biggest blocks first. In this example that means retrieval and history before prompt wording, because trimming 700 tokens from a system prompt is pleasant, while moving thousands of tokens out of retrieval and history is where the real saving sits. The meter's per-section counts tell you which block that is for your own workload, instead of leaving it to instinct.
Key Technical Challenge: Selecting Context Instead of Stuffing It
Retrieval is usually the biggest single block of input tokens, and the least examined. The tricky part is that "top 10" isn't wrong, it's just unpriced. Passages ranked 5 to 10 rarely change the answer, but they cost the same per token as passage one, and long contexts also make it easier for the model to latch onto the wrong passage.
The fix is to pack by budget, not by count. Take passages in rank order, drop anything below a minimum relevance score, trim each passage to its leading sentences, and stop when the token budget is spent. The budget is the control, and the count falls out of it.
public record Passage(string Id, string Text, double Score);
public class ContextPacker
{
private readonly ITokenCounter _tokens;
private const double MinScore = 0.35;
public ContextPacker(ITokenCounter tokens) => _tokens = tokens;
public List<Passage> Pack(IEnumerable<Passage> ranked, int maxPassages, int tokenBudget)
{
var packed = new List<Passage>();
var remaining = tokenBudget;
foreach (var p in ranked.OrderByDescending(p => p.Score))
{
if (packed.Count >= maxPassages || p.Score < MinScore) break;
var text = TrimToBudget(p.Text, Math.Min(remaining, 400));
var cost = _tokens.Count(text);
if (cost == 0 || cost > remaining) break;
packed.Add(p with { Text = text });
remaining -= cost;
}
return packed;
}
// Keep leading sentences until the per-passage cap is reached.
private string TrimToBudget(string text, int cap)
{
var kept = new StringBuilder();
foreach (var sentence in text.Split(". "))
{
var candidate = kept.Length == 0 ? sentence : kept + ". " + sentence;
if (_tokens.Count(candidate) > cap) break;
kept.Clear().Append(candidate);
}
return kept.ToString();
}
}
from dataclasses import dataclass, replace
MIN_SCORE = 0.35
@dataclass(frozen=True)
class Passage:
id: str
text: str
score: float
class ContextPacker:
def __init__(self, tokens): # tokens: object with .count(str) -> int
self._tokens = tokens
def pack(self, ranked: list[Passage], max_passages: int, token_budget: int) -> list[Passage]:
packed: list[Passage] = []
remaining = token_budget
for p in sorted(ranked, key=lambda p: p.score, reverse=True):
if len(packed) >= max_passages or p.score < MIN_SCORE:
break
text = self._trim_to_budget(p.text, min(remaining, 400))
cost = self._tokens.count(text)
if cost == 0 or cost > remaining:
break
packed.append(replace(p, text=text))
remaining -= cost
return packed
# Keep leading sentences until the per-passage cap is reached.
def _trim_to_budget(self, text: str, cap: int) -> str:
kept = ""
for sentence in text.split(". "):
candidate = sentence if not kept else f"{kept}. {sentence}"
if self._tokens.count(candidate) > cap:
break
kept = candidate
return kept
Trimming Can Remove the Answer
Keeping only the leading sentences of a passage assumes the answer sits near the top. That's usually true for well-written help articles and false for long reference pages. This is exactly the sort of change the eval set has to vet, with questions whose answers sit deep in a passage.
Key Technical Challenge: Summarising History Without Losing the Facts
Conversation history grows linearly, and since it's resent every turn, the total tokens spent across a conversation grow roughly quadratically. A ten-turn support chat resends turn one ten times. Summarising old turns fixes that, but a careless summary is lossy in exactly the wrong places: ticket numbers, device names, and the "I already tried restarting" detail that stops the assistant from suggesting it again.
Two guards help. Only summarise when history crosses a token threshold, so short conversations never pay for a summarisation call. And pull identifiers out of the old turns with a deterministic pattern and carry them forward verbatim, so a summary can lose phrasing but never a ticket number.
public record Turn(string Role, string Content);
public class HistoryCompactor
{
private static readonly Regex Identifiers =
new(@"\b(?:INC|REQ|CHG)\d{6,}\b|\b[A-Z]{2,4}-\d{3,6}\b", RegexOptions.Compiled);
private readonly ITokenCounter _tokens;
private readonly ISummariserClient _summariser; // cheap model
private const int Threshold = 1500;
public HistoryCompactor(ITokenCounter tokens, ISummariserClient summariser)
=> (_tokens, _summariser) = (tokens, summariser);
public async Task<(string? Summary, List<Turn> Recent)> CompactAsync(
List<Turn> history, int recentKept)
{
var total = history.Sum(t => _tokens.Count(t.Content));
if (total <= Threshold || history.Count <= recentKept)
return (null, history); // no summarisation call at all
var older = history.Take(history.Count - recentKept).ToList();
var recent = history.Skip(history.Count - recentKept).ToList();
var pinned = older.SelectMany(t => Identifiers.Matches(t.Content).Select(m => m.Value))
.Distinct().ToList();
var summary = await _summariser.SummariseAsync(older, maxTokens: 250);
if (pinned.Count > 0)
summary += "\nReferences mentioned: " + string.Join(", ", pinned);
return (summary, recent);
}
}
import re
from dataclasses import dataclass
IDENTIFIERS = re.compile(r"\b(?:INC|REQ|CHG)\d{6,}\b|\b[A-Z]{2,4}-\d{3,6}\b")
THRESHOLD = 1500
@dataclass(frozen=True)
class Turn:
role: str
content: str
class HistoryCompactor:
def __init__(self, tokens, summariser): # summariser wraps the cheap model
self._tokens = tokens
self._summariser = summariser
async def compact(self, history: list[Turn], recent_kept: int):
total = sum(self._tokens.count(t.content) for t in history)
if total <= THRESHOLD or len(history) <= recent_kept:
return None, history # no summarisation call at all
older = history[: len(history) - recent_kept]
recent = history[len(history) - recent_kept :]
pinned = list(dict.fromkeys(
m for t in older for m in IDENTIFIERS.findall(t.content)))
summary = await self._summariser.summarise(older, max_tokens=250)
if pinned:
summary += "\nReferences mentioned: " + ", ".join(pinned)
return summary, recent
Key Technical Challenge: Prompt Shape, Caching and Bounded Output
Two things happen at the edges of the prompt. At the front, providers that support prompt caching reuse a stable prefix, but only if the prefix is byte-identical across requests. One timestamp, one user name, or one reordered instruction early in the prompt and every request is a cache miss. At the back, output tokens cost more per token than input tokens and dominate latency, so free-form prose is the most expensive way to return four fields.
The assembler enforces both. Everything stable (instructions, schema, policy) goes first, everything per-request goes last, and a hash of the prefix is exposed so a test can prove it never drifts. The response is a small schema with short enum values, capped by max_tokens.
public record HelpdeskAnswer(
string Answer, // 1-3 sentences
string Category, // "access" | "hardware" | "software" | "other"
bool NeedsHuman,
List<string> SourceIds);
public class PromptBuilder
{
// Stable: no dates, no user names, no per-request values. Ever.
private const string StablePrefix = """
You are an internal IT helpdesk assistant. Answer only from the passages provided.
If they do not contain the answer, set needs_human to true.
Reply as JSON: answer (max 3 sentences), category, needs_human, source_ids.
""";
public string PrefixHash => Convert.ToHexString(
SHA256.HashData(Encoding.UTF8.GetBytes(StablePrefix)));
public (string Prefix, string Suffix) Build(
string? summary, IEnumerable<Turn> recent, IEnumerable<Passage> passages, string question)
{
var suffix = new StringBuilder();
if (summary is not null) suffix.AppendLine($"Conversation so far: {summary}");
foreach (var t in recent) suffix.AppendLine($"{t.Role}: {t.Content}");
foreach (var p in passages) suffix.AppendLine($"[{p.Id}] {p.Text}");
suffix.AppendLine($"Question: {question}");
return (StablePrefix, suffix.ToString());
}
}
import hashlib
from pydantic import BaseModel, Field
from typing import Literal
class HelpdeskAnswer(BaseModel):
answer: str = Field(max_length=400) # 1-3 sentences
category: Literal["access", "hardware", "software", "other"]
needs_human: bool
source_ids: list[str]
# Stable: no dates, no user names, no per-request values. Ever.
STABLE_PREFIX = """You are an internal IT helpdesk assistant. Answer only from the passages provided.
If they do not contain the answer, set needs_human to true.
Reply as JSON: answer (max 3 sentences), category, needs_human, source_ids."""
class PromptBuilder:
@property
def prefix_hash(self) -> str:
return hashlib.sha256(STABLE_PREFIX.encode()).hexdigest()
def build(self, summary, recent, passages, question) -> tuple[str, str]:
lines = []
if summary:
lines.append(f"Conversation so far: {summary}")
lines += [f"{t.role}: {t.content}" for t in recent]
lines += [f"[{p.id}] {p.text}" for p in passages]
lines.append(f"Question: {question}")
return STABLE_PREFIX, "\n".join(lines)
What This Actually Costs
The table below is a worked example, not a benchmark. It models the helpdesk assistant with a plausible request profile, and shows how each lever moves tokens and cost per request. Pricing is for the mid-tier model at the time of writing ($2.00 per million input tokens, $10.00 per million output tokens), and cached input is billed at roughly a tenth of the normal input rate. Check current pricing, it changes, and verify cache terms with your provider.
| Step | What changed | Input tokens | Output tokens | Cost / request |
|---|---|---|---|---|
| 0. Baseline | Verbose prompt, full history, 10 passages, free prose | 13,900 | 450 | $0.0323 |
| 1. Tight prompt | System prompt 1,800 → 1,100 tokens | 13,200 | 450 | $0.0309 |
| 2. Retrieval limit | Passages 6,000 → 1,600 tokens | 8,800 | 450 | $0.0221 |
| 3. Summarised history | History 6,000 → 900 tokens | 3,700 | 450 | $0.0119 |
| 4. Structured output | Output 450 → 120 tokens | 3,700 | 120 | $0.0086 |
| 5. Prefix caching | 1,100-token prefix at ~90% hit rate | 3,700 | 120 | $0.0068 |
| 6. Cheap-first routing | ~70% resolved by a small model, 30% escalate | 3,700 | 120 | ~$0.0025 |
Three things stand out. Context work (steps 2 and 3) delivers most of the saving, because that's where most of the tokens were. Caching is modest here only because the stable prefix is small; with a long stable prefix full of examples or policy, it matters far more. And the final figure, about $0.0025 against $0.0323, is roughly a 13x reduction. At 100,000 requests a month that's about $3,230 down to about $250, with the caveat that every number here depends on the assumed profile. Your table will look different, which is exactly why you build the meter first.
One thing the table doesn't show is latency. It generally falls with both input and output tokens, with output tokens dominating, but measure it rather than assuming. The meter records it per request.
Observability
Log token counts by prompt section, never the prompt text. Section counts are enough to answer "why did this request cost so much" (the history section blew up, or retrieval returned long passages), and they carry none of the personal or confidential content that helpdesk conversations contain.
public static class Telemetry
{
private static readonly ActivitySource Source = new("TokenOptimisation");
public static void Tag(Activity? a, string variant, MeterRecord r, string prefixHash, bool summarised)
{
a?.SetTag("variant", variant);
a?.SetTag("tokens.system", r.Usage.Sections.System);
a?.SetTag("tokens.history", r.Usage.Sections.History);
a?.SetTag("tokens.retrieval", r.Usage.Sections.Retrieval);
a?.SetTag("tokens.user", r.Usage.Sections.User);
a?.SetTag("tokens.output", r.Usage.Output);
a?.SetTag("tokens.cached_input", r.Usage.CachedInput);
a?.SetTag("cost_usd", (double)r.CostUsd);
a?.SetTag("latency_ms", r.LatencyMs);
a?.SetTag("prefix_hash", prefixHash); // drift here means cache misses
a?.SetTag("history_summarised", summarised);
// Deliberately not logged: prompt text, user messages, passage text.
}
}
from opentelemetry import trace
tracer = trace.get_tracer("token-optimisation")
def tag(span, variant: str, r: MeterRecord, prefix_hash: str, summarised: bool) -> None:
s = r.usage.sections
span.set_attribute("variant", variant)
span.set_attribute("tokens.system", s.system)
span.set_attribute("tokens.history", s.history)
span.set_attribute("tokens.retrieval", s.retrieval)
span.set_attribute("tokens.user", s.user)
span.set_attribute("tokens.output", r.usage.output)
span.set_attribute("tokens.cached_input", r.usage.cached_input)
span.set_attribute("cost_usd", r.cost_usd)
span.set_attribute("latency_ms", r.latency_ms)
span.set_attribute("prefix_hash", prefix_hash) # drift here means cache misses
span.set_attribute("history_summarised", summarised)
# Deliberately not logged: prompt text, user messages, passage text.
The alerts worth setting are on cost per request at p95 (a handful of huge conversations can hide behind a healthy average), the cache-hit ratio, and the fraction of conversations that trigger summarisation. A sudden drop in cache hits almost always means someone edited the stable prefix.
Testing
The model call is the only non-deterministic piece. Everything else, including budget packing, the summarisation trigger, prefix stability, and cost arithmetic, is ordinary code and is tested with the model clients mocked. The summariser test matters most: it asserts the call count, proving the model is invoked when history is long and not invoked when it's short. A test that only checks the returned history would pass even if the summariser ran on every request and quietly wasted the savings.
[Fact]
public void Packer_NeverExceedsTokenBudget()
{
var packer = new ContextPacker(new WordTokenCounter()); // 1 word = 1 token in tests
var passages = Enumerable.Range(0, 10)
.Select(i => new Passage($"KB-{i}", string.Join(" ", Enumerable.Repeat("word", 300)) + ".", 0.9 - i * 0.01));
var packed = packer.Pack(passages, maxPassages: 10, tokenBudget: 700);
Assert.True(packed.Sum(p => p.Text.Split(' ').Length) <= 700);
}
[Fact]
public async Task ShortHistory_DoesNotCallSummariser()
{
var summariser = new FakeSummariser();
var compactor = new HistoryCompactor(new WordTokenCounter(), summariser);
var (summary, recent) = await compactor.CompactAsync(ShortHistory(turns: 3), recentKept: 2);
Assert.Null(summary);
Assert.Equal(0, summariser.CallCount);
}
[Fact]
public async Task LongHistory_SummarisesOnce_AndKeepsTicketNumbers()
{
var summariser = new FakeSummariser("User has a VPN problem.");
var compactor = new HistoryCompactor(new WordTokenCounter(), summariser);
var history = LongHistory(turns: 12, includes: "INC0012345");
var (summary, recent) = await compactor.CompactAsync(history, recentKept: 2);
Assert.Equal(1, summariser.CallCount);
Assert.Equal(2, recent.Count);
Assert.Contains("INC0012345", summary); // deterministic pin survives a lossy summary
}
[Fact]
public void StablePrefix_HashIsIdenticalAcrossRequests()
{
var a = new PromptBuilder().PrefixHash;
var b = new PromptBuilder().PrefixHash;
Assert.Equal(a, b);
}
[Fact]
public void Meter_BillsCachedInputAtCachedRate()
{
var pricing = new Pricing(2.00m, 10.00m, 0.20m);
var usage = new Usage(new SectionTokens(1000, 0, 0, 0), Output: 0, CachedInput: 1000);
var record = new TokenMeter().Record("t", usage, pricing, 100);
Assert.Equal(0.0002m, record.CostUsd);
}
def test_packer_never_exceeds_token_budget():
packer = ContextPacker(WordTokenCounter()) # 1 word = 1 token in tests
passages = [Passage(f"KB-{i}", " ".join(["word"] * 300) + ".", 0.9 - i * 0.01)
for i in range(10)]
packed = packer.pack(passages, max_passages=10, token_budget=700)
assert sum(len(p.text.split()) for p in packed) <= 700
async def test_short_history_does_not_call_summariser():
summariser = FakeSummariser()
compactor = HistoryCompactor(WordTokenCounter(), summariser)
summary, recent = await compactor.compact(short_history(turns=3), recent_kept=2)
assert summary is None
assert summariser.call_count == 0
async def test_long_history_summarises_once_and_keeps_ticket_numbers():
summariser = FakeSummariser("User has a VPN problem.")
compactor = HistoryCompactor(WordTokenCounter(), summariser)
history = long_history(turns=12, includes="INC0012345")
summary, recent = await compactor.compact(history, recent_kept=2)
assert summariser.call_count == 1
assert len(recent) == 2
assert "INC0012345" in summary # deterministic pin survives a lossy summary
def test_stable_prefix_hash_is_identical_across_requests():
assert PromptBuilder().prefix_hash == PromptBuilder().prefix_hash
def test_meter_bills_cached_input_at_cached_rate():
pricing = Pricing(2.00, 10.00, 0.20)
usage = Usage(SectionTokens(1000, 0, 0, 0), output=0, cached_input=1000)
record = TokenMeter().record("t", usage, pricing, 100)
assert record.cost_usd == pytest.approx(0.0002)
Tests like these prove the mechanics, not the quality. Quality is the job of the eval set, which every optimisation has to pass before it's kept. Keep that set fixed and versioned, so a drop in score means the pipeline got worse, not that the goalposts moved.
Technology Choices
C#/.NET Implementation
Why choose C#: If your backend is .NET, you get first-party Microsoft support and enterprise patterns.
- Native Azure integration — first-party SDKs, Microsoft-maintained, with tokenizer support in
Microsoft.ML.Tokenizers - Enterprise patterns — dependency injection makes the options presets and the swappable
ITokenCountereasy to wire and test - Production-ready — battle-tested at scale, with OpenTelemetry built into the platform
Python Implementation
Why choose Python: If your team writes Python, you get access to the richest AI/ML ecosystem.
- Library ecosystem — tokenizers such as
tiktokenand evaluation tooling are Python-first - Rapid prototyping — quick iteration on prompt variants, with notebooks for comparing before/after runs
- Community — most prompt-caching and context-engineering examples are written in Python first
The Bottom Line
.NET team? Use C#. Python team? Use Python. The pattern is the same either way. The decision that matters is whether you've built the meter and the eval set, not which language runs them.
Engineering Trade-offs
When this pays off: high request volume, long conversations, retrieval-heavy prompts, or prompts that have grown by accretion. The sign to look for is a large gap between the tokens you send and the tokens that influenced the answer.
What it costs you: each lever adds a moving part. Summarisation adds a model call and a failure mode. Aggressive trimming adds a way to lose the answer. Caching makes the stable prefix fragile, so an innocent edit becomes a cost regression. Each of these needs a test and a metric, or you've traded a visible cost for an invisible risk.
When to Skip It
If volume is low, prompts are already short, or the invoice is a rounding error next to engineering time, leave it alone. Don't summarise history in flows where exact wording matters (legal, compliance, anything quoted back to a user). And never optimise without an eval set. Without one, "cheaper" and "worse" look identical.
Key Takeaways
- Measure first. Record tokens by prompt section, plus cost and latency, before changing anything.
- One lever at a time. Named presets let a before/after comparison isolate a single change.
- Pack context by budget, not count. Retrieval and history are usually the biggest blocks and the least examined.
- Protect the facts when you compress. Carry identifiers forward deterministically so lossy summaries can't drop them.
- Design the prompt for the cache. Stable content first, per-request content last, and test that the prefix never drifts.
- Quality gates every change. A saving only counts if the fixed eval set agrees nothing regressed.
Interview Questions
How would you reduce the token cost of an LLM application without hurting quality?
Measure tokens by prompt section first, then apply one lever at a time (tighter prompts, budgeted retrieval, summarised history, bounded structured output, caching, model routing) and re-run a fixed eval set after each change. Keep a change only if quality holds, and report savings per request.
What counts toward an LLM's context window and cost?
Everything sent as input (system prompt, history, retrieved text, the user message, tool definitions) plus everything generated as output. Output tokens are typically priced higher and drive latency, so verbose responses cost more than verbose prompts of the same length.
How does prompt caching work, and how do you structure prompts to benefit from it?
Providers can reuse the computed prefix of a prompt when a later request starts with identical content. Put stable instructions and examples first, per-request content last, and avoid anything volatile (timestamps, user names) in the prefix, since a single changed byte early on defeats the cache.
What are the trade-offs of summarising conversation history?
It cuts the quadratic growth of resent history but is lossy, adds a model call, and can drop exact details like IDs and figures. Mitigate by summarising only past a threshold, keeping recent turns verbatim, and carrying identifiers forward deterministically.
Why not just retrieve more passages to be safe?
Extra passages cost tokens and latency, and they dilute the context, making it easier for the model to focus on the wrong passage. Retrieving by relevance and a token budget gives you a controllable cost, and the eval set shows whether the extra passages were ever contributing.
How do you verify an optimisation didn't degrade quality?
Run the same fixed, versioned eval set before and after, compare correctness and schema validity alongside cost and latency, and include cases that stress the change (answers deep in a passage, facts from early in a long conversation).