Second article in Real-World AI Engineering. The first covered getting a trustworthy response shape out of an LLM; this one covers something that breaks even when every response is perfectly valid: a feature that works correctly can still bankrupt you, quietly, under load you didn't plan for.
What You'll Learn
- Why "it works" and "it's affordable" are two different claims about the same request
- How to build a usage ledger that records model, tokens, latency, and cost per request
- The race condition hiding in naive budget checks, and the atomic-reservation pattern that fixes it
- Why estimated cost and provider-reported actual cost are different numbers, and why you need both
- When a full multi-tenant budget system is worth building, and when a weekly report is enough
1. The Problem: Correct and Expensive Aren't Opposites
Every request in this series so far has assumed the model call itself is a known, bounded cost. It isn't, once more than one customer is calling your system. A support-reply drafting feature that costs fractions of a cent per request looks free right up until one tenant's usage spikes — a bulk-import job, an automation loop gone slightly wrong, a customer who just has ten times the ticket volume of everyone else — and a single tenant quietly accounts for most of the month's model spend.
The usual starting point is no tracking at all: call the model, return the result, and find out what it cost when the Azure bill arrives. That's fine for a single-user side project. It stops being fine the moment more than one tenant shares the same model deployment, because now "what did this cost" and "who should pay for it" are two different questions, and neither has an answer without a record of every call.
The running example here: a multi-tenant support platform where agents draft replies using an LLM. Each tenant is on a subscription with a monthly AI budget baked into their plan. Three model tiers are available — a cheap default for routine drafts, a mid-tier for more nuanced replies, and a premium tier reserved for top-plan tenants. The system needs to know, in real time, what every request costs, whether a tenant can afford the next one, and what happened last month across every tenant and feature.
2. The Shape of a Real Solution
Four pieces sit between "call the model" and "know what it cost, safely":
- A usage ledger — one record per request: tenant, feature, model, tokens, latency, request ID, cost.
- A cost calculator against a configurable per-model pricing table, since rates differ by tier and change over time.
- A budget gate that runs before the call, not after — checking and reserving spend atomically, so two concurrent requests can't both slip through on stale numbers.
- A reporting layer that aggregates the ledger by tenant, feature, and time period, for both the budget gate and a human-facing cost report.
Stack: Azure OpenAI as the model layer, ASP.NET Core with EF Core for the ledger on the C# side, FastAPI with SQLAlchemy on the Python side. Pricing used in examples below is current-generation Azure OpenAI pricing verified at the time of writing — Luna tier at $0.10/$0.50 per million input/output tokens, Sol at $2.00/$10.00, Astra at $10.00/$50.00 (check current Azure pricing before building a budget around this; it moves).
3. Architecture Overview
The flow has two gates before the model is ever called, and one reconciliation step after:
The token-limit check and the budget reservation are deliberately separate gates — a request can be within budget but still too large (a runaway prompt), or small but still unaffordable (a tenant already at their cap). Each needs its own rejection reason. The reservation happens before the call using an estimate, because you don't know the actual output length yet; the ledger write and reservation finalization happen after, using the provider's reported usage. The reporting endpoint reads the ledger independently — it never blocks a request.
4. Core Implementation: The Usage Ledger
Every downstream decision — budget checks, reports, alerts — reads from this one record shape. Get it right before building anything on top of it.
public record UsageRecord
{
public required Guid RequestId { get; init; }
public required string TenantId { get; init; }
public required string Feature { get; init; } // "draft-reply", "summarize-ticket"
public required string Model { get; init; } // "gpt-6-luna", "gpt-6-sol", "gpt-6-astra"
public required int InputTokens { get; init; }
public required int OutputTokens { get; init; }
public required decimal EstimatedCostUsd { get; init; }
public decimal? ActualCostUsd { get; init; } // set once provider usage is known
public required int LatencyMs { get; init; }
public required DateTimeOffset Timestamp { get; init; }
}
from datetime import datetime
from decimal import Decimal
from uuid import UUID
from pydantic import BaseModel
class UsageRecord(BaseModel):
request_id: UUID
tenant_id: str
feature: str # "draft-reply", "summarize-ticket"
model: str # "gpt-6-luna", "gpt-6-sol", "gpt-6-astra"
input_tokens: int
output_tokens: int
estimated_cost_usd: Decimal
actual_cost_usd: Decimal | None = None # set once provider usage is known
latency_ms: int
timestamp: datetime
Two fields that aren't obvious until you've built this once. EstimatedCostUsd and ActualCostUsd are separate fields, not one field updated in place — Section 6 explains why keeping both matters. And Feature is a plain string tag, not an enum, deliberately: new features get added to the product faster than you want to ship a schema migration for a budget ledger, and feature-level cost breakdown is a reporting concern, not a validation one.
5. Configurable Pricing and Cost Calculation
Rates vary by model and change over time, so they live in one lookup, not scattered through the codebase as magic numbers.
public static class ModelPricing
{
// USD per 1M tokens — update when Azure's price list changes
private static readonly Dictionary<string, (decimal InPerM, decimal OutPerM)> Rates = new()
{
["gpt-6-luna"] = (0.10m, 0.50m),
["gpt-6-sol"] = (2.00m, 10.00m),
["gpt-6-astra"] = (10.00m, 50.00m),
};
public static decimal EstimateCost(string model, int inputTokens, int outputTokens)
{
var (inRate, outRate) = Rates[model];
return (inputTokens * inRate + outputTokens * outRate) / 1_000_000m;
}
}
from decimal import Decimal
# USD per 1M tokens — update when Azure's price list changes
RATES: dict[str, tuple[Decimal, Decimal]] = {
"gpt-6-luna": (Decimal("0.10"), Decimal("0.50")),
"gpt-6-sol": (Decimal("2.00"), Decimal("10.00")),
"gpt-6-astra": (Decimal("10.00"), Decimal("50.00")),
}
def estimate_cost(model: str, input_tokens: int, output_tokens: int) -> Decimal:
in_rate, out_rate = RATES[model]
return (input_tokens * in_rate + output_tokens * out_rate) / Decimal("1000000")
The detail worth noting: this computes an estimate, because at the point you need a cost figure to check against budget, you only know the input token count for certain — you're about to send the request, not read its response. Output tokens have to be assumed from a configured ceiling (typically your max_tokens request parameter) until the model actually responds.
6. Budget Enforcement and the Race Condition
The naive budget check reads the tenant's current spend, compares it to their limit, and proceeds if there's room. Under concurrency, that's a textbook time-of-check-to-time-of-use race: two requests both read "spend: $48 of $50," both decide there's room, both proceed, and the tenant ends up at $54. Individually harmless; at any real request volume, a tenant with a tight budget will blow through it regularly.
The fix is to make the check-and-commit a single atomic operation against the database, not two separate steps in application code.
public async Task<bool> TryReserveAsync(string tenantId, decimal estimatedCost, CancellationToken ct)
{
// Single atomic statement: the check and the reservation happen together.
// Returns 0 rows affected if the tenant doesn't have room — no separate read-then-write.
const string sql = """
UPDATE tenant_budgets
SET reserved = reserved + @cost
WHERE tenant_id = @tenantId
AND spent + reserved + @cost <= monthly_budget
""";
var rows = await db.ExecuteAsync(sql, new { tenantId, cost = estimatedCost }, ct);
return rows > 0;
}
async def try_reserve(tenant_id: str, estimated_cost: Decimal) -> bool:
# Single atomic statement: the check and the reservation happen together.
# rowcount == 0 means the tenant doesn't have room — no separate read-then-write.
sql = """
UPDATE tenant_budgets
SET reserved = reserved + :cost
WHERE tenant_id = :tenant_id
AND spent + reserved + :cost <= monthly_budget
"""
result = await db.execute(text(sql), {"tenant_id": tenant_id, "cost": estimated_cost})
return result.rowcount > 0
The WHERE clause doing the check and the SET clause doing the write are the same database operation, so there's no window between "check" and "use" for a second request to land in. If the reservation succeeds, reserved now includes this request's estimated cost — a concurrent request reading the row a microsecond later sees the updated total, not the stale one. If the model call later fails outright, release the reservation explicitly (subtract it back out); don't leave reserved budget stranded.
7. Estimated vs. Actual Usage
The reservation in Section 6 used an estimate. Once the model responds, most providers — Azure OpenAI included — report the actual input and output token counts used, and that number can differ from the estimate in either direction: prompt caching can make the actual input cost lower than a naive token count would suggest, and a verbose completion can make actual output higher than your average-case estimate. Treat the provider-reported figure as the one that goes on the bill, and reconcile the reservation to match it rather than leaving the estimate as the permanent record.
public async Task FinalizeAsync(string tenantId, decimal reservedCost, UsageRecord record, CancellationToken ct)
{
var actualCost = ModelPricing.EstimateCost(record.Model, record.InputTokens, record.OutputTokens);
const string sql = """
UPDATE tenant_budgets
SET reserved = reserved - @reserved,
spent = spent + @actual
WHERE tenant_id = @tenantId
""";
await db.ExecuteAsync(sql, new { tenantId, reserved = reservedCost, actual = actualCost }, ct);
await ledger.InsertAsync(record with { ActualCostUsd = actualCost }, ct);
}
async def finalize(tenant_id: str, reserved_cost: Decimal, record: UsageRecord) -> None:
actual_cost = estimate_cost(record.model, record.input_tokens, record.output_tokens)
sql = """
UPDATE tenant_budgets
SET reserved = reserved - :reserved,
spent = spent + :actual
WHERE tenant_id = :tenant_id
"""
await db.execute(text(sql), {"tenant_id": tenant_id, "reserved": reserved_cost, "actual": actual_cost})
record.actual_cost_usd = actual_cost
await ledger.insert(record)
This is also exactly why EstimatedCostUsd and ActualCostUsd stay separate fields in the ledger rather than one field overwritten in place — losing the estimate makes it impossible to later audit how accurate your pre-call estimates actually are, which is the number you'd want before tightening or loosening the estimation logic.
8. What Each Failure Mode Looks Like
| Scenario | What happens |
|---|---|
| Tenant at 70% of budget | Request proceeds normally — no gate triggered |
| Tenant at 92% of budget | Request proceeds, but crosses an alert threshold — a near-limit notification fires (Section 9) |
| Tenant at 100%+ of budget | Reservation fails — request rejected before the model is ever called |
| Unusually large prompt | Rejected by the token-limit gate, independent of remaining budget — a well-funded tenant can still submit a request that's simply too large |
| Estimated vs. actual mismatch | Request succeeds; the ledger's ActualCostUsd differs from EstimatedCostUsd, and the budget is finalized against the actual figure |
9. The Cost-Reporting Endpoint
The ledger exists to be queried, not just written. A minimal reporting endpoint aggregates it by whatever dimension someone's asking about — usually tenant and time period, sometimes feature or model.
app.MapGet("/tenants/{tenantId}/usage", async (string tenantId, DateOnly from, DateOnly to, ILedgerDb db) =>
{
var rows = await db.UsageRecords
.Where(r => r.TenantId == tenantId && r.Timestamp >= from.ToDateTime(TimeOnly.MinValue))
.GroupBy(r => r.Feature)
.Select(g => new
{
Feature = g.Key,
RequestCount = g.Count(),
TotalCostUsd = g.Sum(r => r.ActualCostUsd ?? r.EstimatedCostUsd)
})
.ToListAsync();
return Results.Ok(rows);
});
@app.get("/tenants/{tenant_id}/usage")
async def tenant_usage(tenant_id: str, from_: date, to: date, db: Session = Depends(get_db)):
rows = (
db.query(
UsageRecord.feature,
func.count(UsageRecord.request_id).label("request_count"),
func.sum(func.coalesce(UsageRecord.actual_cost_usd, UsageRecord.estimated_cost_usd)).label("total_cost_usd"),
)
.filter(UsageRecord.tenant_id == tenant_id, UsageRecord.timestamp >= from_)
.group_by(UsageRecord.feature)
.all()
)
return [dict(row._mapping) for row in rows]
Note the ActualCostUsd ?? EstimatedCostUsd fallback — a request that's still mid-flight (reserved but not yet finalized) only has an estimate, and a report that silently excludes in-flight spend will under-report a tenant's real exposure.
10. Testing
The model call is still the only non-deterministic piece — mock it, same as the previous article. What's new here is testing the budget gate's concurrency behavior directly, not just its happy path.
[Fact]
public async Task ConcurrentReservations_NeverExceedBudget()
{
await SeedBudget(tenantId: "t1", budget: 10.00m);
// Fire 20 concurrent reservations of $1 each against a $10 budget
var tasks = Enumerable.Range(0, 20)
.Select(_ => gate.TryReserveAsync("t1", 1.00m, default));
var results = await Task.WhenAll(tasks);
Assert.Equal(10, results.Count(r => r)); // exactly 10 should succeed, not more
}
[Fact]
public async Task OverBudgetRequest_IsRejectedBeforeModelCall()
{
await SeedBudget(tenantId: "t2", budget: 1.00m, alreadySpent: 1.00m);
var approved = await gate.TryReserveAsync("t2", 0.01m, default);
Assert.False(approved);
}
@pytest.mark.asyncio
async def test_concurrent_reservations_never_exceed_budget(seeded_budget):
await seeded_budget("t1", budget=Decimal("10.00"))
# Fire 20 concurrent reservations of $1 each against a $10 budget
results = await asyncio.gather(
*[try_reserve("t1", Decimal("1.00")) for _ in range(20)]
)
assert sum(results) == 10 # exactly 10 should succeed, not more
@pytest.mark.asyncio
async def test_over_budget_request_is_rejected_before_model_call(seeded_budget):
await seeded_budget("t2", budget=Decimal("1.00"), already_spent=Decimal("1.00"))
approved = await try_reserve("t2", Decimal("0.01"))
assert approved is False
The first test is the one that actually matters: it fires real concurrent requests against the gate rather than asserting on logic read in isolation, because a race condition by definition doesn't show up in a single-threaded test. If the atomic-SQL approach from Section 6 were replaced with a naive read-then-write, this test would fail intermittently — exactly the signature of a race condition, as opposed to a deterministic bug that fails every time.
11. Observability
Track the things finance and on-call both care about: cost per tenant per day (trend, not just total), near-limit events (the 92% case from Section 8), rejection counts by reason (budget vs. token-limit — they mean different things operationally), and the gap between estimated and actual cost over time, which tells you whether your estimation logic needs tightening. As before, log metadata, not prompt or completion content — a cost ledger is exactly the kind of thing finance teams get read access to, and prompt content is not something finance needs or should see.
Alert on Trend, Not Just Threshold
A tenant crossing 92% of budget on day 3 of the month is a very different signal from crossing it on day 29. A pure threshold alert treats both the same; a trend-aware one catches the runaway case days earlier, when there's still time to reach out to the customer before a hard rejection surprises them.
12. Technology Choices
C#/.NET Implementation
Why choose C#: EF Core's raw-SQL execution path makes the atomic reservation pattern straightforward, and a billing-adjacent system benefits from the compile-time guarantees around the money-handling types (decimal, not float, throughout).
- Transactional guarantees — EF Core's execution strategy composes cleanly with retry-on-transient-failure for the budget SQL
- Decimal by default — harder to accidentally introduce floating-point rounding into a cost ledger
Python Implementation
Why choose Python: if the reporting consumer is a finance or data team already living in Python/pandas, keeping the ledger and the reporting layer in the same ecosystem removes a translation step.
- Reporting ecosystem — straight line from a SQLAlchemy query to a pandas DataFrame for ad-hoc finance analysis
- Decimal discipline still required — Python's
Decimaleverywhere, neverfloat, for exactly the same reason
The Bottom Line
Either stack works — the pattern (atomic reservation, separate estimate/actual fields, metadata-only logging) is what matters, not the language. Match it to whoever maintains the billing-adjacent code already.
13. Engineering Trade-offs
The full pattern — atomic reservations, separate estimate/actual tracking, per-feature aggregation — earns its complexity once you have more than one tenant whose usage you can't fully trust to stay small, or once AI spend is a line item someone outside engineering is accountable for. For an internal tool with one user and a fixed monthly Azure budget you're personally watching, a daily cron job that emails a usage summary is enough; the atomic reservation pattern is solving a concurrency problem that doesn't exist with a single caller.
Skip the Full Pattern When...
- There's exactly one caller, or all callers trust each other completely
- Nobody outside engineering needs a cost breakdown — a provider-level monthly bill is sufficient
- You can tolerate finding out about a cost spike a day late rather than blocking it in real time
14. Key Takeaways
- A usage ledger — one row per request, with model, tokens, latency, and cost — is the foundation every budget decision and every report reads from.
- Never check-then-write a budget across two steps in application code. Make the check and the reservation one atomic database operation, or concurrent requests will race past the limit.
- Estimated cost (pre-call) and actual cost (post-call, provider-reported) are different numbers for a reason — keep both, and reconcile the budget against the actual figure.
- Token-limit rejections and budget rejections are different failure modes with different causes — don't collapse them into one error.
- Test concurrency directly, with real concurrent calls against the gate — a race condition doesn't show up in a single-threaded test.
- Build the full pattern once more than one tenant or stakeholder depends on accurate, real-time cost control — not before.
Interview Questions
- What's the FinOps problem this solves that a provider's monthly invoice doesn't? A monthly invoice tells you total spend after the fact. It doesn't tell you which tenant or feature drove it, and it can't stop an overspend in progress — by the time it arrives, the money's already spent.
- How do you meter usage per tenant without adding latency to every request? The ledger write and reservation finalization happen after the model responds, off the response's critical path where possible — only the budget-gate check before the call needs to block, and that's a single indexed atomic update, not a full aggregation query.
- How would you enforce a hard quota without over-provisioning or under-utilizing it? An atomic reserve-then-finalize pattern: reserve the estimate before the call so concurrent requests can't double-spend the same headroom, then true up to the actual cost afterward so the tenant isn't left with phantom reserved budget they can't actually use.
- How do you allocate shared infrastructure cost across tenants and features fairly? Tag every usage record with tenant and feature at write time, not after the fact — retroactively attributing cost from aggregate logs is unreliable, while a ledger designed for allocation from day one makes any breakdown a straightforward query.
- Where exactly is the race condition in budget enforcement, and how do you prove you've fixed it? It's in a read-then-write budget check split across two steps — two concurrent requests both read "under budget" before either writes. Fixing it means making the check and the write one atomic statement; proving it means a test that fires genuinely concurrent requests and asserts the total approved never exceeds the limit, since the bug is intermittent by nature and won't show up in a sequential test.
Want More Practical AI Tutorials?
I write about building production AI systems with Azure, Python, and C#. Subscribe for practical tutorials delivered twice a month.
Subscribe to Newsletter →