The Problem
The first version of every PDF-extraction feature is the same: read the PDF, paste all of the text into a prompt, ask for JSON, save it. On the three sample documents in the demo it works beautifully. Then it meets real uploads. A scanned agreement with no text layer returns an empty string and the model cheerfully invents a plausible vendor name. A 140-page master agreement blows past the context window. A document where the renewal clause is simply absent comes back with a confident "auto-renews annually" that appears nowhere in the file. And when the model API times out at 2 a.m., the upload request that triggered it has long since failed in the user's browser.
The running example is a vendor-agreement intake service: users upload a PDF, and the service extracts the parties, effective date, end date, contract value, and renewal terms, then stores the result alongside a processing status. None of that is hard to demo. What makes it an engineering problem is everything around the model call: what to do with documents that have no text, how to handle input that doesn't fit, how to refuse to guess, and how to survive failure without losing or duplicating work.
"Just send the whole document to the model" fails in all four ways at once. It's expensive, it breaks on size, it can't tell you which page a value came from, and it has no concept of a document that should be rejected rather than processed. The pipeline below treats the LLM as one stage among several, each with a clear contract about what it accepts and what happens when it can't proceed.
The Shape of a Real Solution
The fix is a pipeline of small, separately testable stages, run asynchronously so a slow or failing model never blocks an upload.
- Accept fast: validate the upload, store the original in blob storage, record a row with status
Queued, enqueue work, and return202 Accepted. - Extract text per page: keep page numbers from the start, because everything downstream (selection, citations, review) depends on them.
- Gate on text quality: an empty or scanned document stops here with a clear reason, before any model call is made.
- Select what to send: choose the relevant pages within a token budget instead of sending everything.
- Extract with a schema: every field is nullable and carries the page and quote it came from.
- Validate before trusting: check that quotes exist on the cited pages, then apply business rules. Anything missing or conflicting goes to
NeedsReview, never silently toCompleted.
The principle running through all of it is that a refusal is a valid output. A document that can't be processed, or a field that can't be found, is an expected result with its own status, not an exception to hide.
Architecture Overview
The API tier does almost nothing: it validates, stores, records, and enqueues. A background worker owns the slow, failure-prone work, and every path through the worker ends in an explicit, persisted status.
Two details are worth noticing. The quality gate sits before the model, so the cheapest way to handle a bad document is to never spend a token on it. And the retry arrow goes back to the queue, not into a loop inside the request: transient failures are retried with backoff, permanent ones fail fast with a reason a human can read.
Core Implementation
The data model encodes the "refusal is valid" principle. Every extracted field is a value that may be null, plus the page and verbatim quote that support it. The document record carries a status and a machine-readable reason, so the system can always explain itself.
public enum DocStatus { Queued, Processing, Completed, NeedsReview, Failed }
// A value the model claims to have found, with the evidence for it.
public record Extracted<T>(T? Value, int? Page, string? Quote);
public record AgreementExtraction(
Extracted<List<string>> Parties,
Extracted<DateOnly?> EffectiveDate,
Extracted<DateOnly?> EndDate,
Extracted<decimal?> ContractValue,
Extracted<string> Currency,
Extracted<bool?> AutoRenew,
Extracted<int?> RenewalNoticeDays);
public record Issue(string Code, string Message);
public record DocumentRecord(
Guid Id, string FileName, string BlobName, DocStatus Status,
string? Reason = null, AgreementExtraction? Result = null,
List<Issue>? Issues = null);
from datetime import date
from enum import Enum
from typing import Generic, TypeVar
from pydantic import BaseModel
T = TypeVar("T")
class DocStatus(str, Enum):
QUEUED = "queued"
PROCESSING = "processing"
COMPLETED = "completed"
NEEDS_REVIEW = "needs_review"
FAILED = "failed"
# A value the model claims to have found, with the evidence for it.
class Extracted(BaseModel, Generic[T]):
value: T | None = None
page: int | None = None
quote: str | None = None
class AgreementExtraction(BaseModel):
parties: Extracted[list[str]]
effective_date: Extracted[date]
end_date: Extracted[date]
contract_value: Extracted[float]
currency: Extracted[str]
auto_renew: Extracted[bool]
renewal_notice_days: Extracted[int]
class Issue(BaseModel):
code: str
message: str
class DocumentRecord(BaseModel):
id: str
file_name: str
blob_name: str
status: DocStatus
reason: str | None = None
result: AgreementExtraction | None = None
issues: list[Issue] = []
The upload endpoint is deliberately thin. It rejects obviously bad input immediately (empty, oversized, not actually a PDF), stores the original, records the row, enqueues the work, and returns. Nothing slow happens in the request.
const long MaxBytes = 20 * 1024 * 1024;
app.MapPost("/documents", async (IFormFile file, IBlobStore blobs,
IDocumentRepository repo, IWorkQueue queue) =>
{
if (file.Length == 0) return Results.BadRequest(new { error = "empty_file" });
if (file.Length > MaxBytes) return Results.StatusCode(413);
await using var stream = file.OpenReadStream();
var magic = new byte[5];
await stream.ReadExactlyAsync(magic);
if (Encoding.ASCII.GetString(magic) != "%PDF-")
return Results.BadRequest(new { error = "not_a_pdf" });
stream.Position = 0;
var id = Guid.NewGuid();
var blobName = $"{id}.pdf";
await blobs.UploadAsync(blobName, stream);
await repo.CreateAsync(new DocumentRecord(id, file.FileName, blobName, DocStatus.Queued));
await queue.EnqueueAsync(id);
return Results.Accepted($"/documents/{id}", new { id, status = "queued" });
});
app.MapGet("/documents/{id:guid}", async (Guid id, IDocumentRepository repo) =>
await repo.GetAsync(id) is { } doc ? Results.Ok(doc) : Results.NotFound());
MAX_BYTES = 20 * 1024 * 1024
@app.post("/documents", status_code=202)
async def upload(file: UploadFile, blobs: BlobStore = Depends(get_blobs),
repo: DocumentRepository = Depends(get_repo),
queue: WorkQueue = Depends(get_queue)):
data = await file.read()
if not data:
raise HTTPException(400, "empty_file")
if len(data) > MAX_BYTES:
raise HTTPException(413, "file_too_large")
if not data.startswith(b"%PDF-"):
raise HTTPException(400, "not_a_pdf")
doc_id = str(uuid.uuid4())
blob_name = f"{doc_id}.pdf"
await blobs.upload(blob_name, data)
await repo.create(DocumentRecord(id=doc_id, file_name=file.filename,
blob_name=blob_name, status=DocStatus.QUEUED))
await queue.enqueue(doc_id)
return {"id": doc_id, "status": "queued"}
@app.get("/documents/{doc_id}")
async def get_document(doc_id: str, repo: DocumentRepository = Depends(get_repo)):
doc = await repo.get(doc_id)
if doc is None:
raise HTTPException(404)
return doc
Key Technical Challenge: Getting Text Out With Page Numbers, and Knowing When There Isn't Any
Text extraction looks like a solved problem until you meet a scanned PDF. A scan is a set of images, so a text extractor returns pages successfully, with nothing on them. If that empty text reaches the model, it's asked to extract a contract from a blank prompt. A model asked for a JSON object will often produce one.
The quality gate catches this with a simple heuristic: count how many pages have a meaningful amount of text. Zero pages means an empty document. Mostly empty pages means a probable scan. Either way, the document stops before any model call with a specific reason. In practice, you'll find the threshold needs tuning against your own documents, since cover pages and signature pages are legitimately sparse. That's why the gate looks at the fraction of pages with text, not a single page.
using UglyToad.PdfPig;
public record PageText(int Number, string Text);
public enum TextQuality { Usable, Empty, LikelyScanned }
public class PdfTextExtractor
{
public List<PageText> Extract(Stream pdf)
{
using var doc = PdfDocument.Open(pdf);
return doc.GetPages().Select(p => new PageText(p.Number, p.Text)).ToList();
}
}
public static class QualityGate
{
private const int MinCharsPerPage = 80; // sparse pages exist; tune on real documents
private const double MinTextPageRatio = 0.4;
public static TextQuality Assess(IReadOnlyList<PageText> pages)
{
if (pages.Count == 0) return TextQuality.Empty;
var withText = pages.Count(p => p.Text.Trim().Length >= MinCharsPerPage);
return (double)withText / pages.Count < MinTextPageRatio
? TextQuality.LikelyScanned
: TextQuality.Usable;
}
}
from dataclasses import dataclass
from enum import Enum
from io import BytesIO
from pypdf import PdfReader
@dataclass(frozen=True)
class PageText:
number: int
text: str
class TextQuality(str, Enum):
USABLE = "usable"
EMPTY = "empty"
LIKELY_SCANNED = "likely_scanned"
def extract_pages(pdf_bytes: bytes) -> list[PageText]:
reader = PdfReader(BytesIO(pdf_bytes))
return [PageText(i + 1, page.extract_text() or "") for i, page in enumerate(reader.pages)]
MIN_CHARS_PER_PAGE = 80 # sparse pages exist; tune on real documents
MIN_TEXT_PAGE_RATIO = 0.4
def assess_quality(pages: list[PageText]) -> TextQuality:
if not pages:
return TextQuality.EMPTY
with_text = sum(1 for p in pages if len(p.text.strip()) >= MIN_CHARS_PER_PAGE)
if with_text / len(pages) < MIN_TEXT_PAGE_RATIO:
return TextQuality.LIKELY_SCANNED
return TextQuality.USABLE
A scanned document isn't necessarily a failure forever. The natural extension is an OCR stage that runs when the gate reports LikelyScanned, but that's a separate stage with its own cost and its own accuracy problems. Until it exists, the honest outcome is a clear Failed status with the reason no_extractable_text.
Key Technical Challenge: Context Limits and Oversized Input
Even with good text, "send everything" is the wrong default. Most of a vendor agreement is boilerplate that never contains the fields you want: indemnities, notices, governing law. Parties and dates usually appear on the first page, term and renewal near the front or in a dedicated section, and fees in a schedule. Sending 140 pages to find six values is slow and expensive, and gives the model a lot of irrelevant text to be distracted by.
The page selector scores each page by how many field-related terms it contains, always includes the first page, and packs the highest-scoring pages into a token budget, returned in page order so the model reads them as the document is written. Documents that still don't fit are processed in windows, with each field taking its first confident value and any disagreement between windows sent to review.
public class PageSelector
{
private static readonly string[] Signals =
{
"between", "party", "parties", "effective date", "commencement", "term",
"renew", "expire", "terminate", "notice", "fees", "price", "total", "payment", "$", "usd", "aud", "eur"
};
public static int EstimateTokens(string text) => text.Length / 4; // rough; use the provider's count to bill
public List<PageText> Select(IReadOnlyList<PageText> pages, int tokenBudget)
{
var chosen = new List<PageText>();
var remaining = tokenBudget;
// First page almost always names the parties and date.
var first = pages[0];
chosen.Add(first);
remaining -= EstimateTokens(first.Text);
foreach (var page in pages.Skip(1).OrderByDescending(Score))
{
var cost = EstimateTokens(page.Text);
if (Score(page) == 0 || cost > remaining) continue;
chosen.Add(page);
remaining -= cost;
}
return chosen.OrderBy(p => p.Number).ToList(); // keep document order
}
private static int Score(PageText p)
{
var lower = p.Text.ToLowerInvariant();
return Signals.Sum(s => lower.Split(s).Length - 1);
}
}
SIGNALS = ["between", "party", "parties", "effective date", "commencement", "term",
"renew", "expire", "terminate", "notice", "fees", "price", "total",
"payment", "$", "usd", "aud", "eur"]
def estimate_tokens(text: str) -> int:
return len(text) // 4 # rough; use the provider's count to bill
def score(page: PageText) -> int:
lower = page.text.lower()
return sum(lower.count(s) for s in SIGNALS)
def select_pages(pages: list[PageText], token_budget: int) -> list[PageText]:
# First page almost always names the parties and date.
chosen = [pages[0]]
remaining = token_budget - estimate_tokens(pages[0].text)
for page in sorted(pages[1:], key=score, reverse=True):
cost = estimate_tokens(page.text)
if score(page) == 0 or cost > remaining:
continue
chosen.append(page)
remaining -= cost
return sorted(chosen, key=lambda p: p.number) # keep document order
Selection Is a Recall Risk
If the renewal clause sits on a page the scorer ranks low, it's never sent, and the field comes back null. That's the safe failure: the validator flags the missing field and the document goes to review instead of passing with a wrong value. Still, track how often selection causes missing fields, and widen the signal list or the budget when it does.
Key Technical Challenge: Validating Extraction Before Trusting It
A schema guarantees shape, not truth. The model can return perfectly valid JSON containing a date that appears nowhere in the document. Two layers of validation close that gap.
The first is grounding: every extracted value must carry a quote, and that quote must actually appear on the page the model cited. This is a plain string check, and it catches most fabricated values, because a model inventing a field rarely invents a matching quote at the right page. The second is business rules, which are ordinary code: at least two distinct parties, an end date after the effective date, a positive value with a known currency, and a notice period whenever auto-renewal is true. Missing required fields aren't errors in the extraction. They're a reason to route to human review, listed by name.
public class AgreementValidator
{
private static readonly HashSet<string> Currencies = new() { "USD", "AUD", "EUR", "GBP", "NZD", "CAD" };
public List<Issue> Validate(AgreementExtraction x, IReadOnlyList<PageText> pages)
{
var issues = new List<Issue>();
// 1. Required fields present.
if (x.Parties.Value is not { Count: >= 2 }) issues.Add(new("missing_parties", "Fewer than two parties found."));
if (x.EffectiveDate.Value is null) issues.Add(new("missing_effective_date", "No effective date found."));
if (x.ContractValue.Value is null) issues.Add(new("missing_value", "No contract value found."));
// 2. Grounding: each quote must exist on the cited page.
CheckQuote(issues, "parties", x.Parties.Page, x.Parties.Quote, pages);
CheckQuote(issues, "effective_date", x.EffectiveDate.Page, x.EffectiveDate.Quote, pages);
CheckQuote(issues, "contract_value", x.ContractValue.Page, x.ContractValue.Quote, pages);
CheckQuote(issues, "auto_renew", x.AutoRenew.Page, x.AutoRenew.Quote, pages);
// 3. Business rules.
if (x.EffectiveDate.Value is { } start && x.EndDate.Value is { } end && end <= start)
issues.Add(new("end_before_start", "End date is not after the effective date."));
if (x.ContractValue.Value is <= 0)
issues.Add(new("non_positive_value", "Contract value must be positive."));
if (x.ContractValue.Value is not null && !Currencies.Contains(x.Currency.Value ?? ""))
issues.Add(new("unknown_currency", $"Currency '{x.Currency.Value}' is not recognised."));
if (x.AutoRenew.Value == true && x.RenewalNoticeDays.Value is null)
issues.Add(new("missing_notice_period", "Auto-renewal found but no notice period."));
return issues;
}
private static void CheckQuote(List<Issue> issues, string field, int? page, string? quote,
IReadOnlyList<PageText> pages)
{
if (quote is null) return; // absence is reported above
var text = pages.FirstOrDefault(p => p.Number == page)?.Text ?? "";
if (!Normalise(text).Contains(Normalise(quote)))
issues.Add(new($"ungrounded_{field}", $"Quote for {field} not found on page {page}."));
}
private static string Normalise(string s) => Regex.Replace(s, @"\s+", " ").Trim().ToLowerInvariant();
}
import re
CURRENCIES = {"USD", "AUD", "EUR", "GBP", "NZD", "CAD"}
def normalise(s: str) -> str:
return re.sub(r"\s+", " ", s).strip().lower()
def check_quote(issues: list[Issue], field: str, f, pages: list[PageText]) -> None:
if f.quote is None:
return # absence is reported elsewhere
text = next((p.text for p in pages if p.number == f.page), "")
if normalise(f.quote) not in normalise(text):
issues.append(Issue(code=f"ungrounded_{field}",
message=f"Quote for {field} not found on page {f.page}."))
def validate(x: AgreementExtraction, pages: list[PageText]) -> list[Issue]:
issues: list[Issue] = []
# 1. Required fields present.
if not x.parties.value or len(x.parties.value) < 2:
issues.append(Issue(code="missing_parties", message="Fewer than two parties found."))
if x.effective_date.value is None:
issues.append(Issue(code="missing_effective_date", message="No effective date found."))
if x.contract_value.value is None:
issues.append(Issue(code="missing_value", message="No contract value found."))
# 2. Grounding: each quote must exist on the cited page.
for name, f in [("parties", x.parties), ("effective_date", x.effective_date),
("contract_value", x.contract_value), ("auto_renew", x.auto_renew)]:
check_quote(issues, name, f, pages)
# 3. Business rules.
if x.effective_date.value and x.end_date.value and x.end_date.value <= x.effective_date.value:
issues.append(Issue(code="end_before_start", message="End date is not after the effective date."))
if x.contract_value.value is not None and x.contract_value.value <= 0:
issues.append(Issue(code="non_positive_value", message="Contract value must be positive."))
if x.contract_value.value is not None and (x.currency.value or "") not in CURRENCIES:
issues.append(Issue(code="unknown_currency", message=f"Currency '{x.currency.value}' is not recognised."))
if x.auto_renew.value is True and x.renewal_notice_days.value is None:
issues.append(Issue(code="missing_notice_period", message="Auto-renewal found but no notice period."))
return issues
Key Technical Challenge: Async Processing and Failures That Aren't All the Same
Not every failure deserves a retry. A model API returning 429 or a timeout is transient: wait and try again. A corrupt PDF, a document with no text, or output that isn't valid JSON will fail identically every time, so retrying only burns tokens. The processor treats the two classes differently and gives each a distinct, persisted outcome.
Transient errors retry in-process with exponential backoff up to a fixed attempt cap, then fail with llm_unavailable. Unparseable model output goes straight to NeedsReview. A worker crash mid-job is the queue's responsibility (a message that isn't acknowledged becomes visible again), which is why status is written to the database at each step and processing is safe to repeat.
public class DocumentProcessor
{
private const int MaxAttempts = 3;
private const int TokenBudget = 6000;
// Dependencies injected: _repo, _blobs, _extractor, _selector, _llm, _validator, _delay.
public async Task ProcessAsync(Guid id)
{
var doc = await _repo.GetAsync(id);
await _repo.SetStatusAsync(id, DocStatus.Processing);
var pages = _extractor.Extract(await _blobs.OpenAsync(doc!.BlobName));
var quality = QualityGate.Assess(pages);
if (quality != TextQuality.Usable)
{
await _repo.FailAsync(id, $"no_extractable_text:{quality}"); // no model call made
return;
}
var selected = _selector.Select(pages, TokenBudget);
AgreementExtraction? result = null;
for (var attempt = 1; attempt <= MaxAttempts; attempt++)
{
try
{
var raw = await _llm.ExtractAsync(selected);
result = JsonSerializer.Deserialize<AgreementExtraction>(raw);
break;
}
catch (TransientLlmException) when (attempt < MaxAttempts)
{
await _delay.WaitAsync(TimeSpan.FromSeconds(Math.Pow(2, attempt)));
}
catch (TransientLlmException)
{
await _repo.FailAsync(id, "llm_unavailable");
return;
}
catch (JsonException)
{
await _repo.NeedsReviewAsync(id, new() { new("unparseable_output", "Model output was not valid JSON.") });
return;
}
}
var issues = _validator.Validate(result!, pages);
if (issues.Count == 0) await _repo.CompleteAsync(id, result!);
else await _repo.NeedsReviewAsync(id, issues, result);
}
}
import json
MAX_ATTEMPTS = 3
TOKEN_BUDGET = 6000
class DocumentProcessor:
# Dependencies injected: repo, blobs, llm, delay.
async def process(self, doc_id: str) -> None:
doc = await self.repo.get(doc_id)
await self.repo.set_status(doc_id, DocStatus.PROCESSING)
pages = extract_pages(await self.blobs.read(doc.blob_name))
quality = assess_quality(pages)
if quality != TextQuality.USABLE:
await self.repo.fail(doc_id, f"no_extractable_text:{quality.value}") # no model call made
return
selected = select_pages(pages, TOKEN_BUDGET)
result = None
for attempt in range(1, MAX_ATTEMPTS + 1):
try:
raw = await self.llm.extract(selected)
result = AgreementExtraction.model_validate(json.loads(raw))
break
except TransientLlmError:
if attempt < MAX_ATTEMPTS:
await self.delay.wait(2 ** attempt)
continue
await self.repo.fail(doc_id, "llm_unavailable")
return
except (ValueError, ValidationError):
await self.repo.needs_review(doc_id, [Issue(code="unparseable_output",
message="Model output was not valid JSON.")])
return
issues = validate(result, pages)
if not issues:
await self.repo.complete(doc_id, result)
else:
await self.repo.needs_review(doc_id, issues, result)
What This Actually Costs
The table is a worked example, not a benchmark: a 30-page agreement is roughly 18,000 tokens of text, and a well-formed extraction response is about 300 tokens. Prices are for a mid-tier and a small model at the time of writing ($2.00 / $10.00 and $0.10 / $0.50 per million input / output tokens respectively). Check current pricing, it changes.
| Approach | Input tokens | Output tokens | Cost / document | Per 10,000 documents |
|---|---|---|---|---|
| Whole document, mid-tier model | 18,400 | 300 | $0.0398 | $398 |
| Selected pages, mid-tier model | 4,000 | 300 | $0.0110 | $110 |
| Selected pages, small model | 4,000 | 300 | $0.0006 | $6 |
| Small model first, 25% escalate to mid-tier | — | — | ~$0.0033 | ~$33 |
Page selection alone cuts the model bill by about 72%, and the extraction task (find a few fields, quote them) is well suited to a small model, with escalation reserved for documents the validator flags. The cheapest document of all is the one the quality gate rejects before any model call. Two caveats apply. OCR, if you add it, is billed separately by the OCR service per page, so check its pricing rather than assuming it's negligible. And some providers charge a higher per-token rate once a prompt crosses a length threshold, so an oversized input can cost more per token as well as more tokens.
Observability
Log what happened to each document, never what was in it. Agreements contain names, amounts, and commercial terms that don't belong in a log aggregator. Page counts, token estimates, status transitions, attempt counts, and issue codes are enough to debug almost everything.
public static class PipelineTelemetry
{
public static readonly ActivitySource Source = new("PdfPipeline");
private static readonly Meter AppMeter = new("PdfPipeline");
private static readonly Counter<long> Outcomes = AppMeter.CreateCounter<long>("documents_processed");
public static void Record(Activity? a, Guid id, int pages, int selectedPages,
int estimatedTokens, int attempts, DocStatus status, IEnumerable<Issue>? issues)
{
a?.SetTag("document_id", id.ToString());
a?.SetTag("pages.total", pages);
a?.SetTag("pages.selected", selectedPages);
a?.SetTag("tokens.estimated_input", estimatedTokens);
a?.SetTag("llm.attempts", attempts);
a?.SetTag("status", status.ToString());
a?.SetTag("issue_codes", string.Join(",", issues?.Select(i => i.Code) ?? Array.Empty<string>()));
// Deliberately not logged: extracted values, quotes, page text, file names.
Outcomes.Add(1, new KeyValuePair<string, object?>("status", status.ToString()));
}
}
from opentelemetry import metrics, trace
tracer = trace.get_tracer("pdf-pipeline")
outcomes = metrics.get_meter("pdf-pipeline").create_counter("documents_processed")
def record(span, doc_id: str, pages: int, selected_pages: int,
estimated_tokens: int, attempts: int, status: DocStatus, issues: list[Issue]) -> None:
span.set_attribute("document_id", doc_id)
span.set_attribute("pages.total", pages)
span.set_attribute("pages.selected", selected_pages)
span.set_attribute("tokens.estimated_input", estimated_tokens)
span.set_attribute("llm.attempts", attempts)
span.set_attribute("status", status.value)
span.set_attribute("issue_codes", ",".join(i.code for i in issues))
# Deliberately not logged: extracted values, quotes, page text, file names.
outcomes.add(1, {"status": status.value})
The metrics worth watching are the status mix (a rising share of NeedsReview means the model, the selector, or the documents changed), the most frequent issue codes, and retry rate. A spike in no_extractable_text usually means a new upload source is sending scans.
Testing
The LLM client is the only non-deterministic component, so it's the only thing mocked. Text extraction, the quality gate, page selection, validation, and the retry control flow are all deterministic and get tested like ordinary code. The scenarios map straight onto the failure cases above: scanned and empty documents, missing fields, oversized input, extraction errors, and retryable failures. Where a path shouldn't call the model, the test asserts the call count is zero. Where a retry should happen, it asserts the exact number of calls, because a test that only checks the final status would pass even if the retry loop never ran.
[Fact]
public async Task ScannedDocument_FailsWithoutCallingModel()
{
var llm = new FakeLlm();
var h = new Harness(pages: BlankPages(count: 10), llm: llm);
await h.Processor.ProcessAsync(h.DocId);
Assert.Equal(DocStatus.Failed, h.Repo.Status);
Assert.StartsWith("no_extractable_text", h.Repo.Reason);
Assert.Equal(0, llm.CallCount); // never spent a token
}
[Fact]
public async Task EmptyDocument_ZeroPages_Fails()
{
var h = new Harness(pages: new List<PageText>(), llm: new FakeLlm());
await h.Processor.ProcessAsync(h.DocId);
Assert.Equal(DocStatus.Failed, h.Repo.Status);
}
[Fact]
public async Task MissingRenewalNotice_GoesToReview_NotCompleted()
{
var llm = new FakeLlm(Json(autoRenew: true, noticeDays: null));
var h = new Harness(pages: AgreementPages(), llm: llm);
await h.Processor.ProcessAsync(h.DocId);
Assert.Equal(DocStatus.NeedsReview, h.Repo.Status);
Assert.Contains(h.Repo.Issues, i => i.Code == "missing_notice_period");
}
[Fact]
public async Task FabricatedQuote_IsCaughtByGroundingCheck()
{
var llm = new FakeLlm(Json(effectiveQuote: "This text is not in the document"));
var h = new Harness(pages: AgreementPages(), llm: llm);
await h.Processor.ProcessAsync(h.DocId);
Assert.Contains(h.Repo.Issues, i => i.Code == "ungrounded_effective_date");
}
[Fact]
public void OversizedDocument_SelectionStaysWithinBudget()
{
var pages = Enumerable.Range(1, 140).Select(n => new PageText(n, "term renew fees " + new string('x', 3000))).ToList();
var selected = new PageSelector().Select(pages, tokenBudget: 6000);
Assert.True(selected.Sum(p => PageSelector.EstimateTokens(p.Text)) <= 6000 + 800); // first page always included
Assert.Equal(1, selected.First().Number);
}
[Fact]
public async Task TransientFailureTwice_ThenSuccess_RetriesExactlyThreeCalls()
{
var llm = new FakeLlm(failTransientTimes: 2, thenReturn: Json());
var h = new Harness(pages: AgreementPages(), llm: llm);
await h.Processor.ProcessAsync(h.DocId);
Assert.Equal(3, llm.CallCount); // proves the loop actually re-invoked the model
Assert.Equal(DocStatus.Completed, h.Repo.Status);
Assert.Equal(2, h.Delay.WaitCount);
}
[Fact]
public async Task PersistentTransientFailure_FailsAfterMaxAttempts()
{
var llm = new FakeLlm(failTransientTimes: int.MaxValue);
var h = new Harness(pages: AgreementPages(), llm: llm);
await h.Processor.ProcessAsync(h.DocId);
Assert.Equal(3, llm.CallCount);
Assert.Equal(DocStatus.Failed, h.Repo.Status);
Assert.Equal("llm_unavailable", h.Repo.Reason);
}
[Fact]
public async Task InvalidJson_GoesToReview_WithoutRetry()
{
var llm = new FakeLlm("not json at all");
var h = new Harness(pages: AgreementPages(), llm: llm);
await h.Processor.ProcessAsync(h.DocId);
Assert.Equal(1, llm.CallCount); // permanent error: no blind retry
Assert.Equal(DocStatus.NeedsReview, h.Repo.Status);
}
async def test_scanned_document_fails_without_calling_model():
llm = FakeLlm()
h = Harness(pages=blank_pages(10), llm=llm)
await h.processor.process(h.doc_id)
assert h.repo.status == DocStatus.FAILED
assert h.repo.reason.startswith("no_extractable_text")
assert llm.call_count == 0 # never spent a token
async def test_empty_document_zero_pages_fails():
h = Harness(pages=[], llm=FakeLlm())
await h.processor.process(h.doc_id)
assert h.repo.status == DocStatus.FAILED
async def test_missing_renewal_notice_goes_to_review_not_completed():
llm = FakeLlm(json_response(auto_renew=True, notice_days=None))
h = Harness(pages=agreement_pages(), llm=llm)
await h.processor.process(h.doc_id)
assert h.repo.status == DocStatus.NEEDS_REVIEW
assert any(i.code == "missing_notice_period" for i in h.repo.issues)
async def test_fabricated_quote_is_caught_by_grounding_check():
llm = FakeLlm(json_response(effective_quote="This text is not in the document"))
h = Harness(pages=agreement_pages(), llm=llm)
await h.processor.process(h.doc_id)
assert any(i.code == "ungrounded_effective_date" for i in h.repo.issues)
def test_oversized_document_selection_stays_within_budget():
pages = [PageText(n, "term renew fees " + "x" * 3000) for n in range(1, 141)]
selected = select_pages(pages, token_budget=6000)
assert sum(estimate_tokens(p.text) for p in selected) <= 6000 + 800 # first page always included
assert selected[0].number == 1
async def test_transient_failure_twice_then_success_retries_exactly_three_calls():
llm = FakeLlm(fail_transient_times=2, then_return=json_response())
h = Harness(pages=agreement_pages(), llm=llm)
await h.processor.process(h.doc_id)
assert llm.call_count == 3 # proves the loop actually re-invoked the model
assert h.repo.status == DocStatus.COMPLETED
assert h.delay.wait_count == 2
async def test_persistent_transient_failure_fails_after_max_attempts():
llm = FakeLlm(fail_transient_times=10**9)
h = Harness(pages=agreement_pages(), llm=llm)
await h.processor.process(h.doc_id)
assert llm.call_count == 3
assert h.repo.status == DocStatus.FAILED
assert h.repo.reason == "llm_unavailable"
async def test_invalid_json_goes_to_review_without_retry():
llm = FakeLlm("not json at all")
h = Harness(pages=agreement_pages(), llm=llm)
await h.processor.process(h.doc_id)
assert llm.call_count == 1 # permanent error: no blind retry
assert h.repo.status == DocStatus.NEEDS_REVIEW
The fake delay matters as much as the fake model. Injecting it keeps the retry tests instant, and counting its waits proves backoff actually happened rather than a tight loop hammering a struggling API.
Technology Choices
C#/.NET Implementation
Why choose C#: If your backend is .NET, you get first-party Microsoft support and enterprise patterns.
- Native Azure integration — first-party SDKs for Blob Storage, Service Bus or Storage Queues, and Azure SQL, with PdfPig for text extraction
- Enterprise patterns — dependency injection makes the injected delay, repository, and LLM client trivial to fake
- Production-ready —
BackgroundServiceand hosted workers are first-class for the queue consumer
Python Implementation
Why choose Python: If your team writes Python, you get access to the richest AI/ML ecosystem.
- Library ecosystem — pypdf, pdfplumber, and the OCR libraries are Python-first, which matters once you add a scan-handling stage
- Rapid prototyping — Pydantic models double as the extraction schema and the validator
- Community — FastAPI plus a worker such as Celery or an Azure Functions queue trigger is a well-trodden path
The Bottom Line
.NET team? Use C#. Python team? Use Python. One thing to check either way: PDF text extraction quality varies a lot between libraries, so test candidates on your own documents before committing.
Engineering Trade-offs
When this pays off: any workload where documents arrive from outside your control, volume is more than a few a day, and a wrong value has downstream consequences. The status model and the grounding check are what let you automate the easy documents and still catch the hard ones.
What it costs you: more moving parts than a single function. There's a queue, a worker, a status machine, and a review path that somebody has to staff. Page selection trades recall for cost. Grounding checks will occasionally reject a correct value because the model paraphrased the quote. And a heuristic quality gate will misclassify some documents in both directions.
When to Skip It
If you process a handful of documents a month, have a human read them anyway. If every document is a clean, uniform template, a deterministic parser is cheaper and more reliable than a model. And if a wrong extraction can't be caught by validation or review (payments, legal commitments), don't automate the final step.
Key Takeaways
- Accept fast, process async. Validate, store, record, enqueue, and return 202. Never run the model inside the upload request.
- Keep page numbers from the start. Selection, citations, and human review all depend on them.
- Gate before you spend. Empty and scanned documents should stop before any model call.
- Don't send everything. Select pages within a token budget, and treat selection misses as a missing-field problem, not a silent one.
- Validate in layers. Quotes must exist on the cited page, then business rules apply. Missing or conflicting fields mean review, not completion.
- Classify failures. Retry transient errors with backoff and a cap, fail permanent ones fast, and persist an explicit status for every path.
Interview Questions
How would you design document ingestion for user-uploaded PDFs?
Validate cheaply at the edge (size, emptiness, file signature), store the original in blob storage, record a row with a status, enqueue the work, and return 202. All slow processing runs in a background worker, so upload latency never depends on the model.
How do you handle documents that exceed the model's context limit?
Select the relevant pages by scoring them against field-related terms within a token budget, and keep page order. For documents that still don't fit, process in windows, take the first confident value per field, and route disagreements between windows to review.
How do you get reliable structured data out of an LLM?
Use a schema where every field is nullable and carries its evidence (page and quote), then verify the output in code. Check that each quote appears on the cited page, parse into typed models, and treat anything missing or ungrounded as a review case instead of guessing.
Why make the pipeline asynchronous, and what do you have to get right?
Model calls are slow and fail intermittently, so tying them to a request makes uploads fragile. The price is a status machine and idempotent steps: persist status at each stage, make reprocessing safe, and rely on the queue's redelivery for worker crashes.
How do you validate extracted data before storing it?
In two layers: grounding (every quote must exist on the cited page) and business rules written as ordinary code (date ordering, positive amounts, known currencies, conditional required fields). Failures produce named issue codes and a NeedsReview status, so a person sees exactly what to check.
Which failures should be retried, and which shouldn't?
Retry transient ones (timeouts, 429s, 5xx) with exponential backoff and an attempt cap. Don't retry deterministic ones (corrupt files, no text layer, invalid model output), because they'll fail the same way and only cost tokens.