Code examples:

PDF → LLM → Structured Data Pipeline

November 12, 2026 16 min read By Jaffar Kazi
Real-World AI Engineering Document Processing Data Pipelines
C# Python Azure

The Problem

The first version of every PDF-extraction feature is the same: read the PDF, paste all of the text into a prompt, ask for JSON, save it. On the three sample documents in the demo it works beautifully. Then it meets real uploads. A scanned agreement with no text layer returns an empty string and the model cheerfully invents a plausible vendor name. A 140-page master agreement blows past the context window. A document where the renewal clause is simply absent comes back with a confident "auto-renews annually" that appears nowhere in the file. And when the model API times out at 2 a.m., the upload request that triggered it has long since failed in the user's browser.

The running example is a vendor-agreement intake service: users upload a PDF, and the service extracts the parties, effective date, end date, contract value, and renewal terms, then stores the result alongside a processing status. None of that is hard to demo. What makes it an engineering problem is everything around the model call: what to do with documents that have no text, how to handle input that doesn't fit, how to refuse to guess, and how to survive failure without losing or duplicating work.

"Just send the whole document to the model" fails in all four ways at once. It's expensive, it breaks on size, it can't tell you which page a value came from, and it has no concept of a document that should be rejected rather than processed. The pipeline below treats the LLM as one stage among several, each with a clear contract about what it accepts and what happens when it can't proceed.

The Shape of a Real Solution

The fix is a pipeline of small, separately testable stages, run asynchronously so a slow or failing model never blocks an upload.

  • Accept fast: validate the upload, store the original in blob storage, record a row with status Queued, enqueue work, and return 202 Accepted.
  • Extract text per page: keep page numbers from the start, because everything downstream (selection, citations, review) depends on them.
  • Gate on text quality: an empty or scanned document stops here with a clear reason, before any model call is made.
  • Select what to send: choose the relevant pages within a token budget instead of sending everything.
  • Extract with a schema: every field is nullable and carries the page and quote it came from.
  • Validate before trusting: check that quotes exist on the cited pages, then apply business rules. Anything missing or conflicting goes to NeedsReview, never silently to Completed.

The principle running through all of it is that a refusal is a valid output. A document that can't be processed, or a field that can't be found, is an expected result with its own status, not an exception to hide.

Architecture Overview

The API tier does almost nothing: it validates, stores, records, and enqueues. A background worker owns the slow, failure-prone work, and every path through the worker ends in an explicit, persisted status.

Architecture diagram showing a PDF upload API writing to blob storage and SQL Server and enqueuing work, a background worker running text extraction, a quality gate, page selection, schema-constrained LLM extraction and validation, with outcomes of completed, needs review, failed, or retry with backoff

Two details are worth noticing. The quality gate sits before the model, so the cheapest way to handle a bad document is to never spend a token on it. And the retry arrow goes back to the queue, not into a loop inside the request: transient failures are retried with backoff, permanent ones fail fast with a reason a human can read.

Core Implementation

The data model encodes the "refusal is valid" principle. Every extracted field is a value that may be null, plus the page and verbatim quote that support it. The document record carries a status and a machine-readable reason, so the system can always explain itself.

Models.cs / models.py

public enum DocStatus { Queued, Processing, Completed, NeedsReview, Failed }

// A value the model claims to have found, with the evidence for it.
public record Extracted<T>(T? Value, int? Page, string? Quote);

public record AgreementExtraction(
    Extracted<List<string>> Parties,
    Extracted<DateOnly?> EffectiveDate,
    Extracted<DateOnly?> EndDate,
    Extracted<decimal?> ContractValue,
    Extracted<string> Currency,
    Extracted<bool?> AutoRenew,
    Extracted<int?> RenewalNoticeDays);

public record Issue(string Code, string Message);

public record DocumentRecord(
    Guid Id, string FileName, string BlobName, DocStatus Status,
    string? Reason = null, AgreementExtraction? Result = null,
    List<Issue>? Issues = null);

from datetime import date
from enum import Enum
from typing import Generic, TypeVar
from pydantic import BaseModel

T = TypeVar("T")

class DocStatus(str, Enum):
    QUEUED = "queued"
    PROCESSING = "processing"
    COMPLETED = "completed"
    NEEDS_REVIEW = "needs_review"
    FAILED = "failed"

# A value the model claims to have found, with the evidence for it.
class Extracted(BaseModel, Generic[T]):
    value: T | None = None
    page: int | None = None
    quote: str | None = None

class AgreementExtraction(BaseModel):
    parties: Extracted[list[str]]
    effective_date: Extracted[date]
    end_date: Extracted[date]
    contract_value: Extracted[float]
    currency: Extracted[str]
    auto_renew: Extracted[bool]
    renewal_notice_days: Extracted[int]

class Issue(BaseModel):
    code: str
    message: str

class DocumentRecord(BaseModel):
    id: str
    file_name: str
    blob_name: str
    status: DocStatus
    reason: str | None = None
    result: AgreementExtraction | None = None
    issues: list[Issue] = []

The upload endpoint is deliberately thin. It rejects obviously bad input immediately (empty, oversized, not actually a PDF), stores the original, records the row, enqueues the work, and returns. Nothing slow happens in the request.

UploadEndpoint.cs / upload_endpoint.py

const long MaxBytes = 20 * 1024 * 1024;

app.MapPost("/documents", async (IFormFile file, IBlobStore blobs,
    IDocumentRepository repo, IWorkQueue queue) =>
{
    if (file.Length == 0) return Results.BadRequest(new { error = "empty_file" });
    if (file.Length > MaxBytes) return Results.StatusCode(413);

    await using var stream = file.OpenReadStream();
    var magic = new byte[5];
    await stream.ReadExactlyAsync(magic);
    if (Encoding.ASCII.GetString(magic) != "%PDF-")
        return Results.BadRequest(new { error = "not_a_pdf" });
    stream.Position = 0;

    var id = Guid.NewGuid();
    var blobName = $"{id}.pdf";
    await blobs.UploadAsync(blobName, stream);
    await repo.CreateAsync(new DocumentRecord(id, file.FileName, blobName, DocStatus.Queued));
    await queue.EnqueueAsync(id);

    return Results.Accepted($"/documents/{id}", new { id, status = "queued" });
});

app.MapGet("/documents/{id:guid}", async (Guid id, IDocumentRepository repo) =>
    await repo.GetAsync(id) is { } doc ? Results.Ok(doc) : Results.NotFound());

MAX_BYTES = 20 * 1024 * 1024

@app.post("/documents", status_code=202)
async def upload(file: UploadFile, blobs: BlobStore = Depends(get_blobs),
                 repo: DocumentRepository = Depends(get_repo),
                 queue: WorkQueue = Depends(get_queue)):
    data = await file.read()
    if not data:
        raise HTTPException(400, "empty_file")
    if len(data) > MAX_BYTES:
        raise HTTPException(413, "file_too_large")
    if not data.startswith(b"%PDF-"):
        raise HTTPException(400, "not_a_pdf")

    doc_id = str(uuid.uuid4())
    blob_name = f"{doc_id}.pdf"
    await blobs.upload(blob_name, data)
    await repo.create(DocumentRecord(id=doc_id, file_name=file.filename,
                                     blob_name=blob_name, status=DocStatus.QUEUED))
    await queue.enqueue(doc_id)
    return {"id": doc_id, "status": "queued"}

@app.get("/documents/{doc_id}")
async def get_document(doc_id: str, repo: DocumentRepository = Depends(get_repo)):
    doc = await repo.get(doc_id)
    if doc is None:
        raise HTTPException(404)
    return doc

Key Technical Challenge: Getting Text Out With Page Numbers, and Knowing When There Isn't Any

Text extraction looks like a solved problem until you meet a scanned PDF. A scan is a set of images, so a text extractor returns pages successfully, with nothing on them. If that empty text reaches the model, it's asked to extract a contract from a blank prompt. A model asked for a JSON object will often produce one.

The quality gate catches this with a simple heuristic: count how many pages have a meaningful amount of text. Zero pages means an empty document. Mostly empty pages means a probable scan. Either way, the document stops before any model call with a specific reason. In practice, you'll find the threshold needs tuning against your own documents, since cover pages and signature pages are legitimately sparse. That's why the gate looks at the fraction of pages with text, not a single page.

PdfTextExtractor.cs / pdf_text_extractor.py

using UglyToad.PdfPig;

public record PageText(int Number, string Text);

public enum TextQuality { Usable, Empty, LikelyScanned }

public class PdfTextExtractor
{
    public List<PageText> Extract(Stream pdf)
    {
        using var doc = PdfDocument.Open(pdf);
        return doc.GetPages().Select(p => new PageText(p.Number, p.Text)).ToList();
    }
}

public static class QualityGate
{
    private const int MinCharsPerPage = 80;      // sparse pages exist; tune on real documents
    private const double MinTextPageRatio = 0.4;

    public static TextQuality Assess(IReadOnlyList<PageText> pages)
    {
        if (pages.Count == 0) return TextQuality.Empty;

        var withText = pages.Count(p => p.Text.Trim().Length >= MinCharsPerPage);
        return (double)withText / pages.Count < MinTextPageRatio
            ? TextQuality.LikelyScanned
            : TextQuality.Usable;
    }
}

from dataclasses import dataclass
from enum import Enum
from io import BytesIO
from pypdf import PdfReader

@dataclass(frozen=True)
class PageText:
    number: int
    text: str

class TextQuality(str, Enum):
    USABLE = "usable"
    EMPTY = "empty"
    LIKELY_SCANNED = "likely_scanned"

def extract_pages(pdf_bytes: bytes) -> list[PageText]:
    reader = PdfReader(BytesIO(pdf_bytes))
    return [PageText(i + 1, page.extract_text() or "") for i, page in enumerate(reader.pages)]

MIN_CHARS_PER_PAGE = 80          # sparse pages exist; tune on real documents
MIN_TEXT_PAGE_RATIO = 0.4

def assess_quality(pages: list[PageText]) -> TextQuality:
    if not pages:
        return TextQuality.EMPTY
    with_text = sum(1 for p in pages if len(p.text.strip()) >= MIN_CHARS_PER_PAGE)
    if with_text / len(pages) < MIN_TEXT_PAGE_RATIO:
        return TextQuality.LIKELY_SCANNED
    return TextQuality.USABLE

A scanned document isn't necessarily a failure forever. The natural extension is an OCR stage that runs when the gate reports LikelyScanned, but that's a separate stage with its own cost and its own accuracy problems. Until it exists, the honest outcome is a clear Failed status with the reason no_extractable_text.

Key Technical Challenge: Context Limits and Oversized Input

Even with good text, "send everything" is the wrong default. Most of a vendor agreement is boilerplate that never contains the fields you want: indemnities, notices, governing law. Parties and dates usually appear on the first page, term and renewal near the front or in a dedicated section, and fees in a schedule. Sending 140 pages to find six values is slow and expensive, and gives the model a lot of irrelevant text to be distracted by.

The page selector scores each page by how many field-related terms it contains, always includes the first page, and packs the highest-scoring pages into a token budget, returned in page order so the model reads them as the document is written. Documents that still don't fit are processed in windows, with each field taking its first confident value and any disagreement between windows sent to review.

PageSelector.cs / page_selector.py

public class PageSelector
{
    private static readonly string[] Signals =
    {
        "between", "party", "parties", "effective date", "commencement", "term",
        "renew", "expire", "terminate", "notice", "fees", "price", "total", "payment", "$", "usd", "aud", "eur"
    };

    public static int EstimateTokens(string text) => text.Length / 4;   // rough; use the provider's count to bill

    public List<PageText> Select(IReadOnlyList<PageText> pages, int tokenBudget)
    {
        var chosen = new List<PageText>();
        var remaining = tokenBudget;

        // First page almost always names the parties and date.
        var first = pages[0];
        chosen.Add(first);
        remaining -= EstimateTokens(first.Text);

        foreach (var page in pages.Skip(1).OrderByDescending(Score))
        {
            var cost = EstimateTokens(page.Text);
            if (Score(page) == 0 || cost > remaining) continue;
            chosen.Add(page);
            remaining -= cost;
        }
        return chosen.OrderBy(p => p.Number).ToList();     // keep document order
    }

    private static int Score(PageText p)
    {
        var lower = p.Text.ToLowerInvariant();
        return Signals.Sum(s => lower.Split(s).Length - 1);
    }
}

SIGNALS = ["between", "party", "parties", "effective date", "commencement", "term",
           "renew", "expire", "terminate", "notice", "fees", "price", "total",
           "payment", "$", "usd", "aud", "eur"]

def estimate_tokens(text: str) -> int:
    return len(text) // 4          # rough; use the provider's count to bill

def score(page: PageText) -> int:
    lower = page.text.lower()
    return sum(lower.count(s) for s in SIGNALS)

def select_pages(pages: list[PageText], token_budget: int) -> list[PageText]:
    # First page almost always names the parties and date.
    chosen = [pages[0]]
    remaining = token_budget - estimate_tokens(pages[0].text)

    for page in sorted(pages[1:], key=score, reverse=True):
        cost = estimate_tokens(page.text)
        if score(page) == 0 or cost > remaining:
            continue
        chosen.append(page)
        remaining -= cost
    return sorted(chosen, key=lambda p: p.number)      # keep document order

Selection Is a Recall Risk

If the renewal clause sits on a page the scorer ranks low, it's never sent, and the field comes back null. That's the safe failure: the validator flags the missing field and the document goes to review instead of passing with a wrong value. Still, track how often selection causes missing fields, and widen the signal list or the budget when it does.

Key Technical Challenge: Validating Extraction Before Trusting It

A schema guarantees shape, not truth. The model can return perfectly valid JSON containing a date that appears nowhere in the document. Two layers of validation close that gap.

The first is grounding: every extracted value must carry a quote, and that quote must actually appear on the page the model cited. This is a plain string check, and it catches most fabricated values, because a model inventing a field rarely invents a matching quote at the right page. The second is business rules, which are ordinary code: at least two distinct parties, an end date after the effective date, a positive value with a known currency, and a notice period whenever auto-renewal is true. Missing required fields aren't errors in the extraction. They're a reason to route to human review, listed by name.

AgreementValidator.cs / agreement_validator.py

public class AgreementValidator
{
    private static readonly HashSet<string> Currencies = new() { "USD", "AUD", "EUR", "GBP", "NZD", "CAD" };

    public List<Issue> Validate(AgreementExtraction x, IReadOnlyList<PageText> pages)
    {
        var issues = new List<Issue>();

        // 1. Required fields present.
        if (x.Parties.Value is not { Count: >= 2 }) issues.Add(new("missing_parties", "Fewer than two parties found."));
        if (x.EffectiveDate.Value is null) issues.Add(new("missing_effective_date", "No effective date found."));
        if (x.ContractValue.Value is null) issues.Add(new("missing_value", "No contract value found."));

        // 2. Grounding: each quote must exist on the cited page.
        CheckQuote(issues, "parties", x.Parties.Page, x.Parties.Quote, pages);
        CheckQuote(issues, "effective_date", x.EffectiveDate.Page, x.EffectiveDate.Quote, pages);
        CheckQuote(issues, "contract_value", x.ContractValue.Page, x.ContractValue.Quote, pages);
        CheckQuote(issues, "auto_renew", x.AutoRenew.Page, x.AutoRenew.Quote, pages);

        // 3. Business rules.
        if (x.EffectiveDate.Value is { } start && x.EndDate.Value is { } end && end <= start)
            issues.Add(new("end_before_start", "End date is not after the effective date."));
        if (x.ContractValue.Value is <= 0)
            issues.Add(new("non_positive_value", "Contract value must be positive."));
        if (x.ContractValue.Value is not null && !Currencies.Contains(x.Currency.Value ?? ""))
            issues.Add(new("unknown_currency", $"Currency '{x.Currency.Value}' is not recognised."));
        if (x.AutoRenew.Value == true && x.RenewalNoticeDays.Value is null)
            issues.Add(new("missing_notice_period", "Auto-renewal found but no notice period."));

        return issues;
    }

    private static void CheckQuote(List<Issue> issues, string field, int? page, string? quote,
        IReadOnlyList<PageText> pages)
    {
        if (quote is null) return;                        // absence is reported above
        var text = pages.FirstOrDefault(p => p.Number == page)?.Text ?? "";
        if (!Normalise(text).Contains(Normalise(quote)))
            issues.Add(new($"ungrounded_{field}", $"Quote for {field} not found on page {page}."));
    }

    private static string Normalise(string s) => Regex.Replace(s, @"\s+", " ").Trim().ToLowerInvariant();
}

import re

CURRENCIES = {"USD", "AUD", "EUR", "GBP", "NZD", "CAD"}

def normalise(s: str) -> str:
    return re.sub(r"\s+", " ", s).strip().lower()

def check_quote(issues: list[Issue], field: str, f, pages: list[PageText]) -> None:
    if f.quote is None:
        return                                           # absence is reported elsewhere
    text = next((p.text for p in pages if p.number == f.page), "")
    if normalise(f.quote) not in normalise(text):
        issues.append(Issue(code=f"ungrounded_{field}",
                            message=f"Quote for {field} not found on page {f.page}."))

def validate(x: AgreementExtraction, pages: list[PageText]) -> list[Issue]:
    issues: list[Issue] = []

    # 1. Required fields present.
    if not x.parties.value or len(x.parties.value) < 2:
        issues.append(Issue(code="missing_parties", message="Fewer than two parties found."))
    if x.effective_date.value is None:
        issues.append(Issue(code="missing_effective_date", message="No effective date found."))
    if x.contract_value.value is None:
        issues.append(Issue(code="missing_value", message="No contract value found."))

    # 2. Grounding: each quote must exist on the cited page.
    for name, f in [("parties", x.parties), ("effective_date", x.effective_date),
                    ("contract_value", x.contract_value), ("auto_renew", x.auto_renew)]:
        check_quote(issues, name, f, pages)

    # 3. Business rules.
    if x.effective_date.value and x.end_date.value and x.end_date.value <= x.effective_date.value:
        issues.append(Issue(code="end_before_start", message="End date is not after the effective date."))
    if x.contract_value.value is not None and x.contract_value.value <= 0:
        issues.append(Issue(code="non_positive_value", message="Contract value must be positive."))
    if x.contract_value.value is not None and (x.currency.value or "") not in CURRENCIES:
        issues.append(Issue(code="unknown_currency", message=f"Currency '{x.currency.value}' is not recognised."))
    if x.auto_renew.value is True and x.renewal_notice_days.value is None:
        issues.append(Issue(code="missing_notice_period", message="Auto-renewal found but no notice period."))
    return issues

Key Technical Challenge: Async Processing and Failures That Aren't All the Same

Not every failure deserves a retry. A model API returning 429 or a timeout is transient: wait and try again. A corrupt PDF, a document with no text, or output that isn't valid JSON will fail identically every time, so retrying only burns tokens. The processor treats the two classes differently and gives each a distinct, persisted outcome.

Transient errors retry in-process with exponential backoff up to a fixed attempt cap, then fail with llm_unavailable. Unparseable model output goes straight to NeedsReview. A worker crash mid-job is the queue's responsibility (a message that isn't acknowledged becomes visible again), which is why status is written to the database at each step and processing is safe to repeat.

DocumentProcessor.cs / document_processor.py

public class DocumentProcessor
{
    private const int MaxAttempts = 3;
    private const int TokenBudget = 6000;

    // Dependencies injected: _repo, _blobs, _extractor, _selector, _llm, _validator, _delay.

    public async Task ProcessAsync(Guid id)
    {
        var doc = await _repo.GetAsync(id);
        await _repo.SetStatusAsync(id, DocStatus.Processing);

        var pages = _extractor.Extract(await _blobs.OpenAsync(doc!.BlobName));
        var quality = QualityGate.Assess(pages);
        if (quality != TextQuality.Usable)
        {
            await _repo.FailAsync(id, $"no_extractable_text:{quality}");   // no model call made
            return;
        }

        var selected = _selector.Select(pages, TokenBudget);
        AgreementExtraction? result = null;

        for (var attempt = 1; attempt <= MaxAttempts; attempt++)
        {
            try
            {
                var raw = await _llm.ExtractAsync(selected);
                result = JsonSerializer.Deserialize<AgreementExtraction>(raw);
                break;
            }
            catch (TransientLlmException) when (attempt < MaxAttempts)
            {
                await _delay.WaitAsync(TimeSpan.FromSeconds(Math.Pow(2, attempt)));
            }
            catch (TransientLlmException)
            {
                await _repo.FailAsync(id, "llm_unavailable");
                return;
            }
            catch (JsonException)
            {
                await _repo.NeedsReviewAsync(id, new() { new("unparseable_output", "Model output was not valid JSON.") });
                return;
            }
        }

        var issues = _validator.Validate(result!, pages);
        if (issues.Count == 0) await _repo.CompleteAsync(id, result!);
        else await _repo.NeedsReviewAsync(id, issues, result);
    }
}

import json

MAX_ATTEMPTS = 3
TOKEN_BUDGET = 6000

class DocumentProcessor:
    # Dependencies injected: repo, blobs, llm, delay.

    async def process(self, doc_id: str) -> None:
        doc = await self.repo.get(doc_id)
        await self.repo.set_status(doc_id, DocStatus.PROCESSING)

        pages = extract_pages(await self.blobs.read(doc.blob_name))
        quality = assess_quality(pages)
        if quality != TextQuality.USABLE:
            await self.repo.fail(doc_id, f"no_extractable_text:{quality.value}")   # no model call made
            return

        selected = select_pages(pages, TOKEN_BUDGET)
        result = None

        for attempt in range(1, MAX_ATTEMPTS + 1):
            try:
                raw = await self.llm.extract(selected)
                result = AgreementExtraction.model_validate(json.loads(raw))
                break
            except TransientLlmError:
                if attempt < MAX_ATTEMPTS:
                    await self.delay.wait(2 ** attempt)
                    continue
                await self.repo.fail(doc_id, "llm_unavailable")
                return
            except (ValueError, ValidationError):
                await self.repo.needs_review(doc_id, [Issue(code="unparseable_output",
                                             message="Model output was not valid JSON.")])
                return

        issues = validate(result, pages)
        if not issues:
            await self.repo.complete(doc_id, result)
        else:
            await self.repo.needs_review(doc_id, issues, result)

What This Actually Costs

The table is a worked example, not a benchmark: a 30-page agreement is roughly 18,000 tokens of text, and a well-formed extraction response is about 300 tokens. Prices are for a mid-tier and a small model at the time of writing ($2.00 / $10.00 and $0.10 / $0.50 per million input / output tokens respectively). Check current pricing, it changes.

ApproachInput tokensOutput tokensCost / documentPer 10,000 documents
Whole document, mid-tier model18,400300$0.0398$398
Selected pages, mid-tier model4,000300$0.0110$110
Selected pages, small model4,000300$0.0006$6
Small model first, 25% escalate to mid-tier——~$0.0033~$33

Page selection alone cuts the model bill by about 72%, and the extraction task (find a few fields, quote them) is well suited to a small model, with escalation reserved for documents the validator flags. The cheapest document of all is the one the quality gate rejects before any model call. Two caveats apply. OCR, if you add it, is billed separately by the OCR service per page, so check its pricing rather than assuming it's negligible. And some providers charge a higher per-token rate once a prompt crosses a length threshold, so an oversized input can cost more per token as well as more tokens.

Observability

Log what happened to each document, never what was in it. Agreements contain names, amounts, and commercial terms that don't belong in a log aggregator. Page counts, token estimates, status transitions, attempt counts, and issue codes are enough to debug almost everything.

Telemetry.cs / telemetry.py

public static class PipelineTelemetry
{
    public static readonly ActivitySource Source = new("PdfPipeline");
    private static readonly Meter AppMeter = new("PdfPipeline");
    private static readonly Counter<long> Outcomes = AppMeter.CreateCounter<long>("documents_processed");

    public static void Record(Activity? a, Guid id, int pages, int selectedPages,
        int estimatedTokens, int attempts, DocStatus status, IEnumerable<Issue>? issues)
    {
        a?.SetTag("document_id", id.ToString());
        a?.SetTag("pages.total", pages);
        a?.SetTag("pages.selected", selectedPages);
        a?.SetTag("tokens.estimated_input", estimatedTokens);
        a?.SetTag("llm.attempts", attempts);
        a?.SetTag("status", status.ToString());
        a?.SetTag("issue_codes", string.Join(",", issues?.Select(i => i.Code) ?? Array.Empty<string>()));
        // Deliberately not logged: extracted values, quotes, page text, file names.

        Outcomes.Add(1, new KeyValuePair<string, object?>("status", status.ToString()));
    }
}

from opentelemetry import metrics, trace

tracer = trace.get_tracer("pdf-pipeline")
outcomes = metrics.get_meter("pdf-pipeline").create_counter("documents_processed")

def record(span, doc_id: str, pages: int, selected_pages: int,
           estimated_tokens: int, attempts: int, status: DocStatus, issues: list[Issue]) -> None:
    span.set_attribute("document_id", doc_id)
    span.set_attribute("pages.total", pages)
    span.set_attribute("pages.selected", selected_pages)
    span.set_attribute("tokens.estimated_input", estimated_tokens)
    span.set_attribute("llm.attempts", attempts)
    span.set_attribute("status", status.value)
    span.set_attribute("issue_codes", ",".join(i.code for i in issues))
    # Deliberately not logged: extracted values, quotes, page text, file names.

    outcomes.add(1, {"status": status.value})

The metrics worth watching are the status mix (a rising share of NeedsReview means the model, the selector, or the documents changed), the most frequent issue codes, and retry rate. A spike in no_extractable_text usually means a new upload source is sending scans.

Testing

The LLM client is the only non-deterministic component, so it's the only thing mocked. Text extraction, the quality gate, page selection, validation, and the retry control flow are all deterministic and get tested like ordinary code. The scenarios map straight onto the failure cases above: scanned and empty documents, missing fields, oversized input, extraction errors, and retryable failures. Where a path shouldn't call the model, the test asserts the call count is zero. Where a retry should happen, it asserts the exact number of calls, because a test that only checks the final status would pass even if the retry loop never ran.

DocumentProcessorTests.cs / test_document_processor.py

[Fact]
public async Task ScannedDocument_FailsWithoutCallingModel()
{
    var llm = new FakeLlm();
    var h = new Harness(pages: BlankPages(count: 10), llm: llm);

    await h.Processor.ProcessAsync(h.DocId);

    Assert.Equal(DocStatus.Failed, h.Repo.Status);
    Assert.StartsWith("no_extractable_text", h.Repo.Reason);
    Assert.Equal(0, llm.CallCount);                       // never spent a token
}

[Fact]
public async Task EmptyDocument_ZeroPages_Fails()
{
    var h = new Harness(pages: new List<PageText>(), llm: new FakeLlm());
    await h.Processor.ProcessAsync(h.DocId);
    Assert.Equal(DocStatus.Failed, h.Repo.Status);
}

[Fact]
public async Task MissingRenewalNotice_GoesToReview_NotCompleted()
{
    var llm = new FakeLlm(Json(autoRenew: true, noticeDays: null));
    var h = new Harness(pages: AgreementPages(), llm: llm);

    await h.Processor.ProcessAsync(h.DocId);

    Assert.Equal(DocStatus.NeedsReview, h.Repo.Status);
    Assert.Contains(h.Repo.Issues, i => i.Code == "missing_notice_period");
}

[Fact]
public async Task FabricatedQuote_IsCaughtByGroundingCheck()
{
    var llm = new FakeLlm(Json(effectiveQuote: "This text is not in the document"));
    var h = new Harness(pages: AgreementPages(), llm: llm);

    await h.Processor.ProcessAsync(h.DocId);

    Assert.Contains(h.Repo.Issues, i => i.Code == "ungrounded_effective_date");
}

[Fact]
public void OversizedDocument_SelectionStaysWithinBudget()
{
    var pages = Enumerable.Range(1, 140).Select(n => new PageText(n, "term renew fees " + new string('x', 3000))).ToList();

    var selected = new PageSelector().Select(pages, tokenBudget: 6000);

    Assert.True(selected.Sum(p => PageSelector.EstimateTokens(p.Text)) <= 6000 + 800); // first page always included
    Assert.Equal(1, selected.First().Number);
}

[Fact]
public async Task TransientFailureTwice_ThenSuccess_RetriesExactlyThreeCalls()
{
    var llm = new FakeLlm(failTransientTimes: 2, thenReturn: Json());
    var h = new Harness(pages: AgreementPages(), llm: llm);

    await h.Processor.ProcessAsync(h.DocId);

    Assert.Equal(3, llm.CallCount);                       // proves the loop actually re-invoked the model
    Assert.Equal(DocStatus.Completed, h.Repo.Status);
    Assert.Equal(2, h.Delay.WaitCount);
}

[Fact]
public async Task PersistentTransientFailure_FailsAfterMaxAttempts()
{
    var llm = new FakeLlm(failTransientTimes: int.MaxValue);
    var h = new Harness(pages: AgreementPages(), llm: llm);

    await h.Processor.ProcessAsync(h.DocId);

    Assert.Equal(3, llm.CallCount);
    Assert.Equal(DocStatus.Failed, h.Repo.Status);
    Assert.Equal("llm_unavailable", h.Repo.Reason);
}

[Fact]
public async Task InvalidJson_GoesToReview_WithoutRetry()
{
    var llm = new FakeLlm("not json at all");
    var h = new Harness(pages: AgreementPages(), llm: llm);

    await h.Processor.ProcessAsync(h.DocId);

    Assert.Equal(1, llm.CallCount);                       // permanent error: no blind retry
    Assert.Equal(DocStatus.NeedsReview, h.Repo.Status);
}

async def test_scanned_document_fails_without_calling_model():
    llm = FakeLlm()
    h = Harness(pages=blank_pages(10), llm=llm)

    await h.processor.process(h.doc_id)

    assert h.repo.status == DocStatus.FAILED
    assert h.repo.reason.startswith("no_extractable_text")
    assert llm.call_count == 0                       # never spent a token

async def test_empty_document_zero_pages_fails():
    h = Harness(pages=[], llm=FakeLlm())
    await h.processor.process(h.doc_id)
    assert h.repo.status == DocStatus.FAILED

async def test_missing_renewal_notice_goes_to_review_not_completed():
    llm = FakeLlm(json_response(auto_renew=True, notice_days=None))
    h = Harness(pages=agreement_pages(), llm=llm)

    await h.processor.process(h.doc_id)

    assert h.repo.status == DocStatus.NEEDS_REVIEW
    assert any(i.code == "missing_notice_period" for i in h.repo.issues)

async def test_fabricated_quote_is_caught_by_grounding_check():
    llm = FakeLlm(json_response(effective_quote="This text is not in the document"))
    h = Harness(pages=agreement_pages(), llm=llm)

    await h.processor.process(h.doc_id)

    assert any(i.code == "ungrounded_effective_date" for i in h.repo.issues)

def test_oversized_document_selection_stays_within_budget():
    pages = [PageText(n, "term renew fees " + "x" * 3000) for n in range(1, 141)]

    selected = select_pages(pages, token_budget=6000)

    assert sum(estimate_tokens(p.text) for p in selected) <= 6000 + 800   # first page always included
    assert selected[0].number == 1

async def test_transient_failure_twice_then_success_retries_exactly_three_calls():
    llm = FakeLlm(fail_transient_times=2, then_return=json_response())
    h = Harness(pages=agreement_pages(), llm=llm)

    await h.processor.process(h.doc_id)

    assert llm.call_count == 3                       # proves the loop actually re-invoked the model
    assert h.repo.status == DocStatus.COMPLETED
    assert h.delay.wait_count == 2

async def test_persistent_transient_failure_fails_after_max_attempts():
    llm = FakeLlm(fail_transient_times=10**9)
    h = Harness(pages=agreement_pages(), llm=llm)

    await h.processor.process(h.doc_id)

    assert llm.call_count == 3
    assert h.repo.status == DocStatus.FAILED
    assert h.repo.reason == "llm_unavailable"

async def test_invalid_json_goes_to_review_without_retry():
    llm = FakeLlm("not json at all")
    h = Harness(pages=agreement_pages(), llm=llm)

    await h.processor.process(h.doc_id)

    assert llm.call_count == 1                       # permanent error: no blind retry
    assert h.repo.status == DocStatus.NEEDS_REVIEW

The fake delay matters as much as the fake model. Injecting it keeps the retry tests instant, and counting its waits proves backoff actually happened rather than a tight loop hammering a struggling API.

Technology Choices

C#/.NET Implementation

Why choose C#: If your backend is .NET, you get first-party Microsoft support and enterprise patterns.

  • Native Azure integration — first-party SDKs for Blob Storage, Service Bus or Storage Queues, and Azure SQL, with PdfPig for text extraction
  • Enterprise patterns — dependency injection makes the injected delay, repository, and LLM client trivial to fake
  • Production-ready — BackgroundService and hosted workers are first-class for the queue consumer

Python Implementation

Why choose Python: If your team writes Python, you get access to the richest AI/ML ecosystem.

  • Library ecosystem — pypdf, pdfplumber, and the OCR libraries are Python-first, which matters once you add a scan-handling stage
  • Rapid prototyping — Pydantic models double as the extraction schema and the validator
  • Community — FastAPI plus a worker such as Celery or an Azure Functions queue trigger is a well-trodden path

The Bottom Line

.NET team? Use C#. Python team? Use Python. One thing to check either way: PDF text extraction quality varies a lot between libraries, so test candidates on your own documents before committing.

Engineering Trade-offs

When this pays off: any workload where documents arrive from outside your control, volume is more than a few a day, and a wrong value has downstream consequences. The status model and the grounding check are what let you automate the easy documents and still catch the hard ones.

What it costs you: more moving parts than a single function. There's a queue, a worker, a status machine, and a review path that somebody has to staff. Page selection trades recall for cost. Grounding checks will occasionally reject a correct value because the model paraphrased the quote. And a heuristic quality gate will misclassify some documents in both directions.

When to Skip It

If you process a handful of documents a month, have a human read them anyway. If every document is a clean, uniform template, a deterministic parser is cheaper and more reliable than a model. And if a wrong extraction can't be caught by validation or review (payments, legal commitments), don't automate the final step.

Key Takeaways

  • Accept fast, process async. Validate, store, record, enqueue, and return 202. Never run the model inside the upload request.
  • Keep page numbers from the start. Selection, citations, and human review all depend on them.
  • Gate before you spend. Empty and scanned documents should stop before any model call.
  • Don't send everything. Select pages within a token budget, and treat selection misses as a missing-field problem, not a silent one.
  • Validate in layers. Quotes must exist on the cited page, then business rules apply. Missing or conflicting fields mean review, not completion.
  • Classify failures. Retry transient errors with backoff and a cap, fail permanent ones fast, and persist an explicit status for every path.

Interview Questions

How would you design document ingestion for user-uploaded PDFs?
Validate cheaply at the edge (size, emptiness, file signature), store the original in blob storage, record a row with a status, enqueue the work, and return 202. All slow processing runs in a background worker, so upload latency never depends on the model.

How do you handle documents that exceed the model's context limit?
Select the relevant pages by scoring them against field-related terms within a token budget, and keep page order. For documents that still don't fit, process in windows, take the first confident value per field, and route disagreements between windows to review.

How do you get reliable structured data out of an LLM?
Use a schema where every field is nullable and carries its evidence (page and quote), then verify the output in code. Check that each quote appears on the cited page, parse into typed models, and treat anything missing or ungrounded as a review case instead of guessing.

Why make the pipeline asynchronous, and what do you have to get right?
Model calls are slow and fail intermittently, so tying them to a request makes uploads fragile. The price is a status machine and idempotent steps: persist status at each stage, make reprocessing safe, and rely on the queue's redelivery for worker crashes.

How do you validate extracted data before storing it?
In two layers: grounding (every quote must exist on the cited page) and business rules written as ordinary code (date ordering, positive amounts, known currencies, conditional required fields). Failures produce named issue codes and a NeedsReview status, so a person sees exactly what to check.

Which failures should be retried, and which shouldn't?
Retry transient ones (timeouts, 429s, 5xx) with exponential backoff and an attempt cap. Don't retry deterministic ones (corrupt files, no text layer, invalid model output), because they'll fail the same way and only cost tokens.

Written by Jaffar Kazi, a software engineer in Sydney building AI-powered applications. Connect on LinkedIn.