AI Contract Review Assistant

I'm building a FastAPI backend for legal/compliance-style contract review: upload a contract, extract the fields that actually matter (parties, payment terms, termination, renewal, liability cap, governing law), run them against a policy rule engine, and surface risk flags for a human reviewer to approve or override — with a full audit trail on every AI output and every reviewer action. This is a project in progress, not a finished product, and this article says so plainly rather than dressing it up.

The problem and context

This project is Project 1 of a planned 3-project portfolio, and it's being built as the first four weeks of a structured, self-directed study plan closing specific, self-identified gaps against forward-deployed-engineering work. I'd already built and rewritten an AI resume/job-description matcher twice in production, so this project deliberately reuses those proven skills — LLM-based field extraction, rule-based scoring, chunking for retrieval — rather than relearning them. The genuinely new ground it's meant to break is the ground the resume-matcher project never had to cover: a formal eval/scoring harness, RAG citations, MCP tool exposure, Node/TypeScript, and Key Vault/Managed Identity.

The build

The backend is a Python/FastAPI service backed by SQL Server, with contract files stored in Azure Blob Storage. The core flow: a contract is uploaded and stored in Blob with a Document row created for it; analysis extracts the text (with real PDF support, not just plain-text), chunks it into paragraphs, runs LLM-based field extraction, links each extracted field back to the specific chunk it came from, and runs the results through a policy rule engine that raises named risk flags — MISSING_LIABILITY_CAP, UNAPPROVED_GOVERNING_LAW, AUTO_RENEWAL_NO_TERM, LONG_PAYMENT_TERMS — each one traceable back to the source clause. A reviewer can approve or override any flag, and a document can be corrected and re-uploaded as a new version without losing the analysis history on the previous one. Every AI extraction and every reviewer action is written to an audit trail.

The backend today: upload/analyze/review/reupload endpoints, an extraction-to-rule-engine pipeline, and a SQL Server + Blob Storage split for structured data vs. original files.

Phases 0–6 are done: the core data layer, Blob storage and upload, field extraction and the rule engine wired into /analyze, review actions, and the PDF/chunking/document-versioning work described above.

Key production decisions

1. Building the eval harness neither prior production build ever had

Both of my earlier production builds (the resume-matcher's two rewrites) judged output quality by eye — there was never a scoring harness. This project closed that gap directly: a labeled 25-contract eval set and a scoring script (run_eval.py + test_scorer.py) that runs extraction against every contract and reports per-field and overall accuracy. It took four iterative rounds to go from a 50.5% baseline to 98.0% overall accuracy.

2. Using that harness to make a real model/cost decision, not swap models on instinct

I'd swapped LLM models before (GPT-3.5 to GPT-4, in the resume-matcher project) without ever measuring what actually changed. Here, the same 25-contract set was run against the production baseline (gpt-4.1-mini) and two cheaper candidates (gpt-4.1-nano, gpt-5-nano), extending the harness to capture real cost-per-contract (from actual token usage) and latency-per-contract alongside accuracy. The verdict was to keep the baseline: gpt-5-nano was closer on accuracy (97.0% vs. 98.0%) but both pricier and nearly twice as slow — a direct counter-example to assuming a newer or smaller-named model is automatically cheaper or faster. gpt-4.1-nano was a real trade-off (4.4x cheaper, 3.4x faster) but collapsed renewal_term accuracy to 60%, the exact field the AUTO_RENEWAL_NO_TERM rule depends on — disqualifying, despite the cost savings. This is the same shape of numbers-backed keep/reverse judgment as the resume-matcher project's own cost audit, this time about model choice instead of vector-store choice.

Current state

Stated plainly: this project is roughly eight working days into a structured four-week plan. Phases 0–6 (core backend, rule engine, PDF extraction, chunking, document versioning) are done. The Week 2 eval harness and model/cost comparison described above are done. Everything from MCP exposure onward has not been built yet — it's planned, not implemented, and this article isn't claiming otherwise.

What's coming — the full roadmap

Rather than a vague "more features coming soon," here's every remaining stage, in the order it's planned:

• MCP tool exposure — wrapping the real backend's upload/analyze/get-document endpoints as MCP tools using the FastMCP pattern, pointed at the live, DB-backed app rather than local files.
• A Node.js/TypeScript port — a bounded port of an existing Python MCP server to Node + the official TypeScript MCP SDK, verified side-by-side against the Python original, to close a confirmed zero-prior-exposure gap in a stack much of the AI-agent tooling world ships first.
• RAG with citations — wiring contract Q&A answers to cite the specific document chunk they came from, using the chunk-linkage already built in Phase 6, plus extending the eval harness to score whether an answer actually cites the right chunk and is grounded in it.
• A systematic unhappy-path pass — deliberately enumerating and handling every failure class (malformed PDFs, invalid JSON, out-of-schema values, duplicate uploads, concurrent review actions) as its own discipline, not case-by-case patching as bugs show up.
• Hardening — request-ID middleware and structured logging done properly, a cost-conscious OpenTelemetry setup, and a real multi-stage CI/CD pipeline with gates.
• Production-grade secrets and identity — moving API keys and connection strings into Key Vault, and authenticating to Key Vault/Blob/SQL via Managed Identity instead of keys or connection strings.
• The business and engineering narrative — writing up the architecture decisions and the eval/cost numbers above into both a technical narrative and a business-facing one, compressed down to a short pitch.
• Publishing and a capstone pitch — this portfolio write-up is part of that step, folding the eval-harness and model-comparison numbers in as the evidence.

Skills and technologies this demonstrates

• Building an eval/scoring discipline from nothing — a labeled dataset, a deterministic scorer, and the judgment to tell a real bug apart from a scorer bug, converging from 50.5% to 98.0% accuracy across four documented rounds.
• Numbers-backed model selection — a real accuracy/cost/latency comparison across three LLM models, resulting in a documented decision to keep the existing model rather than switch.
• LLM application engineering — structured field extraction, clause-level citation linkage via chunking, and a configurable rule engine operating on extracted fields.
• Human-in-the-loop system design — every AI output is reviewable and overridable, with a full audit trail on both the AI's outputs and the human's actions.
• Deliberate, gap-driven learning — reusing proven skills from a prior production build instead of relearning them, and spending effort specifically on the confirmed, self-identified gaps.

Closing

This one is unfinished on purpose — it's being built in public, gap by confirmed gap, with the eval harness and model-comparison numbers above as the first solid proof that the approach works. The roadmap above is what's next, in order.