AI Contract Review Assistant

I'm building a FastAPI backend for legal/compliance-style contract review: upload a contract, extract the fields that actually matter (parties, payment terms, termination, renewal, liability cap, governing law), run them against a policy rule engine, and surface risk flags for a human reviewer to approve or override — with a full audit trail on every AI output and every reviewer action. Since the first draft of this article, it's grown into more than the extraction pipeline: contract Q&A grounded in real citations, and an autonomous agent that triages the review queue over the same MCP interface a human can use to resolve escalations. This is a project in progress, not a finished product, and this article says so plainly rather than dressing it up.

The problem and context

This project is Project 1 of a planned 3-project portfolio, and it's being built as the first four weeks of a structured, self-directed study plan closing specific, self-identified gaps against forward-deployed-engineering work. I'd already built and rewritten an AI resume/job-description matcher twice in production, so this project deliberately reuses those proven skills — LLM-based field extraction, rule-based scoring, chunking for retrieval — rather than relearning them. The genuinely new ground it's meant to break is the ground the resume-matcher project never had to cover: a formal eval/scoring harness, RAG citations, MCP tool exposure, Node/TypeScript, and Key Vault/Managed Identity.

The build

The backend is a Python/FastAPI service backed by SQL Server, with contract files stored in Azure Blob Storage. The core flow: a contract is uploaded and stored in Blob with a Document row created for it; analysis extracts the text (with real PDF support, not just plain-text), chunks it into paragraphs, runs LLM-based field extraction, links each extracted field back to the specific chunk it came from, and runs the results through a policy rule engine that raises named risk flags — MISSING_LIABILITY_CAP, UNAPPROVED_GOVERNING_LAW, AUTO_RENEWAL_NO_TERM, LONG_PAYMENT_TERMS — each one traceable back to the source clause. A reviewer can approve or override any flag, and a document can be corrected and re-uploaded as a new version without losing the analysis history on the previous one. Every AI extraction and every reviewer action is written to an audit trail.

The backend today: upload/analyze/review/reupload endpoints, an extraction-to-rule-engine pipeline, and a SQL Server + Blob Storage split for structured data vs. original files.

Phases 0–6 are done: the core data layer, Blob storage and upload, field extraction and the rule engine wired into /analyze, review actions, and the PDF/chunking/document-versioning work described above.

Since that first draft, two more weeks of the plan shipped: an MCP server (backend/mcp_server/server.py) wraps the real, running backend's upload, analyze, and get_document routes as MCP tools — calling over real HTTP rather than importing route functions in-process, so a tool call has the same real, persisted side effects a genuine external client would produce — plus a standalone evaluate_rules_preview tool so an agent can test which flags would fire against a hand-built fields dict without a full upload-and-analyze round trip. A new /documents/{id}/qa endpoint answers free-text questions grounded in one document's own chunks, with citations that reference real chunk_id/chunk_index/page_number rows rather than free-text quotes the model could fabricate; a question the contract doesn't answer returns "Not found in the provided excerpts." with zero citations, not a guess. A systematic unhappy-path design pass followed, enumerating 13 failure modes in writing rather than patching them case by case as they surfaced. All three were already named in the original roadmap below. What wasn't originally scoped at this granularity is the biggest addition: a full agents + MCP autonomous triage system, covered in its own section below.

"An MCP server is a standard, documented front door onto a system's existing capabilities... The business value isn't the protocol itself, it's what it removes: a support rep, a compliance reviewer, or an ops person can ask an assistant to 'check this contract's governing law' ... instead of learning an internal tool's UI or waiting on engineering to wire up a one-off integration."

Key production decisions

1. Building the eval harness neither prior production build ever had

Both of my earlier production builds (the resume-matcher's two rewrites) judged output quality by eye — there was never a scoring harness. This project closed that gap directly: a labeled 25-contract eval set and a scoring script (run_eval.py + test_scorer.py) that runs extraction against every contract and reports per-field and overall accuracy. It took four iterative rounds to go from a 50.5% baseline to 98.0% overall accuracy.

2. Using that harness to make a real model/cost decision, not swap models on instinct

I'd swapped LLM models before (GPT-3.5 to GPT-4, in the resume-matcher project) without ever measuring what actually changed. Here, the same 25-contract set was run against the production baseline (gpt-4.1-mini) and two cheaper candidates (gpt-4.1-nano, gpt-5-nano), extending the harness to capture real cost-per-contract (from actual token usage) and latency-per-contract alongside accuracy. The verdict was to keep the baseline: gpt-5-nano was closer on accuracy (97.0% vs. 98.0%) but both pricier and nearly twice as slow — a direct counter-example to assuming a newer or smaller-named model is automatically cheaper or faster. gpt-4.1-nano was a real trade-off (4.4x cheaper, 3.4x faster) but collapsed renewal_term accuracy to 60%, the exact field the AUTO_RENEWAL_NO_TERM rule depends on — disqualifying, despite the cost savings. This is the same shape of numbers-backed keep/reverse judgment as the resume-matcher project's own cost audit, this time about model choice instead of vector-store choice.

3. RAG grounding bugs that only real testing surfaced

The RAG Q&A feature shipped clean on its own eval harness, then broke in two different ways once I actually used it. First: citation excerpts were technically correct but misleading — the displayed excerpt was always the first 300 characters of the cited chunk, regardless of where the relevant sentence actually sat. The fix was to have the model return a verbatim supporting_quote per citation and center the displayed excerpt on it, but that surfaced a second bug: exact-substring matching silently failed on roughly 80% of real cases, because the source .txt contracts hard-wrap long sentences, so a quote spanning a wrapped line has a plain space where the chunk has a newline. A whitespace-tolerant regex matcher, shared by both the production code and the eval harness, fixed it — the harness's "quotes verbatim-verified" metric went from 2/10 to 10/10, with grounded accuracy and citation rate holding steady at 10/10 and 9/9 throughout.

Second, after adding cross-document Q&A (POST /documents/qa-portfolio, ranking chunks across multiple documents at once) and verifying it against a real two-document comparison question, a follow-up test with a large, unrelated document already in the database exposed a subtler bug: with top_k shared across the whole corpus, a 273-chunk unrelated document had far more chances to land in the top-k purely from volume, crowding out the actually-relevant document's chunks entirely. With no source-document labeling in the prompt, the model had no way to notice none of what it saw was actually about the named company — so it confidently attributed the wrong document's number to it. The fix was two changes together: a per-document retrieval fairness cap (3 chunks per document by default, backfilled from what's left if under-filled) and source-filename labels on every excerpt shown to the model, plus an explicit system-prompt rule not to attribute a fact unless the excerpt actually supports it. I re-verified against the exact original failure case, with the interfering document still present in the database, not a fresh, easier re-test.

4. An autonomous triage agent, built as a real MCP client

The largest thing built since the first draft: an autonomous agent (backend/agent/triage_agent.py) that clears the document review queue — discovers documents awaiting review, analyzes them, decides approve vs. escalate based on risk-flag severity, and submits that decision back — driving the real backend entirely through the same MCP server from Day 9-10, spawned as a subprocess and talked to over the actual MCP protocol (a real ClientSession, stdio transport). I deliberately chose a true MCP client over the simpler direct-HTTP alternative, for the stronger, more literal "agent that speaks MCP" story.

A few decisions locked in along the way: the escalation threshold (medium and high severity both escalate) is switchable to high-only via one config value, no code change needed; the escalation reason is mandatory and specific, naming the triggering risk flag's rule code rather than a generic "needs review"; and two different vendors do two different jobs, not redundancy — OpenAI's gpt-4.1-mini stays the field-extraction model unchanged from Day 4/8, while Claude (claude-sonnet-5, configurable) runs the agent's own judgment/tool-use loop. It ships as a CLI script, not a worker or scheduled service — a deliberate scope boundary, not an oversight.

The part that makes this a genuine human-in-the-loop system rather than just automation is the resolution loop: the agent doesn't block waiting for a human — unlike an earlier prior-art build of mine that pauses a whole review live in a console chat until a human types a decision — it's fire-and-forget: decide, record, move to the next document. Two new MCP tools, list_escalated_documents and qa_document, let a human resolve escalations later, asynchronously, through the exact same MCP interface the autonomous agent uses. I proved this "one interface, two consumers" design for real, not just designed it on paper: a human resolved a live escalation conversationally through MCP tools — list escalated documents, ask why, ask a grounded follow-up question about the actual clause, then submit a reject decision with a real reviewer identity — with zero new bespoke resolution UI, and confirmed identical results via both the raw API and the MCP tool afterward.

One MCP server, two consumers: an autonomous agent that decides and moves on, and a human who resolves escalations later through the same interface.

I verified this live, end to end, twice. First, direct MCP calls bypassing Claude's reasoning entirely: all 8 tools discovered correctly via session.list_tools() with real JSON-schema inputSchemas, and get_document, submit_review_action, list_escalated_documents, and qa_document each independently confirmed against real data. Second, a real Claude-driven reasoning loop, run for real: triage_agent.py against a deliberately-built adversarial test document (a synthetic contract engineered to collide with a real one on provider name, termination period, and governing law) correctly escalated it with a specific reason naming the exact triggering rule and clause — then a human resolved that same escalation via the conversational MCP flow above and rejected it for a real business reason (jurisdiction mismatch).

Honestly flagged as open, not blockers: the agent isn't yet exposed as its own MCP tool a human could trigger conversationally ("run triage now"); a full batch run against the whole pending queue hasn't been observed yet, though every underlying mechanic is independently verified; and approving or rejecting a document doesn't auto-resolve its underlying risk flag — that's a separate, deliberate override_flag action, since document-level and flag-level status are tracked independently by design, not by accident.

Current state

Stated plainly: Weeks 1 through 3 of the structured four-week plan are now done, with one explicit exception — the Node.js/TypeScript port (originally planned for Day 11-12, before the RAG work above) was skipped over and hasn't been built. No node-port folder exists anywhere in the repo yet; it stays in the roadmap below rather than getting marked complete. On top of the plan itself, the agents + MCP triage system described above goes meaningfully beyond what the original roadmap scoped for that stage. One infrastructure note along the way: the original Azure Storage account went unreachable mid-project (a DNS failure, almost certainly deleted and recreated under a new name) and had to be re-provisioned under a new name with an idempotent CLI script — a small reminder that early infra choices need to be reproducible, not just working once. Week 4 — hardening, Key Vault/Managed Identity, the business narrative, and publishing — hasn't started, and this article isn't claiming otherwise.

What's coming — the full roadmap

Rather than a vague "more features coming soon," here's every remaining stage, in the order it's planned:

• A Node.js/TypeScript port — a bounded port of the existing Python MCP server to Node + the official TypeScript MCP SDK, verified side-by-side against the Python original, to close a confirmed zero-prior-exposure gap in a stack much of the AI-agent tooling world ships first. Still not started, out of its original sequence — skipped over while the RAG and agents work above happened, not built yet.
• Hardening — request-ID middleware and structured logging done properly, a cost-conscious OpenTelemetry setup, and a real multi-stage CI/CD pipeline with gates.
• Production-grade secrets and identity — moving API keys and connection strings into Key Vault, and authenticating to Key Vault/Blob/SQL via Managed Identity instead of keys or connection strings.
• The business and engineering narrative — writing up the architecture decisions and the eval/cost numbers above into both a technical narrative and a business-facing one, compressed down to a short pitch.
• Publishing and a capstone pitch — this portfolio write-up is part of that step, folding the eval-harness, model-comparison, and RAG/agent numbers above in as the evidence.

Skills and technologies this demonstrates

• Building an eval/scoring discipline from nothing — a labeled dataset, a deterministic scorer, and the judgment to tell a real bug apart from a scorer bug, converging from 50.5% to 98.0% accuracy across four documented rounds.
• Numbers-backed model selection — a real accuracy/cost/latency comparison across three LLM models, resulting in a documented decision to keep the existing model rather than switch.
• LLM application engineering — structured field extraction, clause-level citation linkage via chunking, and a configurable rule engine operating on extracted fields.
• Human-in-the-loop system design — every AI output is reviewable and overridable, with a full audit trail on both the AI's outputs and the human's actions, extended into a genuine async resolution loop between an autonomous agent and a human reviewer.
• RAG grounding discipline — citations that reference real chunk rows instead of free-text quotes, verbatim-quote verification, and retrieval-fairness fixes found by testing against a real multi-document corpus rather than a small, forgiving demo set.
• MCP protocol implementation, both directions — exposing a real backend as MCP tools with FastMCP, and building an autonomous agent that consumes those same tools as a true MCP client (a real ClientSession over stdio, not a simpler direct-HTTP shortcut).
• Deliberate, gap-driven learning — reusing proven skills from a prior production build instead of relearning them, and spending effort specifically on the confirmed, self-identified gaps.

Closing

This one is unfinished on purpose — it's being built in public, gap by confirmed gap. The eval harness and model-comparison numbers were the first solid proof the approach works; the RAG grounding fixes and the MCP triage agent's verified human-in-the-loop resolution loop are the next round of it. The roadmap above is what's left, in order.