Code examples:

Contract Risk & Clause Negotiation Agent — Building an AI That Flags Risky Clauses and Drafts Redlines

November 9, 2026 20-22 min read By Jaffar Kazi
AI Development Legal Tech Contract Negotiation
Python C# Azure

I built a contract risk agent that could find a bad indemnity clause in about eight seconds. Then a lawyer asked me a question I couldn't answer: "Risky compared to what?" That question reshaped the whole system, and it's why this article is about playbooks and verification far more than it's about prompts.

What You'll Learn

  • Why "is this clause risky?" is the wrong question, and how to score clauses against a structured negotiation playbook instead
  • The agent architecture: segmentation, classification, playbook retrieval, risk scoring, redline drafting, verification, and a lawyer review gate
  • How to stop a model from inventing legal language by constraining redlines to a pre-approved clause library and verifying the diff
  • Real per-contract token costs and the monthly bill for a team reviewing 150 contracts
  • When this system is the wrong tool, including the deal types where I'd keep it switched off

Not Legal Advice

This article describes a software architecture for assisting legal professionals. It is not legal advice, and the system described is designed so that a qualified lawyer makes every final decision. Nothing here replaces that review.

1. The Problem With "Just Review It Quickly"

Every commercial team has the same ritual. Sales closes a deal, the customer sends their paper (or the vendor sends theirs), and the contract lands in a legal queue. A lawyer reads 40 pages looking for the same dozen things they looked for in the last 40 contracts: the liability cap, whether indemnities are uncapped, how termination works, who owns the IP, where the data goes, how auto-renewal is worded.

Most of this isn't legal brilliance. It's pattern matching against a mental checklist that lives in a senior lawyer's head, and occasionally in a half-maintained Word document called "Negotiation Playbook v7 FINAL". The lawyer's real value is judgment on the three clauses that matter on this particular deal. The other 57 clauses are a tax on their time.

This is the gap I built this system to close. It isn't about replacing the lawyer. It's about making sure their first hour on a contract starts at the three clauses that need judgment, with the other 57 already classified, scored, and (where they deviate) paired with a proposed fix. Existing tools don't quite do this. Generic contract summarizers tell you what a document says. They don't tell you whether it's acceptable to your company, and that's the only question the business cares about.

2. How This Gets Done Today (and Where It Breaks)

Teams usually handle contract review in one of three ways, and each fails differently.

The manual playbook approach: a lawyer or contracts manager reads the document with the playbook open in another window. It works, and it's the standard everything else gets measured against. It also scales linearly with headcount. At roughly two to three hours per mid-complexity contract, a team seeing 150 contracts a month needs several full-time reviewers, and the queue still backs up at quarter end, which is exactly when sales needs answers fastest.

Clause-extraction tools: many contract platforms extract and tag clauses ("this is a limitation of liability clause"). That's useful, but it stops at labelling. A tag that says "indemnity" doesn't tell you the indemnity is uncapped, covers third-party IP claims and data breaches, and sits outside the liability cap. That interaction is the actual risk.

The "paste it into a chatbot" approach: this is what happens when the queue gets long enough. Someone pastes a clause into a general-purpose model and asks whether it's risky. The answer is fluent and plausible, and it's calibrated to nothing. The model has no idea your company accepts a cap of 12 months' fees but will walk away from anything under 6. It also may confidently suggest language your lawyers have never approved. And pasting confidential counterparty contracts into an unsanctioned tool is its own problem.

The cost of the status quo shows up in two places: reviewer hours, and deals that stall. Neither is dramatic on its own. Together they're a steady drag, and that's the kind of cost that justifies engineering effort without needing a heroic ROI story.

3. The Solution Approach: Deviation From a Playbook, Not Vibes

Here's what surprised me most building this: the model's ability to read a clause was never the bottleneck. GPT-4o reads contract language well. The bottleneck was giving it a defensible standard to compare against.

So the core design idea is to turn "risk" into deviation from a position your company has already decided. Every clause type in the playbook has three positions:

  • Preferred: what you'd write if you held the pen.
  • Fallback: what you'll accept after negotiation, usually with conditions.
  • Walk-away: the line where the deal needs senior sign-off or doesn't happen.

The agent's job for each clause is then narrow and checkable: classify the clause type, retrieve the matching playbook entry, decide which band the clause falls into, and cite the playbook entry it relied on. If it's outside preferred, it drafts a redline using language from an approved clause library, not language it invented.

The Design Principle

Let the model do what it's good at (reading messy language, matching it to a structured position, drafting within constraints) and let deterministic code and humans do what they're good at (verifying, deciding, and being accountable). Every place the model could improvise is a place a lawyer would have to re-check.

The stack: Azure OpenAI (GPT-4o) for classification, scoring, and drafting; Azure AI Document Intelligence for turning DOCX and PDF into structured text; Azure AI Search to hold the playbook and the clause library; and either LangGraph (Python) or Semantic Kernel (C#) for orchestration. The lawyer review gate is a first-class node in the workflow, not a UI feature added later.

4. Architecture Overview

The pipeline has four phases: intake, analysis, drafting and verification, and human review.

Architecture diagram showing contract intake through Document Intelligence and clause segmentation, a clause classifier, playbook retrieval from Azure AI Search, risk scoring, a redline drafting agent checked by a verification guard, a cross-clause checker, and a lawyer review gate producing tracked-changes DOCX, a negotiation brief, and an audit log, with lawyer overrides recalibrating the playbook

Intake. Document Intelligence extracts layout and structure, preserving numbering, headings, and tables. The segmenter then splits the contract into clauses and, importantly, pulls out the definitions section separately. Definitions matter more than people expect: "Confidential Information" or "Fees" can quietly change what a liability cap actually covers.

Analysis. The classifier assigns each clause a type from your taxonomy (limitation of liability, indemnity, termination, IP ownership, data protection, governing law, and so on). Azure AI Search then retrieves the matching playbook entry using a hybrid of keyword and vector search, filtered by the classified clause type. The scoring agent compares the clause against the playbook's three positions and returns a band plus the ID of the playbook entry it used.

Drafting and verification. Clauses outside the preferred band go to the redline drafting agent, which proposes edits assembled from the approved clause library. A verification guard then checks the proposed text with deterministic code before anything moves on. Failures loop back to the drafter with the specific reason. A cross-clause checker looks at interactions: does the proposed cap conflict with an uncapped indemnity elsewhere?

Human review. Nothing leaves the system without a lawyer approving, editing, or rejecting it. Approved redlines become a tracked-changes DOCX. The system also produces a short negotiation brief (must-haves versus tradeable items) and writes a full audit log. Lawyer overrides feed back into the playbook review process, which I'll come back to.

5. Core Implementation

The state model carries a clause through the pipeline. I keep the extracted contract text, the model's inferences, and the lawyer's decisions in clearly separate fields, so an audit can always tell which is which.

models.py

from dataclasses import dataclass, field
from enum import Enum

class RiskBand(str, Enum):
    PREFERRED = "preferred"
    FALLBACK = "fallback"        # acceptable with conditions
    WALK_AWAY = "walk_away"      # needs senior sign-off
    NO_PLAYBOOK = "no_playbook"  # clause type not covered

class ReviewDecision(str, Enum):
    APPROVE = "approve"
    EDIT = "edit"
    REJECT = "reject"

@dataclass
class Clause:
    clause_id: str               # e.g. "14.2"
    heading: str
    text: str                    # verbatim from the contract, never rewritten
    clause_type: str | None = None
    classifier_confidence: float = 0.0

@dataclass
class ClauseAssessment:
    clause: Clause
    playbook_entry_id: str | None = None   # which playbook entry was applied
    band: RiskBand = RiskBand.NO_PLAYBOOK
    rationale: str = ""
    cited_text: str = ""                   # exact span that drove the band
    proposed_redline: str | None = None
    library_clause_ids: list[str] = field(default_factory=list)
    verification_passed: bool = False
    decision: ReviewDecision | None = None  # lawyer's call, set at the gate
    lawyer_edit: str | None = None

@dataclass
class ContractReview:
    contract_id: str
    definitions: dict[str, str] = field(default_factory=dict)
    assessments: list[ClauseAssessment] = field(default_factory=list)
    cross_clause_flags: list[str] = field(default_factory=list)
    redraft_count: int = 0       # caps verify-and-redraft loops

public enum RiskBand { Preferred, Fallback, WalkAway, NoPlaybook }

public enum ReviewDecision { Approve, Edit, Reject }

public record Clause(
    string ClauseId,               // e.g. "14.2"
    string Heading,
    string Text)                   // verbatim from the contract, never rewritten
{
    public string? ClauseType { get; set; }
    public double ClassifierConfidence { get; set; }
}

public class ClauseAssessment
{
    public required Clause Clause { get; init; }
    public string? PlaybookEntryId { get; set; }      // which playbook entry was applied
    public RiskBand Band { get; set; } = RiskBand.NoPlaybook;
    public string Rationale { get; set; } = "";
    public string CitedText { get; set; } = "";       // exact span that drove the band
    public string? ProposedRedline { get; set; }
    public List<string> LibraryClauseIds { get; set; } = new();
    public bool VerificationPassed { get; set; }
    public ReviewDecision? Decision { get; set; }     // lawyer's call, set at the gate
    public string? LawyerEdit { get; set; }
}

public class ContractReview
{
    public required string ContractId { get; init; }
    public Dictionary<string, string> Definitions { get; set; } = new();
    public List<ClauseAssessment> Assessments { get; set; } = new();
    public List<string> CrossClauseFlags { get; set; } = new();
}

Two details are worth pointing out. text is verbatim and never modified, because the redline is always a separate artifact. And NO_PLAYBOOK is a real band, not an error. When a clause type isn't covered, I want the system to say so plainly instead of guessing.

The playbook itself is data, not code. Each entry is a record your legal team owns. A simplified example for a limitation of liability clause looks like this:

PositionRule (playbook entry LOL-001)Approved language
PreferredMutual cap at 12 months of fees paid or payable; consequential loss excludedLIB-LOL-A
FallbackCap no lower than 6 months of fees; super-cap permitted for data breach up to 2x feesLIB-LOL-B
Walk-awayUncapped liability for us; cap below 6 months of fees; one-sided cap favouring counterpartyEscalate to General Counsel

Orchestration runs the steps in sequence with a conditional branch: only clauses outside the preferred band go to the drafter. The review gate is an explicit node. In LangGraph it's an interrupt, so the workflow pauses and waits for the lawyer's decision before any output is produced.

orchestrator.py

from langgraph.graph import StateGraph, END
from langgraph.checkpoint.memory import MemorySaver

MAX_REDRAFTS = 2

def needs_redline(state: ContractReview) -> str:
    flagged = [a for a in state.assessments
               if a.band in (RiskBand.FALLBACK, RiskBand.WALK_AWAY)]
    return "draft_redlines" if flagged else "cross_clause_check"

def verification_route(state: ContractReview) -> str:
    failed = [a for a in state.assessments
              if a.proposed_redline and not a.verification_passed]
    if failed and state.redraft_count < MAX_REDRAFTS:
        return "draft_redlines"          # loop back with failure reasons
    return "cross_clause_check"          # unresolved failures go to the lawyer flagged

def build_contract_pipeline():
    graph = StateGraph(ContractReview)

    graph.add_node("classify_clauses", classify_clauses_node)
    graph.add_node("score_risk", score_risk_node)
    graph.add_node("draft_redlines", draft_redlines_node)
    graph.add_node("verify_redlines", verify_redlines_node)
    graph.add_node("cross_clause_check", cross_clause_check_node)
    graph.add_node("await_lawyer_review", await_lawyer_review_node)  # pauses here

    graph.set_entry_point("classify_clauses")
    graph.add_edge("classify_clauses", "score_risk")
    graph.add_conditional_edges("score_risk", needs_redline)
    graph.add_edge("draft_redlines", "verify_redlines")
    graph.add_conditional_edges("verify_redlines", verification_route)
    graph.add_edge("cross_clause_check", "await_lawyer_review")
    graph.add_edge("await_lawyer_review", END)

    return graph.compile(
        checkpointer=MemorySaver(),   # use a durable checkpointer in production
        interrupt_before=["await_lawyer_review"],
    )

public class ContractPipeline
{
    private const int MaxRedrafts = 2;

    private readonly ClauseClassifier _classifier;
    private readonly RiskScorer _scorer;
    private readonly RedlineDrafter _drafter;
    private readonly RedlineVerifier _verifier;
    private readonly CrossClauseChecker _crossChecker;

    public ContractPipeline(ClauseClassifier c, RiskScorer s, RedlineDrafter d,
                            RedlineVerifier v, CrossClauseChecker x)
        => (_classifier, _scorer, _drafter, _verifier, _crossChecker) = (c, s, d, v, x);

    public async Task<ContractReview> RunUntilReviewGateAsync(ContractReview review)
    {
        await _classifier.ClassifyAsync(review);
        await _scorer.ScoreAsync(review);

        for (var attempt = 0; attempt <= MaxRedrafts; attempt++)
        {
            var flagged = review.Assessments
                .Where(a => a.Band is RiskBand.Fallback or RiskBand.WalkAway
                            && !a.VerificationPassed)
                .ToList();
            if (flagged.Count == 0) break;

            await _drafter.DraftAsync(flagged);      // receives prior failure reasons
            await _verifier.VerifyAsync(flagged);
        }

        await _crossChecker.CheckAsync(review);

        // Pipeline halts here. No output is generated until the lawyer's
        // decisions are submitted through ApplyLawyerDecisionsAsync.
        return review;
    }
}

Note the redraft cap. In practice, you'll find that a drafter that fails verification twice usually fails a third time for the same reason, usually because the playbook entry and the clause library disagree. Better to surface that to a human than to burn tokens looping.

6. Key Technical Challenge #1: Scoring Against the Playbook, Not Against Intuition

The tricky part about risk scoring is that a model asked "how risky is this clause?" will produce a confident answer that varies run to run and can't be audited. A lawyer who sees "risk: high" next to a liability cap has no way to know whether that means "below our floor" or "I've read scary things about caps".

The fix is to make the model do a comparison instead of an opinion. It receives the clause and the retrieved playbook entry, and it must pick one of the three bands, quote the exact span of the contract that justifies the choice, and cite the playbook entry ID. If it can't quote a span, the result is rejected in code. That one rule removes most of the "plausible but unsupported" failures.

risk_scoring.py

RISK_SCORING_PROMPT = """
You compare ONE contract clause against ONE company playbook entry.

Rules:
- Choose exactly one band: preferred, fallback, or walk_away.
- Use walk_away if ANY walk-away condition is met, even if others are fine.
- Quote the exact words from the clause that decided the band (cited_text).
- If the clause is silent on something the playbook requires, say so in rationale
  and treat the silence as a deviation.
- Do not use outside legal knowledge. Judge only against the playbook entry.

Clause {clause_id}: {clause_text}

Playbook entry {entry_id}:
PREFERRED: {preferred}
FALLBACK: {fallback}
WALK_AWAY: {walk_away}

Return JSON: band, rationale, cited_text.
"""

def score_clause(assessment: ClauseAssessment, entry: dict) -> ClauseAssessment:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": RISK_SCORING_PROMPT.format(
            clause_id=assessment.clause.clause_id,
            clause_text=assessment.clause.text,
            entry_id=entry["id"], preferred=entry["preferred"],
            fallback=entry["fallback"], walk_away=entry["walk_away"])}],
        temperature=0.0,
        response_format={"type": "json_object"},
    )
    result = json.loads(response.choices[0].message.content)

    # Deterministic guard: the quoted span must actually exist in the clause.
    if normalise(result["cited_text"]) not in normalise(assessment.clause.text):
        assessment.band = RiskBand.WALK_AWAY      # fail safe, not fail open
        assessment.rationale = "Model citation could not be verified; escalated."
        return assessment

    assessment.playbook_entry_id = entry["id"]
    assessment.band = RiskBand(result["band"])
    assessment.rationale = result["rationale"]
    assessment.cited_text = result["cited_text"]
    return assessment

public class RiskScorer
{
    private const string PromptTemplate = """
        You compare ONE contract clause against ONE company playbook entry.

        Rules:
        - Choose exactly one band: preferred, fallback, or walk_away.
        - Use walk_away if ANY walk-away condition is met, even if others are fine.
        - Quote the exact words from the clause that decided the band (cited_text).
        - If the clause is silent on something the playbook requires, say so in rationale
          and treat the silence as a deviation.
        - Do not use outside legal knowledge. Judge only against the playbook entry.

        Clause {0}: {1}

        Playbook entry {2}:
        PREFERRED: {3}
        FALLBACK: {4}
        WALK_AWAY: {5}

        Return JSON: band, rationale, cited_text.
        """;

    private readonly IChatCompletionService _chat;

    public RiskScorer(IChatCompletionService chat) => _chat = chat;

    public async Task ScoreClauseAsync(ClauseAssessment a, PlaybookEntry entry)
    {
        var prompt = string.Format(PromptTemplate, a.Clause.ClauseId, a.Clause.Text,
            entry.Id, entry.Preferred, entry.Fallback, entry.WalkAway);

        var result = await _chat.GetChatMessageContentAsync(prompt,
            new OpenAIPromptExecutionSettings { Temperature = 0.0, ResponseFormat = "json_object" });
        var parsed = JsonSerializer.Deserialize<ScoreResult>(result.Content!)!;

        // Deterministic guard: the quoted span must actually exist in the clause.
        if (!Normalise(a.Clause.Text).Contains(Normalise(parsed.CitedText)))
        {
            a.Band = RiskBand.WalkAway;               // fail safe, not fail open
            a.Rationale = "Model citation could not be verified; escalated.";
            return;
        }

        a.PlaybookEntryId = entry.Id;
        a.Band = Enum.Parse<RiskBand>(parsed.Band, ignoreCase: true);
        a.Rationale = parsed.Rationale;
        a.CitedText = parsed.CitedText;
    }
}

Fail Safe, Not Fail Open

When the citation check fails, the clause is escalated rather than quietly passed. A contract risk tool that fails open trains reviewers to trust a green tick it hasn't earned. I'd rather a lawyer look at a few extra clauses than miss one because the model's evidence didn't hold up.

7. Key Technical Challenge #2: Redlines Without Invented Legal Language

Drafting is where legal AI gets genuinely dangerous. A model asked to "fix this clause" will write something that reads like law and might not be. It might create ambiguity, drop a carve-out, or introduce a term your company has never used. A lawyer reviewing 30 redlines will catch most problems, but "most" isn't a standard you want to rely on.

My approach has three layers. First, the drafter doesn't write from scratch. It selects and adapts from the approved clause library, language your lawyers have already cleared, and it must return the library IDs it used. Second, edits should be minimal: change the number, the party, or the carve-out, and leave the surrounding text alone. Third, a verification guard checks the result with plain code before a human ever sees it.

The guard does things a model is bad at and code is good at: confirming that every number in the redline appears in the approved library language or the playbook, confirming that defined terms used are defined in the contract, and confirming that the redline's non-edited text is character-for-character unchanged from the original.

redline_verifier.py

import re
import difflib

NUMBER_PATTERN = re.compile(r"\b\d+(?:[.,]\d+)?\s*(?:%|months?|days?|years?|x)?", re.I)

def verify_redline(a: ClauseAssessment, library: dict[str, str],
                   definitions: dict[str, str]) -> list[str]:
    problems: list[str] = []
    allowed_text = " ".join(library[i] for i in a.library_clause_ids if i in library)

    # 1. Every ID the model claims to have used must exist in the library.
    missing = [i for i in a.library_clause_ids if i not in library]
    if missing:
        problems.append(f"Unknown library clause IDs: {missing}")

    # 2. Every number in the redline must come from approved language
    #    or from the original clause. No invented figures.
    source = allowed_text + " " + a.clause.text
    for num in NUMBER_PATTERN.findall(a.proposed_redline):
        if num.strip() and num.strip() not in source:
            problems.append(f"Figure '{num.strip()}' not found in approved sources")

    # 3. Defined terms used in the redline must be defined in this contract.
    for term in re.findall(r"\b[A-Z][a-z]+(?: [A-Z][a-z]+)+\b", a.proposed_redline):
        if term not in definitions and term not in allowed_text:
            problems.append(f"Capitalised term '{term}' is undefined")

    # 4. Unchanged text must stay unchanged. Measure how much was rewritten.
    ratio = difflib.SequenceMatcher(None, a.clause.text, a.proposed_redline).ratio()
    if ratio < 0.45:
        problems.append(f"Redline rewrites too much of the clause (similarity {ratio:.2f})")

    a.verification_passed = not problems
    return problems   # fed back to the drafter on the next attempt

using System.Text.RegularExpressions;

public class RedlineVerifier
{
    private static readonly Regex NumberPattern =
        new(@"\b\d+(?:[.,]\d+)?\s*(?:%|months?|days?|years?|x)?", RegexOptions.IgnoreCase);
    private static readonly Regex TermPattern =
        new(@"\b[A-Z][a-z]+(?: [A-Z][a-z]+)+\b");

    public List<string> Verify(ClauseAssessment a,
        IReadOnlyDictionary<string, string> library,
        IReadOnlyDictionary<string, string> definitions)
    {
        var problems = new List<string>();
        var redline = a.ProposedRedline ?? "";
        var allowedText = string.Join(" ",
            a.LibraryClauseIds.Where(library.ContainsKey).Select(id => library[id]));

        // 1. Every ID the model claims to have used must exist in the library.
        var missing = a.LibraryClauseIds.Where(id => !library.ContainsKey(id)).ToList();
        if (missing.Count > 0)
            problems.Add($"Unknown library clause IDs: {string.Join(", ", missing)}");

        // 2. Every number in the redline must come from approved language
        //    or from the original clause. No invented figures.
        var source = allowedText + " " + a.Clause.Text;
        foreach (Match m in NumberPattern.Matches(redline))
        {
            var figure = m.Value.Trim();
            if (figure.Length > 0 && !source.Contains(figure))
                problems.Add($"Figure '{figure}' not found in approved sources");
        }

        // 3. Defined terms used in the redline must be defined in this contract.
        foreach (Match m in TermPattern.Matches(redline))
            if (!definitions.ContainsKey(m.Value) && !allowedText.Contains(m.Value))
                problems.Add($"Capitalised term '{m.Value}' is undefined");

        // 4. Unchanged text must stay unchanged (similarity check omitted for
        //    brevity; use a diff library such as DiffPlex to compute it).
        a.VerificationPassed = problems.Count == 0;
        return problems;   // fed back to the drafter on the next attempt
    }
}

The trade-off here is that the verifier is deliberately conservative. It will occasionally reject a perfectly good redline because a number appears in a slightly different format. I accept that. A false rejection costs one extra drafting attempt. A false pass costs a lawyer's trust. Over time, you tune the patterns using the rejection log.

The cross-clause checker deserves a brief mention because it catches what clause-by-clause review misses. A redline that lowers the liability cap to 12 months' fees looks fine in isolation, but if clause 18 makes the indemnity uncapped, the cap is largely decorative. The checker takes the final set of assessments plus the definitions and asks the model one narrow question: do any of these clauses undermine each other? Its output is a list of flags for the lawyer, never an automatic edit.

8. Cost Analysis: Real Numbers for 150 Contracts a Month

Here's the token math for a typical contract of about 40 pages (~18,000 tokens) with roughly 60 clauses, using GPT-4o list pricing at the time of writing ($2.50 per million input tokens, $10 per million output tokens). Prices change, so check current rates before budgeting.

StageTokens per contractCost per contract
Clause classification (6 batches of 10 clauses)~21,000 in / 3,600 out$0.089
Playbook retrieval embeddings (60 queries)~9,000 tokens$0.0002
Risk scoring (~25 playbook-covered clauses)~22,500 in / 6,250 out$0.119
Redline drafting (~10 flagged clauses)~14,000 in / 4,500 out$0.080
Cross-clause check~6,000 in / 800 out$0.023
Negotiation brief~3,000 in / 900 out$0.017
Total model cost per contract—~$0.33

Redrafts after failed verification add roughly 10-15%, so budget about $0.37 per contract. For 150 contracts a month, that's about $55 in model tokens. The larger costs are the surrounding infrastructure.

Monthly line item (150 contracts)Estimate
Azure OpenAI tokens~$55
Document Intelligence (150 × ~30 pages = 4,500 pages at ~$10 per 1,000)~$45
Azure AI Search (Basic tier, small playbook and library index)~$75
App hosting, state store, Key Vault, monitoring~$150
Total~$325 per month (~$2.20 per contract)

The Cost You Won't See on the Azure Bill

Building and maintaining the playbook and clause library is the real investment. Expect your legal team to spend several weeks codifying positions the first time, then a few hours a month maintaining them. If the playbook is vague, the agent's output will be too, and no amount of prompt engineering fixes that.

9. Observability & Debugging

Every scoring and drafting call is traced with the prompt version, the playbook entry ID, the library clause IDs, and the verification result. This isn't generic AI-observability hygiene. Six months after a contract is signed, someone may ask why a clause was accepted, and "the model said it was fine" is not an answer a legal department can use. The trace is the answer.

observability.py

from opentelemetry import trace

tracer = trace.get_tracer("contract-risk-agent")

def traced_score(assessment: ClauseAssessment, entry: dict) -> ClauseAssessment:
    with tracer.start_as_current_span("score_clause") as span:
        span.set_attribute("clause_id", assessment.clause.clause_id)
        span.set_attribute("clause_type", assessment.clause.clause_type or "unknown")
        span.set_attribute("prompt_version", "risk-scoring-v4")
        span.set_attribute("playbook_entry_id", entry["id"])
        span.set_attribute("playbook_version", entry["version"])

        result = score_clause(assessment, entry)

        span.set_attribute("band", result.band.value)
        span.set_attribute("citation_verified", result.cited_text != "")
        return result

def record_lawyer_decision(a: ClauseAssessment):
    # The metric that matters most: how often does a lawyer disagree?
    meter.create_counter("lawyer_decisions").add(1, {
        "clause_type": a.clause.clause_type,
        "band": a.band.value,
        "decision": a.decision.value,          # approve / edit / reject
    })

using System.Diagnostics;
using System.Diagnostics.Metrics;

public static class ContractTelemetry
{
    private static readonly ActivitySource Source = new("ContractRiskAgent");
    private static readonly Meter AppMeter = new("ContractRiskAgent");
    private static readonly Counter<long> LawyerDecisions =
        AppMeter.CreateCounter<long>("lawyer_decisions");

    public static async Task ScoreTracedAsync(RiskScorer scorer, ClauseAssessment a, PlaybookEntry entry)
    {
        using var activity = Source.StartActivity("score_clause");
        activity?.SetTag("clause_type", a.Clause.ClauseType ?? "unknown");
        activity?.SetTag("prompt_version", "risk-scoring-v4");
        activity?.SetTag("playbook_entry_id", entry.Id);
        activity?.SetTag("playbook_version", entry.Version);

        await scorer.ScoreClauseAsync(a, entry);

        activity?.SetTag("band", a.Band.ToString());
        activity?.SetTag("citation_verified", !string.IsNullOrEmpty(a.CitedText));
    }

    public static void RecordLawyerDecision(ClauseAssessment a) =>
        // The metric that matters most: how often does a lawyer disagree?
        LawyerDecisions.Add(1,
            new KeyValuePair<string, object?>("clause_type", a.Clause.ClauseType),
            new KeyValuePair<string, object?>("band", a.Band.ToString()),
            new KeyValuePair<string, object?>("decision", a.Decision?.ToString()));
}

The lawyer override rate, broken down by clause type, is the single most useful production metric. If lawyers edit 40% of the indemnity redlines but 3% of the governing-law ones, you know exactly where the playbook or the library needs work. That's also the feedback loop in the architecture diagram: overrides don't retrain the model, they trigger a playbook review. The fix is usually a clearer rule or a better library clause, not a different prompt.

Beyond that, I track the citation-verification failure rate (it should be low and stable), the redraft rate, and the proportion of clauses landing in NO_PLAYBOOK. A rising NO_PLAYBOOK share means counterparties are sending clause types your playbook doesn't yet cover.

10. Technology Choices: Python vs C#

Python Implementation

Why choose Python: If your team writes Python, you get access to the richest AI/ML ecosystem.

  • Library ecosystem — LangChain, LangGraph, and mature document-processing libraries (python-docx, difflib) for generating tracked changes
  • Rapid prototyping — quick iteration on prompts and playbook structure, with Jupyter support for evaluating against labelled contracts
  • Community — most AI tutorials and evaluation tooling are Python-first

C#/.NET Implementation

Why choose C#: If your backend is .NET, you get first-party Microsoft support and enterprise patterns.

  • Native Azure integration — first-party SDKs, Microsoft-maintained, with strong support for Azure AI Search and Document Intelligence
  • Enterprise patterns — dependency injection, strong typing for playbook and clause models, and the Open XML SDK for reading and writing DOCX with real tracked changes
  • Production-ready — battle-tested at scale, and a natural fit if your contract management system is already a .NET application

The Bottom Line

Python team? Use Python. C#/.NET team? Use C#. Don't fight your stack. One practical note: DOCX tracked changes are the fiddliest part of the whole build in either language, so budget more time for it than for the LLM work.

11. Azure Infrastructure

The services you need are straightforward. Azure OpenAI hosts the models. Azure AI Document Intelligence handles layout extraction. Azure AI Search holds the playbook and clause library as an index with both vector and keyword fields. A durable store (Azure SQL or Cosmos DB) keeps review state and the audit log. Key Vault holds secrets, and Azure Monitor with Application Insights collects the traces.

For legal workloads, the network and data posture matters as much as the features. Use private endpoints for Azure OpenAI and AI Search, keep everything in one region that satisfies your data-residency obligations, and confirm your data-handling terms for Azure OpenAI with your legal and security teams. Counterparty contracts routinely contain confidentiality clauses that restrict where the text can go, so this isn't optional diligence.

Azure AI Foundry Agent Service

Azure AI Foundry Agent Service is now generally available, providing managed orchestration for AI systems.

  • Built-in routing and workflows
  • Managed state persistence
  • Native Azure OpenAI integration
  • Observability through Azure Monitor

It's worth evaluating if you'd rather not run your own orchestration host. The managed state persistence is particularly relevant here, because a review that waits two days for a lawyer is a long-running workflow, and durable state matters more than raw speed.

Check Azure AI Foundry Agent Service for current pricing.

12. ROI & Business Value

Take the 150-contract team from the cost section. Assume a lawyer's first-pass review takes about 2.5 hours at an internal blended cost of $250 per hour. That's roughly $625 per contract, or $94,000 a month, before counting external counsel overflow. If the agent shortens the first pass by 1.5 hours per contract, the gross saving is about $56,000 a month. These are assumptions, so substitute your own volumes and rates.

A Conservative Reading

Even if realised savings are only a third of that estimate (about $18,000 a month), the running cost of roughly $325 a month is under 2% of it. The real constraint isn't whether it pays back. It's whether your playbook is good enough for the agent to be useful, and whether your lawyers trust the output enough to change how they work.

The metrics I'd track from day one: median time from contract received to first lawyer touch, lawyer override rate by clause type, percentage of contracts closed without escalation, and turnaround time at quarter end. The last one is where the business feels the benefit most. Don't promise a headcount reduction. In practice, the better outcome is the same team handling more contracts with fewer stalled deals, and lawyers spending their time on judgment instead of checklist work.

13. When NOT to Use This

Be Honest About the Boundaries

There are contract types and organisations where this system will cost more than it saves, or create risk it was meant to remove.

  • You don't have a playbook. If your negotiating positions live in people's heads and have never been written down, build that first. The agent can't be more consistent than the standard it's given, and writing the playbook is valuable on its own.
  • High-value or bespoke deals. M&A agreements, major joint ventures, and heavily negotiated strategic partnerships depend on context and commercial judgment that no playbook captures. Use it, if at all, as a clause-finder.
  • Low volume. If you review fewer than 20 or so contracts a month, the effort of building and maintaining the playbook and library is hard to recoup. A good template and a checklist will serve you better.
  • Highly regulated or contentious contexts. Employment disputes, litigation, and regulated-industry agreements carry obligations that go beyond clause deviation. Follow your counsel's guidance on where automated assistance is appropriate.
  • Unsupported languages and jurisdictions. If your playbook and library are written for one jurisdiction, the agent will score foreign-law contracts with false confidence. Restrict it to the jurisdictions your playbook actually covers.
  • Nobody to own the review gate. If lawyers will rubber-stamp the output under time pressure, you've built a risk-laundering machine. The human gate only works if reviewers have the time and authority to disagree.

A simpler alternative worth trying first: a well-structured checklist plus clause-extraction tagging from your existing contract platform. If that already gets your lawyers to the important clauses quickly, you may not need an agent at all.

14. Key Takeaways

  • Score deviation, not risk. Anchor every assessment to a preferred, fallback, and walk-away position your company has already decided, and make the model cite the playbook entry and quote the contract.
  • Constrain the drafter. Redlines should come from an approved clause library with minimal edits, and plain code should verify numbers, defined terms, and how much text changed.
  • Fail safe. When evidence doesn't verify, escalate to a human. Never pass a clause on the model's say-so.
  • The lawyer gate is the product. Everything upstream exists to make that review faster and better informed, not to bypass it.
  • Measure overrides. The lawyer override rate by clause type tells you precisely where the playbook, the library, or the prompts need work.
  • The costs are mostly not tokens. About $0.37 per contract in model spend, but the real work is codifying and maintaining the playbook.

Want More Practical AI Tutorials?

I write about building production AI systems with Azure, Python, and C#. Subscribe for practical tutorials delivered twice a month.

Subscribe to Newsletter →

Written by Jaffar Kazi, a software engineer in Sydney building AI-powered applications. Connect on LinkedIn.