Building a Clinical Notes & Care Pathway Assistant with Azure OpenAI

Introduction

I built this because I kept running into the same complaint from clinicians I talked to while scoping healthcare AI work: the ambient scribe tools solve half the problem. They listen to the visit, draft a note, and stop. The clinician still has to read the note, decide what happens next — order a referral, start a med, schedule a follow-up — and separately go look up whatever protocol applies. The time saved on typing gets spent again on chart review and manual pathway lookup, and none of it is captured anywhere for later audit.

This hits hardest in high-volume primary care and multi-site specialty clinics, where a clinician might see 20-30 patients a day and where "pajama time" — documentation done after hours — is a well-documented driver of burnout. It also matters anywhere care pathway adherence is tracked for quality metrics or payer contracts, because a note that isn't linked to a concrete next action tends to get the next action delayed or skipped.

Generic ambient documentation products fail here for a structural reason, not a quality reason: they're built to draft notes, not to reason about an institution's specific care pathways. Doing that well means grounding recommendations in the clinic's actual guideline library — not the model's general medical training — and it means being honest that an LLM should never be the one deciding what happens to a patient. This article walks through the system I'd build to do both: draft the note, extract the structured clinical facts, retrieve the relevant pathway guidance, and put all of it in front of a clinician as a reviewable suggestion.

Current Approach & Limitations

Most clinics today sit somewhere between two baselines. The first is manual: the clinician types or dictates into an EHR template during or after the visit, using macros for common phrases. The second is a commercial ambient scribe: a microphone picks up the visit conversation, a model drafts a SOAP note, and the clinician edits and signs it.

Both have real limitations once you look past the note itself:

• No connection to institutional pathways. The note describes what happened. It doesn't tell the clinician which order set, referral pathway, or follow-up interval the institution's own protocols call for given this presentation.
• No structured, codeable output. A well-written paragraph of prose doesn't populate billing codes, problem lists, or med reconciliation fields. Someone still has to extract that by hand.
• No grounding and no audit trail. If a commercial scribe's model infers a detail that wasn't actually said — a dosage, a duration, a negative finding — there's often no way to trace that sentence back to a specific moment in the transcript.
• Generic suggestions, if any. Where these tools do suggest next steps, they're drawing on the model's general training, not the specific clinical pathway documents your institution has approved and audited.

The cost of these gaps isn't abstract. Inconsistent pathway adherence shows up in quality metrics and payer scorecards. Manual extraction work eats the time the scribe was supposed to save. And ungrounded suggestions are a liability problem waiting to happen the first time a clinician trusts one that's wrong.

The Solution Approach

The system I'm describing has five stages, and the design choice that matters most is where the human sits in the loop: at the very end, reviewing everything before anything reaches the patient's record.

1. Note drafting. An LLM drafts a structured SOAP note from the encounter transcript plus relevant EHR context (current meds, problem list, recent vitals).
2. Structured extraction. A second pass pulls discrete, codeable fields out of the draft — diagnoses, medication changes, suggested orders — each one tagged with the transcript span or EHR field it came from.
3. Contradiction check. Extracted fields are checked against the patient's existing EHR data. A new medication that conflicts with a documented allergy, or a vital sign the model "wrote" but never appeared in the transcript or chart, gets flagged rather than silently accepted.
4. Care pathway retrieval and recommendation. A RAG step retrieves the relevant sections of the institution's own clinical guideline corpus and a second LLM call proposes next steps — but only ones it can cite directly to a retrieved passage.
5. Clinician review. Everything — the note, the extracted fields, the flags, the cited pathway suggestions — lands in a review queue. Nothing writes to the EHR until a clinician approves or edits it.

The benefit of structuring it this way is that the two places where an LLM is most likely to cause real harm — inventing clinical facts and suggesting care that isn't actually in the approved guideline — both have a technical control attached to them (the contradiction check and the citation requirement), and both still end in a human decision regardless of how well those controls work. The tech stack: Azure OpenAI (GPT-4o) for drafting and extraction, Azure AI Search for the guideline RAG index, Azure Speech for transcription, LangGraph or Semantic Kernel for orchestration, and Azure Health Data Services for the FHIR write-back once a clinician signs off.

Architecture Overview

The diagram below shows the full pipeline, from ambient capture through to the FHIR write-back.

Reading it left to right: Encounter capture combines the ambient audio transcript with whatever EHR context is already on file — current medications, problem list, last set of vitals. Azure Speech-to-Text turns the audio into a timestamped transcript. The note drafting LLM uses both inputs to produce a SOAP-structured draft. Structured extraction pulls codeable fields out of that draft, each tagged with its source span. The contradiction check cross-references those fields against the EHR before anything moves forward. The care pathway engine retrieves relevant guideline passages from an Azure AI Search index and generates next-step suggestions that cite those passages directly. Everything converges on the clinician review queue, and only approved items reach FHIR write-back through Azure Health Data Services. Rejected or edited items loop back to note drafting rather than disappearing silently. Observability and audit logging attach to every stage, not just the end — that matters as much for debugging as for compliance.

Core Implementation

The orchestration here is a graph, not a chain, because the review step can loop back to drafting. I modeled the whole encounter as a single state object that accumulates fields as it moves through the pipeline, which makes it straightforward to persist mid-review state and resume after a clinician acts on it.

from enum import Enum
from typing import Optional
from pydantic import BaseModel

class ReviewStatus(str, Enum):
    PENDING_DRAFT = "pending_draft"
    PENDING_REVIEW = "pending_review"
    APPROVED = "approved"
    REJECTED = "rejected"

class ExtractedField(BaseModel):
    field_name: str          # e.g. "diagnosis", "medication_change"
    value: str
    source_span: str         # transcript excerpt or "ehr:"
    confidence: float
    flagged: bool = False
    flag_reason: Optional[str] = None

class PathwaySuggestion(BaseModel):
    action: str
    rationale: str
    citation_doc: str        # guideline document id
    citation_excerpt: str
    similarity_score: float

class EncounterState(BaseModel):
    encounter_id: str
    transcript: str
    ehr_context: dict
    draft_note: Optional[str] = None
    extracted_fields: list[ExtractedField] = []
    pathway_suggestions: list[PathwaySuggestion] = []
    status: ReviewStatus = ReviewStatus.PENDING_DRAFT

public enum ReviewStatus
{
    PendingDraft,
    PendingReview,
    Approved,
    Rejected
}

public record ExtractedField(
    string FieldName,          // e.g. "diagnosis", "medication_change"
    string Value,
    string SourceSpan,         // transcript excerpt or "ehr:"
    double Confidence,
    bool Flagged = false,
    string? FlagReason = null
);

public record PathwaySuggestion(
    string Action,
    string Rationale,
    string CitationDoc,        // guideline document id
    string CitationExcerpt,
    double SimilarityScore
);

public class EncounterState
{
    public string EncounterId { get; init; } = default!;
    public string Transcript { get; init; } = default!;
    public Dictionary<string, object> EhrContext { get; init; } = new();
    public string? DraftNote { get; set; }
    public List<ExtractedField> ExtractedFields { get; set; } = new();
    public List<PathwaySuggestion> PathwaySuggestions { get; set; } = new();
    public ReviewStatus Status { get; set; } = ReviewStatus.PendingDraft;
}

The orchestration graph has five nodes matching the pipeline stages, with one edge that loops back from review to drafting when a clinician rejects or substantially edits a note.

from langgraph.graph import StateGraph, END

def build_encounter_graph():
    graph = StateGraph(EncounterState)

    graph.add_node("draft_note", draft_note_node)
    graph.add_node("extract_fields", extract_fields_node)
    graph.add_node("check_contradictions", contradiction_check_node)
    graph.add_node("retrieve_pathway", pathway_retrieval_node)
    graph.add_node("await_review", await_review_node)

    graph.set_entry_point("draft_note")
    graph.add_edge("draft_note", "extract_fields")
    graph.add_edge("extract_fields", "check_contradictions")
    graph.add_edge("check_contradictions", "retrieve_pathway")
    graph.add_edge("retrieve_pathway", "await_review")

    graph.add_conditional_edges(
        "await_review",
        lambda s: "draft_note" if s.status == ReviewStatus.REJECTED else END,
    )
    return graph.compile()

public class EncounterProcess
{
    private readonly INoteDrafter _drafter;
    private readonly IFieldExtractor _extractor;
    private readonly IContradictionChecker _checker;
    private readonly IPathwayRetriever _pathway;

    public EncounterProcess(INoteDrafter drafter, IFieldExtractor extractor,
        IContradictionChecker checker, IPathwayRetriever pathway)
    {
        _drafter = drafter; _extractor = extractor;
        _checker = checker; _pathway = pathway;
    }

    public async Task<EncounterState> RunAsync(EncounterState state)
    {
        do
        {
            state.DraftNote = await _drafter.DraftAsync(state);
            state.ExtractedFields = await _extractor.ExtractAsync(state);
            state.ExtractedFields = await _checker.CheckAsync(state);
            state.PathwaySuggestions = await _pathway.RetrieveAsync(state);
            state.Status = ReviewStatus.PendingReview;

            state = await AwaitClinicianReviewAsync(state);
        } while (state.Status == ReviewStatus.Rejected);

        return state;
    }
}

Key Technical Challenge #1: Keeping the Draft Note Honest

The tricky part with note drafting isn't getting the model to write fluent clinical prose — GPT-4o does that easily. It's stopping it from quietly filling gaps. If the transcript doesn't mention a dosage and the chart doesn't have one on file, a model asked to "complete a SOAP note" will sometimes supply a plausible-sounding one anyway, because plausible completion is what language models are built to do.

I handle this with two controls stacked together. First, the drafting prompt is explicitly extractive: every clinical fact in the note must trace to either the transcript or the supplied EHR context, and the model is instructed to write "not documented" rather than infer. Second, the extraction pass that follows tags every field with its source span and a confidence score, and anything below a threshold — or anything with no traceable source at all — gets flagged for the clinician rather than silently included.

EXTRACTION_PROMPT = """
Extract discrete clinical fields from this draft note. For each field,
quote the exact transcript or EHR excerpt that supports it. If a field
cannot be traced to the transcript or EHR context, set source_span to
"UNSUPPORTED" and confidence to 0.0. Never invent a value to fill a gap.
"""

def extract_fields_node(state: EncounterState) -> EncounterState:
    response = azure_openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": EXTRACTION_PROMPT},
            {"role": "user", "content": f"Draft note:\n{state.draft_note}\n\n"
                                         f"Transcript:\n{state.transcript}\n\n"
                                         f"EHR context:\n{state.ehr_context}"},
        ],
        response_format={"type": "json_schema", "json_schema": FIELD_SCHEMA},
    )
    fields = parse_extracted_fields(response)
    for field in fields:
        if field.source_span == "UNSUPPORTED" or field.confidence < 0.6:
            field.flagged = True
            field.flag_reason = "No traceable source in transcript or EHR"
    state.extracted_fields = fields
    return state

public class FieldExtractor : IFieldExtractor
{
    private const string ExtractionPrompt = @"
Extract discrete clinical fields from this draft note. For each field,
quote the exact transcript or EHR excerpt that supports it. If a field
cannot be traced to the transcript or EHR context, set SourceSpan to
'UNSUPPORTED' and Confidence to 0.0. Never invent a value to fill a gap.";

    public async Task<List<ExtractedField>> ExtractAsync(EncounterState state)
    {
        var response = await _chatClient.CompleteAsync(new ChatRequest
        {
            Model = "gpt-4o",
            Messages = new[]
            {
                new ChatMessage("system", ExtractionPrompt),
                new ChatMessage("user",
                    $"Draft note:\n{state.DraftNote}\n\nTranscript:\n{state.Transcript}\n\n" +
                    $"EHR context:\n{JsonSerializer.Serialize(state.EhrContext)}")
            },
            ResponseFormat = FieldSchema.AsJsonSchema()
        });

        var fields = ParseExtractedFields(response);
        foreach (var field in fields.Where(f => f.SourceSpan == "UNSUPPORTED" || f.Confidence < 0.6))
        {
            field.Flagged = true;
            field.FlagReason = "No traceable source in transcript or EHR";
        }
        return fields;
    }
}

The contradiction check runs next, comparing every extracted field against the patient's existing EHR record — allergies, active medications, documented conditions. This is deliberately simple logic, not another LLM call, because the failure mode you're guarding against here (a new med that conflicts with a logged allergy, a vital that doesn't match anything in the transcript) needs to be deterministic and auditable, not another layer of probabilistic judgment.

Key Technical Challenge #2: Grounding Pathway Suggestions in Approved Guidelines

This is the part I was most cautious about. A model asked "what should happen next for this patient" will answer from its general medical training, which is broad, frequently correct, and completely unsuitable to ship — because it isn't the institution's approved pathway, isn't version-controlled against the current protocol, and isn't something a clinician can check against a source document in ten seconds.

The fix is to never let the pathway LLM answer from its own knowledge at all. It only sees retrieved chunks from the institution's guideline corpus, indexed in Azure AI Search, and it's instructed to produce a suggestion only when it can cite a specific retrieved passage with a similarity score above threshold. Below that threshold, it returns nothing rather than a hedged guess — an empty result is a much safer failure than a plausible-sounding one with no real source.

SIMILARITY_THRESHOLD = 0.78

PATHWAY_PROMPT = """
You recommend next-step clinical actions based ONLY on the retrieved
guideline passages below. Every suggestion must quote the exact passage
it is based on. If no retrieved passage clearly supports an action for
this presentation, return an empty suggestions list. Do not use general
medical knowledge to fill gaps.
"""

def pathway_retrieval_node(state: EncounterState) -> EncounterState:
    query = build_query_from_fields(state.extracted_fields)
    results = ai_search_client.search(
        search_text=query,
        vector_queries=[{"vector": embed(query), "fields": "contentVector", "k": 5}],
    )
    passages = [r for r in results if r["@search.score"] >= SIMILARITY_THRESHOLD]

    if not passages:
        state.pathway_suggestions = []
        return state

    response = azure_openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": PATHWAY_PROMPT},
            {"role": "user", "content": format_passages_and_fields(passages, state.extracted_fields)},
        ],
        response_format={"type": "json_schema", "json_schema": PATHWAY_SCHEMA},
    )
    state.pathway_suggestions = parse_pathway_suggestions(response, passages)
    return state

public class PathwayRetriever : IPathwayRetriever
{
    private const double SimilarityThreshold = 0.78;

    private const string PathwayPrompt = @"
You recommend next-step clinical actions based ONLY on the retrieved
guideline passages below. Every suggestion must quote the exact passage
it is based on. If no retrieved passage clearly supports an action for
this presentation, return an empty suggestions list. Do not use general
medical knowledge to fill gaps.";

    public async Task<List<PathwaySuggestion>> RetrieveAsync(EncounterState state)
    {
        var query = BuildQueryFromFields(state.ExtractedFields);
        var results = await _searchClient.SearchAsync<GuidelinePassage>(query,
            new SearchOptions { VectorQueries = { new VectorizedQuery(await EmbedAsync(query)) } });

        var passages = results.GetResults()
            .Where(r => r.Score >= SimilarityThreshold)
            .ToList();

        if (passages.Count == 0)
            return new List<PathwaySuggestion>();

        var response = await _chatClient.CompleteAsync(new ChatRequest
        {
            Model = "gpt-4o",
            Messages = new[]
            {
                new ChatMessage("system", PathwayPrompt),
                new ChatMessage("user", FormatPassagesAndFields(passages, state.ExtractedFields))
            },
            ResponseFormat = PathwaySchema.AsJsonSchema()
        });

        return ParsePathwaySuggestions(response, passages);
    }
}

Every suggestion that reaches the clinician review queue carries its source document ID and the exact excerpt it was based on. In practice, this means a nurse or physician can verify a suggestion against the source guideline in seconds rather than trusting it blind — which also happens to be exactly what you want for a compliance audit trail.

Cost Analysis

Here's the real math for a mid-size clinic doing 300 encounters a day, using Azure OpenAI GPT-4o pricing as of late 2026 — check current Azure OpenAI pricing before budgeting, since rates shift.

At 300 encounters/day across 22 clinic days a month, that's roughly 6,600 encounters/month and about $178/month in model inference cost alone. The infrastructure layer costs more than the tokens do:

All in, that's roughly $1,800/month for a 300-encounter/day clinic — call it $0.27 per encounter fully loaded. The speech-to-text line is the biggest surprise for most people budgeting this; it dwarfs the LLM cost by nearly 6x. If encounters are already captured as text (portal messages, structured intake) rather than ambient audio, that line disappears and the economics look very different.

Observability & Debugging

In a system where every stage can be the source of a wrong clinical suggestion, you need to be able to answer "why did it say that" for any single encounter, months after the fact. I log every pipeline stage as a structured event in Application Insights, tagged with the encounter ID, so a full trace reconstructs the entire decision path — transcript in, note out, extraction confidence scores, contradiction flags, retrieval similarity scores, and the clinician's final decision.

from opencensus.ext.azure.log_exporter import AzureLogHandler
import logging

logger = logging.getLogger("encounter_pipeline")
logger.addHandler(AzureLogHandler(connection_string=APPINSIGHTS_CONNECTION_STRING))

def log_stage(state: EncounterState, stage: str, **extra):
    logger.info(
        "pipeline_stage",
        extra={
            "custom_dimensions": {
                "encounter_id": state.encounter_id,
                "stage": stage,
                "flagged_fields": sum(1 for f in state.extracted_fields if f.flagged),
                **extra,
            }
        },
    )

# Called after each node, e.g.:
log_stage(state, "pathway_retrieval",
          suggestions=len(state.pathway_suggestions),
          min_similarity=min((s.similarity_score for s in state.pathway_suggestions), default=None))

public class PipelineTelemetry
{
    private readonly TelemetryClient _telemetry;

    public PipelineTelemetry(TelemetryClient telemetry) => _telemetry = telemetry;

    public void LogStage(EncounterState state, string stage, IDictionary<string, string>? extra = null)
    {
        var props = new Dictionary<string, string>
        {
            ["encounter_id"] = state.EncounterId,
            ["stage"] = stage,
            ["flagged_fields"] = state.ExtractedFields.Count(f => f.Flagged).ToString()
        };
        if (extra != null)
            foreach (var kv in extra) props[kv.Key] = kv.Value;

        _telemetry.TrackEvent("pipeline_stage", props);
    }
}

Two things matter beyond the trace itself. First, the review decision — approved, edited, rejected, and what changed — gets written to an append-only audit store, not just the application log, because that's the record a compliance review will actually ask for. Second, I track flag rates and rejection rates as ongoing metrics, not just per-incident traces: a sudden jump in "unsupported field" flags on a given day is usually the first sign something upstream (a transcription quality issue, a prompt regression) broke before anyone would notice it any other way.

Technology Choices

Python Implementation

Why choose Python: If your team writes Python, you get access to the richest AI/ML ecosystem, and most of the retrieval and prompt-engineering patterns in this space are documented Python-first.

• Library ecosystem — LangChain, LangGraph, thousands of AI tools, plus mature clinical NLP libraries if you need them later
• Rapid prototyping — quick iteration on prompts and retrieval tuning, Jupyter support for reviewing extraction quality against sample transcripts
• Community — most healthcare-AI reference implementations and research code you'll want to borrow from are Python-first

C#/.NET Implementation

Why choose C#: If your EHR integration layer or hospital backend is already .NET — common in enterprise healthcare IT — you get first-party Microsoft support and patterns your team already knows.

• Native Azure integration — first-party SDKs for Azure OpenAI, AI Search, and Health Data Services, Microsoft-maintained
• Enterprise patterns — dependency injection, strong typing on clinical data models, which matters a lot when a typo in a field name could misroute a medication field
• Production-ready — battle-tested at scale, and typically the easier sell to a hospital IT security review that already runs a .NET stack

Azure Infrastructure

The services this system actually needs: Azure OpenAI (GPT-4o) for drafting, extraction, and pathway generation; Azure AI Search for the guideline RAG index; Azure Speech for transcription if you're working from ambient audio; Azure Health Data Services for the FHIR-native write-back; and Azure Monitor / Application Insights for the trace and audit layer described above.

Azure AI Foundry Agent Service

Azure AI Foundry Agent Service is now generally available, providing managed orchestration for AI systems like this one.

• Built-in routing and workflows, which can replace a hand-rolled graph for the simpler stages of this pipeline
• Managed state persistence — useful here since encounters can sit "pending review" for hours
• Native Azure OpenAI integration
• Observability through Azure Monitor, which lines up with the tracing approach described above

Check Azure AI Foundry Agent Service for current pricing.

One point specific to healthcare: whichever services you provision, confirm a Business Associate Agreement (BAA) is in place and that the Azure OpenAI resource is configured for no data retention for training — this is a standard option in Azure OpenAI for healthcare workloads, but it has to be explicitly configured, not assumed.

ROI / Business Value

The metric that matters most to a clinic deciding whether to build this is minutes of documentation time saved per encounter, multiplied by encounter volume. In practice, ambient-scribe-style note drafting alone tends to save clinicians several minutes per encounter versus manual typing; adding structured extraction and pathway suggestions saves additional time on the chart-review and protocol-lookup side, which is the part generic scribes don't touch.

A simple framework: estimate baseline documentation minutes per encounter today, estimate the reduction from this system, multiply by daily encounter volume and a reasonable dollar value per clinician-minute (fully loaded cost, not just salary), and compare that monthly figure against the ~$1,800/month infrastructure cost calculated above. For a 300-encounter/day clinic, even a conservative 3-4 minutes saved per encounter at a modest per-minute clinician cost clears that bar within the first month — the economics here tend to favor the build once you're above roughly 100-150 encounters/day.

When NOT to Use This

I'd talk a clinic out of this in a few specific situations. Low volume is the first: under roughly 100 encounters/day, the fixed infrastructure cost and the engineering and compliance overhead of building and maintaining this pipeline usually isn't worth it compared to a commercial ambient scribe product or just dictation with templates.

Second, if your specialty's presentations are highly idiosyncratic and your guideline corpus is thin or inconsistent, the RAG grounding has little to retrieve against, and you'll get an empty-suggestion system that doesn't add much over plain note drafting.

Third, and this one is non-negotiable rather than a judgment call: if your organization isn't ready to staff a genuine clinician review step — meaning someone with the time and authority to actually read and decide on every suggestion, not rubber-stamp a queue — don't build this. The entire safety case rests on that review happening for real.

Key Takeaways

• Ambient note drafting alone solves the typing problem but leaves the pathway and extraction problems untouched — this architecture addresses both by adding structured extraction and guideline-grounded RAG recommendations on top of drafting.
• The two places an LLM can cause real clinical harm — inventing facts in a note and suggesting ungrounded care actions — each get a specific technical control: extractive prompting with confidence flags, and citation-required RAG with a similarity floor.
• Every output is a suggestion for a clinician, never an autonomous action; the review step and its audit trail are as much a part of the architecture as the LLM calls.
• Speech-to-text, not LLM inference, is usually the largest line item in the cost model — budget accordingly.
• This only makes sense above a certain encounter volume, with a real guideline corpus to ground against, and with genuine clinician review capacity. Below any of those thresholds, simpler tools win.