The Adaptive Assessment Engine — Generating Personalized Quizzes That Track Real Mastery

Introduction

Most classroom quizzes are the same for every student in the room. The student who mastered fractions three weeks ago answers the same ten questions as the student who's still confused about denominators. Both walk away with a score, but neither gets much signal about what to study next — the quiz was calibrated to the average student, which means it's miscalibrated for almost everyone.

Adaptive testing solves this in theory: give each student questions matched to their current ability, and use their responses to refine that estimate in real time. It's not a new idea — computerized adaptive testing (CAT) has existed since the GRE went digital in the 1990s. What's new is that generating the questions themselves used to require a huge, expensive, pre-validated item bank built by psychometricians. Most schools, training providers, and internal L&D teams don't have that. What they do have is a curriculum, a set of learning objectives, and now, an LLM that can generate an unlimited supply of questions on demand.

I built this because a colleague running an online tutoring platform had exactly this problem: they had rich course content but a static question bank that ran out of good questions for their strongest students within a few sessions. The fix wasn't "add more questions" — it was "generate questions calibrated to where each student actually is, and trust the answer key enough to grade on it automatically." That second part turned out to be the hard part.

Current Approach & Limitations

The default approach in most LMS platforms is a static item bank: a few hundred pre-written questions per topic, shuffled and served in a fixed or lightly-randomized order. Three limitations show up quickly once you scale past a pilot cohort:

• Item exhaustion. Strong students burn through the "hard" questions fast, then start repeating them, which teaches them the answer key rather than the skill.
• Flat difficulty. Without a mechanism to estimate ability, most banks either serve everyone the same fixed set or use crude rules ("3 wrong in a row → easier"), which converges slowly and doesn't generalize across skills.
• Manual authoring cost. Writing, reviewing, and tagging a psychometrically sound question bank is genuinely expensive — commercial adaptive assessment vendors quote per-student licensing partly because someone had to hand-write and validate thousands of items.

None of this is a knock on existing platforms — building a properly normed item bank the traditional way is slow, specialist work. The opportunity is that an LLM can generate items against a skill taxonomy on demand, at a cost per item that's a rounding error compared to authoring it by hand. The catch is trust: an auto-graded assessment is only as good as its answer key, and LLMs get answer keys wrong often enough that you can't ship them ungated.

The Solution Approach

The system I'll walk through has three moving parts, each solving a distinct problem:

• A question generator with a verification gate — GPT-4o drafts a question and answer key against a specific learning objective and target difficulty, then a second, independent pass tries to solve the question from scratch. The item only enters the bank if both answers agree.
• An adaptive selector — a simplified Elo-style rating system tracks each student's mastery per skill and picks (or triggers generation of) the next question at a difficulty that keeps expected success around 60%, the sweet spot for measuring ability without frustrating the student.
• A grading agent — multiple-choice questions grade for free with a string comparison; open-ended responses get graded by GPT-4o against a rubric attached to the question at generation time.

The trade-off here is that this isn't a certified psychometric instrument — it's not going to produce a defensible IRT-calibrated theta score you can put in front of an accreditation board. What it is good at is the much more common case: internal practice assessments, formative quizzes, corporate training checks, and tutoring platforms where "good enough calibration, unlimited fresh questions, near-zero marginal cost" beats "gold-standard calibration, a few hundred fixed questions, six-figure licensing."

Tech stack: Azure OpenAI (GPT-4o) for generation, verification, and grading; a skill-tagged item bank and per-student mastery store in Azure SQL or Cosmos DB; LangGraph (Python) or a lightweight orchestration loop (C#) to drive the select → present → grade → update cycle.

Architecture Overview

The system is organized around four groups of responsibility: curriculum content that defines what can be tested, a generation pipeline that turns learning objectives into verified questions, a session loop that runs the actual assessment, and a per-student state store that mastery estimates persist to between sessions.

Two design decisions matter most here. First, generation and verification happen before a question ever reaches a student — nothing ungated goes live. Second, the mastery store is the only thing that needs to survive between sessions; the item bank grows over time but is shared across all students, so the marginal cost of a new student is close to zero once the bank has reasonable coverage of a skill.

Core Implementation

The data model is intentionally small: a learning objective, a question tied to a skill and a difficulty, a per-student-per-skill mastery record, and a turn that captures what happened when a student answered.

from pydantic import BaseModel
from typing import Literal, Optional

class LearningObjective(BaseModel):
    skill_id: str
    standard_code: str
    description: str

class Question(BaseModel):
    question_id: str
    skill_id: str
    difficulty: float  # Elo-style rating, roughly 800-1600
    stem: str
    question_type: Literal["multiple_choice", "open_response"]
    options: Optional[list[str]] = None
    answer_key: str
    rubric: Optional[str] = None
    verified: bool = False

class SkillState(BaseModel):
    student_id: str
    skill_id: str
    mastery: float = 1200.0  # same scale as difficulty
    attempts: int = 0

class AssessmentTurn(BaseModel):
    session_id: str
    student_id: str
    question: Question
    response_text: str
    is_correct: Optional[bool] = None
    score: Optional[float] = None

public enum QuestionType { MultipleChoice, OpenResponse }

public record LearningObjective(
    string SkillId,
    string StandardCode,
    string Description);

public record Question(
    string QuestionId,
    string SkillId,
    double Difficulty,       // Elo-style rating, roughly 800-1600
    string Stem,
    QuestionType Type,
    IReadOnlyList<string>? Options,
    string AnswerKey,
    string? Rubric,
    bool Verified = false);

public record SkillState(
    string StudentId,
    string SkillId,
    double Mastery = 1200.0, // same scale as difficulty
    int Attempts = 0);

public record AssessmentTurn(
    string SessionId,
    string StudentId,
    Question Question,
    string ResponseText,
    bool? IsCorrect,
    double? Score);

The session loop is a small state machine: pick a skill and a target difficulty from the student's current mastery, select or generate a matching question, present it, grade the response, update mastery, and decide whether that skill is "converged" enough to move on.

from typing import TypedDict
from langgraph.graph import StateGraph, END

class SessionState(TypedDict):
    student_id: str
    skill_queue: list[str]
    current_question: Question | None
    turns: list[AssessmentTurn]
    done: bool

def select_question(state: SessionState) -> SessionState:
    skill_id = state["skill_queue"][0]
    mastery = mastery_store.get(state["student_id"], skill_id).mastery
    state["current_question"] = item_bank.select_or_generate(
        skill_id, target_difficulty=mastery
    )
    return state

def grade_turn(state: SessionState) -> SessionState:
    turn = state["turns"][-1]
    q = turn.question
    if q.question_type == "multiple_choice":
        turn.is_correct = turn.response_text.strip() == q.answer_key.strip()
    else:
        turn.score = grade_open_response(q, turn.response_text)
        turn.is_correct = turn.score >= 0.7
    return state

def update_mastery(state: SessionState) -> SessionState:
    turn = state["turns"][-1]
    new_mastery = update_skill_mastery(
        state["student_id"], turn.question.skill_id,
        turn.is_correct, turn.question.difficulty
    )
    skill_state = mastery_store.get(state["student_id"], turn.question.skill_id)
    if skill_state.attempts >= 5:
        state["skill_queue"].pop(0)
    state["done"] = len(state["skill_queue"]) == 0
    return state

graph = StateGraph(SessionState)
graph.add_node("select", select_question)
graph.add_node("grade", grade_turn)
graph.add_node("update", update_mastery)
graph.set_entry_point("select")
graph.add_edge("select", "grade")
graph.add_edge("grade", "update")
graph.add_conditional_edges(
    "update", lambda s: END if s["done"] else "select"
)
session_graph = graph.compile()

public class AssessmentSession
{
    private readonly IMasteryStore _masteryStore;
    private readonly IItemBank _itemBank;
    private readonly IGradingService _grading;

    public AssessmentSession(
        IMasteryStore masteryStore, IItemBank itemBank, IGradingService grading)
    {
        _masteryStore = masteryStore;
        _itemBank = itemBank;
        _grading = grading;
    }

    public async Task RunAsync(string studentId, Queue<string> skillQueue)
    {
        while (skillQueue.Count > 0)
        {
            var skillId = skillQueue.Peek();
            var state = await _masteryStore.GetAsync(studentId, skillId);
            var question = await _itemBank.SelectOrGenerateAsync(
                skillId, targetDifficulty: state.Mastery);

            var responseText = await PresentAndCaptureResponseAsync(question);
            var turn = await GradeAsync(studentId, question, responseText);

            var updated = await _masteryStore.UpdateAsync(
                studentId, skillId, turn.IsCorrect ?? false, question.Difficulty);

            if (updated.Attempts >= 5)
                skillQueue.Dequeue();
        }
    }

    private async Task<AssessmentTurn> GradeAsync(
        string studentId, Question question, string responseText)
    {
        if (question.Type == QuestionType.MultipleChoice)
        {
            var isCorrect = responseText.Trim() == question.AnswerKey.Trim();
            return new AssessmentTurn(
                Guid.NewGuid().ToString(), studentId, question,
                responseText, isCorrect, null);
        }

        var score = await _grading.GradeOpenResponseAsync(question, responseText);
        return new AssessmentTurn(
            Guid.NewGuid().ToString(), studentId, question,
            responseText, score >= 0.7, score);
    }
}

Key Technical Challenge #1: Trusting the Answer Key

Here's what surprised me most building this: GPT-4o writes plausible-looking multiple-choice questions easily, but it mislabels the correct option more often than you'd expect — especially on questions involving arithmetic, date logic, or multi-step word problems. If you auto-grade against a wrong key, you're not just introducing noise, you're actively teaching students the wrong answer is right. That's worse than no adaptive system at all.

The fix is a generate-then-verify pipeline: after the model drafts a question and its proposed answer key, a second, independent prompt is asked to solve the question from scratch — it never sees the proposed answer key, only the stem and options. If the two answers agree, the item is marked verified and enters the bank. If they disagree, the draft is discarded and regenerated. In practice, about 10-15% of drafts fail verification on the first pass, almost entirely on multi-step math and questions with subtly overlapping options.

MAX_GENERATION_ATTEMPTS = 3

async def generate_verified_question(
    skill: LearningObjective, difficulty: float, question_type: str
) -> Question:
    for attempt in range(MAX_GENERATION_ATTEMPTS):
        draft = await llm_generate_question(skill, difficulty, question_type)

        # Independent solver never sees the proposed answer key
        solved = await llm_solve_independently(draft.stem, draft.options)

        if solved.strip().lower() == draft.answer_key.strip().lower():
            draft.verified = True
            return draft

        log.warning(
            "verification_mismatch",
            skill_id=skill.skill_id, attempt=attempt,
            proposed=draft.answer_key, solved=solved,
        )

    raise QuestionGenerationError(
        f"Could not verify a question for {skill.skill_id} "
        f"after {MAX_GENERATION_ATTEMPTS} attempts"
    )

private const int MaxGenerationAttempts = 3;

public async Task<Question> GenerateVerifiedQuestionAsync(
    LearningObjective skill, double difficulty, QuestionType type)
{
    for (int attempt = 0; attempt < MaxGenerationAttempts; attempt++)
    {
        var draft = await GenerateQuestionAsync(skill, difficulty, type);

        // Independent solver never sees the proposed answer key
        var solved = await SolveIndependentlyAsync(draft.Stem, draft.Options);

        if (string.Equals(
                solved.Trim(), draft.AnswerKey.Trim(),
                StringComparison.OrdinalIgnoreCase))
        {
            return draft with { Verified = true };
        }

        _logger.LogWarning(
            "Verification mismatch for skill {SkillId} on attempt {Attempt}: " +
            "proposed={Proposed}, solved={Solved}",
            skill.SkillId, attempt, draft.AnswerKey, solved);
    }

    throw new QuestionGenerationException(
        $"Could not verify a question for {skill.SkillId} " +
        $"after {MaxGenerationAttempts} attempts");
}

Key Technical Challenge #2: Calibrating Difficulty Without a Cold Start Problem

Real IRT calibration needs response data from hundreds of students per item before difficulty estimates are trustworthy. You don't have that on day one, and a brand-new skill taxonomy has zero response history. The trade-off I made was to skip full IRT and use an Elo-style rating instead — the same mechanism chess uses to rate players, adapted so "difficulty" and "mastery" live on the same scale and update from a single win/loss signal.

Every student starts at a neutral mastery rating (1200) per skill. Every question gets an initial difficulty rating assigned by the generator itself (easy ≈ 1000, medium ≈ 1200, hard ≈ 1400), then that rating drifts toward reality as real students answer it — a question that everyone gets right despite being tagged "hard" will lose difficulty points over time. The adaptive selector always targets a question whose difficulty is close to the student's current mastery, which keeps expected success around 50-60%: hard enough to be informative, not so hard it's discouraging.

K_FACTOR = 32.0

def expected_success(mastery: float, difficulty: float) -> float:
    return 1.0 / (1.0 + 10 ** ((difficulty - mastery) / 400.0))

def update_skill_mastery(
    student_id: str, skill_id: str, is_correct: bool, difficulty: float
) -> float:
    state = mastery_store.get(student_id, skill_id)
    expected = expected_success(state.mastery, difficulty)
    actual = 1.0 if is_correct else 0.0

    state.mastery += K_FACTOR * (actual - expected)
    state.attempts += 1
    mastery_store.save(state)

    # The question's own difficulty drifts too, using a smaller K
    # so a single outlier response doesn't swing it much.
    update_item_difficulty(skill_id, difficulty, actual, expected, k=8.0)
    return state.mastery

def next_target_difficulty(mastery: float) -> float:
    # Aim slightly above current mastery: ~55-60% expected success
    return mastery + 60

public class MasteryUpdater
{
    private const double KFactor = 32.0;
    private const double ItemKFactor = 8.0;

    public static double ExpectedSuccess(double mastery, double difficulty) =>
        1.0 / (1.0 + Math.Pow(10, (difficulty - mastery) / 400.0));

    public async Task<SkillState> UpdateAsync(
        string studentId, string skillId, bool isCorrect, double difficulty)
    {
        var state = await _masteryStore.GetAsync(studentId, skillId);
        var expected = ExpectedSuccess(state.Mastery, difficulty);
        var actual = isCorrect ? 1.0 : 0.0;

        var updated = state with
        {
            Mastery = state.Mastery + KFactor * (actual - expected),
            Attempts = state.Attempts + 1
        };
        await _masteryStore.SaveAsync(updated);

        // The question's own difficulty drifts too, using a smaller K
        // so a single outlier response doesn't swing it much.
        await _itemBank.UpdateDifficultyAsync(
            skillId, difficulty, actual, expected, ItemKFactor);

        return updated;
    }

    public static double NextTargetDifficulty(double mastery) => mastery + 60;
}

Cost Analysis

Real numbers, using Azure OpenAI GPT-4o pricing of roughly $2.50 per million input tokens and $10 per million output tokens (check current pricing — this moves).

Generating one verified question (draft + independent solve): draft prompt ≈ 750 input tokens (learning objective, difficulty target, few-shot format example), ≈ 280 output tokens (stem, options, answer key, rubric as JSON). Solve pass ≈ 180 input tokens, ≈ 40 output tokens (answer only). Total ≈ 930 input / 320 output tokens ≈ $0.0055 per verified item, plus roughly 12% retry overhead for failed verifications, so call it $0.006 per item that makes it into the bank.

Grading one open-ended response: ≈ 520 input tokens (question, rubric, student response), ≈ 120 output tokens (score + short feedback) ≈ $0.0025 per graded response. Multiple-choice questions grade for free — it's a string comparison.

Add infrastructure — a small Cosmos DB or Azure SQL tier for state, and Azure Functions or a modest App Service plan for orchestration — and the whole system runs a 5,000-student deployment for roughly $500-650/month, or about $0.10-0.13 per student per month. Commercial adaptive assessment add-ons commonly license in the $8-15 per student per year range, so even with engineering time factored in, the economics favor building this in-house once you're past a few hundred students.

Observability & Debugging

The two numbers I watch most closely in production are the verification mismatch rate per skill and the grading score distribution for open-ended responses. A skill whose mismatch rate climbs above the usual 10-15% baseline almost always means the learning objective's description drifted too vague for the model to generate against consistently — that's a content problem, not a model problem, and it shows up in the data before a teacher ever notices.

Every generation, verification, and grading call gets wrapped in an OpenTelemetry span tagged with skill ID, difficulty, and token counts, exported to Azure Monitor / Application Insights. That gives you three views for free: cost per skill over time, latency percentiles per call type, and — most useful for debugging a "this grade looks wrong" complaint from a teacher — the full prompt/response trace for that specific turn, so you can see exactly what the grading model saw and said without re-running anything.

Technology Choices

Python Implementation

Why choose Python: If your team writes Python, you get access to the richest AI/ML ecosystem, and LangGraph's explicit state machine is a natural fit for the select → grade → update cycle.

• Library ecosystem — LangChain, LangGraph, structured-output helpers for JSON-mode generation
• Rapid prototyping — quick to iterate on prompts and rubric wording, Jupyter support for eyeballing generated items
• Community — most adaptive testing and IRT-adjacent tutorials are Python-first

C#/.NET Implementation

Why choose C#: If your LMS backend is already .NET, you get first-party Microsoft support, strong typing on the mastery/difficulty math, and no cross-language deployment overhead.

• Native Azure integration — first-party SDKs, Microsoft-maintained
• Enterprise patterns — dependency injection, strongly-typed records for question and state models
• Production-ready — battle-tested at scale, easy to slot into an existing ASP.NET Core LMS

Azure Infrastructure

The system needs surprisingly little infrastructure: Azure OpenAI (GPT-4o) for generation, verification, and grading; a small Cosmos DB or Azure SQL instance for the item bank and per-student mastery state; and Azure Functions (or a lightweight App Service) to host the session orchestration. Azure Monitor covers tracing and cost dashboards.

Azure AI Foundry Agent Service

Azure AI Foundry Agent Service is now generally available, providing managed orchestration for AI systems.

• Built-in routing and workflows
• Managed state persistence
• Native Azure OpenAI integration
• Observability through Azure Monitor

Check Azure AI Foundry Agent Service for current pricing.

ROI / Business Value

The value shows up in two places. First, direct licensing avoidance: at $8-15/student/year for a commercial adaptive assessment add-on versus roughly $1.20-1.60/student/year running this in-house at the volumes above, a 5,000-student deployment saves five to six figures annually even after accounting for engineering and infrastructure. Second, and harder to put a number on but arguably more important: an item bank that never runs dry means strong students keep getting genuinely new, appropriately hard questions instead of repeating a fixed set — which is the actual point of adaptive testing, not just the cost savings.

When NOT to Use This

Be honest about the boundaries here. This is not a substitute for a certified, psychometrically-validated instrument in a high-stakes context — state standardized testing, professional licensing exams, or anything requiring defensible year-over-year score comparability needs a real IRT calibration process and human psychometric review, full stop. Don't route accountability testing through an LLM-generated item bank.

• No digitized skill taxonomy or learning objectives — if your curriculum isn't broken into discrete, tagged skills, there's nothing for the generator to target. Build the taxonomy first.
• Very small cohorts — under a few hundred students, a well-curated static question bank plus a spreadsheet is simpler and the Elo model won't have enough response volume to converge meaningfully.
• High-stakes, legally defensible scoring — anywhere a score needs to hold up to external audit or legal challenge, use a validated commercial instrument.

Key Takeaways

• Static item banks run out of good questions for strong students and can't adapt difficulty per skill — an LLM generator with unlimited output fixes the supply problem, but only if you gate what reaches students.
• A generate-then-verify pipeline — an independent solver pass that never sees the proposed answer — is the single highest-leverage control in the whole system, and it costs a fraction of a cent per item.
• An Elo-style mastery model, not full IRT, is the right amount of machinery for a growing item bank with sparse per-item response history.
• Real costs land around $0.10-0.13 per student per month at 5,000-student scale, dominated by grading open-ended responses rather than question generation.
• This is a fit for internal practice assessments and formative quizzes with a digitized skill taxonomy — not a replacement for validated instruments in high-stakes, legally defensible testing.