Code examples:

Building a Basic Vector-Search RAG Pipeline — With Thresholds That Know When to Abstain

By Jaffar Kazi
Real-World AI Engineering RAG Vector Search
C# Python Azure

What You'll Learn

  • How the ingestion and query paths of a vector-search RAG pipeline fit together
  • How embeddings and cosine similarity find paraphrases that keyword search misses
  • Why vector search always returns results, and how a calibrated threshold lets the pipeline abstain
  • How metadata filtering and exact versus approximate search behave, with measured timings
  • How to test the pipeline offline with a fake embedder and a mocked LLM

The Problem

Keyword search breaks the moment a user phrases a question differently from the source. Someone asks "Am I allowed to log in from my sofa?" and the policy says "Employees may work remotely up to three days per week." The two share no useful words. Synonym lists patch individual cases, but you never finish writing them.

Vector search fixes the paraphrase problem by comparing meaning instead of words. I built this pipeline because the part most tutorials skip is the one that bites in production: vector search always returns something. Ask it for the capital of France and it hands back the four nearest chunks of your HR policy, ranked, with scores. Pass those to an LLM and you get a fluent answer built on irrelevant evidence.

So the goal here is not just "retrieve similar text." It is a pipeline that finds paraphrases, shows its retrieval results, and says "I couldn't find that" when the evidence isn't there.

The Shape of a Real Solution

The pipeline has two paths that meet at the vector store.

  • Ingestion (offline): split documents into chunks, embed them in batches, normalise each vector, and store it with metadata (chunk ID, document ID, page, text).
  • Query (online): embed the question with the same model, search with an optional metadata filter, apply a similarity threshold, then either build a prompt from numbered sources or abstain.

Two principles run through it. Retrieval results are an output, not an internal detail: every answer carries the chunks, documents, pages and scores behind it. And "no answer" is a valid result with its own code path, one that costs almost nothing because it never calls the LLM.

Architecture Overview

Architecture diagram showing an offline ingestion path from documents through a chunker, embedding model and vector normalisation into a vector store, and an online query path from a question through the same embedding model and a filtered cosine top-k search to a threshold decision that either calls the LLM or abstains, both returning the answer with visible retrieval results
  • One embedding model on both sides. Vectors from different models aren't comparable, so changing the model means re-embedding the whole corpus.
  • Normalise once, at ingest. With unit-length vectors, cosine similarity is just a dot product.
  • The threshold decision sits before the LLM, so irrelevant retrievals never reach the prompt.
  • The embedder and the LLM are injected, which is what makes the whole pipeline testable offline.

Core Implementation: Ingestion

An embedding is a list of numbers (a vector) produced by a model so that texts with similar meaning land close together. Cosine similarity measures the angle between two vectors: 1 means the same direction, 0 means unrelated, negative means opposed. Scores are not percentages, and their useful range depends on the model, which matters later when we pick a threshold.

Ingestion.cs / vector_rag.py

public interface IEmbedder
{
    Task<IReadOnlyList<float[]>> EmbedAsync(IReadOnlyList<string> texts, CancellationToken ct = default);
}

public interface ILlm
{
    Task<string> CompleteAsync(string prompt, CancellationToken ct = default);
}

public record Chunk(string Id, string DocId, int Page, string Text);
public record ChunkRecord(Chunk Chunk, float[] Vector);

public static class Vectors
{
    // Normalise once at ingest; after that, cosine similarity is just a dot product.
    public static float[] Normalise(float[] v)
    {
        double norm = Math.Sqrt(v.Sum(x => (double)x * x));
        if (norm == 0) throw new ArgumentException("zero vector");
        return v.Select(x => (float)(x / norm)).ToArray();
    }

    public static double Dot(float[] a, float[] b)
    {
        double sum = 0;
        for (int i = 0; i < a.Length; i++) sum += (double)a[i] * b[i];
        return sum;
    }
}

public static class Ingestion
{
    public static async Task IngestAsync(IReadOnlyList<Chunk> chunks, IEmbedder embedder,
                                         VectorStore store, int batchSize = 64,
                                         CancellationToken ct = default)
    {
        for (int i = 0; i < chunks.Count; i += batchSize)
        {
            var batch = chunks.Skip(i).Take(batchSize).ToList();
            var vectors = await embedder.EmbedAsync(batch.Select(c => c.Text).ToList(), ct);
            store.Add(batch.Zip(vectors, (c, v) => new ChunkRecord(c, Vectors.Normalise(v))));
        }
    }
}

import math
from dataclasses import dataclass
from typing import Protocol


class Embedder(Protocol):
    def embed(self, texts: list[str]) -> list[list[float]]: ...


class Llm(Protocol):
    def complete(self, prompt: str) -> str: ...


@dataclass(frozen=True)
class Chunk:
    id: str
    doc_id: str
    page: int
    text: str


@dataclass(frozen=True)
class ChunkRecord:
    chunk: Chunk
    vector: tuple[float, ...]


@dataclass(frozen=True)
class Hit:
    chunk: Chunk
    score: float


@dataclass(frozen=True)
class Answer:
    text: str
    hits: list[Hit]
    abstained: bool


def normalise(v: list[float]) -> tuple[float, ...]:
    norm = math.sqrt(sum(x * x for x in v))
    if norm == 0:
        raise ValueError("zero vector")
    return tuple(x / norm for x in v)


def ingest(chunks: list[Chunk], embedder: Embedder, store: VectorStore, batch_size: int = 64) -> None:
    for i in range(0, len(chunks), batch_size):
        batch = chunks[i:i + batch_size]
        vectors = embedder.embed([c.text for c in batch])
        store.add([ChunkRecord(c, normalise(v)) for c, v in zip(batch, vectors)])

The tricky part is small but easy to miss. Embedding APIs accept many inputs per call, and batching cuts latency and request counts, so ingestion sends chunks in batches rather than one at a time. Each stored record keeps the chunk text and its source metadata next to the vector, so a hit can be shown and cited without a second lookup.

One thing this in-memory store doesn't do, and a real one must: ingestion should be idempotent. Key records by chunk ID and replace them on re-run (an upsert). Otherwise re-ingesting a document silently doubles its chunks, and every question about it returns the same passage twice and crowds a different one out of the top k.

Core Implementation: Search

The store holds records and answers one question: which chunks are closest to this query vector? This version is exact. It scores every record and sorts, which is the simplest thing that is also correct.

VectorStore.cs / vector_rag.py

public record Hit(Chunk Chunk, double Score);

public class VectorStore
{
    private readonly List<ChunkRecord> _records = new();

    public int Count => _records.Count;

    public void Add(IEnumerable<ChunkRecord> records) => _records.AddRange(records);

    // Exact search. Filter first, then rank, so a filter can't push good matches out of the top k.
    public List<Hit> Search(float[] query, int k, string? docId = null) =>
        _records.Where(r => docId is null || r.Chunk.DocId == docId)
                .Select(r => new Hit(r.Chunk, Vectors.Dot(query, r.Vector)))
                .OrderByDescending(h => h.Score)
                .Take(k)
                .ToList();
}

class VectorStore:
    def __init__(self) -> None:
        self._records: list[ChunkRecord] = []

    def __len__(self) -> int:
        return len(self._records)

    def add(self, records: list[ChunkRecord]) -> None:
        self._records.extend(records)

    def search(self, query: tuple[float, ...], k: int, doc_id: str | None = None) -> list[Hit]:
        # Exact search. Filter first, then rank, so a filter can't push good matches out of the top k.
        candidates = (r for r in self._records if doc_id is None or r.chunk.doc_id == doc_id)
        hits = [Hit(r.chunk, sum(a * b for a, b in zip(query, r.vector))) for r in candidates]
        return sorted(hits, key=lambda h: -h.score)[:k]

Notice the order: filter first, then rank. If you rank the whole corpus and filter afterwards, a document filter can leave you with fewer than k results, or none, because the top hits all belonged to other documents.

Key Technical Challenge: Vector Search Always Answers

Here's the part that surprised me when I first ran it. There is no "no match" in nearest-neighbour search. A question about something your documents never mention still returns the k closest chunks. Those chunks have lower scores, but nothing in the retrieval step tells you they are irrelevant. Only a threshold does.

The answer function applies one. Hits below the minimum score are dropped, and if none remain the pipeline abstains without calling the LLM. The abstention still returns the raw hits so you can see what was retrieved and how low the scores were.

RagService.cs / vector_rag.py

public record Answer(string Text, IReadOnlyList<Hit> Hits, bool Abstained);

public class RagService(IEmbedder embedder, VectorStore store, ILlm llm,
                        double minScore = 0.35, int k = 4)
{
    public const string NotFound = "I couldn't find that in the documents.";

    public async Task<Answer> AskAsync(string question, string? docId = null,
                                       CancellationToken ct = default)
    {
        var query = Vectors.Normalise((await embedder.EmbedAsync(new[] { question }, ct))[0]);
        var hits = store.Search(query, k, docId);          // always returns up to k, however irrelevant
        var relevant = hits.Where(h => h.Score >= minScore).ToList();
        if (relevant.Count == 0)
            return new Answer(NotFound, hits, Abstained: true);   // no LLM call: nothing to ground on

        var context = string.Join("\n\n", relevant.Select((h, i) =>
            $"[{i + 1}] {h.Chunk.DocId}, page {h.Chunk.Page}: {h.Chunk.Text}"));
        var prompt = "Answer using only the context. If it does not contain the answer, say so.\n\n" +
                     $"Context:\n{context}\n\nQuestion: {question}";
        return new Answer(await llm.CompleteAsync(prompt, ct), relevant, Abstained: false);
    }
}

NOT_FOUND = "I couldn't find that in the documents."


def answer(question: str, embedder: Embedder, store: VectorStore, llm: Llm,
           k: int = 4, min_score: float = 0.35, doc_id: str | None = None) -> Answer:
    query = normalise(embedder.embed([question])[0])
    hits = store.search(query, k, doc_id)            # always returns up to k, however irrelevant
    relevant = [h for h in hits if h.score >= min_score]
    if not relevant:
        return Answer(NOT_FOUND, hits, abstained=True)   # no LLM call: nothing to ground on

    context = "\n\n".join(f"[{i}] {h.chunk.doc_id}, page {h.chunk.page}: {h.chunk.text}"
                          for i, h in enumerate(relevant, 1))
    prompt = ("Answer using only the context. If it does not contain the answer, say so.\n\n"
              f"Context:\n{context}\n\nQuestion: {question}")
    return Answer(llm.complete(prompt), relevant, abstained=False)

The threshold can't be copied from a blog post. Scores depend on the embedding model and your text, so calibrate it. Collect 30 to 50 real questions, label each as answerable or not, record the best retrieval score for each, and pick the threshold that separates them best. The function below does that, and it prefers the higher threshold on ties because a false answer usually costs more than a false abstention.

Calibration.cs / vector_rag.py

public static class Calibration
{
    // samples: (best retrieval score for a question, whether the corpus can answer it)
    public static double CalibrateThreshold(IReadOnlyList<(double Score, bool Answerable)> samples)
    {
        double bestT = 0;
        int bestCorrect = -1;
        foreach (var t in samples.Select(s => s.Score).Distinct().OrderBy(s => s))
        {
            int correct = samples.Count(s => (s.Score >= t) == s.Answerable);
            if (correct >= bestCorrect) { bestT = t; bestCorrect = correct; }   // prefer the higher threshold on ties
        }
        return bestT;
    }
}

def calibrate_threshold(samples: list[tuple[float, bool]]) -> float:
    """samples: (best retrieval score for a question, whether the corpus can answer it)."""
    best_t, best_correct = 0.0, -1
    for t in sorted({score for score, _ in samples}):
        correct = sum((score >= t) == answerable for score, answerable in samples)
        if correct >= best_correct:                  # prefer the higher threshold on ties
            best_t, best_correct = t, correct
    return best_t

Two failure modes pull in opposite directions. Too low a threshold lets irrelevant context through and the model answers confidently from it. Too high and the pipeline refuses questions it could have answered. The prompt instruction "say so if the context does not contain the answer" is a second line of defence, not a substitute for the threshold.

The visible hits are also your debugging tool. When an answer is wrong, the retrieval results tell you which of three problems you have. If the right chunk isn't in the hits at all, the problem is upstream: chunking, or an embedding model that doesn't represent your text well. If it is present but ranked low, the question and the passage are phrased too differently, and a better query or a hybrid search may help. If the right chunk is the top hit and the answer is still wrong, retrieval did its job and the fault is in the prompt or the model. Without the hits you can't tell these apart.

Key Technical Challenge: Filtering and Indexing

Real corpora need metadata filters: only this document, only this department, only this year. Stores differ in whether they filter before or after the nearest-neighbour search, so check your store's behaviour rather than assuming. Post-filtering is the one that quietly returns too few results.

Indexing is the other decision. Exact search compares the query to every vector. Approximate indexes such as HNSW trade a little recall for speed on large collections, and add parameters to tune. I measured exact search before deciding I needed anything else. These are normalised random 1,536-dimension float32 vectors, top 4, using numpy on one machine, so they show speed and memory, not retrieval quality:

VectorsMemoryMedian search95th percentile
10,00061 MB2.3 ms2.6 ms
50,000307 MB11.4 ms12.2 ms

Time grew about five times for five times the vectors, so it scales linearly. If that holds, a million vectors would take roughly 230 ms and about 6 GB, which is the extrapolated point where approximate indexes start to earn their complexity. For a corpus of tens of thousands of chunks, exact search is fast enough, and it can't miss a nearest neighbour.

What This Costs

Cost has three parts: embedding the corpus once, embedding each question, and the LLM call for answered questions. I use OpenAI's published list prices at the time of writing, with about 1.3 tokens per English word as a rough conversion.

ItemAssumptionCost
Embed the corpus once10 million words, about 13 million tokens$0.26 with text-embedding-3-small, $1.69 with text-embedding-3-large
Vector memory (in-memory store)about 66,700 chunks of 150 words, 1,536 dimensions, float32about 410 MB
Answered questionabout 900 input and 150 output tokens (4 chunks plus prompt)about $0.00135 with GPT-5.4 mini; about $0.0045 with GPT-5.4
Abstained questionone query embedding of about 15 tokensabout $0.0000003

Cost Reality

Per 1,000 answered questions that is about $1.35 with GPT-5.4 mini and $4.50 with GPT-5.4, so generation dominates and embedding is nearly free. The expensive event is changing the embedding model, because every vector must be recomputed. Prices are OpenAI's direct list prices, not Azure's, and they change, so check current pricing.

The cheapest lever on the generation side is k. Each extra 150-word chunk adds roughly 195 tokens to every prompt, so moving from 4 chunks to 8 about doubles the context cost per answered question. The threshold helps here too, because chunks that don't clear it never reach the prompt, which keeps k a ceiling instead of a fixed spend.

Observability

Log what the retrieval step did, not what it read. For each question, record the filter used, the chunk IDs with their scores and ranks, how many cleared the threshold, whether the pipeline abstained, and whether an LLM call was made.

Two aggregates are worth watching: the abstention rate, which shifts when your documents or your users' questions change, and the distribution of top-1 scores, which drifts if you swap embedding models by accident. Avoid logging question text or chunk text by default, since both can contain personal or confidential data.

Testing

The embedder and the LLM are the only parts that depend on a model, so I replace both. The embedder returns hand-built four-dimensional vectors, and the LLM is a fake that counts its calls. Everything else (batching, normalisation, search, filtering, thresholding, prompt building) runs as ordinary code.

The tests cover the brief's two demonstrations. A paraphrase with no shared words is found by vector search, and a keyword baseline finds nothing for it. An off-topic question still returns k hits, all below the threshold. The call-count assertions prove the control flow: an abstained question makes zero LLM calls, and an answered one makes exactly one.

The vectors encode an assumption, that a good model places paraphrases close together. These tests prove the pipeline handles that geometry correctly, not that any real model embeds well. I haven't included similarity scores from a real model in this article, because they depend on the model and your text. Run the pipeline against your own deployment and look at the scores for your own questions.

VectorRagTests.cs / test_vector_rag.py

using System.Text.RegularExpressions;
using Xunit;

public class VectorRagTests
{
    static readonly HashSet<string> Stop = new() { "am", "i", "to", "in", "from", "my", "a", "the", "is", "of" };

    static readonly List<Chunk> Chunks = new()
    {
        new("c1", "remote-work", 1, "Employees may work remotely up to three days per week."),
        new("c2", "leave", 1, "Full-time employees accrue 20 days of annual leave."),
        new("c3", "travel", 1, "Flights over 10 hours may be booked in business class."),
    };

    // Hand-built 4-d vectors: axes are remote-work, leave, travel and "something else".
    // They encode the ASSUMPTION that a good embedding model places paraphrases close together.
    static readonly Dictionary<string, float[]> Vecs = new()
    {
        [Chunks[0].Text] = new float[] { 1, 0, 0, 0 },
        [Chunks[1].Text] = new float[] { 0, 1, 0, 0 },
        [Chunks[2].Text] = new float[] { 0, 0, 1, 0 },
        ["Am I allowed to log in from my sofa?"] = new[] { 0.9f, 0.1f, 0.1f, 0f },   // paraphrase
        ["What is the capital of France?"] = new[] { 0.2f, 0.1f, 0.1f, 0.97f },      // off topic
    };

    class FakeEmbedder : IEmbedder
    {
        public int Calls;
        public Task<IReadOnlyList<float[]>> EmbedAsync(IReadOnlyList<string> texts, CancellationToken ct = default)
        {
            Calls++;
            return Task.FromResult<IReadOnlyList<float[]>>(texts.Select(t => Vecs[t]).ToList());
        }
    }

    class FakeLlm : ILlm
    {
        public int Calls;
        public List<string> Prompts = new();
        public Task<string> CompleteAsync(string prompt, CancellationToken ct = default)
        {
            Calls++; Prompts.Add(prompt);
            return Task.FromResult("You may work remotely three days a week.");
        }
    }

    static IEnumerable<string> Tokens(string s) => Regex.Matches(s.ToLowerInvariant(), "[a-z]+").Select(m => m.Value);

    static List<Chunk> KeywordHits(string query, IEnumerable<Chunk> chunks)
    {
        var q = Tokens(query).Where(t => !Stop.Contains(t)).ToHashSet();
        return chunks.Where(c => Tokens(c.Text).Any(q.Contains)).ToList();
    }

    static async Task<(FakeEmbedder, FakeLlm, VectorStore, RagService)> PipelineAsync()
    {
        var embedder = new FakeEmbedder(); var llm = new FakeLlm(); var store = new VectorStore();
        await Ingestion.IngestAsync(Chunks, embedder, store);
        embedder.Calls = 0;                                 // count only query-time calls
        return (embedder, llm, store, new RagService(embedder, store, llm));
    }

    [Fact]
    public async Task Ingest_BatchesAndStoresUnitVectors()
    {
        var embedder = new FakeEmbedder(); var store = new VectorStore();
        await Ingestion.IngestAsync(Chunks, embedder, store, batchSize: 2);
        Assert.Equal(2, embedder.Calls);                     // 3 chunks in batches of 2
        Assert.Equal(3, store.Count);
        Assert.Equal(1.0, Vectors.Dot(Vectors.Normalise(new float[] { 3, 4, 0, 0 }), Vectors.Normalise(new float[] { 3, 4, 0, 0 })), 5);
    }

    [Fact]
    public async Task SemanticMatch_IsFound_WhereKeywordSearchFindsNothing()
    {
        var (_, _, _, rag) = await PipelineAsync();
        const string q = "Am I allowed to log in from my sofa?";
        Assert.Empty(KeywordHits(q, Chunks));                // no shared words at all
        var result = await rag.AskAsync(q);
        Assert.Equal("c1", result.Hits[0].Chunk.Id);
        Assert.False(result.Abstained);
    }

    [Fact]
    public async Task VectorSearch_ReturnsKHits_EvenForAnIrrelevantQuestion()
    {
        var (_, _, store, _) = await PipelineAsync();
        var hits = store.Search(Vectors.Normalise(Vecs["What is the capital of France?"]), 3);
        Assert.Equal(3, hits.Count);                         // it never returns "nothing"
        Assert.True(hits.Max(h => h.Score) < 0.35);          // but none of it is relevant
    }

    [Fact]
    public async Task Abstains_WithoutCallingTheLlm_WhenNothingClearsTheThreshold()
    {
        var (_, llm, _, rag) = await PipelineAsync();
        var result = await rag.AskAsync("What is the capital of France?");
        Assert.True(result.Abstained);
        Assert.Equal(RagService.NotFound, result.Text);
        Assert.Equal(0, llm.Calls);                          // nothing to ground an answer on
        Assert.Equal(3, result.Hits.Count);                  // retrieval stays visible
    }

    [Fact]
    public async Task Answers_WithExactlyOneLlmCall_AndCitesTheSource()
    {
        var (_, llm, _, rag) = await PipelineAsync();
        var result = await rag.AskAsync("Am I allowed to log in from my sofa?");
        Assert.Equal(1, llm.Calls);
        Assert.Contains("[1] remote-work, page 1:", llm.Prompts[0]);
        Assert.Contains(Chunks[0].Text, llm.Prompts[0]);
        Assert.Equal(new[] { "c1" }, result.Hits.Select(h => h.Chunk.Id));
    }

    [Fact]
    public async Task MetadataFilter_RestrictsResultsToOneDocument()
    {
        var (_, _, _, rag) = await PipelineAsync();
        var result = await rag.AskAsync("Am I allowed to log in from my sofa?", docId: "leave");
        Assert.All(result.Hits, h => Assert.Equal("leave", h.Chunk.DocId));
    }

    [Fact]
    public void ZeroVector_IsRejected() =>
        Assert.Throws<ArgumentException>(() => Vectors.Normalise(new float[] { 0, 0, 0, 0 }));

    [Fact]
    public void Calibration_SeparatesAnswerableFromUnanswerableQuestions()
    {
        var samples = new List<(double, bool)>
            { (0.82, true), (0.71, true), (0.55, true), (0.48, false), (0.30, false), (0.22, false) };
        Assert.Equal(0.55, Calibration.CalibrateThreshold(samples));   // the lowest score that still separates them
    }
}

import re

import pytest

from vector_rag import Chunk, NOT_FOUND, VectorStore, answer, calibrate_threshold, ingest, normalise

STOP = {"am", "i", "to", "in", "from", "my", "a", "the", "is", "of"}

CHUNKS = [
    Chunk("c1", "remote-work", 1, "Employees may work remotely up to three days per week."),
    Chunk("c2", "leave", 1, "Full-time employees accrue 20 days of annual leave."),
    Chunk("c3", "travel", 1, "Flights over 10 hours may be booked in business class."),
]

# Hand-built 4-d vectors: axes are remote-work, leave, travel and "something else".
# They encode the ASSUMPTION that a good embedding model places paraphrases close together.
VECTORS = {
    CHUNKS[0].text: [1, 0, 0, 0],
    CHUNKS[1].text: [0, 1, 0, 0],
    CHUNKS[2].text: [0, 0, 1, 0],
    "Am I allowed to log in from my sofa?": [0.9, 0.1, 0.1, 0],   # paraphrase of remote work
    "What is the capital of France?": [0.2, 0.1, 0.1, 0.97],      # off topic
    "Can I take annual leave?": [0.1, 0.9, 0, 0],
}


class FakeEmbedder:
    def __init__(self):
        self.calls = 0

    def embed(self, texts):
        self.calls += 1
        return [VECTORS[t] for t in texts]


class FakeLlm:
    def __init__(self):
        self.calls, self.prompts = 0, []

    def complete(self, prompt):
        self.calls += 1
        self.prompts.append(prompt)
        return "You may work remotely three days a week."


def keyword_hits(query, chunks):
    q = set(re.findall(r"[a-z]+", query.lower())) - STOP
    return [c for c in chunks if q & set(re.findall(r"[a-z]+", c.text.lower()))]


@pytest.fixture
def pipeline():
    embedder, llm, store = FakeEmbedder(), FakeLlm(), VectorStore()
    ingest(CHUNKS, embedder, store)
    embedder.calls = 0                                  # count only query-time calls
    return embedder, llm, store


def test_ingest_batches_and_stores_unit_vectors():
    embedder, store = FakeEmbedder(), VectorStore()
    ingest(CHUNKS, embedder, store, batch_size=2)
    assert embedder.calls == 2 and len(store) == 3      # 3 chunks in batches of 2
    assert abs(sum(x * x for x in normalise([3, 4, 0, 0])) - 1) < 1e-9


def test_semantic_match_is_found_where_keyword_search_finds_nothing(pipeline):
    embedder, llm, store = pipeline
    question = "Am I allowed to log in from my sofa?"
    assert keyword_hits(question, CHUNKS) == []         # no shared words at all
    result = answer(question, embedder, store, llm)
    assert result.hits[0].chunk.id == "c1" and not result.abstained


def test_vector_search_returns_k_hits_even_for_an_irrelevant_question(pipeline):
    embedder, _, store = pipeline
    hits = store.search(normalise(VECTORS["What is the capital of France?"]), k=3)
    assert len(hits) == 3                               # it never returns "nothing"
    assert max(h.score for h in hits) < 0.35            # but none of it is relevant


def test_abstains_without_calling_the_llm_when_nothing_clears_the_threshold(pipeline):
    embedder, llm, store = pipeline
    result = answer("What is the capital of France?", embedder, store, llm)
    assert result.abstained and result.text == NOT_FOUND
    assert llm.calls == 0                               # nothing to ground an answer on
    assert len(result.hits) == 3                        # retrieval stays visible


def test_answers_with_exactly_one_llm_call_and_cites_the_source(pipeline):
    embedder, llm, store = pipeline
    result = answer("Am I allowed to log in from my sofa?", embedder, store, llm)
    assert llm.calls == 1
    assert "[1] remote-work, page 1:" in llm.prompts[0]
    assert CHUNKS[0].text in llm.prompts[0]
    assert [h.chunk.id for h in result.hits] == ["c1"]


def test_metadata_filter_restricts_results_to_one_document(pipeline):
    embedder, llm, store = pipeline
    result = answer("Am I allowed to log in from my sofa?", embedder, store, llm, doc_id="leave")
    assert all(h.chunk.doc_id == "leave" for h in result.hits)


def test_zero_vector_is_rejected():
    with pytest.raises(ValueError):
        normalise([0, 0, 0, 0])


def test_calibration_separates_answerable_from_unanswerable_questions():
    samples = [(0.82, True), (0.71, True), (0.55, True), (0.48, False), (0.30, False), (0.22, False)]
    assert calibrate_threshold(samples) == 0.55         # the lowest score that still separates them

Technology Choices

C#/.NET Implementation

Why choose C#: If your backend is .NET, you get first-party Microsoft support and enterprise patterns.

  • Native Azure integration — the Azure OpenAI SDK provides an embedding client, as in the adapter below
  • Enterprise patterns — dependency injection for the embedder and LLM, immutable records for chunks and hits
  • Production-ready — ingestion fits a background service and the query path an ASP.NET Core endpoint

Python Implementation

Why choose Python: If your team writes Python, you get access to the richest AI/ML ecosystem.

  • Library ecosystem — numpy for fast exact search, plus most vector stores' client libraries
  • Rapid prototyping — quick iteration on thresholds and chunking in a notebook
  • Community — most RAG examples and evaluation tooling are Python-first

The Bottom Line

C#/.NET team? Use C#. Python team? Use Python. Don't fight your stack. Both versions pass the same eight tests. The in-memory store here is the part to replace: Azure AI Search or another vector store can provide the retrieval layer behind the same search interface.

The adapters below connect the embedder interface to Azure OpenAI. I compiled the C# one against the Azure SDK and constructed the Python client offline, but I haven't called a live endpoint, so treat them as a starting point and check the current API version.

AzureEmbedder.cs / azure_embedder.py

using Azure.AI.OpenAI;
using OpenAI.Embeddings;

public class AzureEmbedder(AzureOpenAIClient client, string deployment) : IEmbedder
{
    private readonly EmbeddingClient _embeddings = client.GetEmbeddingClient(deployment);

    public async Task<IReadOnlyList<float[]>> EmbedAsync(IReadOnlyList<string> texts,
                                                         CancellationToken ct = default)
    {
        var result = await _embeddings.GenerateEmbeddingsAsync(texts, cancellationToken: ct);
        return result.Value.Select(e => e.ToFloats().ToArray()).ToList();
    }
}

from openai import AzureOpenAI


class AzureEmbedder:
    def __init__(self, endpoint: str, api_key: str, deployment: str,
                 api_version: str = "2024-10-21"):        # use the current GA version
        self._client = AzureOpenAI(azure_endpoint=endpoint, api_key=api_key, api_version=api_version)
        self._deployment = deployment

    def embed(self, texts: list[str]) -> list[list[float]]:
        response = self._client.embeddings.create(model=self._deployment, input=texts)
        return [item.embedding for item in response.data]

A managed store such as Azure AI Search adds what the in-memory one lacks: persistence, concurrent updates, approximate indexes and built-in filters. The cost is another service to run, plus its own filter semantics to learn. Even then I'd keep the in-memory store for tests, because it makes the retrieval logic checkable in milliseconds with no network.

Engineering Trade-offs

Vector search pays off when users paraphrase, the corpus is too big to read in one prompt, and answers live in specific passages. It adds an ingestion pipeline, a model dependency and a threshold to maintain.

When to Skip or Rethink This

  • The corpus fits in the context window. Send the documents and skip retrieval.
  • Queries are identifiers or exact strings. Part numbers, error codes and names are better served by keyword search, and often by a hybrid of both.
  • You need aggregation. "How many policies mention X?" is a query, not a retrieval problem.
  • You trust a default threshold. Any number you didn't calibrate on your own questions is a guess.

Many teams end up at hybrid search, running keyword and vector search together and merging the results, because each covers the other's blind spot: vectors handle paraphrase, keywords handle exact identifiers. I'd start with vector search alone, measure where it fails on your own questions, and add the keyword leg only if the failures look like identifiers and exact phrases.

The next steps are citations with source verification and a retrieval evaluation harness. Both build on the visible hits this pipeline already returns.

Key Takeaways

  • Vector search matches meaning, so it finds paraphrases that keyword search misses.
  • It always returns k results, relevant or not. A calibrated threshold is what lets the pipeline abstain.
  • Normalise vectors at ingest so cosine similarity is a dot product, and use one embedding model on both paths.
  • Filter before ranking so a metadata filter can't empty your results.
  • Exact search is fast enough for tens of thousands of chunks: 11.4 ms for 50,000 in my measurement.
  • Inject the embedder and the LLM, and assert call counts to prove the control flow.

Interview Questions

What is an embedding? A vector produced by a model so that texts with similar meaning are close together. Search then becomes a geometry problem instead of a string-matching one.

Why cosine similarity? It compares direction and ignores vector length, which suits text embeddings. With normalised vectors it reduces to a dot product, which is cheap to compute.

When do you need a vector index? When exact search gets too slow or memory-heavy. Exact search scaled linearly in my test (11.4 ms for 50,000 vectors), and approximate indexes like HNSW trade a little recall for speed beyond that.

How do you choose a retrieval threshold? Calibrate it. Label real questions as answerable or not, record the top score for each, and pick the value that separates them, favouring abstention when a wrong answer is costly.

How does metadata filtering interact with search? Filtering before ranking guarantees k results from the allowed set. Filtering afterwards can return too few, so check how your store behaves.

Want More Practical AI Tutorials?

I write about building production AI systems with Azure, Python, and C#. Subscribe for practical tutorials delivered twice a month.

Subscribe to Newsletter →

Written by Jaffar Kazi, a software engineer in Sydney building AI-powered applications. Connect on LinkedIn.