← All posts

Tuning Chunking, Indexing, and Retrieval in Azure AI Foundry RAG Pipelines

7 min read4.3k
#Azure#Foundry#RAG#LLMOps#Generative AI#AI Architecture#Microsoft

If you've already shipped a first RAG pipeline in Azure AI Foundry, you've probably hit the point where "it works on the demo doc" stops being good enough. This post is about the tuning work that happens after that: chunking strategy, index configuration, hybrid retrieval, and — the part most teams skip — actually measuring retrieval quality instead of eyeballing generation quality.

The retrieval pipeline, end to end

Embedding model choice matters more than teams assume

Chunking and hybrid search get most of the attention, but the embedding model underneath is doing the heavy lifting for the vector half of retrieval, and swapping it isn't free. A few things worth knowing before you pick one and move on:

  • Dimension count trades off storage/latency against recall. A 1536-dimension embedding model captures more nuance than a 384-dimension one, but every additional dimension increases index size and query latency linearly. For most enterprise document sets, the jump from 384 to 1536 dimensions shows a real recall improvement; the jump from 1536 to 3072 usually doesn't justify the cost.
  • Domain-specific fine-tuned embeddings beat general-purpose ones on jargon-heavy content. If your documents are full of internal product names, acronyms, or industry-specific terminology, a general embedding model will cluster those terms poorly in vector space — they weren't well-represented in its training data. Fine-tuning an embedding model on your own document corpus (even a lightweight contrastive fine-tune) often produces a bigger recall gain than any amount of chunking optimization.
  • Re-embedding is expensive and easy to forget. If you change embedding models after your index is already built, every existing chunk needs to be re-embedded and re-indexed — old vectors are meaningless against a new model's vector space. Teams sometimes discover this the hard way after upgrading a model version and getting degraded results because half the index is old vectors and half is new.
python
def validate_embedding_consistency(index_client, sample_size=20):
    """Sanity check: are all vectors in the index from the same embedding model/version?"""
    sample = index_client.search("*", top=sample_size, select=["id", "embedding_model_version"])
    versions = set(doc["embedding_model_version"] for doc in sample)
    if len(versions) > 1:
        raise ValueError(f"Index contains mixed embedding versions: {versions} — re-index required")
    return versions

Tag every indexed chunk with the embedding model version that produced it, and run a check like this before trusting retrieval quality metrics — a silently mixed index is one of the more confusing failure modes to debug after the fact, because retrieval looks "randomly" worse rather than obviously broken.

Why generation quality is the wrong thing to optimize first

When a RAG answer is wrong, the instinct is to blame the model or the prompt. In practice, a large fraction of "bad answers" are retrieval failures wearing a generation costume: the right chunk never made it into context, so the model either hallucinates or answers from a semantically-adjacent-but-wrong passage. Before touching the prompt, isolate retrieval:

python
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

client = AIProjectClient(
    endpoint="https://your-foundry-resource.services.ai.azure.com/api/projects/your-project",
    credential=DefaultAzureCredential(),
)

def retrieve_only(query, index_name, top_k=5):
    search_client = client.inference.get_search_client(index_name)
    results = search_client.search(query, top=top_k)
    return [{"content": r.content, "score": r.score, "source": r.source} for r in results]

Run this against a labeled set of (query, expected_source) pairs before you ever look at a generated answer. If the expected chunk isn't in the top-k, no amount of prompt engineering fixes it.

Chunking strategy: fixed-size is a starting point, not a destination

The default fixed-token chunking (e.g., 512 tokens with 50-token overlap) is fine for homogeneous prose but breaks down on:

  • Tables and structured data — a table split mid-row loses meaning entirely. Detect table boundaries and keep them atomic, even if that means a larger-than-average chunk.
  • Hierarchical documents (policy docs, contracts) — a clause referencing "Section 4.2" is meaningless without that section's heading. Prepend heading context to each chunk rather than relying on overlap alone.
  • Code and config files — chunk on function/class boundaries using an AST-aware splitter, not token count.

A pattern that works well in practice: chunk semantically first (by section/heading/function), then apply a token-count ceiling within each semantic unit, splitting only if it exceeds the limit.

Content typeNaive fixed-token chunkingRecommended strategy
TablesSplits mid-row, loses meaningKeep atomic, allow larger chunk size
Hierarchical docs (policies, contracts)Loses section context on splitPrepend heading path to each chunk
Code / configBreaks functions/blocks arbitrarilySplit on AST/function boundaries
Plain proseWorks reasonably wellFixed-token with modest overlap
python
def semantic_chunk(document, max_tokens=512):
    sections = split_by_headings(document)
    chunks = []
    for section in sections:
        if token_count(section.text) <= max_tokens:
            chunks.append({"text": section.text, "heading": section.heading})
        else:
            for sub in split_by_tokens(section.text, max_tokens, overlap=50):
                chunks.append({"text": f"{section.heading}\n{sub}", "heading": section.heading})
    return chunks

Hybrid retrieval: don't choose between vector and keyword search

Pure vector search misses exact-match cases (product SKUs, error codes, proper nouns that weren't well-represented in training). Pure keyword search misses paraphrase and conceptual queries. Azure AI Search's hybrid mode — combining BM25 keyword scoring with vector similarity via Reciprocal Rank Fusion — outperforms either alone on mixed query workloads, and the config cost is low:

json
{
  "search": "*",
  "vectorQueries": [
    { "vector": [0.02, -0.13, ...], "k": 20, "fields": "content_vector" }
  ],
  "queryType": "semantic",
  "semanticConfiguration": "default"
}

Turn on semantic ranking too — it re-ranks the fused result set using a cross-encoder, which is where most of the precision gain in the top-3 results actually comes from.

Measuring retrieval quality, not vibes

Build a small eval set — 50-100 real or realistic queries with known correct source chunks — and track:

  • Recall@k: did the correct chunk appear in the top k results?
  • MRR (Mean Reciprocal Rank): how high did it rank when it did appear?
python
def recall_at_k(eval_set, retrieve_fn, k=5):
    hits = 0
    for query, expected_source in eval_set:
        results = retrieve_fn(query, top_k=k)
        if any(r["source"] == expected_source for r in results):
            hits += 1
    return hits / len(eval_set)

Re-run this every time you change chunking strategy, embedding model, or index config. It's the only way to know whether a change actually helped or just felt like it should.

Common failure mode: overlap masking a chunking problem

If you find yourself increasing overlap to "fix" retrieval misses, stop — that's usually a signal your chunk boundaries are wrong, not that you need more redundancy. Overlap is a band-aid for boundary noise; it doesn't fix chunks that split coherent ideas in half. Fix the boundaries first.

What's next

Once retrieval is solid, the next failure mode shows up in the orchestration layer itself — Prompt Flow DAGs that work in testing but degrade under production traffic in ways that are hard to trace. That's Part 2.

References