Building a Voice RAG System Under a 200ms Latency Budget

For HH Goa 2026, I worked on Task 2: building a Retrieval-Augmented Generation system over the AI4Bharat MSMARCO-XI dataset.

At first glance, it sounded like a standard RAG problem:
Take a question → retrieve relevant documents → generate an answer.

But the actual engineering challenge was much more interesting. The system needed to handle:

• Voice input
• Speech-to-text
• Large-scale chunking
• Multilingual retrieval
• Dense + sparse search
• Query classification
• Hallucination prevention
• Prompt-injection protection
• Grounding verification
• Failure handling

And, most importantly, a 200ms core latency budget. That last requirement changed almost every architectural decision.

The result was a voice-enabled RAG pipeline that achieved:

• P50: 7.87ms
• P70: 9.62ms
• P100: 40.71ms
• 100% of 927 benchmark runs within 200ms
• 241 automated tests
• Hybrid dense + BM25+ retrieval
• Five chunking strategies
• Multi-layer safety and grounding checks

Repository: github.com/im-shourya/HHGOA-TASK2

1. The Problem

The system was built around the ai4bharat/MSMARCO-XI dataset. The goal was to create a system where a user could essentially:
Speak → Retrieve → Understand → Answer

The initial architecture looked straightforward:
Voice Input ↓ Speech-to-Text ↓ Query Processing ↓ Vector Search ↓ RAG ↓ Answer

But this architecture hides several problems. A conventional RAG pipeline can easily spend hundreds of milliseconds on embedding generation, retrieval, reranking, and LLM inference. A hard 200ms budget meant that simply throwing a larger embedding model and an LLM at the problem wasn't going to work.

So the first principle became:
Optimize the architecture before optimizing the code.

2. The Final Architecture

The final pipeline became considerably more defensive:

┌─────────────────┐
│   Voice Input   │
└────────┬────────┘
         ↓
┌─────────────────┐
│ Speech-to-Text  │
└────────┬────────┘
         ↓
┌─────────────────┐
│   Input Guard   │
└────────┬────────┘
         ↓
┌─────────────────┐
│Query Classifier │
└────────┬────────┘
         ↓
┌─────────────────┐
│ Query Embedding │
└────────┬────────┘
         ↓
┌──────────────┴──────────────┐
↓                             ↓
Dense Retrieval          BM25+ Retrieval
│                             │
└──────────────┬──────────────┘
               ↓
              RRF
               ↓
              MMR
               ↓
        Retrieval Guard
            ↙     ↘
      Decline     Answer
                  ↓
           Grounding Check
                  ↓
                Output

One important architectural decision was separating answerability from grounding. The system doesn't simply ask: "Can I support this generated answer?" It first asks: "Do I actually have enough evidence to answer this question?" That distinction turned out to be extremely important.

3. Chunking Was More Complicated Than Expected

One requirement of the task was to explore different chunking strategies rather than relying on a single fixed-size splitter. So I implemented five approaches:

Running all five independently would have produced almost 40,000 chunks. After reconciliation and deduplication, this became 18,416 chunks. That gave us a 53.9% reduction while preserving provenance.

This was one of the interesting parts of the project because chunking isn't simply "What chunk size gives the best score?" It is also "How can multiple views of the same text coexist without exploding memory and retrieval cost?"

4. Hybrid Retrieval

The retrieval system combines two fundamentally different approaches.

• Dense retrieval: Semantic similarity is calculated using static embeddings.
• Sparse retrieval: BM25+ handles exact lexical matches.

Then the results are combined using Reciprocal Rank Fusion (RRF). Finally, Maximal Marginal Relevance (MMR) is used to diversify the results.

Why not just use vector search? Because semantic search and keyword search fail differently. Dense retrieval can find something conceptually similar but miss an important exact term. BM25 can match the exact words while completely misunderstanding the intent. Combining both gives the system two different retrieval signals.

5. The 200ms Constraint Changed Everything

This was probably the most interesting part of the project. The initial instinct in an RAG system is usually: Better model → better embeddings → better answers.

Under a strict latency budget, the equation changes. The question becomes: What is the cheapest representation that gives us enough retrieval quality?

The embedding layer therefore became a major optimization target. The final system uses minishlab/potion-retrieval-32M. The important property is that it is a token-lookup based model rather than requiring a conventional transformer forward pass for every query.

The measured query embedding latency was approximately 0.31ms P50. That made the 200ms target much more realistic.

6. Measuring Every Stage

Instead of measuring request → response, I instrumented the pipeline at each stage.

And the P100 was 40.71ms. Across 927 benchmark runs, every request stayed within the 200ms core budget.

7. But Benchmarking Is Easy to Get Wrong

One thing I learned from this project is that reporting a latency number is meaningless unless you explain exactly how it was measured.

For example, enabling a query cache could make repeated benchmark queries appear extremely fast. So the benchmark was intentionally run with:

• Query cache: OFF
• Warm-up: completed
• Repeated queries: 3 times
• Distinct queries: 309
• Total runs: 927

The trace was also required to satisfy: core_latency_ms = sum(all measured stages). Without trace accounting, it becomes surprisingly easy to accidentally "lose" latency between stages.

8. The Live HTTP Path Was Measured Separately

Another important distinction was between in-process latency and actual HTTP latency.

The in-process benchmark achieved: P50 7.87ms | P70 9.62ms | P100 40.71ms

But a real user doesn't call the Python function directly. They hit an HTTP endpoint. So I separately benchmarked the deployed service over real HTTP. The result:

• Server core P50: 6.04ms
• Client wall P50: 7.58ms
• Client wall P100: 60.49ms

This distinction matters because benchmarking the internal function and calling the actual deployed service are two different experiments.

9. Guardrails: Sometimes the Correct Answer Is "I Don't Know"

One of the biggest design goals was preventing the system from confidently answering questions that it couldn't support. The pipeline therefore has multiple refusal paths:

• Unsafe request ↓ Decline
• Prompt injection ↓ Decline
• Malformed query ↓ Decline
• Insufficient retrieval confidence ↓ Decline
• Unsupported generated claim ↓ Repair / Decline

The input guard runs before retrieval. This means an unsafe or malformed request can terminate before the system spends time performing expensive retrieval.

10. Retrieval Confidence

Retrieval confidence isn't based on a single number. The system combines: Dense similarity, BM25 evidence, and Query-term coverage.

The reason is simple: Each individual signal can be wrong. Dense similarity can be fooled by topical similarity. BM25 can be fooled by common words. Term coverage can fail on paraphrases. But agreement between independent signals provides stronger evidence.

11. Grounding Is Not the Same as Answerability

This was probably my favorite finding from the project. It is tempting to think:
Retrieved context ↓ Generated answer ↓ Does answer match context? ↓ Safe

But that isn't sufficient. Imagine the retrieved passage says something completely unrelated to the question. The model can generate an answer that is perfectly consistent with that passage. The answer is grounded. But it is still answering the wrong question.

So the system separates Answerability ("Does my retrieved evidence actually contain enough information to answer this question?") from Grounding ("Are the claims in my final answer supported by the evidence I cited?"). These are different problems. The retrieval guard handles the first. The output verification layer handles the second.

12. Extractive Generation as a Safety Mechanism

Another deliberate choice was making extractive answering the default mode. Instead of always asking an LLM to generate arbitrary text, the system can compose an answer directly from retrieved evidence.

This gives us an important property: The default answer cannot invent information that isn't present in the retrieved context. An optional LLM path can polish the answer when the latency budget allows it. But if there isn't enough budget, the system falls back to the extractive path. So instead of LLM failed ↓ Error the system does LLM unavailable / too slow ↓ Extractive answer.

13. Handling External Services

Speech-to-text and LLM providers introduce another problem: They are external dependencies. External APIs can timeout, return errors, become temporarily unavailable, or exceed the latency budget.

So external calls are wrapped with bounded retries, exponential backoff, jitter, and circuit-breaker behavior. The system also supports multiple STT providers: Sarvam ↓ failure ElevenLabs ↓ failure Browser Speech API.

14. A Small Bug That Taught Me a Big Lesson

During development, one of the interesting problems was related to multilingual text processing. A conventional Python regex tokenizer was splitting Devanagari text incorrectly. For example, Hindi words could be broken into individual character fragments.

That silently damaged the sparse retrieval index. English queries appeared fine, which made the problem particularly easy to miss.

The lesson was simple: Multilingual systems need multilingual assumptions at every layer, not just multilingual models. Tokenization, normalization, chunking, metadata, evaluation, and IDs all matter.

15. Another Lesson: Dataset Leakage Can Hide in Metadata

During evaluation, I also became much more cautious about metadata. The dataset contains information derived from the original query. It would have been possible to include query-derived information in chunk metadata and accidentally make retrieval look better.

So chunk metadata was deliberately generated from the passage itself rather than injecting gold-query information. The principle: Never let evaluation labels leak into the retrieval representation.

16. What Didn't Work

The most useful parts of this project were not necessarily the things that worked. Several approaches looked promising but didn't survive measurement. For example:

• More retrieval strategies don't automatically mean better retrieval.
• A larger ensemble isn't automatically worth the memory cost.
• Faster-looking benchmarks can simply be measuring caching.
• Grounding scores don't necessarily measure answerability.
• LLM generation isn't always the best solution.
• A single multilingual component doesn't solve multilingual retrieval.
• HTTP latency and in-process latency are different measurements.

17. Tech Stack

The project uses: Python, FastAPI, Pydantic, BM25+, Dense retrieval, RRF, MMR, Sarvam Saaras, ElevenLabs, Anthropic Claude, ai4bharat/MSMARCO-XI, Docker, pytest.

18. What I Learned

1. Architecture beats brute force: A better architecture can matter more than a larger model.
2. Latency is a design constraint: Don't optimize after building the system. Design around the latency budget from the beginning.
3. Retrieval quality and answer quality are different: A great generator cannot compensate for bad retrieval.
4. "I don't know" is a valid output: A RAG system should know when the evidence isn't sufficient.
5. Benchmarks need methodology: A latency number without knowing the benchmark conditions is almost meaningless.
6. Multilingual systems require end-to-end thinking: Tokenizer → chunker → retriever → evaluator all need to understand the language.
7. Measure before adding complexity: If an optimization doesn't produce a measurable improvement, it may not be worth shipping.

Closing

This project started as a hackathon-style RAG challenge. It ended up teaching me something much more useful: Building an AI system isn't just about making it answer correctly. You also need to know when it should answer, when it shouldn't, where its evidence came from, how fast every stage is, what happens when dependencies fail, and whether your benchmark is actually measuring what you think it is.

If you'd like to explore the implementation, the complete project is available here:
GitHub: HHGOA-TASK2