Building Production-Ready RAG Systems
Take a RAG prototype to production: evaluation harnesses, incremental indexing, caching, latency budgets, and the failure modes that only appear at scale.
Course Overview
A RAG demo takes an afternoon. A RAG system that stays correct as the corpus grows, documents change daily, and users ask questions nobody anticipated is a different engineering problem. This course is about that gap. It assumes you can already build a basic pipeline and focuses entirely on what breaks after launch.
The recurring theme is measurement. Teams tune chunk sizes and prompts by intuition because they have no evaluation harness, so every change is a guess and regressions are invisible until a user complains. You will build a golden dataset, wire up automated retrieval and generation metrics, and put the whole thing in CI so a prompt change that degrades groundedness fails the build.
The rest of the course covers the operational concerns that determine whether the system is affordable and fast enough to keep: incremental and delta indexing so you are not re-embedding the corpus nightly, semantic and exact-match caching, query rewriting for conversational follow-ups, latency budgets across the retrieve-rerank-generate path, and cost control as token spend scales with traffic.
Duration: 3 days|Delivery: onsite, virtual, or hybrid
Prerequisites
- Having built at least a basic RAG pipeline, or completion of Introduction to RAG
- Strong Python; comfort with async programming and testing frameworks
- Working knowledge of Docker and a CI system such as GitHub Actions
- Familiarity with at least one vector store
Who Should Attend
- Engineers whose RAG prototype has been approved for production and now needs an SLA
- ML and platform engineers who own retrieval quality as a measurable metric
- Tech leads who need to justify RAG infrastructure cost and latency to stakeholders
- SREs asked to monitor and support an LLM-backed feature
Course Outline
- 1From prototype to product: the failure modes that only appear after launch
- 2Building a golden evaluation set: query selection, labeling, and avoiding test-set leakage
- 3Retrieval metrics in depth: precision@k, recall@k, MRR, NDCG, and which to optimize
- 4Generation metrics: groundedness, answer relevance, and LLM-as-judge with its pitfalls
- 5Automated evaluation in CI: regression gates on prompt and model changes
- 6Incremental and delta indexing: change data capture, re-embedding policy, and index versioning
- 7Advanced retrieval: query rewriting, HyDE, multi-query expansion, and parent-document retrieval
- 8Reranking at scale: cross-encoders, cost-quality tradeoffs, and when to skip reranking
- 9Caching layers: exact-match, semantic, and embedding caches
- 10Latency budgets: profiling the retrieve-rerank-generate path and shaving p95
- 11Cost modeling: token accounting, embedding refresh cost, and cost per resolved query
- 12Observability: tracing, logging retrieved context, and debugging a bad answer in production
Learning Outcomes
- Build an evaluation harness that catches retrieval and groundedness regressions automatically
- Choose retrieval metrics that reflect your actual user task rather than defaults
- Design an indexing strategy that handles changing documents without full rebuilds
- Cut p95 latency with informed caching and reranking decisions
- Model and control the per-query cost of a RAG system at production traffic
- Instrument a pipeline so any wrong answer can be traced to its cause
- Run a structured rollout with A/B comparison between pipeline versions
What You Will Build
- A golden evaluation dataset and automated scoring harness for your use case
- A CI pipeline that blocks changes which regress retrieval or groundedness
- An incremental indexing design that handles document updates and deletes
- A latency and cost model for your expected query volume
Frequently Asked Questions
- How is this different from Introduction to RAG?
- Introduction to RAG teaches you to build a pipeline. This course assumes you have one and covers what it takes to operate it: evaluation harnesses, incremental indexing, caching, latency budgets, cost control, and observability. If you have never built a RAG pipeline, take the introduction first.
- Do we need a golden dataset before attending?
- No. Building one is part of the course, including how to choose queries, label them without biasing the set, and keep it from going stale. Bringing a list of real user questions makes the exercise more useful, but it is not required.
- Is LLM-as-judge evaluation reliable?
- It is useful and it is also easy to misuse. The course covers where judge models agree with human raters, where they systematically do not, how position and verbosity bias affect scores, and how to calibrate a judge against a human-labeled subset before trusting it in CI.
- Which frameworks and tools are used?
- Labs use Python with LangChain or LlamaIndex for the pipeline, pgvector and FAISS for retrieval, and RAGAS-style metrics for evaluation. The techniques are framework-independent, and we cover how to apply them if your stack differs.
- Can this be tailored to our existing system?
- Yes, and for private cohorts it usually is. We adapt labs to your vector store, model provider, and document types so the tuning work carries straight back into your codebase.
Related Courses
Introduction to RAG
Learn Retrieval-Augmented Generation from first principles: chunking, embeddings, vector search, and grounding LLM answers in your own documents.
Enterprise RAG: Security and Scale
Multi-tenant RAG with document-level access control, PII handling, prompt injection defense, audit trails, and architectures that hold up under regulatory review.
Agentic RAG Patterns
Combine the power of RAG with agentic reasoning. Build AI systems that dynamically retrieve, reason over, and synthesize information from multiple sources.
MLOps for Production AI
Operationalize machine learning: reproducible pipelines, model registries, automated retraining, drift detection, and the practices that keep models correct after launch.
Ready to Get Started?
Contact us to schedule training for your team or inquire about upcoming sessions.