The Technology Stack
Behind Production RAG + MCP
A companion to the RAG system design playbook — that post covers the decisions (chunking, hybrid search, editorial guardrails). This one covers the infrastructure those decisions run on: concrete AWS services, data schemas, and how they wire together into an MCP server that lets an AI agent query a proprietary corpus with citations.
Why a Separate Stack Post
Most RAG writeups stop at the algorithm: chunk, embed, retrieve, generate. That's necessary but it isn't what an engineering team actually has to build. They have to decide where 200,000 source documents physically live, what service resolves a hybrid query in under 300ms, how a nightly ingestion job differs from a streaming one, and — increasingly — how an external AI agent (a Claude, a Perplexity, a Copilot) is allowed to call into that corpus at all without becoming a security or compliance liability.
That last piece is now its own category: the Model Context Protocol (MCP). Instead of a bespoke chat UI, the system exposes a small set of typed tools — search_documents, get_source — that any MCP-compatible agent can call directly, with citations flowing back to the original record. This post walks the stack end to end using AWS as the reference cloud, then closes with a cloud-agnostic mapping for teams standardized on Azure/Databricks instead.
Reference Architecture
Seven layers, each owned by a small number of managed AWS services. No layer is exotic — the value is in how they're wired together and governed.
Source & Storage Layer
Two stores doing two different jobs — raw content is not the same problem as queryable metadata.
2.1 — S3: the system of record for content
Every source document — a transcript, an article, a policy PDF — lands in S3 first, untouched. This is the layer that makes the rest of the system reproducible: if you change your chunking strategy or embedding model, you re-run ingestion against S3, you don't re-collect data. Versioning is on; nothing is ever overwritten, only superseded.
| Bucket / prefix | Contents | Why it's separate |
|---|---|---|
| raw/{source}/{yyyy}/{mm}/{doc_id}.json | Original document as received — full transcript or article body, unmodified | Immutable audit source; re-ingestion never touches this |
| processed/{doc_id}/chunks.jsonl | Chunked, cleaned text ready for embedding | Decouples chunking iteration from raw ingestion |
| exports/compliance/{date}.parquet | Nightly Athena export for legal/compliance review | Compliance queries never hit the hot path |
2.2 — DynamoDB: the document registry
DynamoDB holds one item per source document — not the content itself, but everything you need to know about it without touching S3 or the index: publication status, compliance review state, citation metadata, and which chunks belong to it. Single-digit-millisecond lookups by document ID make this the layer that powers "show me the source" when a citation is clicked.
{
"PK": "DOC#a1b2c3d4",
"SK": "META",
"source_id": "a1b2c3d4",
"title": "Q2 Semiconductor Supply Chain Outlook — Interview",
"source_type": "expert_transcript",
"author_or_expert": "expert_9231",
"published_at": "2026-04-11T00:00:00Z",
"domain": "semiconductors",
"content_type": "interview_transcript",
"compliance_status": "reviewed",
"compliance_reviewed_by": "content_ops_team",
"compliance_reviewed_at": "2026-04-12T00:00:00Z",
"chunk_ids": ["a1b2c3d4#0", "a1b2c3d4#1", "a1b2c3d4#2"],
"s3_raw_key": "raw/transcripts/2026/04/a1b2c3d4.json",
"language": "en",
"entitlement_tier": "institutional",
"GSI1PK": "DOMAIN#semiconductors",
"GSI1SK": "2026-04-11T00:00:00Z"
}domain handles the "all documents in this sector, most recent first" query without a join. A relational store would be the wrong tool unless the metadata needed ad-hoc analytical querying, which is what the S3/Athena export path is for instead.Ingestion Pipeline
The path from a new document landing to it being retrievable — orchestrated, not scripted.
3.1 — Step Functions state machine
S3 PUT → EventBridge rule
New object in raw/ triggers the state machine — no polling, no cron for the hot path
Parse & normalize (Lambda)
Extract clean text, speaker turns (for transcripts), structural metadata; write to processed/
Chunk (Lambda)
Semantic chunking per the strategy chosen in the design playbook — this stage is pluggable and re-runnable
Embed (Bedrock Titan / Cohere)
Batched embedding calls, retried with exponential backoff on throttling
Index (OpenSearch bulk API)
Chunks written with both the vector and the raw text for BM25, in one document
Register (DynamoDB PutItem)
Document metadata item created/updated; chunk_ids populated last, marking the document "live"
3.2 — Batch vs. streaming, and why you need both
Streaming path (per-document)
EventBridge → Step Functions, as above
Handles new documents as they arrive — minutes to searchable. This is the only path for anything time-sensitive.
Batch path (reprocessing)
EventBridge Scheduler → Step Functions Map state over S3 prefix
Re-embeds the entire corpus when you swap embedding models, or backfills a metadata field added after launch. Runs as a fan-out Map state, throttled to stay under Bedrock and OpenSearch rate limits.
The Index Layer
Where hybrid search actually lives — one engine holding both a lexical and a vector index per document.
OpenSearch Service is the workhorse here: it has both a mature BM25 implementation (it's built on Lucene) and native knn_vector field support via HNSW. That means one query can run lexical and vector search in parallel and merge results server-side — no separate vector database to keep in sync with a separate search engine.
{
"mappings": {
"properties": {
"chunk_id": { "type": "keyword" },
"source_id": { "type": "keyword" },
"text": { "type": "text", "analyzer": "english" },
"embedding": {
"type": "knn_vector",
"dimension": 1024,
"method": {
"name": "hnsw",
"engine": "faiss",
"space_type": "cosinesimil",
"parameters": { "ef_construction": 256, "m": 16 }
}
},
"domain": { "type": "keyword" },
"content_type": { "type": "keyword" },
"published_at": { "type": "date" },
"entitlement_tier": { "type": "keyword" },
"compliance_status": { "type": "keyword" }
}
}
}| Alternative | When it fits instead |
|---|---|
| Amazon Kendra | Fastest path to a working semantic search UI with zero index tuning — trades control over ranking and hybrid weighting for managed simplicity |
| Aurora PostgreSQL + pgvector | Team already runs relational workloads on Aurora and wants one engine for transactional + vector data; BM25 needs a separate full-text index (tsvector) rather than being native |
| Pinecone / Weaviate (non-AWS) | Vector-search-only requirement with no lexical/BM25 need, or a genuinely multi-cloud mandate |
ef_construction/m for a small recall hit, before reaching for bigger instances.Models — Bedrock as the Single API
Embeddings, reranking, and generation are three different calls to three different model families, behind one managed endpoint.
| Function | Bedrock model | Called from |
|---|---|---|
| Embeddings (ingest + query) | Titan Text Embeddings v2 (or Cohere Embed via Bedrock for multilingual) | Ingestion Lambda (batch) + retrieval service (per-query, cached) |
| Cross-encoder rerank | Cohere Rerank via Bedrock | Retrieval service, top-50 → top-6 after coarse retrieval |
| Generation | Claude (Anthropic) via Bedrock | MCP server response synthesis, or handed raw to the calling agent |
The Serving Layer — Building the MCP Server
This is the layer that turns a private retrieval pipeline into something an external agent — Claude, Perplexity, an internal Copilot — can call directly, with citations intact.
MCP (Model Context Protocol) standardizes how an AI agent discovers and calls tools exposed by a server. Instead of building a bespoke chat widget, you expose a small, typed tool surface — and any MCP-compatible client can connect to it. The pattern that matters here, and the one behind connectors that expose a proprietary corpus inside a general-purpose AI research tool: the MCP server is a thin, stateless layer in front of the retrieval service, not a reimplementation of it.
Runtime
API Gateway (or ALB) in front of Fargate/Lambda running an MCP server SDK. Fargate if connections are long-lived (streaming responses); Lambda if each tool call is a discrete, short request.
Auth
OAuth 2.1 per the MCP spec, mapped to an entitlement tier (e.g. institutional vs. individual) that gets pushed down as an OpenSearch filter on every query — not a UI-layer restriction.
6.1 — Tool definitions
{
"name": "search_documents",
"description": "Search the expert transcript library for passages relevant to a research question. Returns cited excerpts, not full documents.",
"inputSchema": {
"type": "object",
"properties": {
"query": { "type": "string", "description": "Natural language research question" },
"domain": { "type": "string", "description": "Optional sector filter, e.g. 'semiconductors'" },
"date_from": { "type": "string", "format": "date" },
"max_results": { "type": "integer", "default": 6, "maximum": 15 }
},
"required": ["query"]
}
}{
"results": [
{
"excerpt": "...foundry lead times on advanced nodes have stretched to 9-12 months, driven primarily by...",
"source_id": "a1b2c3d4",
"source_title": "Q2 Semiconductor Supply Chain Outlook — Interview",
"expert_role": "Former VP Supply Chain, Tier-1 Foundry",
"published_at": "2026-04-11",
"citation_url": "https://platform.example.com/transcripts/a1b2c3d4#t=412",
"relevance_score": 0.87,
"compliance_status": "reviewed"
}
],
"trace_id": "req_9f31..."
}6.2 — Request flow through the serving layer
Agent calls search_documents
MCP client (Perplexity, Claude, internal tool) sends the tool call over the MCP transport
Auth + entitlement resolution
OAuth token validated; entitlement tier resolved and attached as a hard filter for this request
Retrieval service call
Hybrid search against OpenSearch (5-stage pipeline from the design playbook), scoped by the entitlement filter
Citation assembly
Each chunk hydrated with source metadata from DynamoDB — title, expert, date, compliance status
Response + audit write
Typed MCP response returned; request/response pair written to the audit trail with a trace_id
Governance, Entitlement & Compliance
The layer that makes a licensed, proprietary corpus safe to expose to a third-party agent at all.
Entitlement Filtering
Every query carries the caller's entitlement tier as a mandatory OpenSearch filter clause, not an app-layer check that can be bypassed by calling the index directly.
Compliance Status Gate
compliance_status must equal 'reviewed' for a chunk to be retrievable at all. Content pending review is indexed but excluded from every query until a reviewer flips the flag.
Immutable Audit Trail
DynamoDB Streams on the documents table + every MCP request/response feed an append-only S3 log via Kinesis Firehose — who asked what, when, and which sources were returned.
Observability Across the Hop
A single user question now spans an MCP call, a retrieval service call, an OpenSearch query, and a Bedrock call. Tracing has to follow it across all four.
| Signal | Service | What it catches |
|---|---|---|
| Distributed trace | AWS X-Ray | One trace_id spans API Gateway → MCP Lambda/Fargate → OpenSearch → Bedrock, so a slow response is attributable to a specific hop, not a black box |
| Latency percentiles | CloudWatch (custom metrics per stage) | P50/P95/P99 broken out per pipeline stage — retrieval vs. rerank vs. generation each get their own SLO |
| Retrieval quality drift | CloudWatch alarm on a scheduled eval Lambda | Recall@5 / faithfulness on a fixed golden set, run nightly — catches silent degradation from an embedding model swap or index remapping |
| Cost attribution | Cost Explorer tags + Bedrock invocation logs | Per-tenant / per-caller token spend, critical once external agents (not just internal users) are calling the MCP server |
The Same Stack on Azure / Databricks
Every layer above maps cleanly onto an Azure-native or Databricks lakehouse stack. The architecture doesn't change — the managed service under each box does.
| Layer | AWS (this post) | Azure / Databricks equivalent |
|---|---|---|
| Raw document storage | S3 | Azure Blob Storage / ADLS Gen2 |
| Document metadata registry | DynamoDB | Azure Cosmos DB (same single-table, partition-key design pattern) |
| Ingestion orchestration | Step Functions + EventBridge + Lambda | Databricks Workflows / Azure Durable Functions, triggered by Event Grid |
| Hybrid index (BM25 + vector) | OpenSearch Service | Azure AI Search (native hybrid search + semantic ranker) or Databricks Vector Search over a Delta table |
| Embeddings / rerank / generation | Bedrock | Azure AI Foundry model endpoints, or MLflow-served models on Databricks |
| MCP serving layer | API Gateway + Fargate/Lambda | Azure API Management + Azure Functions / Container Apps |
| Governance & audit | IAM + DynamoDB Streams + CloudTrail | Microsoft Entra ID + Unity Catalog (Databricks) for lineage and access control + Azure Monitor |
| Observability | CloudWatch + X-Ray | Azure Monitor + Application Insights distributed tracing |
What We Covered
- S3 for immutable raw content, DynamoDB for the queryable document registry — two stores, two jobs
- Step Functions orchestrating a six-stage ingestion pipeline: parse → chunk → embed → index → register
- OpenSearch Service as one hybrid engine for BM25 + HNSW vector search, no separate vector DB to sync
- Bedrock as a single governed API for embeddings, reranking, and generation
- An MCP server as a thin, stateless serving layer — typed tools, cited excerpts, entitlement-scoped queries
- Compliance status and entitlement enforced at the index/query layer, not the UI
- X-Ray tracing a single request across API Gateway, retrieval, OpenSearch, and Bedrock
- A clean AWS ↔ Azure/Databricks mapping — the architecture travels, the vendor underneath doesn't have to
You might also like
Designing a GenAI News Discovery System: Every Decision, Every Tradeoff
The decisions this post's infrastructure serves — RAG vs. fine-tuning, chunking strategy, hybrid retrieval, editorial guardrails, and evaluation. Includes a live interactive demo.
Enterprise Agentic AI: From Chatbot to Autonomous Agent
What changes when your AI stops answering and starts acting. LangGraph, MCP, human-in-the-loop gates, and a credit analyst agent as a reference architecture.
Enterprise GenAI Strategy: From Data to Production
A complete RAG rollout program for enterprise teams — data readiness, RBAC, evaluation frameworks, and organisational change management.
Shashank Padala
Founder, Kirak Labs · AI Product Leader
AI Product & Transformation Leader with 8+ years building production LLM systems. Previously led GenAI integration into an internal content-authoring platform at a Fortune 500 enterprise, serving millions of employees globally — an AI assistant embedded in the CMS that surfaced grounded, cited insight from engagement and support-ticket data to inform what the team published next.