Cut the AI Chatbot Tax
Before It Cuts You
Every engineering leader shipped the enterprise AI assistant. Now the CFO wants the cloud bill explained. This is a companion to the RAG + MCP tech stack post — that post flagged OpenSearch instance-hours as the largest recurring cost once you cross a few million chunks. This one picks up exactly there: a worked case study on where that memory actually goes, the two levers every team reaches for first, and the object-storage-first architecture that cuts the bill by up to 90% for the right workloads.
The Cost Nobody Runs the Math On
Every company that shipped a RAG chatbot in the last two years hit the same unbudgeted line item. Turn your docs, your Slack history, and your codebase into embeddings, and your data footprint doesn't grow a little — it explodes. A rough rule of thumb: 1KB of source text becomes roughly 16KB once it's chunked and embedded (the chunk text itself, plus a 1024-dimension float vector per chunk, plus metadata). Vectorize your whole knowledge base and you've 16x'd what has to be stored — and, in most vector databases, stored hot.
That distinction — hot vs. cold — is the entire story of this post. A team running a $3,000/month Postgres bill can watch it become a $30,000/month bill the moment the same data is fully vectorized into a conventional vector database, because every one of those databases was built on the same unquestioned assumption: the whole inflated footprint has to sit in RAM or on premium SSD, always on, always fast, always expensive — whether anyone is querying it or not.
Case Study: Where the Memory Actually Goes
Back-of-envelope math on a mid-size corpus, using the same OpenSearch index mapping from the companion stack post.
Say a company has 500,000 source documents — support tickets, wiki pages, transcripts — averaging 3 chunks each after semantic chunking. That's 1.5M chunks. Each one carries a 1024-dimension embedding (the same dimension used in the OpenSearch mapping from The Technology Stack Behind Production RAG + MCP), stored as 32-bit floats.
Before the math: almost every mainstream vector database — including OpenSearch — finds those embeddings using HNSW (Hierarchical Navigable Small World), an algorithm that connects every vector to a small set of its nearest neighbors, forming a searchable graph. A query walks that graph hopping toward closer and closer neighbors instead of comparing against every vector in the corpus — which is what makes vector search fast at scale. The tradeoff is the one this whole post is about: that graph only works if it stays resident in memory, so every vector you add doesn't just cost you its own storage, it costs you a permanent slot in RAM.
1.5M chunks × 1024 dimensions × 4 bytes/float
= 1.5M × 4,096 bytes
= ~6.1 GB of raw vector data
HNSW graph overhead (neighbor links, per-vector, at m=16)
adds roughly another 1.0–1.5x on top of raw vectors
→ ~9–14 GB resident in memory for the vector index alone
Add replicas for HA (typically 2–3x for a production cluster)
→ ~20–40 GB of RAM has to be provisioned,
just to hold the vector field — before the text index,
OS overhead, or headroom for growthNone of those numbers move based on how often the corpus is queried. A cluster serving ten queries a day and a cluster serving ten thousand pay the identical memory bill, because HNSW keeps the entire neighbor graph resident to guarantee fast lookups — as covered in the index layer chapter of the companion post. You are paying for the data existing, not for the data being used.
The Two Levers Everyone Pulls First
Both assume the premise is correct: that the whole corpus has to stay resident in memory. They just negotiate the price of that premise.
The first lever is two HNSW knobs on the index mapping: m controls how many neighbor connections each vector keeps in the graph (higher = better recall, more memory, slower to build); ef_construction controls how hard the graph search looks for those neighbors while building the index (higher = better graph quality, slower indexing, more memory). Both are quiet, no-recall-hit-you'd-notice ways to shrink memory per vector — which is exactly why they're usually the first thing tried.
| Lever | What it trades away | Ceiling |
|---|---|---|
| Reduce HNSW m / ef_construction | Fewer neighbor connections per vector and a less thorough graph build → a small, usually invisible drop in recall (occasionally missing a marginally relevant chunk) | Shrinks memory per vector, but the corpus still has to be fully resident — this delays the next instance upgrade, it doesn't remove the need for one |
| Provision bigger / more instances | Nothing algorithmic — pure cost. A permanent, recurring increase, not a one-time expense | Scales linearly with corpus size forever; there's no point at which this gets cheaper on its own |
The Question Nobody Asked: Does It All Need to Be Hot?
Most vectors in a RAG corpus are cold at any given moment. turbopuffer built its entire architecture around that one observation.
A RAG query distribution isn't uniform. A handful of recent documents, popular topics, or actively-open codebases get hit constantly; the long tail of a 5M-chunk corpus might go days without a single query touching it. Every mainstream vector database — OpenSearch, Pinecone, Weaviate, pgvector — ignores that distribution and keeps 100% of the corpus resident in RAM or premium SSD regardless. turbopuffer's architecture is what happens when two infrastructure engineers refuse to accept that as a given.
Instead of RAM as the default home for vectors, object storage (S3, GCS, Azure Blob) is the default — at roughly $0.02/GB versus the ~$1.40/GB effective cost of a managed vector database keeping everything resident. That's a ~70x storage cost gap per gigabyte. Data only gets promoted into faster tiers when something actually asks for it:
The index structure itself is different too — not a memory-resident HNSW graph, but clustered indexes: vectors grouped into semantic clusters with small centroid summaries kept cheap to scan. A query reads the centroids first to identify a handful of relevant clusters, then pulls only those clusters' full vectors from storage — the opposite of HNSW's assumption that the whole graph must already be in memory to be useful.
Why 2020 and 2023 Matter
This architecture wasn't possible five years ago. It needed two specific S3 features to ship first.
You can't build a database on a storage layer that might hand you stale or conflicting data. For most of its history, S3 couldn't make either guarantee strongly enough — two changes fixed that, and both are load-bearing for treating S3 as a database backend rather than just a file dump.
4.1 — Strong read-after-write consistency (Dec 2020)
Before this, S3 was only eventually consistent for overwrites and deletes — write a new version of an object, and a read immediately after could still return the old version, or briefly 404. For a static website, that's a non-issue. For a database index, it's disqualifying.
4.2 — Conditional writes / compare-and-swap (Nov 2023)
Before this, S3 had no atomic "update this object only if it hasn't changed since I last read it" primitive. Two concurrent writers to the same object raced under last-write-wins semantics — whichever write landed second silently clobbered the first, with no error, no conflict signal, nothing to retry.
If-Match / If-None-Match headers), the second writer's request is rejected because the object changed underneath it — it detects the conflict and retries, instead of destroying the first writer's work.Same Corpus, Two Architectures
Applying the published per-GB numbers to the Chapter 01 case study — directional, not a vendor quote.
| Architecture | What's always paid for | Cost driver at ~10–40GB resident |
|---|---|---|
| Always-hot (OpenSearch / managed vector DB) | 100% of the corpus, in RAM/SSD, continuously, regardless of query volume | ~$1.40/GB effective → the full resident footprint priced every month, whether or not it's queried |
| Object-storage-first + tiered cache (turbopuffer-style) | 100% of the corpus in object storage (cold); only the actively-queried working set promoted to SSD/RAM | ~$0.02/GB baseline storage, plus a much smaller hot-tier footprint sized to actual query traffic, not corpus size |
The ~70x per-gigabyte gap doesn't translate into a flat 70x bill reduction — some working set still has to live in the fast tiers, and cold-query latency is a real cost paid in user experience rather than dollars (more on that next). But it explains why teams making this switch report savings in the 10–100x range rather than a marginal 20–30% optimization: the two architectures are pricing fundamentally different things — total corpus size versus actual query traffic.
How the Gap Widens as the Corpus Grows
Same math as Chapter 01, extended across corpus sizes. Costs below are storage-only, order-of-magnitude estimates from the published per-GB figures — not a vendor quote, and real cluster bills also include compute nodes, redundancy, and query capacity on top of this.
| Documents | Chunks (~3/doc) | Resident memory (hot, w/ HNSW + replicas) | Illustrative hot storage cost/mo | Illustrative cold storage cost/mo |
|---|---|---|---|---|
| 500K | 1.5M | ~20–40GB | ~$28–56 | ~$0.12 |
| 2M | 6M | ~80–160GB | ~$112–224 | ~$0.49 |
| 5M | 15M | ~200–400GB | ~$280–560 | ~$1.23 |
| 20M | 60M | ~0.8–1.6TB | ~$1,120–2,240 | ~$4.92 |
| 100M | 300M | ~4–8TB | ~$5,680–11,180 | ~$24.60 |
| 1B | 3B | ~40–80TB | ~$56,770–111,820 | ~$245.76 |
| 100B | 300B | ~4–8PB | ~$5.68M–11.18M | ~$24,576 |
Two things to notice. First, the cold-storage column barely moves — it's pricing raw bytes at $0.02/GB, so even 300B chunks of vector data lands under $25K/month, which is inexpensive at object-storage rates. Second, the hot column scales linearly with corpus size forever, because that architecture has no mechanism to distinguish a chunk queried every second from one that hasn't been touched in a year — both cost the same to keep resident. By the time a corpus reaches billions of chunks, the always-hot bill is in the tens or hundreds of thousands of dollars a month for storage alone, while the cold tier is still a rounding error.
The last three rows are deliberately extreme — no single enterprise corpus realistically reaches 100B chunks — but they make the shape of the divergence obvious: the always-hot column has no ceiling, it just keeps climbing with corpus size, while the cold column grows so slowly it's barely visible on the same chart. The object-storage-first total isn't just the cold-storage column, though — you still pay for whatever working set actually gets promoted to the SSD/RAM cache tiers. That number depends on traffic, not corpus size, which is exactly the point: it can stay small and flat even as the underlying corpus grows without bound.
At What Point Switching Makes Sense
Two conditions, both required — corpus size alone isn't the trigger.
Chapter 06's table makes it tempting to say "switch once you cross N chunks." That's necessary but not sufficient. The second condition is about how the corpus is queried, not just how big it is: object-storage-first only pays off when access is skewed — a small, shifting subset of the corpus gets most of the queries at any given time, and the rest sits idle. A corpus that's large but queried uniformly (every chunk equally likely to be hit at any moment) gets none of the caching benefit, because nothing stays cold long enough to matter.
| Use case | What makes access skewed | Fits object-storage-first? |
|---|---|---|
| Cursor — codebase semantic search | A developer works in one or two repos per session; the rest of their indexed repositories sit untouched between sessions | Yes — per-repo cache warms on open, reported ~95% cost reduction vs. keeping every repo permanently hot |
| Notion — cross-workspace document search | Each workspace has a long tail of pages nobody's opened in months, plus a small set of actively-edited or recently-viewed pages | Yes — recency and edit activity naturally concentrate queries on a small hot subset |
| Linear — issue tracker semantic search | Open/active issues get referenced constantly; resolved issues from a year ago are rarely pulled back up | Yes — issue status is itself a strong skew signal (open vs. closed) |
| Proprietary expert-network / research corpus | Which experts or topics are "hot" tracks active client engagements and current market questions; a multi-year transcript archive is mostly dormant at any moment | Yes — research demand is inherently topical and time-bound, not uniform across the archive |
| Real-time fraud/trading signal lookups | Every incoming transaction or tick needs a comparison against the full reference set, continuously, with no natural "recent vs. stale" split | No — access is close to uniform and latency-critical on every single call; this is the Chapter 08 counter-example |
When Not to Do This
Every 10–100x cost story has a matching latency story. Here's the honest version.
| Situation | Why object-storage-first is the wrong call |
|---|---|
| Hard, tight synchronous latency SLA on every query | Cold-tier reads run 200–500ms (published p90 ≈ 444ms); if every query must return in well under that, even occasionally, a memory-resident index is the safer default |
| Uniformly hot access pattern — the whole corpus is queried constantly | If nothing is ever cold, there's no cost saving to capture and no cache-warming benefit; you'd just be paying object-storage latency with none of its cost upside |
| Small corpus (well under the case study's ~1.5M chunks) | Below a certain size, the whole thing fits comfortably and cheaply in a single memory-resident instance anyway — this problem doesn't exist yet |
| Team has no tolerance for variable latency in the product experience | A cache-miss-shaped tail latency spike needs to be an acceptable UX tradeoff, not a surprise discovered in production |
What We Covered
- Why vectorizing a corpus inflates its footprint ~16x, and why that inflated data defaults to being stored hot
- A worked case study: 1.5M chunks at 1024 dimensions costs 20–40GB of always-resident RAM once HNSW overhead and replicas are counted
- The two levers every team reaches for first — tuning m/ef_construction and buying bigger instances — and why both accept the same flawed premise
- turbopuffer's object-storage-first architecture: cheap cold storage by default, clustered indexes instead of a memory-resident HNSW graph, and automatic tiered caching
- The two specific S3 features (strong consistency in 2020, compare-and-swap in 2023) that had to ship before this architecture was even possible
- A directional cost comparison between always-hot and object-storage-first architectures on the same corpus
- How that cost gap widens with corpus size — the hot column scales linearly forever, the cold column barely moves
- The real breakpoint for switching: corpus size and access-pattern skew both have to hold, illustrated against Cursor, Notion, Linear, and a proprietary research-corpus use case
- When this pattern is the wrong call — tight latency SLAs, uniformly hot access, or corpora too small to have this problem yet
You might also like
The Technology Stack Behind Production RAG + MCP
The companion post — concrete AWS services, data schemas, and the MCP serving layer this cost analysis assumes as its starting architecture.
Designing a GenAI News Discovery System: Every Decision, Every Tradeoff
The algorithmic decisions this post's infrastructure serves — chunking strategy, hybrid retrieval, editorial guardrails, and evaluation.
Enterprise GenAI Strategy: From Data to Production
A complete RAG rollout program for enterprise teams — data readiness, RBAC, evaluation frameworks, and organisational change management.
Shashank Padala
Founder, Kirak Labs · AI Product Leader
AI Product & Transformation Leader with 8+ years building production LLM systems. Previously led GenAI integration into an internal content-authoring platform at a Fortune 500 enterprise, serving millions of employees globally — an AI assistant embedded in the CMS that surfaced grounded, cited insight from engagement and support-ticket data to inform what the team published next.