Chapter 9
RAG Systems · Vector databases — choosing and operating
Why this matters
Storing and querying millions of embeddings cheaply, with millisecond latency, with metadata filters, with frequent updates — every vector DB makes a different set of tradeoffs. Picking poorly locks you into the wrong cost curve for years.
The landscape
| DB | Hosted | OSS | Strengths | Caveats |
|---|---|---|---|---|
| Pinecone | ✓ | ✗ | Zero ops, best-in-class index management, multi-tenant native | Pricey at scale; vendor lock |
| Weaviate | ✓ | ✓ | Hybrid search (BM25 + dense) built in | Resource hungry on self-host |
| Qdrant | ✓ | ✓ | Fast, Rust, excellent filtering, payload indexing | Newer, smaller ecosystem |
| Chroma | ✓ | ✓ | Best DX for prototypes; Python-native | Not battle-tested above ~10M vectors |
| pgvector (Postgres) | via RDS/Supabase/Neon | ✓ | Reuses existing Postgres; ACID + joins + vectors in one query | Slower than dedicated DBs above ~5M vectors; harder to scale writes |
| Vespa / OpenSearch | ✓ | ✓ | Massive scale; hybrid search + ranking; ML serving | Operationally heavy; overkill for <100M vectors |
Indexing algorithms — what you should know
- HNSW: hierarchical graph index. Default in most DBs. Tradeoff knobs:
M(graph degree, more = recall up, memory up),efConstruction(build quality),efSearch(query recall vs latency). Good for <100M vectors. - IVF (+ PQ): clusters vectors, searches relevant clusters. Used by FAISS at scale; memory-cheap, slightly lower recall.
- Flat (brute force): exact, slow. Fine up to ~100K vectors or for ground-truth evaluation.
Metadata filtering is where pain lives
"Find similar docs, but only those by user_id=X, dated after Y, in language Z." Filters interact poorly with ANN indexes — they can drop recall to zero if applied naively (the candidates from the index don't match the filter, so you get nothing back).
Three patterns DBs use:
- Pre-filter: restrict candidates first, then ANN search the subset. Good for selective filters; bad if the filter is loose (you index a lot).
- Post-filter: ANN search, then filter results. Fast but loses recall when the filter is selective.
- Filtered HNSW / Qdrant payload indexing: filter during graph traversal. Best of both — Qdrant and Weaviate are particularly good at this.
War story: team built RAG on Pinecone with
user_id filtering. Most users had ~50 docs each. ANN with post-filter regularly returned 0 results because the top-50 ANN candidates were from other users. Switched to Qdrant with payload index on user_id — same workload, recall jumped from 30% to 95%.
Operational concerns
- Write throughput. Inserting 10M vectors? Some DBs choke during HNSW build. Test before you commit.
- Updates and deletes. HNSW doesn't love deletes (marks tombstones; rebuild required eventually).
- Replication and durability. Hosted offerings handle this; self-host means you handle it.
- Hybrid search (BM25 + dense). If your data has acronyms, product SKUs, exact-match queries — you want hybrid. Weaviate and OpenSearch ship this out of the box.