RAG Systems · Embeddings and semantic search
Why this matters
Embeddings are the workhorse of every RAG system, every "find similar" feature, every clustering / classification pipeline. Picking the wrong model or the wrong dimension wastes years of latency budget on every query.
What is an embedding, mechanically?
A neural network reads a chunk of text and outputs a fixed-length vector (e.g., 1536 floats). Models are trained so that semantically similar texts produce similar vectors (high cosine similarity). The model that produces embeddings is different from the model that produces chat completions — usually smaller, encoder-only, and cheap.
Choosing an embedding model in 2026
| Model | Dim | $/1M tokens | Notes |
|---|---|---|---|
| OpenAI text-embedding-3-small | 1536 (truncatable) | $0.02 | Default for most apps; cheap, good |
| OpenAI text-embedding-3-large | 3072 (truncatable) | $0.13 | Best for nuanced retrieval; truncate to 1024 for storage savings |
| Cohere embed-english-v3 | 1024 | $0.10 | Strong on retrieval-tuned tasks; multilingual variant available |
| Voyage AI voyage-3 | 1024 | $0.06 | Often top of MTEB leaderboard for retrieval |
| BGE-M3 (open source) | 1024 | self-host | Free; competitive with closed; great for multilingual + long-context |
Always normalise (when you can)
If your vectors are unit-length, cosine similarity = dot product. Dot product indexes are faster and more widely supported in vector DBs. Most providers return normalised vectors; if you're working with raw outputs, normalise before storing.
import numpy as np
def normalise(v): return v / np.linalg.norm(v)
Distance metrics — when each one matters
- Cosine — semantic similarity (most common).
- Dot product — equivalent to cosine if normalised; faster.
- Euclidean (L2) — only correct if vectors aren't normalised AND magnitudes matter (rare for text).
Takeaway
Start with OpenAI text-embedding-3-small, truncate to 1024 dims, normalise. Benchmark something better only after you have an eval set. Most teams over-optimise the embedding model and under-optimise chunking strategy (chapter 10), which matters far more.