Chapter 3

Foundations · Transformers from first principles

📖 2 min read · 5 sections · May 2026

Why this matters

Every LLM you'll touch in 2026 is a transformer or a transformer descendant. You don't need to implement one, but you should know what's happening inside the box you're calling — because every "weird behaviour" in production has its roots in the architecture.

The core idea: attention over tokens

Old RNN/LSTM models processed text one token at a time, threading state through a hidden vector. Transformers throw that out: every token can directly look at every other token in the context, weighing them by relevance. That's attention.

flowchart LR
    T1[Token 1] --> A((Attention))
    T2[Token 2] --> A
    T3[Token 3] --> A
    Tn[Token n] --> A
    A --> O1[New repr 1]
    A --> O2[New repr 2]
    A --> O3[New repr 3]
    A --> On[New repr n]

Each output token is a weighted blend of every input token. The weights are learned. That's it — there's a feed-forward MLP between attention layers, and you stack many layers (GPT-3.5 has 96 layers; GPT-4 ~120; Llama 3.1 70B has 80), but conceptually that's the architecture.

Why this matters operationally

Context window cost is quadratic in attention. Doubling context length quadruples compute for attention. That's why "long context" models cost more per token and get slow.
Position is information. Transformers have no inherent notion of order — positional encodings inject it. This is why "needle in a haystack" tests reveal model weaknesses at certain positions.
Causal mask in decoder LLMs means each token only attends to previous tokens. That's why generation is left-to-right and you can't "ask the model to fill in the middle" without a different model type (e.g., FIM-trained code models).

Decoder vs encoder vs both

Type	Examples	What it's good at
Decoder-only	GPT, Claude, Llama, Mistral	Generation. Everything you call "an LLM" is this.
Encoder-only	BERT, RoBERTa, all embedding models	Producing fixed-size representations (embeddings, classifications).
Encoder-decoder	T5, BART	Translation, summarisation. Used less in 2026 — decoder-only LLMs do these well.

Q: Why can't I get GPT-4 to fill in the middle of a paragraph?

Decoder-only causal models attend only to previous tokens, so they can't condition the middle on text that appears after it. You'd need a Fill-In-the-Middle (FIM) trained variant. Most code completion models (Codex, Copilot's older models) were FIM-trained for this exact reason.

Takeaway

You don't need to memorise multi-head attention math. You need to know: generation is left-to-right, costs scale quadratically with context, position matters, and the model has no memory between calls. Those four facts explain 90% of "why is this LLM behaving weirdly?" questions.

Further reading: Attention is All You Need · The Illustrated Transformer · Karpathy — Let's build GPT