Chapter 5
LLM Engineering · Prompt engineering that works
Why this matters
Two prompts that look 90% the same can have 30% accuracy difference. There's a lot of nonsense online about "prompt magic" — most of it is cargo culting. The real practices are mundane: be explicit, give examples, structure the output, and test.
The message anatomy
Every modern chat API uses three roles:
- system: who the model is, what task it's doing, constraints. Set once at the top.
- user: the actual input/question for this turn.
- assistant: prior assistant responses (for multi-turn) — or you can prefill the start of the response to bias output (Claude supports this).
The six patterns that actually help
| Pattern | What it is | When to use |
|---|---|---|
| Be specific about format | "Respond as a JSON object with keys X, Y, Z" | Always for downstream parsing |
| Few-shot examples | 2-5 input→output pairs in the prompt | When zero-shot is borderline; massive accuracy jump |
| Chain of thought | "Let's think step by step" or explicit reasoning steps | Math, multi-step reasoning, complex extraction |
| Role priming | "You are a senior compiler engineer" | Mostly placebo for SOTA models; modest help on smaller ones |
| Negative constraints | "Do not return prose; only the JSON" | When the model loves to be helpful and won't shut up |
| Decompose | Split into 2-3 LLM calls, each with one job | When one prompt tries to do too much and fails on edge cases |
Few-shot — the cheapest accuracy gain
If you only do one thing: add 3-5 examples to your prompt. Pick examples that look like the inputs your real users send. This single change often moves accuracy more than switching from GPT-4o-mini to GPT-4o.
SYSTEM: Classify sentiment as positive / negative / neutral. Respond with one word only.
USER: This phone's battery dies in 4 hours. Garbage.
ASSISTANT: negative
USER: It's a phone. Does the things phones do.
ASSISTANT: neutral
USER: Camera is unbelievable for this price.
ASSISTANT: positive
USER: {actual_input}
Common anti-patterns
- "You are an expert..." with no follow-up specifics. The model is already expert-level. The system prompt should describe the task, not flatter the model.
- Asking for both reasoning AND structured output. Choose. If you need both, do two calls or use a "thinking" field in your schema that the parser ignores.
- Putting variables in the system prompt. System prompts should be cacheable. User-specific data belongs in the user message — otherwise you defeat prompt caching and pay 10× per call.
- "Important: do not hallucinate". Hallucinations come from the model not having grounding, not from forgetting to behave. Give it the data; the magic words don't work.
Q: When should I use chain-of-thought vs structured output?
CoT improves accuracy on multi-step problems (math, multi-hop reasoning). Structured output makes parsing reliable. They fight each other — strict JSON mode disables the model's reasoning. Solution: ask for a
"reasoning" string field in the JSON schema (model writes its thinking there, you ignore it at parse time) or do two calls: one for reasoning, one to format the answer.Cross-Q: Doesn't the reasoning field cost tokens? A: Yes, ~2-5× the output tokens. Worth it for hard tasks; overkill for trivial ones.
Takeaway
The best prompt is the one with examples close to your real data and an output format your code can parse without regex. Everything else is theatre.