Chapter 22
Career · The AI engineering interview
Why this matters
You can know all 21 prior chapters cold and still flunk the interview. The interview rewards a specific pattern of communication — structured, confidence-calibrated, tradeoff-aware. This chapter is the meta-skill.
The typical loop (2026)
| Stage | What they assess | How long |
|---|---|---|
| Resume screen | Past production AI work, depth signals | passive |
| Recruiter call | Comp, fit, role expectations | 30 min |
| Technical screen | Foundations: tokenisation, embeddings, RAG basics, transformer mechanics | 1 hour |
| Coding | Build something with an LLM — RAG, agent, structured extraction | 1-2 hours |
| System design | Design a real LLM product end-to-end | 1 hour |
| Behavioural / leadership (senior) | Project ownership, cross-team work, ambiguity handling | 1 hour |
Topics by level — what's expected
| Level | Bar |
|---|---|
| L4 (mid) | Strong at prompts, structured output, basic RAG, can ship one feature |
| L5 (senior) | Owns RAG end-to-end: chunking, eval, observability. Picks the right vector DB, defends the choice. Knows when not to use agents. |
| L6 (staff) | Architects production AI systems: cost, safety, multi-agent orchestration, evaluation strategy, organisational governance. |
The system-design interview pattern
Mirrors the HLD system-design method:
- Clarify (3 min): scale, latency, accuracy target, budget, what's in/out of scope.
- Estimate (3 min): QPS, tokens/request, cost/month, vector store size.
- Sketch (10 min): blocks — entry point, retrieval, LLM, output filter, observability. Name the tools you'd use.
- Deepen (15-20 min): pick two boxes the interviewer probes; show real tradeoffs. The interviewer will steer here.
- Defend under cross-examination: "what if X breaks?", "what does this cost at 10× scale?", "how do you evaluate?".
Common questions with real answers
Q: How do you evaluate a RAG system?
Three layers: (1) Golden set — 100-500 hand-curated Q&A pairs covering easy/medium/hard/edge. (2) RAGAS metrics on every deploy — faithfulness, answer relevance, context precision/recall. (3) Online sampling — 1-2% of production traffic gets LLM-judge eval; track rolling-window scores; alert on 2σ drift. The golden set runs in CI; RAGAS and online go to dashboards.
Q: Hybrid retrieval — explain when it beats dense alone.
Hybrid (BM25 + dense) wins when the data contains exact-match tokens that semantic embeddings underweight: SKU codes, error codes, function names, acronyms, named entities. BM25 catches those; dense catches paraphrased semantic similarity. Combine with reciprocal rank fusion. Pure dense is fine when your data is prose and queries are conceptual.
Q: A user reports your bot is confidently wrong. Walk me through diagnosing.
(1) Pull the trace — what did retrieval return? (2) Were the right chunks in the top-5? If not, retrieval bug — check chunking or query rewriting. (3) If retrieval was right but model still wrong, check the prompt: was "answer only from context" present? Were chunks clearly delimited? (4) If prompt was right but model still wrong, try Sonnet/GPT-4o instead of the cheaper tier — sometimes the model just isn't capable enough. (5) Add this case to the golden eval set so the next regression catches it.
Behavioural questions for senior roles
- "Tell me about an AI system you shipped and one thing you'd do differently." — Specific. Numbers. What broke. What you learnt.
- "How would you build the eval strategy for a brand new AI feature?" — Talk about building the golden set first, RAGAS for ongoing, online sampling for drift, human review cadence.
- "How do you decide when to fine-tune?" — Decision tree from chapter 19. Most interviewers want to hear "I'd try prompting + RAG first" before they hear "yes fine-tune".
Red flags interviewers watch for
- Vague answers — "I'd use LangChain" without saying what specifically.
- Reaching for the most complex option — multi-agent for a single-shot task, fine-tuning before prompting.
- Ignoring cost / latency tradeoffs.
- No eval story.
- No production incident story.
- Can't articulate why their previous architecture choice was right (or wrong in retrospect).