Production · Fine-tuning when prompting hits a wall

Why this matters

Most teams reach for fine-tuning too early — it's the shiny option. Most don't need it. Knowing when fine-tuning genuinely beats prompting + RAG is a senior-level call.

The decision tree

flowchart TB
    P[Is the task working with prompting?] -->|yes| K[Keep prompting
don't fine-tune]
    P -->|no| R[Have you tried RAG
+ few-shot?]
    R -->|no| RR[Try those first]
    R -->|yes| D[Do you have 500+
high-quality examples?]
    D -->|no| C[Curate data first
fine-tuning won't fix this]
    D -->|yes| S[Is the task pattern that
even GPT-4 / Claude fails at?]
    S -->|no| Q[Re-examine prompts]
    S -->|yes| F[Fine-tune]

When fine-tuning actually helps

• Specific output style not achievable with examples in the prompt (e.g., always respond in a constrained DSL).
• Domain language the base model doesn't grok (medical, legal acronyms; internal product taxonomy).
• Distilling a large model into a small one — generate 10K examples from GPT-4o, fine-tune Llama 8B, run cheap inference at scale.
• Latency-critical use cases — a fine-tuned 8B model beats a 70B model on speed for the specific task.

LoRA vs QLoRA vs full fine-tune

Dataset curation is the hard part

People fixate on the training script. The real work is the data: 500-2000 examples of (input, ideal output) pairs that represent the target behaviour. Mediocre data + fine-tuning is worse than no fine-tuning — the model learns your inconsistencies.

• Diverse inputs covering edge cases
• Consistent output style — if your reviewers disagree on what "good" looks like, the model will too
• Include negative examples (what NOT to do) only via DPO, not naive SFT
• Reserve 10-20% as held-out eval; never train on it

Eval before AND after

Run your golden eval set on the base model. Fine-tune. Run again. If the fine-tune doesn't beat the base by at least 10% on your eval, the cost (operational + ongoing) isn't worth it — go back to better prompts.