Chapter 6

LLM Engineering · Structured output and tool use

📖 2 min read · 5 sections · May 2026

Why this matters

If your application parses LLM output with regex, you've already lost. Modern LLMs let you specify a JSON schema (OpenAI, Gemini) or pass an XML/tool-call structure (Claude) and guarantee the output validates. Use it.

Three mechanisms

Mechanism	How it works	Best for
JSON mode / response_format	Provider constrains generation to valid JSON	Extracting structured data from text
Function / tool calling	You define functions; LLM returns which one to call with what args	Agents, RAG with tools, action selection
Constrained decoding (open-source)	Library forces token choices to match a grammar	Self-hosted models; libraries like Outlines, jsonformer

Function calling — the production pattern

Define a function (with a clear name, description, and parameter schema). Pass it to the LLM. The LLM decides whether to call it and with what arguments. Your code runs the function. You send the result back. Repeat.

sequenceDiagram
    participant User
    participant App
    participant LLM
    participant Tool as Weather API
    User->>App: What's the weather in Bangalore?
    App->>LLM: messages + tool definitions
    LLM-->>App: tool_call(get_weather, {city: "Bangalore"})
    App->>Tool: GET /weather?city=Bangalore
    Tool-->>App: { temp: 28, conditions: "humid" }
    App->>LLM: tool_result { temp: 28, conditions: "humid" }
    LLM-->>App: "It's 28°C and humid in Bangalore."
    App-->>User: "It's 28°C and humid in Bangalore."

Tool design rules

Names like a senior engineer: search_jobs, not doSearch. The model uses the name as a strong signal for when to call.
Descriptions that read like an internal doc: "Use this when the user asks about open roles. Returns up to 20 results. Does not include archived jobs." The model reads these and chooses accordingly.
Strict parameter schemas: required vs optional, enums where you can, descriptions on every field.
Return errors as data: don't throw. Return {"error": "no results", "hint": "try a broader query"}. The model can recover; thrown exceptions kill the loop.
Idempotent where possible: the model may retry; you don't want double-bookings.

War story: an early agent kept calling delete_user by mistake instead of archive_user because both descriptions said "remove the user". Renamed to permanently_delete_user_irreversible with a warning in the description. Zero mistaken calls after that. Names matter to LLMs more than to humans.

Takeaway

Treat your tool schema as a public API contract. Document it like one. The LLM is the client, and clients with bad docs misuse APIs.