Why structured output matters
Free-text LLM responses are hard to use programmatically. Structured output — JSON, tables, XML — lets you pipe the model's answer directly into your application without fragile string parsing.
Unstructured
"The product is called Widget Pro, it costs $49, and the rating is 4.2 out of 5 stars based on 312 reviews."
You must parse this with regex or another LLM call — fragile and expensive.
Structured
{
"name": "Widget Pro",
"price": 49,
"rating": 4.2,
"reviews": 312
}Direct json.loads() — no parsing needed.
Schema-in-prompt
The simplest approach: describe the output schema directly in your prompt. Include field names, types, and constraints.
Extract product information from the text below.
Return ONLY a JSON object with these exact fields:
- name: string
- price: number (USD, no currency symbol)
- rating: number (0.0–5.0)
- review_count: integer
Text: "Widget Pro sells for $49. Customers love it —
4.2 stars from 312 reviews."
JSON:Works without any API features. Reliable on GPT-4o and Claude 3+ at temperature 0. Less reliable on smaller models — add few-shot examples for those.
JSON mode & structured outputs (API)
Modern LLM APIs provide native structured output features that guarantee valid JSON — no prompt tricks needed.
OpenAI — JSON mode
response = client.chat.completions.create(
model="gpt-4o-mini",
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": "Return JSON only."},
{"role": "user", "content": "Extract: name, price from ..."}
]
)Guarantees valid JSON. You still need to tell the model what fields to include in your prompt.
OpenAI — Structured Outputs (schema enforcement)
from pydantic import BaseModel
class Product(BaseModel):
name: str
price: float
rating: float
review_count: int
response = client.beta.chat.completions.parse(
model="gpt-4o-mini",
messages=[...],
response_format=Product,
)
product = response.choices[0].message.parsed
print(product.name) # typed Python objectEnforces the exact schema — invalid fields are rejected at the API level. You get a typed Pydantic object back.
Adding validation with Pydantic
Even with JSON mode, you should validate the output before using it. Pydantic validators run after parsing and can enforce constraints the schema alone can't express.
from pydantic import BaseModel, field_validator
class Product(BaseModel):
name: str
price: float
rating: float
review_count: int
@field_validator("rating")
@classmethod
def valid_rating(cls, v):
if not 0.0 <= v <= 5.0:
raise ValueError("rating must be between 0 and 5")
return v
@field_validator("price")
@classmethod
def positive_price(cls, v):
if v < 0:
raise ValueError("price must be non-negative")
return vIf the model returns a rating of 5.7, Pydantic raises a ValidationError before the bad data reaches your application. Combine with a retry loop to ask the model to fix the output.
Other structured formats
Markdown tables
Return the comparison as a markdown table with columns:
Feature | Tool A | Tool BReliable for display. Harder to parse programmatically — use JSON for data processing.
Numbered lists with delimiters
Return exactly 5 ideas, one per line, in this format:
1. [TITLE] — [ONE SENTENCE DESCRIPTION]Easy to split by line. Good for generating lists you'll display directly.
XML tags for multi-part responses
Return the analysis in these XML tags:
<summary>...</summary>
<risks>...</risks>
<recommendation>...</recommendation>Useful when different sections need different treatment. Anthropic's Claude responds particularly well to XML tags.
Reliability tips
Always set temperature=0 when extracting structured data. Variance in sampling = variance in field names and values.
Say "Return ONLY the JSON. No explanation, no markdown code block." Models often wrap JSON in backticks — this tells them not to.
Provide a JSON example in your prompt for complex schemas. A concrete example beats abstract field descriptions every time.
Add fallback handling: json.loads() inside a try/except. If parsing fails, retry with a corrective message: "Your last response was not valid JSON. Try again."
Notes
JSON mode guarantees syntax, not semantics
OpenAI's JSON mode ensures the response parses as valid JSON. It does not guarantee your requested fields are present, that values match expected types, or that optional fields are not hallucinated. Always run Pydantic or a JSON Schema validator after parsing — never trust the raw parse alone.
Deeply nested schemas reduce output reliability
Schemas with 4+ levels of nesting produce significantly more hallucinated or missing fields than flat schemas. If possible, flatten your schema and make a second call to extract nested details from the first response. The reliability budget for structured output is lower than it appears from simple examples.
Streaming and JSON mode are incompatible on some providers
OpenAI's JSON mode does not support streaming responses. If you need both streaming and structured output, collect the full response before parsing, or implement an incremental JSON parser. Anthropic's API handles this differently — check the provider's current documentation before assuming compatibility.
Anthropic structured output uses tool definitions, not response_format
On Claude, force structured output by defining a tool with a JSON schema matching your desired output and setting tool_choice to the tool name. This is different from OpenAI's response_format surface but equally reliable. Do not copy OpenAI patterns directly to Anthropic calls.
Structured Output FAQ
How do I get an LLM to always return valid JSON?
Use three approaches in combination: set temperature to 0, include an explicit schema with field names and types in your prompt, and say "Return ONLY the JSON — no explanation, no markdown code block." For guaranteed valid JSON, use the API's JSON mode (OpenAI) or structured outputs feature.
What is JSON mode in the OpenAI and Anthropic APIs?
JSON mode is an API parameter (response_format: json_object in OpenAI) that forces the model to return syntactically valid JSON. It guarantees parseable output but you still need to specify which fields to include in your prompt. OpenAI's Structured Outputs goes further by enforcing a Pydantic schema at the API level.
How do I validate structured output from an LLM?
Parse with json.loads() inside a try/except, then run Pydantic validation with field_validator to enforce business rules (e.g. rating must be 0–5). If validation fails, retry with a corrective message: "Your last response was not valid JSON. Try again." This handles models that wrap JSON in markdown code blocks.
Can LLMs output YAML or XML instead of JSON?
Yes. You can request any structured format by describing it in your prompt. YAML works well for configuration files. XML tags are especially useful for multi-section responses and work particularly reliably with Claude. For programmatic use, JSON is generally the best choice because it is directly parseable in every language.
What should I do when an LLM returns malformed JSON?
Catch the json.JSONDecodeError and retry with a corrective follow-up message telling the model the response was not valid JSON and asking it to try again. After 2–3 retries, fall back to a regex-based extraction or raise an error. Using API-level JSON mode or structured outputs eliminates most malformed-JSON failures.
Quick summary
- Schema-in-prompt: describe fields and types directly — works without special API features
- OpenAI JSON mode guarantees valid JSON; Structured Outputs enforces a Pydantic schema
- Add Pydantic
field_validatorto enforce constraints beyond what the schema expresses - Always use
temperature=0for extraction and say "Return ONLY the JSON" - XML tags work well for multi-section responses, especially with Claude