Domain 4 · Lesson 2 · Prompt Engineering (20%)
Structured Output & Validation Loops
Task Statements 4.3 & 4.4 — tool_use + JSON schema, and when retries help vs don't.
Course progress: Domain 4 ▸ lesson 2 of ~3
tool_use + JSON schema = reliable structure (4.3)
The most reliable way to get schema-compliant output is tool use with a JSON schema as the tool's input parameters — you then read the structured data from the tool_use response. This eliminates JSON syntax errors.
Critical limit
Strict schemas eliminate syntax errors but NOT semantic errors. The model can still emit line items that don't sum to the total, or put a value in the wrong field. Schema validity ≠ correctness. This gap is exactly what validation loops (below) address.
tool_choice for extraction
"any" — guarantee structured output when multiple extraction schemas exist and the doc type is unknown.- forced
{"type":"tool","name":"extract_metadata"} — ensure a specific extraction runs first (before enrichment steps), then continue in follow-up turns.
Schema design to prevent hallucination
Nullable/optional over required
Make a field optional/nullable when the source may not contain it — so the model returns null instead of fabricating a value to satisfy a required field. Use enums with an "other" + detail string (and "unclear" for ambiguous cases) for extensible categorization. Include format-normalization rules in the prompt alongside the strict schema.
Validation, retry & feedback loops (4.4)
Retry-with-error-feedback: when validation fails (Pydantic / JSON schema), send a follow-up containing the original document + the failed extraction + the specific validation error, so the model self-corrects.
| Retry will… | Case |
|---|
| Succeed | Format mismatches, structural output errors — the info is present, just mis-shaped |
| Fail (don't bother) | Required info is simply absent from the source (e.g., it lives in an external doc you didn't provide) |
Self-correction validation flows: extract calculated_total alongside stated_total to flag discrepancies; add a conflict_detected boolean for inconsistent source data; add a detected_pattern field to findings so you can analyze which patterns cause false positives when developers dismiss them.
Check yourself
You use tool_use with a strict JSON schema and still get invoices where line items don't sum to the stated total. This is because strict schemas:
- Only work when tool_choice is forced to a single named tool
- Eliminate syntax errors but not semantic errors like bad sums
- Require all fields to be optional before arithmetic is checked
- Cannot be combined with any prompt-side normalization rules
Correct: option 2. Schemas guarantee well-formed JSON, not correct values. A sum mismatch is a semantic error — catch it with a validation flow (e.g., calculated vs stated total), not the schema alone.
A required field keeps coming back fabricated when the source document lacks that information. The right schema change is to:
- Keep it required and add a prompt line forbidding fabrication
- Force tool_choice so the extractor is guaranteed to run first
- Make the field optional/nullable so the model can return null
- Retry the extraction until the model produces a valid value
Correct: option 3. A nullable field lets the model report absence instead of fabricating to satisfy "required." A prompt plea is weaker; forcing the tool doesn't help; retrying can't conjure absent info.
Validation fails because the needed value exists only in an external document you never supplied. A retry-with-error-feedback loop will:
- Succeed once the specific validation error is included in it
- Fail, because the information is absent from the given source
- Succeed after the schema is switched to strict mode instead
- Succeed if the failed extraction is attached to the retry too
Correct: option 2. Retries fix format/structural errors where the info is present but mis-shaped. When the information is simply absent from the provided source, no retry can recover it.
Decision rules
Need schema-valid output → tool_use + JSON schema. Sums/wrong-field → validation flow (semantic), not schema. May be missing → nullable field. Unknown doc type → "any". Must run first → forced tool. Retry helps for format/structure, never for absent info.
Ask your teacher. Want to see a schema with nullable fields + an "other" enum + a calculated-vs-stated validation? Ask.