Aim: 7/8+.
1. Too many false positives; "only high-confidence findings" didn't help. Fix:
- Write specific categorical criteria for what to report vs skip
- Instruct the model more firmly to be conservative overall
- Lower temperature to reduce the number of findings produced
- Have the model attach a confidence number to each finding
Option 1. Categorical criteria, not vague/confidence filters. (L1)
2. Extraction inconsistent across varied document structures. Strongest lever:
- Write longer prose describing each possible document layout
- Add 2–4 few-shot examples showing correct per-structure output
- Raise max_tokens to give room for every layout variant
- Force tool_choice any so the extractor always gets called
Option 2. Few-shot is the top consistency/generalization lever. (L1)
3. Strict JSON schema, yet totals don't add up. Because schemas:
- Need tool_choice forced to a specific named tool to validate
- Remove syntax errors but not semantic errors like bad sums
- Require every field optional before totals can be checked
- Cannot coexist with any prompt-side normalization rules
Option 2. Valid JSON ≠ correct values. (L2)
4. A required field is fabricated when the source lacks it. Best change:
- Keep it required and forbid fabrication in the prompt text
- Force the extractor tool to run first via tool_choice
- Make the field optional/nullable so it can return null
- Retry extraction until a valid-looking value appears
Option 3. Nullable lets the model report absence. (L2)
5. Needed value lives only in an external doc you didn't provide. A retry loop will:
- Succeed once the exact validation error is fed back to it
- Fail, since the information is absent from the given source
- Succeed after switching the schema into strict mode instead
- Succeed if you also attach the failed extraction on retry
Option 2. Retries fix shape, not absence. (L2)
6. Blocking pre-merge check + overnight report; manager wants both on Batches. Best:
- Move both to batch and poll frequently to stay responsive
- Batch the overnight report only; keep sync for pre-merge
- Keep both synchronous to dodge batch result ordering issues
- Batch both with a real-time fallback when batches run slow
Option 2. Batch has no SLA; unfit for blocking work. (L3)
7. Batch of 100 docs; a few fail from exceeding context limits. Handle by:
- Resubmitting the entire batch of one hundred documents again
- Resubmitting only failed docs by
custom_id, chunked
- Switching the whole job to the synchronous API immediately
- Discarding the failed documents and reporting partial results
Option 2. Resubmit failures by custom_id with fixes like chunking. (L3)
8. 14-file PR reviewed in one pass: uneven, contradictory. Best restructure:
- Move to a larger context window to fit all files at once
- Require developers to submit at most four files per PR
- Per-file local passes plus a cross-file integration pass
- Run three passes; keep issues appearing in at least two
Option 3. Attention dilution → split passes; window size irrelevant. (L3)
Scoring8/8 → Domain 5. Misses → re-read the flagged lesson.
Tell your teacher your score.