Domain 4 · Lesson 3 · Prompt Engineering (20%)
Batch Processing & Multi-Pass Review
Task Statements 4.5 & 4.6 — the Message Batches API, and review architectures that scale.
Course progress: Domain 4 ▸ lesson 3 of ~3
Message Batches API (4.5)
| Property | Detail |
|---|
| Cost | ~50% cheaper than synchronous |
| Latency | Up to 24-hour window, no guaranteed SLA |
| Tool calling | No multi-turn tool calling within a single request |
| Correlation | custom_id matches each request to its response |
The fit test
Batch is right for non-blocking, latency-tolerant work: overnight reports, weekly audits, nightly test generation. It's wrong for blocking workflows — a pre-merge check where a developer is waiting cannot depend on "up to 24h, no SLA." Mixed workloads → batch the overnight job, keep synchronous for the blocking check.
- Failures: resubmit only the failed docs (found by
custom_id) with fixes — e.g., chunk documents that exceeded context limits.
- SLA math: with 24h processing, submit on a schedule (e.g., every 4h) to guarantee a 30h end-to-end SLA.
- Refine first: tune the prompt on a small sample before running a large batch to maximize first-pass success and cut resubmission cost.
Distractor watch
"Switch both workflows to batch for the savings" is wrong when one is a blocking pre-merge check. "Batch results have ordering problems" is a misconception — custom_id correlates them. "Timeout fallback to real-time" adds needless complexity vs simply matching each API to its use case.
⚠ Accuracy note — exam guide vs current product (2026)
The Exam Guide (v0.1, Feb 2025) states the batch API does not support multi-turn tool calling within a single request, and the mock questions follow that. On the exam, answer per the guide. However, the current Message Batches API docs (2026) show that server tools and multi-turn tool loops do now run inside batch requests (the batch worker runs the same server-side agentic loop; a long turn returns pause_turn to continue). The product moved on after the exam was written. The latency/SLA reasoning is unchanged and is what the exam actually tests: batch is asynchronous (most finish <1h, hard cap 24h, no SLA), so it's still wrong for blocking pre-merge checks.
Multi-instance & multi-pass review (4.6)
- Self-review limitation: a model that generated the code retains its reasoning and is less likely to question its own decisions in the same session.
- Independent review (no prior reasoning context) catches subtle issues better than self-review instructions or extended thinking.
- Multi-pass: split large reviews into per-file local passes + a cross-file integration pass to avoid attention dilution and contradictory findings.
- Confidence-reporting pass: have the model self-report confidence per finding to route review attention (calibration detail in Domain 5).
The recurring myth
A bigger context window does not fix attention dilution. More room ≠ more attention quality. The fix is structural: split the passes.
Check yourself
Two workflows: a blocking pre-merge check (developer waits) and an overnight tech-debt report. A manager wants both on the Batches API for 50% savings. Best evaluation:
- Move both to batch and poll for completion to stay responsive
- Batch the overnight report only; keep sync for pre-merge checks
- Keep both synchronous to avoid batch result-ordering problems
- Batch both with a timeout fallback to real-time when slow
Correct: option 2. Batch (up to 24h, no SLA) suits the overnight job, not a blocking check. Polling doesn't make batch acceptable for blocking work; ordering is solved by custom_id; the fallback adds needless complexity.
A 14-file PR reviewed in one pass yields uneven depth and contradictory findings. The best restructure:
- Adopt a model with a larger window to fit all files at once
- Require developers to split PRs into four-file submissions
- Per-file local passes plus a separate cross-file integration pass
- Run three full passes; keep only issues found in two of them
Correct: option 3. The cause is attention dilution; split into focused passes. A bigger window doesn't improve attention quality; shifting to developers doesn't fix the system; consensus voting suppresses intermittently-caught real bugs.
Decision rules
Latency-tolerant / overnight → batch (50% off, custom_id, no SLA, no multi-turn tools). Blocking / pre-merge → synchronous. Batch failures → resubmit by custom_id, chunk oversized. Objective review → independent instance. Big review → per-file + integration passes (window size irrelevant).
Ask your teacher. Ready for the
Domain 4 quiz. Ask if the SLA-window math is unclear.