Domain 4 · Lesson 1 · Prompt Engineering (20%)
Explicit Criteria & Few-Shot Prompting
Task Statements 4.1 & 4.2 — how to actually reduce false positives and force consistency.
Course progress: Domain 4 ▸ lesson 1 of ~3
Explicit criteria beat vague instructions (4.1)
Vague guidance does not improve precision. "Be accurate," "be conservative," "only report high-confidence findings" — these fail to move the false-positive rate. What works is specific categorical criteria.
| Vague (fails) | Explicit (works) |
|---|
| "Check that comments are accurate" | "Flag a comment only when the claimed behavior contradicts the actual code behavior" |
| "Report important bugs" | "Report bugs and security issues; skip minor style and local pattern choices" |
Why it matters
High false-positive categories undermine trust in the accurate ones. Developers stop reading. Fixes: write criteria that name what to report vs skip; temporarily disable a high-false-positive category while you improve its prompt; define explicit severity levels with concrete code examples for consistent classification. Confidence-based filtering ("only high confidence") is not a substitute for categorical criteria.
Few-shot: the strongest consistency lever (4.2)
When detailed instructions alone still produce inconsistent output, few-shot examples are the most effective technique. They do four jobs the exam names:
- Format consistency — demonstrate the exact output shape (location, issue, severity, suggested fix).
- Ambiguous-case handling — show reasoning for why one action beat a plausible alternative.
- Generalization — the model extends judgment to novel patterns, not just the examples shown.
- Hallucination reduction in extraction — show correct handling of varied document structures (inline citations vs bibliographies, narrative vs tables) and informal measurements.
How many, and what kind
Use 2–4 targeted examples for ambiguous scenarios. Include examples that distinguish acceptable patterns from genuine issues — this reduces false positives while enabling generalization, because the model learns the boundary, not a whitelist.
Criteria vs few-shot — which first?
They're complementary. Explicit criteria define what counts; few-shot shows how it looks and handles the fuzzy edges criteria can't fully specify. For a precision problem, tighten criteria; for an inconsistent-format or ambiguous-judgment problem, add few-shot.
Check yourself
A code-review agent has too many false positives. Adding "only report high-confidence issues" to the prompt didn't help. The better fix is to:
- Instruct the model more firmly to be conservative in its reports
- Write specific categorical criteria for what to report vs skip
- Lower the temperature so the model produces fewer total findings
- Ask the model to attach a numeric confidence to each finding
Correct: option 2. Vague/confidence-based instructions don't improve precision; specific categorical criteria (report bugs/security, skip minor style) do. "Be conservative," temperature, and self-reported confidence don't fix the decision boundary.
Extraction output is inconsistent across documents with different structures (inline citations vs bibliographies). The most effective technique is to:
- Write a longer prose description of every document layout variant
- Raise max_tokens so the model has room to handle all layouts
- Add few-shot examples showing correct extraction per structure
- Force tool_choice to any so the model always calls the extractor
Correct: option 3. Few-shot examples demonstrating varied structures are the strongest lever for consistency and for generalizing to novel layouts. Longer prose is what already failed; token limits and forcing a call don't teach structural handling.
Decision rules
Precision problem → specific categorical criteria (not "be conservative"/confidence filters). Inconsistent format or ambiguous judgment → 2–4 few-shot examples showing the boundary. Restore trust fast → disable the noisy category while you fix its prompt.
Ask your teacher. Want a full few-shot block showing the accept-vs-flag boundary? Ask.