Aim: 6/7+.
1. Long session; agent forgets stated refund amounts and dates. Fix:
- Keep a case-facts block of amounts/dates/IDs in every prompt
- Summarize the conversation more aggressively to free up space
- Place the full transcript in the middle of each new request
- Raise max_tokens so the entire history always stays loaded
Option 1. Persistent case-facts block. (L1)
2. Long codebase session; model cites "typical patterns" not specific classes. Use:
- Instruct the model to remember its earlier findings better
- Re-paste every previously read file into the new prompt
- A scratchpad file of key findings, referenced later on
- Switch to a model with a larger maximum context window
Option 3. Scratchpad persists findings across boundaries. (L1)
3. Agent escalates easy cases, attempts hard policy ones. Best calibration fix:
- Explicit escalation criteria plus escalate-vs-resolve few-shot
- Self-reported confidence routing low scores to human agents
- A classifier trained on historical tickets to predict escalation
- Sentiment analysis escalating when negativity exceeds a bound
Option 1. Unclear boundaries → criteria + few-shot. (L2)
4. Customer explicitly demands a human agent. The agent should:
- Investigate first, then escalate only if it cannot resolve it
- Honor the request and escalate immediately without investigating
- Offer a discount to retain the interaction before escalating
- Ask three clarifying questions before deciding to escalate
Option 2. Explicit human request → escalate now. (L2)
5. A web-search subagent times out. Best propagation to the coordinator:
- A generic "search unavailable" status after silent retries
- An empty result marked successful so the pipeline continues
- Structured context: failure type, query, partials, alternatives
- A raised exception that terminates the whole workflow at once
Option 3. Structured error context enables recovery. (L2)
6. 97% aggregate accuracy; you want less human review. First:
- Automate immediately since 97% clears the target threshold
- Segment accuracy by document type and field for weak spots
- Raise the confidence threshold uniformly for every doc type
- Trust the model's self-reported high-confidence extractions
Option 2. Aggregate hides bad segments. (L3)
7. Two credible sources give different statistics. Synthesis should:
- Pick the value from whichever source seems most authoritative
- Average both numbers into one reconciled reported figure
- Present both values annotated with their source attribution
- Omit both figures because the sources disagree on the metric
Option 3. Annotate conflict with attribution; check dates too. (L3)
Tell your teacher your score.