CCAR-P sample question: cutting an agent's run time and token cost
An original sample question by CERTain·exam, based only on Anthropic's public documentation and the public exam guide. CERTain·exam is not affiliated with Anthropic.
Domain 4 · Evaluation, Testing & Optimization (16% of CCAR-P in the exam guide effective until December 6, 2026; 17% from December 7, 2026). Objective covered: "Optimize token usage, latency, and cost-performance trade-offs."
This question tests whether you find what actually drives the time and the cost of an agent run, and pick the change that removes the cause rather than hiding or splitting it. Work through it before you reveal the answer.
The question
Brambleway Medical Supply runs a replenishment agent on Claude Sonnet 5.5. Each morning a planner asks which of 60 critical products are below safety stock at any of the company's 8 depots. For each product, the agent already requests all 8 depots from get_stock_levels as parallel calls in one turn. That makes 60 turns, each adding about 12,000 tokens of JSON that stays in the context, for about 720,000 tokens by the end. The planner needs only the 10 to 20 shortfalls. The answers are correct, but a run takes 40 minutes, the planner needs the list within 10, and input tokens make up most of the agent's cost.
Which change should the architect make? (Select one.)
- A. Split the 60 products between two subagents running in parallel, each reporting its shortfalls to a lead agent.
- B. Stream the agent's output to the planner, so that each shortfall is shown as soon as the agent identifies it.
- C. Turn on tool result clearing so that older stock results are removed from the context once the agent has moved past them.
- D. Let Claude call get_stock_levels from code in a code execution container and pass back only products below safety stock.
Show the answer and explanation
Answer
D. Let Claude call get_stock_levels from code in a code execution container and pass back only products below safety stock.
How to read this question
- Situation. The agent already batches lookups as parallel calls, yet 60 model turns pile about 720,000 tokens of raw results into its context, while the planner needs only a short list. The run is four times too slow, and input tokens dominate the cost.
- What is asked. Which change removes both drivers, the model turns and the tool-result tokens, rather than only hiding or splitting them. One answer.
- Option by option.
- A. Two parallel subagents roughly halve the run to about 20 minutes, still twice the 10-minute limit. Each still takes every 12,000-token result into its own context with one sampled turn per product, so all 60 turns and 720,000 tokens of raw results remain, plus the lead agent's own calls (reasoning from the scenario).
- B. Streaming shows partial results earlier, but the planner needs the complete list. It changes perceived latency, not total latency. The 60 turns and 720,000 tokens stay, and the cost doesn't change.
- C. Clearing old results reduces the context resent on later turns, but every result still enters the context once and Claude is still sampled 60 times, so run time barely changes.
- D (correct). This is a fan-out over many items whose results can be filtered before Claude needs them. With programmatic tool calling, one script runs all 480 lookups without re-sampling the model between calls, and only the short list of products below safety stock enters the context, cutting both latency and input tokens.
- Rule to remember. When an agent fans out over many items and needs only a filtered summary, let code make the calls and filter the results, so the model is not re-sampled per call and raw results never enter its context.
Sources (Claude Docs, quotes checked Oct 28, 2026)
- Programmatic tool calling:
- "This reduces latency for multi-tool workflows and decreases token consumption by allowing Claude to filter or process data before it reaches the model's context window."
- Listed as a strong fit: "Fan-out or parallel operations across many items (for example, checking 50 endpoints or looking up 20 records)".
- "Tool results from programmatic calls are not added to Claude's context - only the final code output is".
- "The model is not re-sampled between the tool calls inside one code execution."
- "Multiple tool calls in one code execution - reduces overhead compared to separate model turns".
- Context editing:
- "When activated, the API automatically clears the oldest tool results in chronological order." (This is what option C clears: the oldest tool results, which are already in the context.)
More practice
- Previous: Domain 3 · An agent with too many overlapping tools
- Next: Domain 5 · Facts that change after the knowledge cutoff
- How to prepare for CCAR-P
Questions
Where does this question come from?
It's an original sample question that CERTain·exam wrote based on Anthropic's public documentation and the public exam guide. Exam content is confidential under Anthropic's candidate agreement, and we do not use it.
What does CCAR-P Domain 4 cover?
According to the exam guide effective until December 6, 2026, Evaluation, Testing & Optimization (16%) covers defining evaluation metrics (accuracy, latency, cost, safety, security), designing evaluation datasets and test frameworks, A/B testing and iterative improvement, diagnosing system issues, optimizing token usage, latency and cost-performance trade-offs, and monitoring with logging and observability tools. From December 7, 2026, the domain weighs 17%, and its objectives are rewritten. See what changes on December 7, 2026.
First, sign in with your email.