CCAR-P sample question: facts that change after the knowledge cutoff

An original sample question by CERTain·exam, based only on Anthropic's public documentation and the public exam guide. CERTain·exam is not affiliated with Anthropic.

Domain 5 · Governance, Safety & Risk Management (14% of CCAR-P in the exam guide effective until December 6, 2026; 16% from December 7, 2026). Objective covered: "Identify risks, limitations, and failure modes of LLM systems."

This question tests whether you name the failure mode correctly and choose the change that removes it, rather than one that only discloses it or postpones it. Work through it before you reveal the answer.

The question

Tollgarth Customs Brokers gives its 200 brokers an internal assistant built on Claude Sonnet 5.5 for questions about import duty rates and required documents. The assistant answers from the model's own knowledge; no documents or tools are attached. Its answers on long-standing procedures are reliable. Last week, however, it quoted a duty rate that the national customs authority had changed in September, after the model's knowledge cutoff, and a broker filed a declaration with the old rate. The authority publishes every current rate and notice on its public website, and rates change several times a month.

Which change best addresses this failure? (Select one.)

  • A. Move the assistant to a newer, larger Claude model and retest it on the duty rates that changed during the last year.
  • B. Give the assistant the web search tool restricted to the customs authority's domain, so rate answers cite its current pages.
  • C. Add a system-prompt rule that every rate answer must state the model's knowledge cutoff and ask the broker to verify the rate.
  • D. Load the authority's rate tables into a retrieval index rebuilt each month, and answer rate questions only from retrieved chunks.
Show the answer and explanation

Answer

B. Give the assistant the web search tool restricted to the customs authority's domain, so rate answers cite its current pages.

How to read this question

  1. Situation. The assistant answers duty-rate questions from model knowledge alone and quoted a rate that changed after the knowledge cutoff. The official rates live on a public website and change several times a month.
  2. What is asked. Which change removes the failure mode (stale knowledge of changing facts), instead of disclosing it or moving it elsewhere. One answer.
  3. Option by option.
    • A. Every model has a knowledge cutoff, and all four models in the current comparison table share the same one (June 2026), which is earlier than the September change. Rates that change several times a month will always outrun training data.
    • B (correct). The failure mode is stale training knowledge for facts that change after the cutoff. Web search fetches the authority's current pages at question time. Restricting it to the authority's domain (the allowed_domains setting) keeps answers tied to the official source, and citations let a broker check the page before filing.
    • C. Stating the cutoff is honest, but the assistant still answers from stale knowledge and shifts the whole burden of verification onto the broker, who uses the assistant precisely to avoid that work.
    • D. Grounding answers in the authority's tables is the right idea, but a monthly rebuild cannot follow rates that change several times a month, and it adds a pipeline the team must maintain.
  4. Rule to remember. Facts that change after a model's knowledge cutoff must be fetched from an authoritative source at request time; a newer model or a periodic snapshot only moves the cutoff.

Sources (Claude Docs, quotes checked Oct 28, 2026)

  • Web search tool:
    • "The web search tool gives Claude direct access to real-time web content, allowing it to answer questions with up-to-date information beyond its knowledge cutoff."
    • Among requests where Claude searches: "Current prices, rates, scores, or statistics".
    • On allowed_domains: "Only include results from these domains."
    • "The response includes citations for sources drawn from search results."
  • Models overview:
    • "Reliable knowledge cutoff: The date through which the model's knowledge is most extensive and reliable."
    • Comparison table: "Reliable knowledge cutoff" is Jun 2026 for Claude Fable 5.1, Opus 5.5, Sonnet 5.5 and Haiku 5.5.

More practice

Questions

Where does this question come from?

It's an original sample question that CERTain·exam wrote based on Anthropic's public documentation and the public exam guide. Exam content is confidential under Anthropic's candidate agreement, and we do not use it.

What does CCAR-P Domain 5 cover?

According to the exam guide effective until December 6, 2026, Governance, Safety & Risk Management (14%) covers guardrails and safety controls, identifying risks, limitations and failure modes of LLM systems, human-in-the-loop validation, compliance with regulations (for example GDPR, HIPAA, FedRAMP), and ethical considerations such as bias, fairness and transparency. From December 7, 2026, the domain weighs 16%, and its objectives are rewritten. See what changes on December 7, 2026.

Get your free practice exam

First, sign in with your email.