Home › Study guides › CCAO-F › Domain 4 › Lesson 4.5
CCAO-F · Domain 4 · 16% of the exam · Lesson 4.5 · 20 min read
Explaining Claude's value and limits to stakeholders
How to report a Claude pilot: value as measured results, limits named with their controls, and one set of facts told to leaders, legal and the team.
Written against objective 4.5 of the official CCAO-F exam guide (Version 1.0, effective July 2026). An independent resource, not affiliated with Anthropic; the practice questions are written from scratch.
4.5.1 Why good pilot results can still be told badly
Nadia is a project manager at a logistics company that delivers freight and parcels for online retailers. For six weeks, six of its eighteen support agents tried a new way of answering customer emails. Claude drafted each reply inside a shared Project, a Claude workspace that keeps its own chats and reference files together. This one held the approved reply guidelines, the claims policy and the answers to frequently asked delivery questions. The agent checked the draft, fixed what was wrong and sent it. Now Nadia has to explain the pilot three times in one week: to leadership on Tuesday, to legal and compliance on Wednesday, and to the support agents on Friday.
Two versions of the story are ready-made. One is the launch slide: "AI now answers our customer emails in seconds." The other is the nervous shrug: "It looks promising, but AI makes mistakes." The first sets leadership up to expect an inbox that runs itself, and leaves legal to find out about the errors later. The second throws away six weeks of evidence. Both describe the technology instead of what happened.
Each room also brings its own question, because each has a different decision to make. So the skill in this lesson is building a balanced case. That is an account of the pilot that gives the value and the limits equal care, backs both with evidence, and answers each room's question.
One pilot, three meetings, three different questions
4.5.2 Value is a measured result, not a feature
Here is the question that trips people up: what counts as Claude's value? It is tempting to answer with what the tool can do: fluent replies in seconds, broad knowledge, an advanced model. Those are features. A leadership team approves outcomes, and the value of the pilot is whatever changed in the work, measured against how the work went before.
Think of a new delivery van. Nobody signs off a fleet order because the brochure praises the engine. They sign it off because, on real routes, the van carried more parcels per trip at a cost they could see. Claude's value shows up the same way, in five business terms, each with a before and an after.
| Value | The hype version | What Nadia's pilot measured |
|---|---|---|
| TIME | "Replies in seconds" | Median handling time per email fell from 9 to 5 minutes |
| TURNAROUND | "Instant customer service" | Average time to first reply fell from 6 hours to 3.5 |
| CONSISTENCY | "On-brand every time" | Sampled replies following the approved claim steps rose from 72% to 94% |
| CAPACITY | "Does the work of three agents" | About 160 agent hours freed over six weeks, spent on the phone backlog |
| QUALITY | "Better answers" | Quality checklist score rose from 81% to 89%; customer satisfaction stayed at 4.2 out of 5 |
Memorise the five terms; the figures are only Nadia's. They rest on a simple design: she measured the same six agents for two weeks before the pilot, then for six weeks with Claude. That baseline, a measurement of the work before anything changed, turns "it felt faster" into "median time per email fell from nine minutes to five". Anthropic's Claude 101 course recommends the same habit on a small scale: test Claude on work you have already done, where you know the right answer, before you rely on it.
Customer satisfaction did not move, and Nadia reports it anyway. A flat result is still evidence, and leaving it out is how a report starts to overclaim. Anecdotes such as "the agents love it" can sit beside the numbers, never in place of them.
4.5.3 Name each limit, with its control
The limits are the part people are tempted to shrink. Some leave them out. Others add a vague line at the foot of the last slide: "As with all AI, outputs may contain errors." That looks honest, but it gives nobody anything to act on. Which errors? How often? What stops one reaching a customer? A medicine leaflet that said only "may cause side effects" would be just as useless. A good one names the common side effects, says how often they happen and tells you what to do about each.
A useful limit statement has three parts: the limit, what the pilot showed, and the control that covers it. Anthropic's Help Center is plain about the main limit. Claude can produce answers that look correct but are wrong, including details it has made up; these are usually called hallucinations. So Claude should not be relied on as a single source of truth, and high-stakes advice needs careful scrutiny. Look at how each line of Nadia's limits slide pairs a problem with what the team does about it.
Limits we found in the pilot, and the control for each
It can be confidently wrong. About 1 in 20 drafts contained a factual error: a wrong delivery window, a refund the claims policy does not offer, a made-up deadline for damage claims. They read as fluently as correct drafts. Control: the agent checks every fact against the tracking system and the policy before sending.
It needs review, and review can miss things. Agents caught all but two of those errors in six weeks; the two reached customers and were corrected within a day. Control: the team lead spot-checks a sample of sent replies every week.
It only knows what it is given. Claude sees the customer's email, the order status the agent pastes in and the policy files in the Project, not the tracking system or last week's phone call. Control: agents paste the current status; the policy owner replaces outdated files.
Data rules apply. Customer emails carry names, addresses and order details. Control: agents paste in only the customer details a reply needs, as our AI policy requires, and work only in the company account.
It assists decisions; it does not make them. A draft that offers a refund reads like a decision even when nobody made one. Control: refunds, claim approvals and exceptions are decided by agents and supervisors; Claude drafts the wording once a person has decided.
Where it helps least: complaints and disputed claims. Agents rewrote most of those drafts.
Anthropic's AI Fluency course makes the general point behind the slide: the most effective uses combine AI with human judgment and oversight. Each line shows where that judgment sits in this workflow. The last line matters too. Saying where Claude helps least is part of an honest case, and it tells leadership where not to expect gains.
4.5.4 Same facts, a different lead for each audience
Should Nadia give the same deck three times, or build three different stories? Neither. One deck bores leadership with procedure and buries the agents in cost figures. Three stories drift apart until legal hears "1 in 20 drafts had a factual error" while leadership hears "near-perfect accuracy". The answer is one set of facts with a different emphasis for each audience.
A house survey works this way. The buyer reads the summary, the lender reads the valuation and the builder reads the list of defects. It is still one report, and nobody gets a copy where the damp has disappeared. For stakeholders, what sets the emphasis is the decision each group has to make.
| Audience | What they decide | Lead with |
|---|---|---|
| Leadership | Whether to expand, and at what cost | Outcomes against the baseline, the cost of seats for the full team, the main risk and its control, the recommendation |
| Legal and compliance | Whether the controls are adequate | What data went in and what stayed out, the account and its terms, who approves each reply, the error record |
| The support team | How their work changes | What changes in their day, what stays their call, training, and how to report a bad draft |
| IT | Whether requests are secure and feasible | Security and access questions, and integration requests that need technical colleagues to build |
The IT row exists because some requests are not Nadia's to grant. The agents asked whether Claude could read order status straight from the tracking system. That is an integration, a connection between two systems that someone has to build and look after. An Associate writes down the need and passes it to the technical colleagues who build such connections, rather than promising it on a slide.
Whatever the audience, the message follows the same spine, and the order matters. Claude's role comes before the numbers, so nobody hears "handling time fell from nine minutes to five" and pictures an inbox with no people in it.
The spine of every stakeholder message
4.5.5 Promise what the pilot showed, then plan to check
Whatever you tell stakeholders becomes a promise the workflow has to keep. Overclaiming ("fully automated", "never wrong") wins the meeting and loses trust at the first error that reaches a customer. It also invites people to drop the review that caught the errors. Underclaiming ("just an experiment", "too risky to rely on") feels safe, but it leaves proven value unused and gives the team no reason to follow the approved workflow.
Three ways to describe the same pilot
Overclaim
Calibrated
Underclaim
The way out of both errors is to make the claim testable. Instead of asking leadership to believe that Claude works, Nadia asks them to approve a larger trial with named measures and a date to look again. Look at the last two lines of her recommendation: a review point, and a condition that would pause the rollout.
Recommendation: extend Claude-drafted replies to all 18 agents for 12 weeks.
What stays the same: an agent reads and approves every reply before it is sent. Refunds, claims and exceptions are decided by agents and supervisors under the current policy.
What we will measure, against the pilot baseline: median handling time, time to first reply, quality checklist score, drafts corrected before sending, errors that reach customers, customer satisfaction.
Owners: Callum, support team lead, for the workflow; the claims policy owner for the policy files in the Project.
Training: a one-hour session for every agent, plus a short checklist of what to verify in each draft.
Review point: week 6, reported in the same format as the pilot.
Pause condition: if errors reach customers at a higher rate than in the pilot (two in six weeks from six agents), or the quality checklist score falls below the 81% baseline, we stop, find the cause and report back before continuing.
A pause condition is not an admission of doubt. It shows that the team knows what failure looks like and will notice it, which makes the rest of the claim believable.
4.5.6 Answer the hard questions with facts
Three concerns come up in most rollouts: jobs, quality and privacy. The tempting answers are reassurance ("don't worry"), evasion ("let's take that offline") and promises you cannot keep ("nobody's job will change"). Each one costs trust, because the person asking can tell it was not built from facts. The better answer uses the pilot's results and the controls in place, and admits what is still undecided.
| Concern | The weak answer | The answer from facts and controls |
|---|---|---|
| Jobs: "Is this replacing us?" | "Don't worry, nothing will change." | Every reply still went through an agent; the freed time went to the phone backlog; no staffing decision has been made, and leadership, not the tool, would make one |
| Quality: "Will customers get wrong answers?" | "Claude is very accurate." | About 1 in 20 drafts had an error; agents caught all but two in six weeks; checks and a pause condition stay in place |
| Privacy: "Where do customer details go?" | "It's secure, it's fine." | Only the details each reply needed; the company account, not personal ones; the provider's published terms and the admin settings |
The jobs question needs the most care. Nadia can say what is true: agents reviewed and sent every reply, and the judgment calls stayed theirs. She cannot promise what leadership has not decided, so she says who decides it and invites the agents to help set the checks for the next phase. Anthropic's AI Fluency course calls this diligence: being honest about AI's role with everyone who needs to know, and taking responsibility for verifying and vouching for what you share. The agent who sends a reply owns it, whoever drafted it.
Privacy answers come from documents, not memory, and not from Claude, which only knows what is in front of it. Nadia brings the provider's published terms and a note from the account's administrator on how the company account is set up. For example, Anthropic's Privacy Center states that, by default, Anthropic does not use inputs or outputs from its commercial products, such as Claude for Work, to train its models. The exceptions include feedback someone explicitly sends, such as a thumbs-up or thumbs-down rating.
Notice how narrow that statement is. It is about training, not about how long data is kept or who can see it. It also covers commercial products only; the Privacy Center deals with consumer plans such as Free, Pro and Max separately. That is why the administrator's note matters: it shows which plan the company is on and how it is set up. Legal can check every one of those points. Nobody can check "it's secure".
4.5.7 The exam traps
Every trap here gives stakeholders more trust, or less, than the evidence supports.
- ✗ Announcing "fully automated replies" or "Claude is never wrong". ✓ Describe Claude as drafting with human approval, and give the error rate you found. Overclaims break at the first error and tempt people to drop the review.
- ✗ Covering the limits with a generic "AI can make mistakes". ✓ Name each limit, what the pilot showed and the control that covers it.
- ✗ Proving value with features or enthusiasm ("advanced model", "the team loves it"). ✓ Show measured change against a baseline in business terms, including results that did not improve.
- ✗ Changing the facts to suit each audience. ✓ Keep one set of facts and change the emphasis to match what each group decides.
- ✗ Presenting Claude, or a Project, as the authority on refunds, claims or policy. ✓ Say which decisions stay with accountable people. Claude drafts; people decide and answer for the result.
- ✗ Meeting concerns with reassurance, or retreating to "too risky" when challenged. ✓ Answer with facts and controls, and propose measures and a review point.
Four tempting messages, one balanced one
4.5.8 Put it together: brief three audiences from one set of facts
You now have every piece. Value is a measured change, each limit comes with evidence and a control, and one set of facts follows one spine for every audience. The quickest way to feel how much the limits carry is to take them out and watch the message change.
This lesson closes Domain 4. Domain 5 covers setting up a shared Project like Nadia's (5.1) and keeping it current (5.4). Organisational AI policies (6.3) are the rules your legal audience will check your controls against. The ethics of AI use (6.4) takes up questions such as whether customers should be told that AI helped draft a reply. When the review point arrives, Domain 7 covers adjusting your approach based on results (7.2) and optimising the workflow (7.3).
Key takeaways
- ✓ Stakeholders should trust Claude exactly as far as the evidence goes, no further and no less.
- ✓ State value as measured change against a baseline in time, turnaround, consistency, capacity and quality, including what did not improve.
- ✓ Name each limit with what the pilot showed and the control that covers it; a generic disclaimer is not a statement of limits.
- ✓ Keep one set of facts and change only the emphasis: outcomes and risk for leadership, data and accountability for legal, daily work and training for the team, integration requests for IT.
- ✓ Follow the same spine for every audience: the problem, Claude's role, limits and controls, evidence, the ask.
- ✓ Avoid both "fully automated" and "too risky"; propose measures, an owner, a review point and a pause condition.
- ✓ Answer concerns about jobs, quality and privacy with facts and controls, and say who decides what is still open.
Check your understanding
4 questions written for this lesson, then one from the CCAO-F question bank on the same topic. Every answer option is explained, including the ones you did not pick. Nothing is stored.
60 CCAO-F questions on Domain 4, free
Every question in the bank is tagged to a domain, so you can drill 60 questions on Workflow Integration and Solution Design alone, or sit the full 60-question timed simulator.
Open the CCAO-F question bank → Back to Domain 4 →
The question bank is free. It asks for an account only because the quiz engine has to store answers to score them and show which domains are weak. The questions on this page need nothing.