Your AI Said “Done.” What Actually Happened?

Show me what happened.

The message says the email was sent. The post was published. The appointment was booked. The follow-up was handled.

Where is the evidence?

That question belongs in the everyday management of an AI system. A confident summary is easy to read and easy to believe, especially when you have twenty other things to do.

It also creates an expensive habit if you discover later that “done” meant drafted, queued or attempted.

How do you know whether an AI agent is working?

Evaluate whether it produces the intended result, stays within its permissions and leaves evidence you can check. Measure the human effort required to initiate, review, correct and follow up on its work. A system can generate more output while increasing the owner’s workload.

Start with a definition of completion that belongs to the job.

A prepared email needs a reviewable draft. A sent email needs a sending record; delivery is a further status, and neither proves the person read it. A booked appointment needs the correct event and attendee details. A published article needs an accessible page.

Treating these states separately makes failures easier to find.

Give the system a way to report failure

Imagine an assistant preparing client briefings. This is an illustrative example.

It reads approved notes, summarizes outstanding questions and prepares a briefing before a call. One morning, the source cannot be reached.

A useful response identifies the missing source and shows what remains unverified. The workflow routes the problem to its owner. It does not quietly assemble a plausible briefing from old information and report success.

This means failure behavior is part of the design. Someone needs to know what can be retried, what must pause and what information a person needs to resolve the issue.

Otherwise, you become the monitoring system, opening every tool because you cannot trust the status.

Count the work that came back

The apparent time saving is only part of the calculation.

Suppose a recurring task used to take 40 minutes. In an illustrative AI-assisted version, preparation takes five minutes of your time, review takes ten, corrections take eight and monitoring adds two. Your active handling has fallen to 25 minutes, a 15-minute reduction, assuming comparable quality.

That may be useful. It is also a different result from claiming the entire 40 minutes disappeared.

Keep software costs, setup work and ongoing maintenance visible as well. Time released can create capacity. It does not automatically become cash saved or revenue earned.

If the owner still starts every run, locates every source, corrects every answer and reminds everyone to follow up, the system may have shifted the shape of the work without providing much relief.

Review a small amount of evidence consistently

For a bounded, low-risk role, a weekly review can be a useful starting rhythm, supplemented by alerts for failures that cannot wait. Higher-consequence work may need closer supervision.

Look at a sample of completed work and every material failure. Check whether expected runs happened. Compare the result with the source. Review the exceptions that reached a person and those that should have.

Then ask what needs to change: the instruction, the information, the permissions, the handoff or the role itself.

This is consistent with the broader principle that AI risk management continues across design, use and evaluation. NIST’s voluntary AI Risk Management Framework provides guidance for organizations managing those responsibilities; it does not certify an individual system as reliable. NIST AI Risk Management Framework

A good demonstration earns a next test

An assistant that performs well on one example has shown you something worth investigating. Give it varied examples, missing information and situations where the correct action is to stop.

Permission can expand only where the evidence and business rules support it. Strong performance in summarizing notes does not establish permission to send messages, change terms or move money.

Keep the boundaries specific enough that you can explain them without saying, “I trust the AI.” Trust it to do what, with which information, under which conditions?

Those questions make ongoing management practical. They also give you a better basis for deciding whether to keep, improve or retire a role.

The next time your AI says “done,” ask to see the result. Then count how much work you still had to do.