The 95% AI agent reliability trap

A 95% score tells you almost nothing until you know what was tested. It might describe one classification, one required step, one completed run, or one final work product. Those are different denominators.

If a hypothetical job has 10 required steps, every step succeeds independently at the same 95% rate, and one missed step makes the job fail, the end-to-end probability is 0.95^10, or 59.9%. That arithmetic is exact under those assumptions. It is not a measured failure rate for AI agents, FidelicAI, or your job.

Define the reliability claim before using the percentage

The 60% result is a diagnostic, not a forecast

The federal reliability handbook states the multiplication rule for independent components: when every component must survive, system reliability is the product of the component reliabilities. Apply that rule to 10 equal 95% steps and the result is about 60%.

Real AI work rarely behaves like 10 identical independent components. An early mistake can make later steps more likely to fail. A shared bad record can affect every step. Some steps may be optional, reversible, or protected by a check. Others may carry more risk than the rest of the path combined.

The data are derived, not observed.

The important buyer move is to stop asking, “Is the agent 95% accurate?” Ask, “What event makes the numerator, what enters the denominator, and what complete state counts as passed?”

End-to-end and repeat-run evidence answer different questions

One complete run can still be lucky. A job that works once and fails when repeated has weak operating reliability even if the successful run looked perfect.

τ-bench, a 2024 research benchmark for AI systems that converse with a simulated user and call domain tools, checks the final database state and required information in the reply. It also repeats tasks and reports whether the system passes consistently. The benchmark used 115 simulated retail tasks and 50 simulated airline tasks. Its historical model results belong to those tasks, models, and 2024 conditions. They are not current product estimates.

The useful design choice transfers: define the accepted end state, test the whole interaction, and repeat it. Do not carry the benchmark’s percentages into a different role.

NIST’s AI Risk Management Framework Core adds the production boundary. Measure 2.3 calls for testing in conditions similar to deployment. Measure 2.4 calls for monitoring the system’s behavior in production. A polished example can show capability. It cannot establish behavior across changing records, permissions, policies, people, and outside services.

A renewal memo exposes the full path

Take one recurring vendor-renewal decision. The accepted result is not a fluent summary. It is a decision record that the owner can use before the notice date.

The path might require the role to find the signed agreement, confirm the current renewal and notice terms, reconcile invoices, attach use and performance evidence, identify the accountable owner, compare the approved requirements, state unresolved risks, and deliver the packet for approval. Those steps are not equally difficult or independent. One wrong agreement version can corrupt several later conclusions.

An end-to-end acceptance check can require:

• the agreement version and clause locator;
• the current billing period and invoice source;
• the renewal date and notice deadline;
• the business owner and approved requirements;
• current use and performance evidence;
• unresolved questions and their owners;
• the approval state and retained decision record.

Done when: a fresh reviewer can open the packet, trace every material term and cost to the source, reproduce the date calculation, see each unresolved exception, and confirm that no renewal, notice, purchase, cancellation, or signature occurred without approval.

IMRA, the production AI vendor manager, is built for that role. IMRA can replace manual vendor-register upkeep, renewal preparation, source reconciliation, and decision-packet drafting. The owner retains vendor selection, negotiation positions, spend, signatures, notices, and exits. Material contract risk stays with counsel.

The outcome guide explains why the accepted decision record matters more than message volume. The finishability question supplies the separate delivery test for one agreed job.

Production evidence needs a stop and repair path

Pre-release tests can find known failures. Production monitoring can find changed conditions and new exceptions. Neither is useful if the role continues past a consequential uncertainty.

The production-survival question goes deeper on change monitoring, containment, repair, and retest. The security page states the access and data boundary. The work-product promise states the current delivery remedy without turning it into a universal reliability claim.

What a vendor should show

Ask for evidence in this order:

1. The work product. What exact state exists at the end?
2. The acceptance rule. What can a fresh reviewer check?
3. The denominator. Does the score include every run started, including stops and timeouts?
4. The conditions. Which records, tools, users, model or system version, and date were tested?
5. The repeats. Does the same case pass more than once?
6. The stop rule. What prevents uncertain state from reaching a binding action?
7. The repair owner. Who corrects the normal method, reruns the case, and reports the result?

FidelicAI does not publish one fleet-wide completion or production-reliability rate. Current production roles publish their work, checks, integrations, limits, and approval boundary. The absence of a universal percentage is not proof of reliability. The role-specific record and accepted work remain the evidence the buyer can inspect.