What I want to see before I trust an AI workflow

An AI workflow is not production-ready because it produces a good answer in a demo. I want to see what happens when it is wrong, uncertain, slow, expensive, or asked to make a decision outside its lane.

The basics are simple: examples, evals, logs, review points, fallback behavior, and a clear line between automation and human judgment.

The goal is not to remove people from the loop. The goal is to put people in the right part of the loop, with enough context to make the system trustworthy.

Start with the consequence of being wrong

The same model behavior can be acceptable in one workflow and unsafe in another. A rough internal summary can tolerate uncertainty that a customer-facing recommendation, financial action, access decision, or irreversible data change cannot.

Before discussing models or prompts, I classify the outcome:

• Is it advisory or does it take action?
• Can a person review it before anyone depends on it?
• Can the result be corrected or rolled back?
• What data does the workflow expose to a model or provider?
• Who is affected when the system is confidently wrong?
• What does a safe refusal or degraded mode look like?

Those answers determine the evaluation threshold, approval boundary, fallback, and evidence the system needs. Trust is not a model property. It is a property of the complete workflow around the model.

Evals need to represent the product contract

An evaluation set should reflect the behavior users rely on, not only examples that make the model look capable.

I want to see several kinds of evidence:

1. Normal cases: representative requests the workflow should handle consistently.
2. Boundary cases: incomplete, ambiguous, multilingual, long, or unusually structured inputs.
3. Known failures: examples collected from production, support, or previous model regressions.
4. Safety and authority cases: requests the system should refuse, escalate, or route to a person.
5. Operational cases: provider errors, timeouts, malformed responses, and missing context.

The scoring method depends on the output. Some behavior can be checked deterministically. Other outputs need rubric-based review, pairwise comparison, or a domain expert. The important part is that the pass condition is written before a candidate model or prompt is evaluated.

Otherwise the team can move the standard after seeing an attractive cost or latency result.

A model change is a release

Changing a model, prompt, retrieval source, tool, or output parser can change product behavior even when application code remains untouched. I treat that as a release with an explicit gate.

The gate should compare the candidate with a known baseline across:

• quality on the product evaluation set;
• critical-case regressions;
• provider and parsing errors;
• latency, including long-tail behavior;
• token or usage cost;
• compatibility with downstream consumers;
• rollback readiness.

The production AI reliability case study shows why these dimensions belong together. A faster and cheaper candidate was not accepted as a global replacement because it did not meet the quality contract. Cost improvement did not get to redefine success.

Human review should have a specific job

“Keep a human in the loop” is too vague to be an operating design. A reviewer needs to know what they are deciding, what evidence is available, and what happens after rejection.

Useful review points tend to sit where one of three things is true:

• the action is consequential or difficult to reverse;
• the system is uncertain or outside the evaluated distribution;
• policy, customer context, or professional judgment matters more than pattern matching.

The interface should expose the source material, model output, relevant confidence or validation signals, and the available actions. A review queue without context simply moves the failure from automation to an overloaded person.

Review should also create learning. Rejections and corrections can become evaluation examples, policy clarifications, or product signals. They should not disappear into an unsearchable manual process.

Observability must avoid collecting the wrong evidence

An AI workflow needs enough telemetry to diagnose quality, reliability, latency, and cost. It does not follow that every prompt and response should be copied into analytics.

I separate content from operating signals where possible. Useful content-free signals include:

• workflow and model version;
• provider and workload type;
• input and output size bands;
• latency and retry count;
• validation or evaluation result;
• fallback or human-review route;
• bounded usage and cost measures;
• error category without sensitive payloads.

When content is needed for evaluation or incident review, collection should have a defined purpose, access boundary, retention period, and redaction strategy. “We may need it later” is not a sufficient data policy.

Fallback behavior is part of the product

Providers fail, quotas change, networks time out, and model output occasionally violates its expected structure. A trustworthy workflow decides what users experience before those conditions occur.

Depending on the consequence, the safe fallback may be:

• retry within a strict time and attempt budget;
• use a validated baseline model;
• return a limited deterministic result;
• preserve the request for later processing;
• ask for clarification;
• route to human review;
• stop without taking the external action.

The fallback should preserve the product contract, not merely keep the technical request green. Silently returning a lower-quality answer can be worse than an explicit delay.

The minimum trust package

Before I would call an AI workflow production-ready, I want to see:

1. A stated outcome, authority boundary, and consequence model.
2. A representative evaluation set with a fixed pass condition.
3. Versioned model, prompt, tool, and parser changes.
4. A release gate that compares quality, failures, latency, and cost.
5. Explicit review and fallback paths.
6. Runtime evidence that can explain failures without indiscriminate data collection.
7. An owner who can stop, roll back, and improve the workflow.

The exact implementation can be small. What matters is that the workflow has an inspectable contract and a way to reject an improvement that is not actually better.

For the engineering process around AI-assisted changes, see How I use AI agents without outsourcing engineering judgment and a practical Codex and Claude Code workflow. Those notes focus on how AI helps build systems. This one focuses on what an AI-powered product itself must prove.