Evaluating AI Outputs Like a BA: Acceptance Criteria for Text

In requirements work, we rarely accept a vague document because it “sounded smart.” We ask whether it is complete for the audience, consistent with known facts, and clear enough to act on. AI-generated text deserves the same treatment. If you only rate outputs on vibe, you will ship fluent junk and call it progress.

I am not a model researcher. I am someone who has spent years writing acceptance criteria and sitting in review meetings. That habit transfers surprisingly well to LLM output.

Treat the output as a deliverable with a job

Start by naming what the text is for. A client email draft is not the same job as a policy summary, a requirements sketch, or a support reply. The job drives the criteria.

Write a one-line purpose: “Draft a reply that answers the customer’s three questions without promising refunds we cannot give.” Then evaluation has something to grip.

This is ordinary BA thinking: actor, goal, constraints, success. Those skills map to AI work more cleanly than a lot of teams expect.

Completeness: did it cover what the ask required?

Completeness is not “long.” It is coverage of required points.

• Were all requested sections or questions addressed?
• Are mandatory fields present (dates, owners, amounts, next steps)?
• Did it omit known constraints that a human SME would never skip?

A short answer that hits every required item often beats a long answer that wanders. Build a checklist from the prompt or ticket, then tick it. If three of five required points are missing, the output fails, even if the prose is smooth.

Consistency: does it fight itself or the source of truth?

Inconsistency shows up in two ways. Internal contradiction (the draft says both “approve” and “deny”). External mismatch (the draft invents a fee that is not in the price list).

Practical checks:

• Scan for conflicting numbers, dates, and recommendations.
• Compare claims against the documents or systems you trust.
• Flag hedging that hides a decision the process needs made explicitly.

When company knowledge is retrieved first, consistency gets easier to check because you have sources to compare. That is one reason grounded retrieval often beats free-form chat for operational text. Data quality still matters; garbage in still produces confident garbage out, as data-first AI work keeps showing.

Sourceability: can a reviewer find where a claim came from?

For operational and client-facing work, “sounds right” is not enough. Ask: if challenged, could someone point to a policy, ticket, or record?

Useful standards:

• Factual claims should map to a cited document, system field, or known process rule.
• If the model is guessing, it should say so, or the workflow should block send.
• Quotes and figures should be copy-checkable, not paraphrased into something new.

Sourceability is also a design choice. If your workflow never attaches sources, reviewers will only have gut feel. Gut feel does not scale.

Definition of done for text

Borrow from delivery discipline. A piece of AI text is done when:

1. It meets the completeness checklist for that job.
2. It has no unresolved contradictions against sources of truth.
3. Claims that need sources have them, or are marked as unverified.
4. Tone and audience fit the channel (internal note vs customer email).
5. A named human has reviewed it when risk requires review.
6. It is stored or sent through the approved path, not left in a personal chat window.

Write fail criteria too. Examples: invents a policy ID, promises a date not in the system, omits required disclaimer, uses data class not allowed in that tool.

How to run this without turning review into theater

Keep evaluation sets small and real. A short list of golden examples from past tickets often beats a huge synthetic suite you never maintain. Re-run them when prompts or models change. Score with the same rubric each time so “better” means something.

Involve the people who own the process. They already know what a bad answer looks like. Your job is to turn that instinct into checkboxes and examples, not to invent academic metrics nobody will use.

Practical takeaway: stop accepting AI text because it reads well. Accept it when it meets job-specific criteria for completeness, consistency, and sourceability, under a definition of done a peer could apply without you in the room. That is BA discipline applied to a new kind of deliverable.