Production voice-agent QA checklist

Before evaluation

• Define the customer outcome for each supported use case.
• Choose representative calls, including failures and edge cases.
• Confirm data access, retention, and evaluator-provider policies.
• Document what pass, fail, and uncertain mean for each metric.

During review

• Inspect transcript and audio together when timing matters.
• Require rationales to reference specific conversation evidence.
• Escalate uncertain and high-impact decisions to a human.
• Separate agent behavior from provider or infrastructure failures.

After review

• Group recurring failures by likely root cause.
• Assign an owner and expected corrective action.
• Retest on both historical examples and new production calls.
• Track whether the failure rate changes after deployment.