How to evaluate AI agents for customer service: Testing agents in pre-production

Evaluating AI agents for customer service comes down to one decision: whether the agent is ready for real customer traffic. Executives saw the agent resolve a clean call and sound natural doing it, and now leadership wants a launch date. Your contact center still has peak-hour queues, compliance scripts, authentication steps, and customers who interrupt because they are already frustrated.

Pre-production testing is the launch gate between an impressive pilot and those customers. A weak gate turns call volume into uncontrolled experimentation. A strong one produces evidence that the agent can act safely before customers become the test environment.

What is pre-production testing?

Pre-production testing is the structured evaluation phase between a working AI agent pilot and introducing live customer traffic. It validates behavior across representative scenarios, edge cases, adversarial inputs, and compliance requirements. It also records the agent's decisions, actions sent to connected systems, and handoffs across repeated runs. A demo proves the agent can handle a cooperative user on a clean path. Pre-production testing measures performance when customers interrupt, change direction, or push against policy.

Two terms matter throughout:

• Containment: The contact was handled without routing to a human agent, which makes it a routing measure.

• Resolution: The customer's issue was confirmed solved, with no escalation or repeat contact within 24-48 hours.

The output is an evidence packet that lets business, technical, compliance, and risk owners judge whether the agent is safe to launch, with thresholds calibrated to the risk of each use case.

Why pre-production testing decides whether a pilot ever ships

Most generative AI pilots produce impressive demos and never survive production traffic. Pre-production testing supplies the missing piece: evidence that compliance owners and risk teams can accept. Several forces make it the deciding factor:

• Executive pressure without proof of value: Only 11% of customer service leaders say their generative AI investments (opens in a new tab) have met their primary business objective, even as pressure to deploy keeps climbing.

• Demos hide failure modes: A pilot shows the cooperative path. Structured testing shows what happens when a caller interrupts or switches intent.

• Governance requires an evidence packet: Each owner needs artifacts they can defend before signing off, and a demo environment produces none of them.

• Incidents end projects: Loose gates turn call volume into uncontrolled experimentation, and the resulting incidents get projects pulled.

Forrester's August 2026 research on trustworthy AI (opens in a new tab) reaches the same conclusion. Traditional testing cannot fully validate how agents use tools, follow multi-step tasks, or change over time, so teams need a continuous testing loop.

What changes when you test agents instead of scripts

Testing an AI agent differs from testing a scripted interactive voice response (IVR) flow, because the agent decides what to do instead of following a fixed menu. Four things change:

• Repeated runs replace single checks: The same input can produce different responses, so each scenario must pass across repeated runs.

• The path matters as much as the answer: Tests check the sequence of decisions and tool calls, such as verifying identity before processing a refund.

• Conversations replace single turns: Tests must catch an agent that loses context by turn seven.

• Back-office steps count: A refund issued but never logged in the compliance system is a failure, even when the customer is satisfied.

These measurements become a launch decision only once each flow has a threshold it must clear. The right threshold depends on what a wrong answer costs in that specific flow.

Setting risk-calibrated thresholds before go-live

There is no universal "good enough" score. A product FAQ agent can tolerate a higher error margin than a billing-dispute or claims agent, because a wrong answer about store hours and a wrong answer about a denied claim carry very different consequences. Business leaders set the threshold for each flow, and each threshold decides whether the flow ships or routes to a human agent. Four tips keep those thresholds meaningful.

Gate on resolution rate instead of containment

High containment can mask unsafe outcomes. Two agents can post the same containment score, but one resolves issues while the other sends customers away to call back. Set the readiness threshold on the share of contacts the agent resolves without a handoff, and add transcript review to confirm those resolutions are real. Leaders then get outcome evidence without suppressing the escalations customers still need.

Cap the hallucination rate by use-case risk

The hallucination ceiling sets the maximum acceptable rate of fabricated responses. Set it far lower for a billing dispute or an insurance claim than for a product FAQ, because one invented figure in those flows creates financial or regulatory exposure. Passing the gate means the agent stays inside approved knowledge before customers hear it.

Require complete context in every escalation

An escalation that drops context turns automation into a longer queue. Check that the agent hands off to the right human agent and that the handoff record carries:

• Transcript: The full conversation.

• Intent: The confirmed intent and any intent still unresolved.

• Entities: Details collected in the conversation, such as account number or authentication status.

• Actions: Steps the agent attempted and their outcomes.

• Escalation reason: The policy applied and the sentiment signal at the moment of transfer.

A handoff missing any of these forces the human agent to re-ask and the customer to repeat themselves.

Test authentication accuracy by channel and language

Identity-sensitive flows fail if the agent acts before it verifies who is speaking. Measure how reliably the agent completes that check before taking action, under sustained load and across every live use case. Run the test per channel and per language variant, because authentication accuracy and intent recognition can diverge across accents, background noise, and digital channels. A gate passed on one channel does not certify the others.

Building red-team and compliance testing into the gate

Functional quality assurance (QA) validates expected behavior with cooperative users. The production gate must also apply the pressure a clean demo never does. These tips build adversarial and compliance testing into the gate.

Attack the agent with prompt injection and jailbreaks

Test inputs that try to override the agent's instructions or push it to act beyond its authority. The Open Worldwide Application Security Project (OWASP) maps these tests to two controls (opens in a new tab): LLM01 for prompt injection and LLM06 for excessive agency, which limits an agent's authority to act. A passing agent ignores the injected instruction and stays inside its permitted actions.

Push for policy and disclosure violations

Write test cases that pressure the agent to skip a required disclosure, misstate a policy, or promise something the business does not offer. Include persistent callers who rephrase the same request several times. A passing agent holds the policy every time and explains it clearly, rather than giving in on the third attempt.

Confirm the agent declines and escalates cleanly

Some requests the agent must refuse. Test that it declines without inventing a reason and explains what it can do instead. When the user pushes beyond the happy path, it should escalate to a human agent with the required disclosure. A clean decline protects the customer relationship and the business.

Run industry-specific compliance scenarios

Build test cases around the regulations that apply to each vertical, because the same behavior can pass in one industry and fail in another.

Under the EU AI Act, Article 50 (opens in a new tab) transparency obligations have applied since 2 August 2026, so callers must be told they are speaking with an AI system at the first interaction. Systems on the market before that date have until 2 December 2026 to meet the marking duty for synthetic outputs.

Red-team the voice channel by phone

Deliver adversarial inputs by phone and in chat, because an attacker on a call behaves differently than one in a chat window. Confirm that the agent delivers required disclosures audibly, at the moment the customer needs them. A disclosure that exists only in the log is a disclosure a regulator will treat as absent.

Well-designed guardrails for AI agents are what make these tests passable. The results matter only when leaders with the right authority can accept or reject the residual risk.

Designing the sign-off sequence and drift baseline

Go-live is a sequenced, cross-functional decision, and each owner holds a veto over the evidence they control:

• CX operations: Reviews transcripts and escalation outcomes to confirm the agent meets contact center standards with impatient or confused customers.

• IT and engineering: Reviews integration and load results across the CRM, authentication provider, and compliance logging under simulated peak load.

• Legal and compliance: Reviews disclosure evidence and the regulatory obligations that govern the industry.

• Risk and finance: Decides whether the residual risk matches the expected return.

During testing, also capture model-drift baselines, the reference measurements used to detect post-launch behavior changes. For phone deployments, include intent recognition and authentication rates under simulated peak volume. Without a baseline, customer complaints reveal drift before your own instruments do.

How to monitor AI agents after go-live

Post-launch monitoring compares live performance against the pre-production baselines, so drift shows up on a dashboard before it reaches the complaint queue. These tips keep the launch gate working after go-live.

Compare every top intent against its launch baseline

Use near-real-time dashboards with week-over-week trend views to compare each high-volume intent with the baseline captured in testing. Model drift is harder to spot than a software crash because an action can work correctly ten times and fail on the eleventh without triggering an error. Trend views surface a slow decline before it becomes a pattern of complaints.

Review the highest-volume intents on a fixed cadence

Set a recurring review, such as a monthly session covering the three to five highest-volume intents. Bring CX operations, engineering, and compliance into the same review, so a failure routes straight to the owner who can fix it. A fixed cadence turns monitoring from an occasional check into an operating rhythm.

Evaluate offline before testing changes on live traffic

Run every proposed change against stored test cases first, then move to online A/B testing with a share of live traffic. Bayesian A/B tests use probability to compare agent configurations and show how likely one is to outperform the other. This two-stage design keeps untested changes away from most customers.

Audit automated scores with human labels

When an LLM acts as judge and scores agent responses, compare a sample of those scores with human labels on a regular schedule. Judges drift and carry their own biases, so unaudited scores can approve failing responses at scale. Recalibrate the judge when agreement drops, so automated evaluation stays trustworthy as volume grows.

Expand QA coverage to every conversation

Start with manual sampling if you must, then widen QA to AI-assisted review across every conversation as the operation matures. Full coverage catches rare intents and single-language failures that a sample misses. It also gives each veto owner continuing evidence instead of a one-time launch packet.

Governed thresholds hold up at real volume. Schwäbisch Hall's AI agent handled 500,000 calls in six months across 16 live use cases, with an authentication rate above 80% and 98% intent recognition accuracy.

How teams test AI agents with Parloa Performance Lab

A launch gate needs a testing environment that can run every scenario at production scale before a real customer calls. Parloa’s Performance Lab gives teams that environment, so they can:

• Generate coverage automatically: Build test coverage directly from the AI agent's configuration.

• Test every component individually: Check each intent, tool call, and agent handoff on its own.

• Blend real and synthetic data: Combine historical transcripts with synthetic tests across scenarios, languages, and channels.

• Keep humans in control: Hold every generated simulation and evaluation in draft until a reviewer approves it.

• Compare agent versions: Run identical tests against two agent configurations and see the results side by side, down to the individual conversation.

• Test fixes before they ship: Validate the fixes Parloa Navigator, an AI agent optimization copilot, recommends through simulated conversations before they are applied.

Every run produces evidence each veto owner can review, so the sign-off packet comes from the same tests that decide whether the agent ships.

Turn AI agent evaluation into governed launch evidence

Pre-production testing decides whether every future release reaches customers safely. A launch-ready team knows which thresholds block release, who owns each decision, and how it will catch drift once the agent is live.

Parloa's AI Agent Management Platform runs that process across the full AI agent lifecycle. Teams build with Agent Builder, test every intent, tool call, and handoff in Performance Lab, and track every production conversation with Parloa Lens. Parloa Navigator traces failures to the responsible configuration and proposes a fix.

Book a demo to give your first live conversation the best chance of solving the customer's problem without making them repeat themselves.

FAQs about pre-launch AI agent evaluation

How long does pre-production testing take?

It depends on the use case's risk and the number of scenarios, but a structured cycle typically spans several weeks. It covers validation, simulation, stress testing, and a monitored canary release that exposes a limited share of traffic first. Higher-risk flows warrant more time on threshold calibration and compliance testing.

How many test scenarios does an AI agent need before launch?

Cover every high-volume intent, the edge cases around each one, adversarial inputs, and each channel and language you plan to support. Run each scenario several times, because a single pass does not prove consistency. The highest-risk flows need the deepest coverage, since one failure there costs the most.

Can you use historical call transcripts to test AI agents?

Yes. Historical transcripts show how real customers phrase requests, interrupt, and change direction, which synthetic tests can miss. Remove personally identifiable information (PII) before using them, and blend them with synthetic scenarios to cover rare intents and adversarial cases that past calls never included.

What is the difference between AI agent testing and AI agent evaluation?

AI agent testing checks specific behaviors against expected results, such as whether the agent verifies identity before a refund. AI agent evaluation is a broader judgment of whether the agent is ready to launch, combining test results with risk thresholds, compliance review, and business outcomes into a single go-live decision.

How do you test AI agents for bias in customer service?

Run the same scenarios with variations in accent, dialect, language, age cues, and phrasing, then compare outcomes such as authentication success, resolution, and escalation rates. Any gap between groups points to bias. Fix it before launch, and keep tracking the same comparisons in production, where new callers can expose new gaps.