Testing
Save conversations as tests, describe what a good reply must do, and run every test against your agent's current configuration to see what a prompt, model or tool change actually changed.
Testing lets you check how an agent performs across a set of scenarios in one go, before customers see a change. A saved test is a conversation that ends with a customer message, plus an expectation: a plain-language description of what a good next reply must do. Run All replays every saved test against the agent as it is configured right now and grades each reply.
Testing lives on the agent page, in the Testing group of the agent navigation: Saved Tests, Future Chats and Run History.
Who can use it
Testing follows the agent permissions. Builders, Admins and Owners can create tests and run them; the chat-only Member role cannot open agent pages. Saved tests are shared with everyone who can see the agent.
Saved tests
A saved test consists of:
- Conversation: 1 to 100 customer and agent messages that start and end with a customer message. The last message is the one the agent has to answer. Earlier messages only provide context and are never graded.
- Expectation (required): what a good reply must do, up to 8,000 characters. See Writing expectations.
- Tags (optional): up to 20 short labels for filtering, such as
hoursorreturns.
Each agent can hold up to 200 saved tests. There is no title; the list shows the customer message that is being tested.
Three ways to create a test
Write one
On Saved Tests, click Add Test, type the customer message and, if you need context, Add earlier messages to insert a customer and agent turn before it. The × next to an exchange removes both of its turns again, so the conversation keeps alternating.
Save a reply from Try Chat
Under any agent reply in Try Chat (Configure), click Save as Test. The conversation up to that customer message becomes the test, and the original reply is kept as reference material only.
Save a real customer conversation
In Chats, every agent reply has the same Save as Test control. The conversation branch that led to the reply is copied; the original reply and any visitor thumbs-up or thumbs-down are shown for reference. A conversation with attachments or one that starts with an agent greeting cannot be imported as is; write a text-only version that starts with a customer message instead.
The editor can draft an expectation and tags for you (Suggest an expectation, Auto). Both call a model and cost credits, and both are drafts you can edit before saving. Saving the same customer reply twice reuses the existing test.
Writing expectations
The expectation is sent only to the evaluator, never to the agent, so it does not change how the agent replies. It works best as a few concrete requirements:
| Kind of requirement | Example | How it is checked |
|---|---|---|
| Facts | "Say that we are closed on Saturday afternoon: on Saturday we are open 7:00 to 13:00." | Only against the fact you supply. "Give the correct opening hours" cannot be checked, because the evaluator has no other source of truth, and the result is Couldn't assess. |
| Behaviour and tone | "Be polite. Answer in the customer's language. Offer to confirm by phone." | By the evaluator, from the reply. |
| Length | "Keep the reply under 300 characters." | A plain whole-reply character limit like this is counted exactly. Limits that are scoped ("each sentence under 100 characters"), conditional ("if the customer asks") or turned into a judgement ("under 100 characters is too short") are left to the evaluator, which sees the exact character count. |
Include the reference facts
The evaluator does not know your business. If a test is about a fact, put the fact in the expectation. "Recommend Product A" checks that Product A is recommended; it cannot check features of Product A you did not state.
Running tests
The Run all tests block on Saved Tests sets how the run works:
- Platform: Website or Internal Chat. This selects the tools and behaviour the agent would have on that channel. Both are available on every paid plan, because a test deploys nothing.
- Model: any text model your organization may use. A new browser starts from the model and platform of the agent's first deployment.
- Tools: one switch per tool. Knowledge search, advanced knowledge browsing and web search are on by default when the agent has them. Connections and other action tools are off by default and marked Can perform real actions: when you switch one on, a test really calls it. Connections that need a personal login, and tools that need a live chat workspace (code execution, scheduled tasks), are unavailable in tests.
These settings are remembered in your browser per agent. They never change the agent itself.
Every test runs against the agent's current saved configuration: prompt, knowledge bases, tools and forms as they are at that moment, with organization connections only. No customer memories and no member identity are involved, so a run is the same for whoever starts it.
Click Run All to run every saved test, even when the list is filtered. Only one run per agent is active at a time. You can leave the page or refresh; the run continues and progress is shown in Run History. Stop after current tests cancels the tests that have not started and lets the ones in flight finish.
To try a single test, open it and click Try This Test. A preview uses the same settings, shows the reply and its assessment in the dialog, and does not add an entry to the history.
Forms in tests
When Form / Lead Capture is switched on for a test, the agent can fill and submit the form as it would with a customer, and the reply is graded on that. The submission itself is not stored and no notification e-mail is sent, so test runs never create leads.
Reading results
Each test ends with one of four outcomes:
| Outcome | Meaning |
|---|---|
| Passed | Every requirement in the expectation was met. |
| Failed | At least one requirement was not met. |
| Couldn't assess | The reply was generated, but at least one requirement could not be judged (usually a fact without a reference in the expectation) and nothing failed. |
| Execution error | No reply was generated: the run was stopped, the model did not answer in time, credits ran out, or the agent could not be loaded. |
Open a result to see the conversation, the expectation, the generated reply and the assessment per requirement, each with a short reason. The result also records the platform, model and tool switches that were used.
Passed, Failed and Couldn't assess are an AI assessment of the reply against your expectation. Treat a surprising verdict as a prompt to sharpen the expectation, not as ground truth.
Run History
Run History lists one entry per Run All, newest first, with the settings and the counts of passed, failed, unassessable and errored tests. Open a run for the per-test results. A run keeps the test inputs and expectations as they were when it started, so editing or deleting a saved test later does not rewrite history; the saved test's row shows Test edited when its latest result no longer matches its current content.
Future Chats
Future Chats predicts questions your customers are likely to ask next, from the agent's recent customer conversations. It offers up to ten editable drafts, each with a conversation, an expectation and tags. Expand a draft, edit it, tick the ones you want and click Save Selected to add them to Saved Tests. Prediction calls a model and costs credits; an agent without recent customer conversations has nothing to predict from.
What a test costs
A test run is billed like normal usage: the agent's reply and its assessment each cost credits, listed separately on the result. Test replies do not count as customer messages. Tools the agent calls during a test (web search, page reads, connections) are billed as they are in a chat.
What testing does not do
- It does not change the agent. There is no automatic prompt rewriting, model selection or "fix" button.
- It does not snapshot the configuration. Two runs of the same tests after a prompt edit compare the old and the new prompt, which is the point.
- It does not use customer memories or a member's personal connections.
- It does not undo real actions. An enabled action tool acts once per test and is not retried; running the tests again acts again.
Last updated on September 13, 2026