Testing Agents and Teams
Test cases let you check, repeatably, that an agent or team still gives good answers after you change its prompt, model or tools.
Each test case sends a prompt to the agent or team. A separate judge model then scores the reply against your expected output using the metrics you choose. A run passes only if every selected metric passes.
Before you start
- You need a model the judge can use. By default the judge is the agent's own model; for a team, it's the first member that has a model. You can choose a different judge per test case.
- Optionally, set Settings → Testing → Default Test Case Model. This pre-fills the Model on new test cases. It's saved in your browser only, so it doesn't follow you to other devices.
Create a test case
- Open the agent or team and go to its Tests tab.
- Click Add Test Case.
- Fill in the fields described below.
- Click Create.
| Field | Required | Description |
|---|---|---|
| Test Name | Yes | A short name for the case. |
| Input Prompt | Yes | The message sent to the agent or team. |
| Expected Output | Yes | What a good answer contains. The judge compares the actual reply with this. |
| Context (one per line, optional) | No | Reference facts, one per line. Context-based metrics such as Faithfulness, Hallucination and the Contextual metrics use it. |
| Expected tool calls (optional) | No | The tools you expect the agent to call, in order. You can pick from the tools mounted on the agent or team, or type a name. These are scored by ToolCorrectness. Leave the field empty to judge only the answer. |
| Rubric (optional) | No | The criterion GEval grades the answer against, for example "Cites a source for every claim". Requires the GEval metric. |
| Model (optional) | No | The judge model. Use system default means the agent's own model. |
| Target Metrics | At least one | Toggle the metrics to score. See the table below. |
| Metric threshold | No | One per selected metric, from 0 to 1 (for example 0.7). Leave it empty to use the metric's default. |
Metrics
| Metric | Checks |
|---|---|
| AnswerRelevancy | The answer addresses the prompt |
| Faithfulness | The answer is consistent with the supplied context |
| ContextualPrecision, ContextualRecall, ContextualRelevancy | How well the supplied context supports the expected answer |
| Hallucination | The answer doesn't contradict the context |
| Bias, Toxicity | The answer is free of bias and toxic content |
| GEval | The answer meets your Rubric. With no rubric, it checks that the answer is factually correct against the expected output |
| Summarization | The answer is a good summary of the input |
| ToolCorrectness | The agent called the Expected tool calls |
Each metric gives a score from 0 to 1. A metric passes when its score reaches its threshold.
Run tests
- Run Test on a case runs that case.
- Run All at the top of the panel runs every case for the agent or team.
Runs happen in the background, and the panel updates while they're running.
Limits
- Generation can take up to 600 seconds and judging up to 180 seconds.
- A run with no result after about 14 minutes is marked failed.
Tools are never actually executed during a test. The agent or team runs normally, but every tool call is recorded and blocked. ToolCorrectness scores the calls the agent tried to make. A test can't send email, write to a database or change anything else.
Read the results
The header of the Tests tab shows counts: passing, failing, running and never run.
Expand a case to see:
- its Input Prompt, Expected Output, Expected Tool Calls, Rubric, Pass thresholds and Context;
- Latest Results, with one card per metric showing its score, pass or fail, and the judge's reason.
If a run failed outright, the error is shown. Common causes:
- the agent produced no output;
- a timeout;
- no usable judge model.
The panel shows only the latest run. Full run history is available through the API (GET /tests/test-cases/{id}/runs).
Good practice
- Cover the job, not just the happy path. Include an edge case and a request the agent should refuse.
- Use GEval with a rubric for open-ended output such as reports. Exact expected outputs rarely match word for word.
- Use a different judge model from the model under test when you can, to reduce self-grading bias.
- Rerun after every change. A result counts as stale once the agent or team has changed since it ran, or after 90 days.
- For high-risk agents, tests can gate promotion. Agents and teams classified as high or critical risk (or not classified at all) need current, passing tests before they can be promoted to another project. See Promoting Between Projects.
Testing through the API
| Method and path | Purpose |
|---|---|
GET /tests/metrics | List available metric names |
POST /tests/agents/{agent_id}/test-cases, GET the same path | Create or list an agent's cases |
POST /tests/teams/{team_id}/test-cases, GET the same path | Create or list a team's cases |
GET / PATCH / DELETE /tests/test-cases/{id} | Read, update or delete a case |
POST /tests/test-cases/{id}/run | Run one case. Returns immediately with status running |
POST /tests/agents/{agent_id}/test-cases/run, POST /tests/teams/{team_id}/test-cases/run | Run every case |
GET /tests/test-cases/{id}/runs | Run history, newest first |
GET /tests/coverage | Test coverage: which agents and teams are passing, failing, stale, never run or have no cases |
The Platform MCP server has matching tools, such as create_agent_test_case, run_test_case and get_test_case_runs.
Workflows have their own test nodes; see Testing Workflows.