Quality checks
Quality checks let AI judges score a sample of an agent’s real answers, so you can see answer quality over time and get an alert when it drops. They’re off unless you pick a set for an agent, because each scored check costs credits.
Create a set
Section titled “Create a set”Open Quality checks → New set:
| Setting | Meaning |
|---|---|
| What to check | Ready-made groups: Answer quality (helpful, correct, on topic), Safety (no harmful content or stereotyping), Tool use (right tool, right details, job done). |
| Your own criteria | One per line, in your own words - e.g. Mentions the 14-day return window when refunds come up. An AI judge checks each sampled answer against each line. |
| How many answers to check | From 5% to every answer. |
| Alert me below | Optional, 1-100. You’re notified when an agent’s average over the last day drops below it. |
The form shows an estimate of the cost per 100 answers and at your recent traffic.
Attach to agents
Section titled “Attach to agents”Pick the set under Quality checks in the Agent Builder. Editing a set later updates every agent using it, with no redeploy.
Read the results
Section titled “Read the results”Scores arrive a few minutes after each answer. On the agent’s page, Answer quality shows:
- the average (0-100, higher is better) over 7 or 30 days,
- each check’s average and its change from the period before,
- the latest scores with the judge’s one-line reason.
The Dashboard lists your agents’ quality, lowest first.
Each scored check costs credits - a judge that returns no score costs nothing. When your workspace runs out of credits or reaches its monthly limit, checks pause instead of running, and resume after a top-up. See Credits.
Test cases before you ship
Section titled “Test cases before you ship”Quality checks score real answers after the fact. Test cases check an agent before you share a change: questions it must answer well, asked of the live version whenever you want, and after every deploy if you turn that on.
On the agent’s page, open Test cases → Test case (up to 20 per agent). Each case is a question and what a good answer needs:
| Expectation | Passes when |
|---|---|
| Must contain | The answer includes your text. Case doesn’t matter, nor do Arabic diacritics, tatweel, alef forms (أ إ آ) or Arabic-Indic digits. |
| Must not contain | The answer doesn’t include your text (matched the same way). |
| Judged by a rubric | The quality checks’ AI judge rates the answer as met against what you wrote, e.g. Explains the 14-day return window politely. Partly met fails, with the judge’s reason. |
Run all shows an estimate first: the number of cases × a typical answer on the agent’s model, plus the judge’s rating for each rubric case. Each case is asked in a new conversation. You see pass or fail per case, the agent’s answer and why, and the score of each version, e.g. v3: 5/5 · v2: 4/5.
Turn on Run after each deploy to run them every time a new version goes live (an edit or a rollback). Nothing runs after an edit that was refused or failed.
Cost: each answer is charged like any request to the agent, and each rubric rating like a scored quality check. A rating the judge didn’t give costs nothing. Runs don’t start without credits or past your monthly limit, and a run that runs out part-way stops there.