Skip to main content
Evaluations provide a structured framework to test, measure, and improve your agents by simulating conversations, scoring responses, and analyzing the results. Create evaluation suites to test how your agent behaves across different scenarios, personas, and evaluators. Review the results to identify issues, validate changes, and continuously improve agent performance. The platform supports both AI-assisted and manual evaluation workflows. We recommend that you use Arch AI to automatically generate evaluation suites based on your project. You can review and customize the generated suite before running it. Navigation: Go to your project and select Evaluate > Evals.

Evaluation workflow

1

Create an evaluation suite

Create an evaluation suite manually or let Arch generate one based on your project. Review and customize the suite before running it.
2

Run evaluations

Run the evaluation suite to simulate conversations and measure your agent’s performance across the configured scenarios, personas, and evaluators.
3

Review Results

Review evaluation scores, conversations, traces, and execution details to identify issues and opportunities for improvement.
4

Repair and Optimize

Use the evaluation results to identify recommended improvements. Depending on your Arch configuration, you can review changes manually or allow Arch to apply eligible improvements automatically.
5

Validate Improvements

Run the evaluation suite again to verify that the applied changes improve the agent’s perfomance.

Key concepts

How Arch AI optimizes your agent

Arch AI helps you continuously improve your agents by analyzing evaluation results, generating repair recommendations, and validating improvements through repeated evaluation cycles. Learn more: To see where Evaluations fit in the Arch AI lifecycle, see Arch AI.

Example: Improve tool call accuracy using Arch

Suppose an evaluation shows that your agent has a low Tool Call Accuracy score. Arch AI analyzes the evaluation results and conversation details to understand why the agent is failing. It may identify that the agent is selecting the correct tool but providing incorrect parameters because the instructions do not clearly specify the required inputs. Based on this analysis, Arch AI recommends an improvement to the agent’s instructions. After the change is reviewed and applied, the evaluation suite is run again to validate the improvement. If the agent now provides the correct parameters and the Tool Call Accuracy score improves, the evaluation cycle is complete. Learn more: For more information about the Arch AI reinforcement loop and continuous optimization, see Optimize with Arch AI.

Create evaluation suites

Evaluation suites define how your project is evaluated. Each suite combines scenarios, personas, evaluators, and run settings to measure your agent’s performance. You can create an evaluation suite in one of the following ways:

Create eval suite with Arch

Use Create with Arch to automatically generate an evaluation suite based on your project.
  1. Go to Evaluate > Evals.
  2. Select Create with Arch.
  3. Wait for Arch to generate the evaluation suite.
  4. Review the generated evaluation suite.
  5. (Optional) Select Edit Configuration to customize the generated components.
  6. Select Create & Run Suite to create the suite and start the evaluation.
Arch analyzes your project and automatically generates the evaluation suite components, including scenarios, personas, evaluators, and run settings. You can review and modify the generated configuration before running the evaluation.

Create a RAG evaluation suite

Create a RAG evaluation suite to evaluate an agent’s retrieval and response quality using a CSV dataset of test questions. Unlike conversation evaluation suites, RAG suites evaluate individual questions against a single agent without using personas or scenarios. For detailed instructions, see RAG evaluation suites.

Create eval suite manually

To create an evaluation suite manually, from the Evals page, select Create test suite or + New suite. Both options open the Create eval suite configuration screen. Configure the suite by defining its:
  • Basics — Specify the suite name, description, scope, and modality.
  • Scenarios — Select or add the scenarios to evaluate.
  • Personas — Select or add the personas that simulate users.
  • Evaluators — Select evaluators and configure their weights and passing score.
  • Arch autonomy — Select how much autonomy Arch has when applying improvements and set the maximum number of runs per loop.
  • Run settings — Configure how conversations are generated and when the suite runs.
Select Create & run suite to create the evaluation suite and start the evaluation.

Evaluation suite components

Every evaluation suite, whether created manually or with Arch, contains the following components. Together, they define what is evaluated, how the evaluation is executed, and how the results are scored. When you create an evaluation suite with Arch, these components are generated automatically based on your project. You can review and modify them before running the evaluation.

Basics

Use the Basics section to define what the evaluation suite covers and which project state it evaluates.
  • Suite name — Enter a name for the evaluation suite.
  • What this suite covers — Describe the purpose or focus of the evaluation suite.
  • Test scope — Choose whether to evaluate the entire project or specific agents.
  • Version to test — Select the project version, deployment, or environment to evaluate. Depending on the selected scope, the available options may include the Working copy, an Environment, a specific Deployment, or a published Agent version.
Select Change scope to change the project or agent scope and configure the corresponding evaluation target.

Arch’s autonomy level

Arch’s autonomy level setting determines how much control Arch has to apply improvements and re-run evaluations when a run does not meet the passing score. You can change this setting at any time.
  • Autopilot — Arch fixes and re-runs the evaluation on its own. Arch patches the agent and keeps re-running until the score clears the target or the configured loop limit is reached.
  • Copilot — Arch automatically applies low-risk fixes. Arch diagnoses regressions, applies low-risk changes, and stages higher-risk changes for approval.
  • Advisor — Arch recommends fixes only. Arch explains what regressed and proposes changes without applying them.
Max runs per loop: Sets the maximum number of fix-and-rerun attempts Arch can make, even if the score has not cleared the target. This limits the number of attempts Arch can make in a single loop.

Scenarios

Scenarios define the conversation flow, user intent, and expected outcomes used during evaluations. Each scenario represents a conversation flow used to evaluate how the agent handles specific tasks, behaviors, or outcomes. To create a scenario:
  1. In the Scenarios section, select Add Scenario.
  2. Complete the scenario details.
  3. Select Create.
Judges score what can’t be checked exactly. Use Agent Path and Expected Milestones for everything that must be exact.
Scripted values let you define exact facts for the simulated user to provide when relevant, such as a PIN or account number. They are separate from Session Variables, which provide context to the agent. Scripted values are supported for text scenarios only. For example, enter the following in Scripted values JSON:
Evaluate

Example scenario

Troubleshoot scenarios


Personas

Personas represent different types of users who interact with your agent. Each persona simulates unique communication styles, domain expertise, goals, behaviors, and constraints to help test how the agent performs across varied user interactions. To create a persona:
  1. In the Personas section, select Add Persona.
  2. Complete the persona details.
  3. Select an adversarial behavior type if you want to simulate edge cases or malicious interactions.
  4. Click Create.
Evaluate

Example persona

Adversarial persona types

You can simulate adversarial or edge-case user behaviors using the Adversarial Type field. To test agent safety and robustness:
  1. Enable Adversarial while creating a persona.
  2. Select the adversarial type.

Troubleshoot personas


Evaluators

Evaluators define how conversations are assessed during an evaluation. Each evaluator measures a specific aspect of the conversation, such as response quality, safety, task completion, or compliance, and assigns a score based on the configured evaluation criteria. To configure an evaluator, follow these steps:
  1. In the Evaluators section, select Add Evaluator.
  2. Enter the evaluator name and description.
  3. Select the Type and Category.
  4. Configure the evaluator based on the selected type.
  5. Select Create.
Lower evaluator temperatures typically produce more consistent scoring results.

Evaluator types

Supported evaluator types include:

LLM Judge evaluators

An LLM Judge evaluator uses a separate LLM to assess the quality of agent responses based on a scoring rubric you define. Evaluate

Write effective judge prompts

The judge prompt is one of the most important evaluator configurations. Well-defined prompts produce more consistent and reliable evaluation results. Effective judge prompts:
  • Clearly define evaluation criteria
  • Focus on observable behavior
  • Avoid ambiguous language
  • Include examples when possible

Example judge prompt

Configure bias mitigation

LLM judges can exhibit scoring biases. Use bias mitigation settings to improve evaluation consistency and reliability.

Trajectory evaluators

Trajectory evaluators assess the agent’s execution behavior rather than response quality. Use them to validate:
  • Milestone completion — did the conversation hit expected checkpoints?
  • Handoff correctness — did the supervisor route to the right agent?
  • Path efficiency — how many unnecessary steps did the agent take?
  • Tool sequence — did the agent call tools in the right order?

Code Check evaluators

Use Code Check evaluators for deterministic validations that do not require an LLM. They use built-in scorers to validate agent responses and runtime behavior. Typical use cases include:
  • Regex matching
  • Keyword validation
  • Latency or response-time thresholds
  • Structured output validation
For text-agent latency, select one of these built-in scorers and set its Max ms budget: A latency check passes when the measured time is within the Max ms budget and fails when it exceeds the budget. It returns a score of 1 when it passes and 0 when it fails. These measurements cover Runtime processing time; they do not include network time, evaluation queue time, or screen rendering.

Human Review evaluators

Use Human Review evaluators for subjective or manual quality assessments. Human Review evaluators flag conversations for manual inspection when evaluation scores fall below configured thresholds, allowing reviewers to validate agent behavior, response quality, or policy compliance before approval or release.
A Human Review evaluator leaves a run provisional. Keep that suite out of automated baselines.

Scoring scale types

The scoring rubric defines how the evaluator assigns scores to conversations. It supports Likert and Binary scales.

Likert scale

Use a 1 to 5 scale to define detailed evaluation criteria for each score level.

Binary scale

Use pass or fail scoring for binary evaluation criteria.

Troubleshoot evaluators


Configure run settings

Run settings determine how the evaluation suite is executed. To configure, follow these steps:
  1. In the Run Settings section, configure the run settings.
  2. Select Create & Run Suite.

Run settings

The total number of conversations in an evaluation is calculated as: Scenarios × Personas × Variations

How evaluations are executed

During execution:
  • Every selected Persona interacts with every selected Scenario
  • Each conversation is independently executed
  • All configured Evaluators score the resulting conversations
This creates a full evaluation matrix across personas, scenarios, and evaluators. In text evaluations, each repeated message from the simulated user reaches the agent as a separate turn, even when the message is identical to the previous one. Example Evaluation Matrix
Each conversation is executed as an independent multi-turn session where the persona LLM simulates the user according to the scenario definition.

View evaluation

After you create and run an evaluation suite, the Evals page displays all evaluation suites in your project. Use this page to monitor suite performance, review evaluation coverage, and open a suite to view detailed results. Each evaluation suite displays:
  • Behavior Score – Overall evaluation score for the latest run.
  • Coverage – Number of scenarios, personas, variations, and generated conversations included in the evaluation.
  • Evaluators – Number of evaluators configured for the suite.
  • Last Run – Shows when the suite was last run and whether it runs manually or on a schedule.
Select an evaluation suite to view its detailed results. The evaluation suite page includes the following tabs: Overview, Latest run, and History.

Overview

Provides a high-level summary of the evaluation suite and its latest execution. Use it to monitor overall performance, review evaluation coverage, and identify areas that require attention.

Alerts

Use the Alerts section on a suite’s Overview page to get notified when a watched run fails, falls below its target score, or regresses. Alerts apply to the suite as a whole; they do not identify a particular scenario, persona, or evaluator as the cause. To set up alerts for an evaluation suite, select Create alert to add a rule for that suite and configure the rule: For regression alerts, the regression note shows the suite’s tolerance and links to the suite settings. When a watched run matches the rule, the alert appears in the project’s alert inbox with a reason, and the platform sends a notification through the configured delivery method. Evaluation alerts support failed runs, below-target scores, and regressions. They do not currently alert on run cost or duration.

Latest run

Displays the results of the most recent execution of the evaluation suite. Use this page to:
  • Review the overall run score and evaluator results.
  • Review the evaluator breakdown.
  • View individual evaluation conversations and their results.
  • Search and filter conversations.
  • Retry conversations that encountered errors using Retry all errors.

History

The History tab displays previous evaluation runs for the suite. Use it to review how the suite’s results change across runs and understand the changes associated with each run. Includes information such as:
  • Run date and duration
  • Changes associated with the run
  • Behavior score and source metric
  • Source of the run
  • View conversations to review conversations from a specific run

Analyze evaluation results

After an evaluation completes, review the results to understand how your agent performed and identify opportunities for improvement. You can:
  • Review conversation transcripts.
  • Analyze evaluator scores and reasoning.
  • Inspect execution traces and tool usage.
  • Compare expected and actual conversation outcomes.
  • Identify patterns across successful and failed conversations.

Analyze conversations

Select a conversation from the Latest Run or History tab to review the evaluation details. For each conversation, you can inspect:

View evaluator reasoning

When Chain-of-Thought Reasoning is enabled for an LLM Judge evaluator, the evaluation results include reasoning that explains how the score was determined. Use evaluator reasoning to identify improvements in:
  • Agent instructions
  • Workflows
  • Tool configuration
An example evaluator reasoning:

Act on results

Use the evaluation results to prioritize improvements to your agent.

Compare runs and set a baseline

Studio has no Compare screen. The Arch MCP platform_eval_runs compare operation returns evaluator-level averages for two complete, compatible runs. Open the run cases to see scenario-level results. A baseline is a complete run that records the starting point of a stable suite. Set it as the suite’s baselineRunId through platform_eval_sets or the API. Comparison is refused when results are incomplete or the scenarios, evaluators, or variations changed. After you change the suite on purpose, create a new baseline. The platform doesn’t measure how often an LLM Judge agrees with people. Before you use a judge as a release gate, hand-score 10 to 20 conversations and compare.
For baselines, judge calibration, and CI gates, see Evaluate agents.

Troubleshoot


RAG evaluation suites

A RAG evaluation suite evaluates an agent’s ability to retrieve relevant information and generate accurate answers. Unlike conversation evaluation suites, which use personas and scenarios to simulate conversations, RAG evaluation suites use a CSV dataset of test questions to evaluate a single agent’s retrieval and response quality.

Create a RAG evaluation suite

  1. Go to Evaluate > Evals and select the option to create a RAG evaluation suite.
  2. Under Basics, enter a suite name and, optionally, a description. Select the target agent and version.
  3. Under Evaluation data, upload a CSV file containing your test questions.
  4. Under Evaluators, weights & passing score, add the evaluators you want to use. Configure their weights and passing scores, and set the suite target score.
  5. Configure the run settings and select Create & run suite.
A RAG suite evaluates individual questions against the selected agent. It does not use personas, scenarios, or conversation simulations.

Prepare the evaluation dataset

Prepare a CSV file with the following columns. Column names must match the sample field names. The agent can generate the answer and retrieve context when these fields are not provided. Supplied answers and contexts are used during evaluation instead of being generated. CSV format requirements:
  • Enter list fields (actual_contexts, actual_context_ids, and expected_context_ids) as JSON arrays of strings. For example, ["chunk 1","chunk 2"].
  • The first element represents retrieval rank 1.
  • Values can contain commas, quotes, and line breaks.
  • Leave an optional field empty if it is not provided for that row.
The uploaded CSV is checked against the specified format before it is used. The user_input column is required. The platform uses the uploaded CSV as provided. It does not maintain a dataset library or version uploaded datasets. To use different evaluation data, upload a new CSV. Dataset limits: A CSV can contain up to 10,000 rows and must not exceed 20 MB.

Configure evaluators and scoring

RAG evaluation supports evaluators from the LLM Judge and Code Check families.
  • LLM Judge: Context Precision, Context Recall, Faithfulness, Factual Correctness, and Noise Sensitivity.
  • Code Check: Precision@K, Recall@K, and Mean Reciprocal Rank (MRR).
Evaluator availability depends on the fields provided in the CSV and the fields the agent can generate. Evaluators that cannot be used with the available data are unavailable, with a reason. Configure a weight and passing score for each evaluator. The evaluator weights must total 100. You can distribute the weights equally. The suite target score defines the overall score the suite is expected to achieve. It is different from the passing score configured for each evaluator. Evaluator limit: You can add up to 12 evaluators to a suite.

Run a RAG evaluation suite

A RAG evaluation run processes each CSV row as an individual sample. For each sample, the run uses the supplied answer and retrieved contexts when available. Otherwise, it runs the target agent to generate them. If a metric requires data that is unavailable for a sample, that metric is skipped for that sample, with a reason. The rest of the evaluation continues. You can run a suite manually or configure a recurring schedule. Run settings also include advanced execution options.

Understand evaluation scores

Each evaluator produces a score for each sample. The evaluator’s passing score determines whether a sample passes or fails that evaluator. Evaluator weights determine how much each evaluator contributes to the overall sample score. If an evaluator is skipped for a sample, the remaining weights are adjusted for that sample. The run’s overall score is the average of the weighted sample scores. The suite target score determines whether the run meets the suite’s target. Most RAG metrics use a higher-is-better scoring direction. Noise Sensitivity uses a lower-is-better direction. Scores are normalized before aggregation so that higher values always represent better performance.
Related articles: