Evaluation workflow
1
Create an evaluation suite
Create an evaluation suite manually or let Arch generate one based on your project. Review and customize the suite before running it.
2
Run evaluations
Run the evaluation suite to simulate conversations and measure your agent’s performance across the configured scenarios, personas, and evaluators.
3
Review Results
Review evaluation scores, conversations, traces, and execution details to identify issues and opportunities for improvement.
4
Repair and Optimize
Use the evaluation results to identify recommended improvements. Depending on your Arch configuration, you can review changes manually or allow Arch to apply eligible improvements automatically.
5
Validate Improvements
Run the evaluation suite again to verify that the applied changes improve the agent’s perfomance.
Key concepts
How Arch AI optimizes your agent
Arch AI helps you continuously improve your agents by analyzing evaluation results, generating repair recommendations, and validating improvements through repeated evaluation cycles.
Learn more: To see where Evaluations fit in the Arch AI lifecycle, see Arch AI.
Example: Improve tool call accuracy using Arch
Suppose an evaluation shows that your agent has a low Tool Call Accuracy score. Arch AI analyzes the evaluation results and conversation details to understand why the agent is failing. It may identify that the agent is selecting the correct tool but providing incorrect parameters because the instructions do not clearly specify the required inputs. Based on this analysis, Arch AI recommends an improvement to the agent’s instructions. After the change is reviewed and applied, the evaluation suite is run again to validate the improvement. If the agent now provides the correct parameters and the Tool Call Accuracy score improves, the evaluation cycle is complete. Learn more: For more information about the Arch AI reinforcement loop and continuous optimization, see Optimize with Arch AI.Create evaluation suites
Evaluation suites define how your project is evaluated. Each suite combines scenarios, personas, evaluators, and run settings to measure your agent’s performance. You can create an evaluation suite in one of the following ways:Create eval suite with Arch
Use Create with Arch to automatically generate an evaluation suite based on your project.- Go to Evaluate > Evals.
- Select Create with Arch.
- Wait for Arch to generate the evaluation suite.
- Review the generated evaluation suite.
- (Optional) Select Edit Configuration to customize the generated components.
- Select Create & Run Suite to create the suite and start the evaluation.
Arch analyzes your project and automatically generates the evaluation suite components, including scenarios, personas, evaluators, and run settings. You can review and modify the generated configuration before running the evaluation.
Create a RAG evaluation suite
Create a RAG evaluation suite to evaluate an agent’s retrieval and response quality using a CSV dataset of test questions. Unlike conversation evaluation suites, RAG suites evaluate individual questions against a single agent without using personas or scenarios. For detailed instructions, see RAG evaluation suites.Create eval suite manually
To create an evaluation suite manually, from the Evals page, select Create test suite or + New suite. Both options open the Create eval suite configuration screen. Configure the suite by defining its:- Basics — Specify the suite name, description, scope, and modality.
- Scenarios — Select or add the scenarios to evaluate.
- Personas — Select or add the personas that simulate users.
- Evaluators — Select evaluators and configure their weights and passing score.
- Arch autonomy — Select how much autonomy Arch has when applying improvements and set the maximum number of runs per loop.
- Run settings — Configure how conversations are generated and when the suite runs.
Evaluation suite components
Every evaluation suite, whether created manually or with Arch, contains the following components. Together, they define what is evaluated, how the evaluation is executed, and how the results are scored. When you create an evaluation suite with Arch, these components are generated automatically based on your project. You can review and modify them before running the evaluation.Basics
Use the Basics section to define what the evaluation suite covers and which project state it evaluates.- Suite name — Enter a name for the evaluation suite.
- What this suite covers — Describe the purpose or focus of the evaluation suite.
- Test scope — Choose whether to evaluate the entire project or specific agents.
- Version to test — Select the project version, deployment, or environment to evaluate. Depending on the selected scope, the available options may include the Working copy, an Environment, a specific Deployment, or a published Agent version.
Arch’s autonomy level
Arch’s autonomy level setting determines how much control Arch has to apply improvements and re-run evaluations when a run does not meet the passing score. You can change this setting at any time.- Autopilot — Arch fixes and re-runs the evaluation on its own. Arch patches the agent and keeps re-running until the score clears the target or the configured loop limit is reached.
- Copilot — Arch automatically applies low-risk fixes. Arch diagnoses regressions, applies low-risk changes, and stages higher-risk changes for approval.
- Advisor — Arch recommends fixes only. Arch explains what regressed and proposes changes without applying them.
Scenarios
Scenarios define the conversation flow, user intent, and expected outcomes used during evaluations. Each scenario represents a conversation flow used to evaluate how the agent handles specific tasks, behaviors, or outcomes. To create a scenario:- In the Scenarios section, select Add Scenario.
- Complete the scenario details.
- Select Create.
Scripted values let you define exact facts for the simulated user to provide when relevant, such as a PIN or account number. They are separate from Session Variables, which provide context to the agent. Scripted values are supported for text scenarios only.
For example, enter the following in Scripted values JSON:

Example scenario
Troubleshoot scenarios
Personas
Personas represent different types of users who interact with your agent. Each persona simulates unique communication styles, domain expertise, goals, behaviors, and constraints to help test how the agent performs across varied user interactions. To create a persona:- In the Personas section, select Add Persona.
-
Complete the persona details.
- Select an adversarial behavior type if you want to simulate edge cases or malicious interactions.
- Click Create.

Example persona
Adversarial persona types
You can simulate adversarial or edge-case user behaviors using the Adversarial Type field. To test agent safety and robustness:- Enable Adversarial while creating a persona.
- Select the adversarial type.
Troubleshoot personas
Evaluators
Evaluators define how conversations are assessed during an evaluation. Each evaluator measures a specific aspect of the conversation, such as response quality, safety, task completion, or compliance, and assigns a score based on the configured evaluation criteria. To configure an evaluator, follow these steps:- In the Evaluators section, select Add Evaluator.
- Enter the evaluator name and description.
- Select the Type and Category.
- Configure the evaluator based on the selected type.
- Select Create.
Lower evaluator temperatures typically produce more consistent scoring results.
Evaluator types
Supported evaluator types include:LLM Judge evaluators
An LLM Judge evaluator uses a separate LLM to assess the quality of agent responses based on a scoring rubric you define.
Write effective judge prompts
The judge prompt is one of the most important evaluator configurations. Well-defined prompts produce more consistent and reliable evaluation results. Effective judge prompts:- Clearly define evaluation criteria
- Focus on observable behavior
- Avoid ambiguous language
- Include examples when possible
Example judge prompt
Configure bias mitigation
LLM judges can exhibit scoring biases. Use bias mitigation settings to improve evaluation consistency and reliability.Trajectory evaluators
Trajectory evaluators assess the agent’s execution behavior rather than response quality. Use them to validate:- Milestone completion — did the conversation hit expected checkpoints?
- Handoff correctness — did the supervisor route to the right agent?
- Path efficiency — how many unnecessary steps did the agent take?
- Tool sequence — did the agent call tools in the right order?
Code Check evaluators
Use Code Check evaluators for deterministic validations that do not require an LLM. They use built-in scorers to validate agent responses and runtime behavior. Typical use cases include:- Regex matching
- Keyword validation
- Latency or response-time thresholds
- Structured output validation
A latency check passes when the measured time is within the Max ms budget and fails when it exceeds the budget. It returns a score of 1 when it passes and 0 when it fails. These measurements cover Runtime processing time; they do not include network time, evaluation queue time, or screen rendering.
Human Review evaluators
Use Human Review evaluators for subjective or manual quality assessments. Human Review evaluators flag conversations for manual inspection when evaluation scores fall below configured thresholds, allowing reviewers to validate agent behavior, response quality, or policy compliance before approval or release.A Human Review evaluator leaves a run provisional. Keep that suite out of automated baselines.
Scoring scale types
The scoring rubric defines how the evaluator assigns scores to conversations. It supports Likert and Binary scales.Likert scale
Use a 1 to 5 scale to define detailed evaluation criteria for each score level.Binary scale
Use pass or fail scoring for binary evaluation criteria.Troubleshoot evaluators
Configure run settings
Run settings determine how the evaluation suite is executed. To configure, follow these steps:- In the Run Settings section, configure the run settings.
- Select Create & Run Suite.
Run settings
The total number of conversations in an evaluation is calculated as: Scenarios × Personas × Variations
How evaluations are executed
During execution:- Every selected Persona interacts with every selected Scenario
- Each conversation is independently executed
- All configured Evaluators score the resulting conversations
View evaluation
After you create and run an evaluation suite, the Evals page displays all evaluation suites in your project. Use this page to monitor suite performance, review evaluation coverage, and open a suite to view detailed results. Each evaluation suite displays:- Behavior Score – Overall evaluation score for the latest run.
- Coverage – Number of scenarios, personas, variations, and generated conversations included in the evaluation.
- Evaluators – Number of evaluators configured for the suite.
- Last Run – Shows when the suite was last run and whether it runs manually or on a schedule.
Overview
Provides a high-level summary of the evaluation suite and its latest execution. Use it to monitor overall performance, review evaluation coverage, and identify areas that require attention.Alerts
Use the Alerts section on a suite’s Overview page to get notified when a watched run fails, falls below its target score, or regresses. Alerts apply to the suite as a whole; they do not identify a particular scenario, persona, or evaluator as the cause. To set up alerts for an evaluation suite, select Create alert to add a rule for that suite and configure the rule:
For regression alerts, the regression note shows the suite’s tolerance and links to the suite settings. When a watched run matches the rule, the alert appears in the project’s alert inbox with a reason, and the platform sends a notification through the configured delivery method.
Evaluation alerts support failed runs, below-target scores, and regressions. They do not currently alert on run cost or duration.
Latest run
Displays the results of the most recent execution of the evaluation suite. Use this page to:- Review the overall run score and evaluator results.
- Review the evaluator breakdown.
- View individual evaluation conversations and their results.
- Search and filter conversations.
- Retry conversations that encountered errors using Retry all errors.
History
The History tab displays previous evaluation runs for the suite. Use it to review how the suite’s results change across runs and understand the changes associated with each run. Includes information such as:- Run date and duration
- Changes associated with the run
- Behavior score and source metric
- Source of the run
- View conversations to review conversations from a specific run
Analyze evaluation results
After an evaluation completes, review the results to understand how your agent performed and identify opportunities for improvement. You can:- Review conversation transcripts.
- Analyze evaluator scores and reasoning.
- Inspect execution traces and tool usage.
- Compare expected and actual conversation outcomes.
- Identify patterns across successful and failed conversations.
Analyze conversations
Select a conversation from the Latest Run or History tab to review the evaluation details. For each conversation, you can inspect:View evaluator reasoning
When Chain-of-Thought Reasoning is enabled for an LLM Judge evaluator, the evaluation results include reasoning that explains how the score was determined. Use evaluator reasoning to identify improvements in:- Agent instructions
- Workflows
- Tool configuration
Act on results
Use the evaluation results to prioritize improvements to your agent.Compare runs and set a baseline
Studio has no Compare screen. The Arch MCPplatform_eval_runs compare operation returns evaluator-level averages for two complete, compatible runs. Open the run cases to see scenario-level results.
A baseline is a complete run that records the starting point of a stable suite. Set it as the suite’s baselineRunId through platform_eval_sets or the API. Comparison is refused when results are incomplete or the scenarios, evaluators, or variations changed. After you change the suite on purpose, create a new baseline.
The platform doesn’t measure how often an LLM Judge agrees with people. Before you use a judge as a release gate, hand-score 10 to 20 conversations and compare.
Troubleshoot
RAG evaluation suites
A RAG evaluation suite evaluates an agent’s ability to retrieve relevant information and generate accurate answers. Unlike conversation evaluation suites, which use personas and scenarios to simulate conversations, RAG evaluation suites use a CSV dataset of test questions to evaluate a single agent’s retrieval and response quality.Create a RAG evaluation suite
- Go to Evaluate > Evals and select the option to create a RAG evaluation suite.
- Under Basics, enter a suite name and, optionally, a description. Select the target agent and version.
- Under Evaluation data, upload a CSV file containing your test questions.
- Under Evaluators, weights & passing score, add the evaluators you want to use. Configure their weights and passing scores, and set the suite target score.
- Configure the run settings and select Create & run suite.
Prepare the evaluation dataset
Prepare a CSV file with the following columns. Column names must match the sample field names.
The agent can generate the answer and retrieve context when these fields are not provided. Supplied answers and contexts are used during evaluation instead of being generated.
CSV format requirements:
- Enter list fields (
actual_contexts,actual_context_ids, andexpected_context_ids) as JSON arrays of strings. For example,["chunk 1","chunk 2"]. - The first element represents retrieval rank 1.
- Values can contain commas, quotes, and line breaks.
- Leave an optional field empty if it is not provided for that row.
user_input column is required.
The platform uses the uploaded CSV as provided. It does not maintain a dataset library or version uploaded datasets. To use different evaluation data, upload a new CSV.
Dataset limits: A CSV can contain up to 10,000 rows and must not exceed 20 MB.
Configure evaluators and scoring
RAG evaluation supports evaluators from the LLM Judge and Code Check families.- LLM Judge: Context Precision, Context Recall, Faithfulness, Factual Correctness, and Noise Sensitivity.
- Code Check: Precision@K, Recall@K, and Mean Reciprocal Rank (MRR).
Run a RAG evaluation suite
A RAG evaluation run processes each CSV row as an individual sample. For each sample, the run uses the supplied answer and retrieved contexts when available. Otherwise, it runs the target agent to generate them. If a metric requires data that is unavailable for a sample, that metric is skipped for that sample, with a reason. The rest of the evaluation continues. You can run a suite manually or configure a recurring schedule. Run settings also include advanced execution options.Understand evaluation scores
Each evaluator produces a score for each sample. The evaluator’s passing score determines whether a sample passes or fails that evaluator. Evaluator weights determine how much each evaluator contributes to the overall sample score. If an evaluator is skipped for a sample, the remaining weights are adjusted for that sample. The run’s overall score is the average of the weighted sample scores. The suite target score determines whether the run meets the suite’s target. Most RAG metrics use a higher-is-better scoring direction. Noise Sensitivity uses a lower-is-better direction. Scores are normalized before aggregation so that higher values always represent better performance.Related articles: