Evaluations - Kore.ai Docs

Evaluations provide a structured framework to test, measure, and improve your agents by simulating conversations, scoring responses, and analyzing the results. Create evaluation suites to test how your agent behaves across different scenarios, personas, and evaluators. Review the results to identify issues, validate changes, and continuously improve agent performance. The platform supports both AI-assisted and manual evaluation workflows. It is recommended to use Arch AI to automatically generate evaluation suites based on your project. You can review and customize the generated suite before running it. Navigation: Go to your project and select Evaluate > Evals.

Evaluation workflow

Create an evaluation suite

Create an evaluation suite manually or let Arch generate one based on your project. Review and customize the suite before running it.

Run evaluations

Run the evaluation suite to simulate conversations and measure your agent’s performance across the configured scenarios, personas, and evaluators.

Review Results

Review evaluation scores, conversations, traces, and execution details to identify issues and opportunities for improvement.

Repair and Optimize

Use the evaluation results to identify recommended improvements. Depending on your Arch configuration, you can review changes manually or allow Arch to apply eligible improvements automatically.

Validate Improvements

Run the evaluation suite again to verify that the applied changes improve the agent’s perfomance.

Key concepts

Concept	Description
Evaluation Suite	A collection of scenarios, personas, evaluators, and run settings that defines how an agent is evaluated.
Eval Library	Stores reusable personas, scenarios, and evaluators that can be shared across multiple evaluation suites within a project.
Arch AI	Generates evaluation suites, analyzes evaluation results, and recommends improvements as part of the evaluation workflow.

How Arch AI optimizes your agent

Arch AI helps you continuously improve your agents by analyzing evaluation results, generating repair recommendations, and validating improvements through repeated evaluation cycles.

Stage	What Happens
Evaluate	Run the evaluation suite to measure your agent’s performance across the configured scenarios and personas.
Analyze	Arch AI analyzes evaluation scores, conversation traces, and execution details to identify failed or low-scoring conversations.
Recommend	Arch AI generates recommendations to improve prompts, workflows, tool usage, or agent behavior based on the identified issues.
Apply	Review and apply recommended changes based on your configured Arch autonomy level.
Validate	Re-run the evaluation suite to verify that the applied changes improve the evaluation results.

Learn more: To see where Evaluations fit in the Arch AI lifecycle, see Arch AI.

Example: Improve tool call accuracy using Arch

Suppose an evaluation shows that your agent has a low Tool Call Accuracy score. Arch AI analyzes the evaluation results and conversation details to understand why the agent is failing. It may identify that the agent is selecting the correct tool but providing incorrect parameters because the instructions do not clearly specify the required inputs. Based on this analysis, Arch AI recommends an improvement to the agent’s instructions. After the change is reviewed and applied, the evaluation suite is run again to validate the improvement. If the agent now provides the correct parameters and the Tool Call Accuracy score improves, the evaluation cycle is complete. Learn more: For more information about the Arch AI reinforcement loop and continuous optimization, see Optimize with Arch AI.

Create evaluation suites

Evaluation suites define how your project is evaluated. Each suite combines scenarios, personas, evaluators, and run settings to measure your agent’s performance. You can create an evaluation suite in one of the following ways:

Option	Description
Create with Arch (Recommended)	Let Arch generate an evaluation suite based on your project.
Create Test Suite	Create and configure an evaluation suite manually.

Create eval suite with Arch

Use Create with Arch to automatically generate an evaluation suite based on your project.

Go to Evaluate > Evals.
Select Create with Arch.
Wait for Arch to generate the evaluation suite.
Review the generated evaluation suite.
(Optional) Select Edit Configuration to customize the generated components.
Select Create & Run Suite to create the suite and start the evaluation.

Arch analyzes your project and automatically generates the evaluation suite components, including scenarios, personas, evaluators, and run settings. You can review and modify the generated configuration before running the evaluation.

Create eval suite manually

To create an evaluation suite manually, from the Evals page, select Create test suite or + New suite. Both options open the Create eval suite configuration screen. Configure the suite by defining its:

Basics — Specify the suite name, description, scope, and modality.
Scenarios — Select or add the scenarios to evaluate.
Personas — Select or add the personas that simulate users.
Evaluators — Select evaluators and configure their weights and passing score.
Arch autonomy — Select how much autonomy Arch has when applying improvements and set the maximum number of runs per loop.
Run settings — Configure how conversations are generated and when the suite runs.

Select Create & run suite to create the evaluation suite and start the evaluation.

Evaluation suite components

Every evaluation suite, whether created manually or with Arch, contains the following components. Together, they define what is evaluated, how the evaluation is executed, and how the results are scored. When you create an evaluation suite with Arch, these components are generated automatically based on your project. You can review and modify them before running the evaluation.

Section	Description
Scenarios	Defines the user tasks or conversations used to evaluate your agent.
Personas	Defines the characteristics and behavior of users participating in the evaluation.
Evaluators	Defines the metrics and criteria used to score each conversation.
Run Settings	Configures how the evaluation is executed, including conversation variations and execution options.

Basics

Use the Basics section to define what the evaluation suite covers and which project state it evaluates.

Suite name — Enter a name for the evaluation suite.
What this suite covers — Describe the purpose or focus of the evaluation suite.
Test scope — Choose whether to evaluate the entire project or specific agents.
Version to test — Select the project version, deployment, or environment to evaluate. Depending on the selected scope, the available options may include the Working copy, an Environment, a specific Deployment, or a published Agent version.

Select Change scope to change the project or agent scope and configure the corresponding evaluation target.

Arch’s autonomy level

Arch’s autonomy level setting determines how much control Arch has to apply improvements and re-run evaluations when a run does not meet the passing score. You can change this setting at any time.

Autopilot — Arch fixes and re-runs the evaluation on its own. Arch patches the agent and keeps re-running until the score clears the target or the configured loop limit is reached.
Copilot — Arch automatically applies low-risk fixes. Arch diagnoses regressions, applies low-risk changes, and stages higher-risk changes for approval.
Advisor — Arch recommends fixes only. Arch explains what regressed and proposes changes without applying them.

Max runs per loop: Sets the maximum number of fix-and-rerun attempts Arch can make, even if the score has not cleared the target. This limits the number of attempts Arch can make in a single loop.

Scenarios

Scenarios define the conversation flow, user intent, and expected outcomes used during evaluations. Each scenario represents a conversation flow used to evaluate how the agent handles specific tasks, behaviors, or outcomes. To create a scenario:

In the Scenarios section, select Add Scenario.
Complete the scenario details.
Select Create.

Field	Description
Name	Unique name for the scenario.
Description	Brief description of what the scenario tests.
Category	Logical grouping used to organize scenarios, such as Billing, Customer Support, Technical, or Onboarding.
Difficulty	Complexity level of the scenario: Easy, Medium, or Hard.
Entry Agent	Agent that starts the conversation. Primarily used in multi-agent projects.
Initial Message	First user message that starts the conversation.
Max Turns	Maximum number of conversation turns before the evaluation stops.
Agent Path	Expected sequence of agent handoffs during the conversation. Applicable to multi-agent projects.
Expected Milestones	Key checkpoints the conversation should achieve.
Expected Outcome	Describes what a successful conversation should accomplish.
Tags	Labels used to organize and filter scenarios by feature area, regression suite, priority, or other categories.

Example scenario

SCENARIO:
  name: "Flight Rebooking After Cancellation"
  category: booking
  difficulty: medium

  initial_message: >
    My flight was cancelled and I need to rebook for tomorrow.

  expected_outcome: >
    Agent identifies the cancelled booking, offers alternatives,
    and confirms a new flight.

  max_turns: 15

  expected_milestones:
    - "Identify cancelled flight"
    - "Present rebooking options"
    - "Confirm new booking"

  agent_path:
    - "Supervisor"
    - "Booking_Manager"

Troubleshoot scenarios

Issue	Recommendation
Duplicate name error	Persona and scenario names must be unique within a project. Use a more specific name or delete the existing one.
Persona not behaving as expected in evals	Refine the Goals and Constraints fields. These are used as system prompt instructions for the simulated user LLM.
Scenario timing out	Increase the Max Turns value or simplify the expected conversation path.

Personas

Personas represent different types of users who interact with your agent. Each persona simulates unique communication styles, domain expertise, goals, behaviors, and constraints to help test how the agent performs across varied user interactions. To create a persona:

In the Personas section, select Add Persona.

Complete the persona details.

Field	Description
Name	Unique name for the persona.
Description	Brief description of the persona and the type of user it represents.
Communication Style	Defines how the persona communicates, for example, casual, terse, or formal.
Domain Knowledge	Defines the persona’s familiarity with the subject, such as beginner, intermediate, or expert.
Behavior Traits	Characteristics that influence how the persona behaves during the conversation. For example, asks follow-up questions, polite, or impatient.
Goals	Defines what the persona is trying to accomplish during the interaction.
Constraints	Rules or limitations that influence the persona’s behavior during the conversation.
Adversarial Type	Specifies whether the persona behaves as a normal user or intentionally challenges the agent to test its robustness.
Session Variables (JSON)	Optional session variables passed to the conversation at runtime.

Select an adversarial behavior type if you want to simulate edge cases or malicious interactions.
Click Create.

Example persona

PERSONA:
  name: "Impatient Business Traveler"
  communication_style: terse
  domain_knowledge: expert
  behavior_traits:
    - impatient
    - asks_follow_up_questions
  goal: "Rebook a cancelled flight quickly"
  constraint: "Avoid unnecessary conversation"

Adversarial persona types

You can simulate adversarial or edge-case user behaviors using the Adversarial Type field. To test agent safety and robustness:

Enable Adversarial while creating a persona.
Select the adversarial type.

Type	Purpose
Prompt Injection	Attempts to override agent instructions
Social Engineering	Attempts to extract sensitive information
Off-topic Derailer	Redirects conversations away from the intended agent goal
Abusive User	Uses hostile or inappropriate language
Edge Case Explorer	Sends unusual or unexpected inputs (empty, very long messages)

Troubleshoot personas

Issue	Recommendation
The persona’s behavior is inconsistent	Refine the goals and constraints fields
The persona’s responses are unrealistic	Add more specific behavioral traits and communication styles
The persona is too passive or aggressive	Adjust goals, constraints, and adversarial settings

Evaluators

Evaluators define how conversations are assessed during an evaluation. Each evaluator measures a specific aspect of the conversation, such as response quality, safety, task completion, or compliance, and assigns a score based on the configured evaluation criteria. To configure an evaluator, follow these steps:

In the Evaluators section, select Add Evaluator.
Enter the evaluator name and description.
Select the Type and Category.
Configure the evaluator based on the selected type.
Select Create.

Lower evaluator temperatures typically produce more consistent scoring results.

Evaluator types

Supported evaluator types include:

Type	Description
LLM Judge	Uses an LLM to evaluate conversations.
Code Scorer	Uses deterministic programmatic scoring.
Trajectory	Evaluates conversation flow and milestones.
Human Review	Flags conversations for manual review.

LLM Judge evaluators

An LLM Judge evaluator uses a separate LLM to assess the quality of agent responses based on a scoring rubric you define.

Field	Description
Judge Model	Language model used to evaluate the conversation.
Temperature	Controls the randomness of the judge model’s responses. Lower values produce more consistent evaluations.
Judge Prompt	Instructions that define the evaluation criteria for the judge model.
Chain-of-Thought Reasoning	Enables the judge model to perform intermediate reasoning before assigning a score.
Scale Type	Specifies the scoring method: Pass/Fail or 1–5 Scale.
Bias Mitigation	Applies techniques to help reduce bias and improve evaluation consistency.

Write effective judge prompts

The judge prompt is one of the most important evaluator configurations. Well-defined prompts produce more consistent and reliable evaluation results. Effective judge prompts:

Clearly define evaluation criteria
Focus on observable behavior
Avoid ambiguous language
Include examples when possible

Example judge prompt

JUDGE_PROMPT:
  You are evaluating an AI agent's response quality in a customer support context.

  Evaluate the conversation on these criteria:
    1. Did the agent correctly identify the customer's intent?
    2. Did the agent provide accurate information?
    3. Did the agent follow the expected conversation flow?
    4. Was the agent's tone appropriate and professional?

  Score each conversation using the provided rubric.

  Focus on the agent's responses, not the simulated user's messages.

Configure bias mitigation

LLM judges can exhibit scoring biases. Use bias mitigation settings to improve evaluation consistency and reliability.

Setting	Description	Default
Position Swap	Evaluates the conversation in both original and reversed order to reduce positional bias.	On
Blind Evaluation	Removes agent or persona identity information before judging.	On
Cross-Model Judge	Uses a different model family than the agent being evaluated.	Off
Evidence-First (RULERS)	Requires the judge to cite evidence before assigning scores.	On

Trajectory evaluators

Trajectory evaluators assess the agent’s execution behavior rather than response quality. Use them to validate:

Milestone completion — did the conversation hit expected checkpoints?
Handoff correctness — did the supervisor route to the right agent?
Path efficiency — how many unnecessary steps did the agent take?
Tool sequence — did the agent call tools in the right order?

Code Scorer evaluators

Use Code Scorer evaluators for deterministic validations that do not require an LLM. Typical use cases include:

Regex matching
Keyword validation
Latency or response-time thresholds
Structured output validation

Code Scorer evaluators execute custom scoring logic to validate agent responses and runtime behavior using deterministic rules.

Human Review evaluators

Use Human Review evaluators for subjective or manual quality assessments. Human Review evaluators flag conversations for manual inspection when evaluation scores fall below configured thresholds, allowing reviewers to validate agent behavior, response quality, or policy compliance before approval or release.

Scoring scale types

The scoring rubric defines how the evaluator assigns scores to conversations. It supports Likert and Binary scales.

Likert scale

Use a 1 to 5 scale to define detailed evaluation criteria for each score level.

Score	Description
5 - Excellent	Addresses the user’s request with accurate and complete information.
4 - Good	Addresses the request with minor omissions.
3 - Adequate	Partially addresses the request but misses important details.
2 - Poor	Mostly misses the request or provides inaccurate information.
1 - Failing	Fails to address the request or provides harmful information.

Binary scale

Use pass or fail scoring for binary evaluation criteria.

Score	Description
1 - Pass	The agent completes the task within the expected flow.
0 - Fail	The agent fails to complete the task or deviates from expected behavior.

Troubleshoot evaluators

Issue	Recommendation
Inconsistent scores across runs	Lower evaluator temperature (try `0.1`) and enable evidence-first mode. Run multiple variants per evaluation to get statistical confidence.
Judge ignores rubric criteria	Make rubric instructions more specific using examples.
The Judge model is too expensive	Use a smaller model for initial screening and reserve larger models for detailed analysis. Set appropriate `maxTokens` limits.
The Evaluation cost is too high	Use smaller judge models during development.
Scores appear random	Increase statistical sample size using variants.

Configure run settings

Run settings determine how the evaluation suite is executed. To configure, follow these steps:

In the Run Settings section, configure the run settings.
Select Create & Run Suite.

Run settings

Setting	Description
Variations per pair	Specifies the number of conversation variations to generate for each scenario and persona combination. Multiple variations help evaluate the consistency of your agent’s responses across different interactions.
Conversation model	Specifies the model used to generate the conversations that are graded.
Conversation budget	Sets the spending limit for the conversation model. You can set the budget in dollars or tokens, or leave it unlimited.
When to run	Specifies whether the evaluation suite runs manually or on a recurring schedule. For scheduled runs, configure the frequency (Daily, Weekly, or Monthly), day and time, start date, and end condition. The next scheduled run is shown in the eval suite, and scheduled runs are identified in the run history.
Max parallel calls	Specifies the maximum number of conversations that can run concurrently.
Retries on failure	Specifies the number of times to retry a conversation when it fails.

The total number of conversations in an evaluation is calculated as: Scenarios × Personas × Variations

How evaluations are executed

During execution:

Every selected Persona interacts with every selected Scenario
Each conversation is independently executed
All configured Evaluators score the resulting conversations

This creates a full evaluation matrix across personas, scenarios, and evaluators. Example Evaluation Matrix

EVAL_SET:
  personas: 3
  scenarios: 4
  evaluators: 2
  variants: 2

TOTAL_CONVERSATIONS:
  formula: "3 Personas × 4 Scenarios × 2 Variants"
  result: 24

TOTAL_EVALUATIONS:
  formula: "24 Conversations × 2 Evaluators"
  result: 48

Each conversation is executed as an independent multi-turn session where the persona LLM simulates the user according to the scenario definition.

View evaluation results

After you create and run an evaluation suite, the Evals page displays all evaluation suites in your project. Use this page to monitor suite performance, review evaluation coverage, and open a suite to view detailed results. Each evaluation suite displays:

Behavior Score – Overall evaluation score for the latest run.
Coverage – Number of scenarios, personas, variations, and generated conversations included in the evaluation.
Evaluators – Number of evaluators configured for the suite.
Last Run – Shows when the suite was last run and whether it runs manually or on a schedule.

Select an evaluation suite to view its detailed results. The evaluation suite page includes the following tabs: Overview, Latest run, and History.

Overview

Provides a high-level summary of the evaluation suite and its latest execution. Use it to monitor overall performance, review evaluation coverage, and identify areas that require attention.

Overview item	Description
Run status and progress	Shows the status of the latest run and its progress.
Behavior score and trend	Shows the latest behavior score and how the score changes across runs.
Latest run summary	Provides details about the most recent evaluation run.
Arch recommendation	Shows the status of recommendations from Arch based on the evaluation results.
Evaluators below target	Highlights evaluators whose results are below the configured passing score.

Latest run

Displays the results of the most recent execution of the evaluation suite. Use this page to:

Review the overall run score and evaluator results.
Review the evaluator breakdown.
View individual evaluation conversations and their results.
Search and filter conversations.
Retry conversations that encountered errors using Retry all errors.

History

The History tab displays previous evaluation runs for the suite. Use it to review how the suite’s results change across runs and understand the changes associated with each run. Includes information such as:

Run date and duration
Changes associated with the run
Behavior score and source metric
Source of the run
View conversations to review conversations from a specific run

Analyze evaluation results

After an evaluation completes, review the results to understand how your agent performed and identify opportunities for improvement. You can:

Review conversation transcripts.
Analyze evaluator scores and reasoning.
Inspect execution traces and tool usage.
Compare expected and actual conversation outcomes.
Identify patterns across successful and failed conversations.

Analyze conversations

Select a conversation from the Latest Run or History tab to review the evaluation details. For each conversation, you can inspect:

Information	Description
Transcript	Complete conversation between the persona and the agent.
Evaluator Scores	Scores assigned by each evaluator, including reasoning when available.
Execution Trace	Tool calls, handoffs, workflow execution, and other runtime details.
Milestones	Expected milestones that were completed or missed.
Tool Usage	Tool invocations, inputs, outputs, and any execution failures.

View evaluator reasoning

When Chain-of-Thought Reasoning is enabled for an LLM Judge evaluator, the evaluation results include reasoning that explains how the score was determined. Use evaluator reasoning to identify improvements in:

Agent instructions
Workflows
Tool configuration

An example evaluator reasoning:

evaluation_result:
  score: 3/5

  reasoning: >
    The agent correctly identified the customer's intent
    but failed to provide complete rebooking options
    before requesting additional information.

Act on results

Use the evaluation results to prioritize improvements to your agent.

Finding	Recommended Action
Low quality scores	Refine agent instructions, goals, personas, or workflow logic.
Low safety scores	Strengthen guardrails and evaluate with adversarial personas.
Low tool accuracy	Review tool configuration, input parameters, and agent instructions.
Handoff issues	Review handoff conditions and validate the expected agent path.

Troubleshoot

Issue	Recommendation
Scores appear inconsistent	Increase the number of Variations per Scenario × Persona and lower the judge model temperature for more consistent results.
Unexpectedly high scores	Review the evaluator prompt and scoring criteria to ensure they accurately reflect the expected behavior.
Higher execution costs	Reduce the number of variations or use smaller judge models during development.

Related articles:

​Evaluation workflow

​Key concepts

​How Arch AI optimizes your agent

​Example: Improve tool call accuracy using Arch

​Create evaluation suites

​Create eval suite with Arch

​Create eval suite manually

​Evaluation suite components

​Basics

​Arch’s autonomy level

​Scenarios

​Example scenario

​Troubleshoot scenarios

​Personas

​Example persona

​Adversarial persona types

​Troubleshoot personas

​Evaluators

​Evaluator types

​LLM Judge evaluators

​Write effective judge prompts

​Example judge prompt

​Configure bias mitigation

​Trajectory evaluators

​Code Scorer evaluators

​Human Review evaluators

​Scoring scale types

​Likert scale

​Binary scale

​Troubleshoot evaluators

​Configure run settings

​Run settings

​How evaluations are executed

​View evaluation results

​Overview

​Latest run

​History

​Analyze evaluation results

​Analyze conversations

​View evaluator reasoning

​Act on results

​Troubleshoot

Evaluation workflow

Key concepts

How Arch AI optimizes your agent

Example: Improve tool call accuracy using Arch

Create evaluation suites

Create eval suite with Arch

Create eval suite manually

Evaluation suite components

Basics

Arch’s autonomy level

Scenarios

Example scenario

Troubleshoot scenarios

Personas

Example persona

Adversarial persona types

Troubleshoot personas

Evaluators

Evaluator types

LLM Judge evaluators

Write effective judge prompts

Example judge prompt

Configure bias mitigation

Trajectory evaluators

Code Scorer evaluators

Human Review evaluators

Scoring scale types

Likert scale

Binary scale

Troubleshoot evaluators

Configure run settings

Run settings

How evaluations are executed

View evaluation results

Overview

Latest run

History

Analyze evaluation results

Analyze conversations

View evaluator reasoning

Act on results

Troubleshoot