Skip to main content
A useful agent design starts with real requests and observable outcomes. Convert the business process into scenarios before you decide how many agents to build. Start with the source material: policies, procedures, system capabilities, sample conversations, channel requirements, and risk constraints. Note conflicts and unknowns. Don’t let a model fill gaps in policy, or invent a target because a table has an empty cell. Arch can read an uploaded SOP section by section and ask about conflicts and gaps before it drafts a design.

Gather journey evidence

Separate evidence from assumptions. Mark an unconfirmed policy, latency target, or data field as a question. Resolve it with an owner before you encode it as agent behavior.

Write scenarios

Each scenario describes a starting situation and a result you can check. Write at least one ordinary case, one boundary case, and one failure or recovery case for every important journey. Write a short sample dialogue before you build. It shows missing decisions: when the identity check happens, what a yes confirms, which facts come from the system, and what the agent says after a failed tool call.

Plan journeys that use files or media

For a journey that involves a document, image, or other uploaded file, include the file in the scenario. Decide which file types the channel accepts, what the agent extracts, whether the content is sensitive, and what to do when a file is unsupported or still processing. Agents can use the built-in list_attachments and get_attachment tools to find a session file and read its extracted text. Project attachment settings control extraction. content and imageDescription can be null while the file processes or when extraction is off. A missing extraction isn’t evidence that the file holds nothing relevant. Ask for confirmation before the agent sends a file to an external service or hands it to a person. Use an approved path for file retention and access. Test a successful extraction and an unavailable or unsupported file.

Set measurable targets

A targets sheet is a short agreement between the people who own the journey and the people who build it. Keep one row per top request, with an observable result and a scenario that proves it. Set a numeric limit only when the owner can explain the measurement and the trade-off. Measure from a defined start and end point, with the same method in every comparison. Give a rate a numerator and a denominator, and define what counts as an eligible case. For each target, record the owner, the measurement method, and the response when the target is missed.

Budget each turn

Write a target for model calls and input tokens per turn for each top request. Check each design choice against it. See Tuning for latency for the levers that reduce these costs.

Sort scenarios by risk tier

Sort scenarios by what the agent can change. Each tier needs different controls and different test data.
A confirmation question isn’t authorization by itself. The system that performs the action still checks identity, permission, current record state, eligibility, and duplicate requests.

Plan the test set with the journey

For each rule, create at least one case that passes and one case that would expose a violation. Make every evaluator check something visible in the transcript, trace, tool result, or resulting record.
  • Cover neighboring intents that could route incorrectly.
  • Include vague wording, interruptions, missing fields, and corrections.
  • Test unavailable tools and empty or conflicting data.
  • Keep read-only scenarios in one eval set and write scenarios in another. Test chats and evals change real records, so reset the rows after each write run.
  • Use a small, stable baseline for routine regression, and a broader set before release.
Turn each diagnosed failure into a scenario. Add the smallest case that would have caught the failure, and keep it after the fix.

Build a scenario portfolio

A scenario portfolio covers the ways a person can enter, continue, correct, and leave a journey. The same journey can sound different on web, chat, and voice, so keep channel-specific input where it affects behavior.

Next steps