Tuning for latency

• Fewer sequential model calls per turn.
• Smaller and more stable prompts.
• Fast tools that return only what the model needs.

Where a turn’s time goes

Fewer model calls per turn

Topology

• Keep routing shallow. Each hop (router to specialist to sub-specialist) adds at least one full model call. Two levels is usually enough.
• Route deterministically where you can. A handoff WHEN that the runtime can evaluate against stored state fires without asking the model. Stored state is a flag that your tools or flow already set. ESCALATE triggers written as expressions work the same way.
• Let coordinators answer simple turns. role: orchestrator can answer greetings and trivial questions directly. role: router always spends a call on routing.
• Delegate only for real sub-tasks. A delegate runs its own full turn while the parent waits, up to 30 seconds by default. If one API call is enough, use a tool.
• Hand back only when it pays off. ON_RETURN: action: resume_intent makes the parent handle a message again right away. Use it only where the customer would otherwise repeat themselves.
• Send each receiver less. A handoff with history: summary_only or last_n and a short summary gives the receiver a smaller first prompt.

Inside an agent

• Answer straight from a tool. on_result: do: - RESPOND: "…" sends the reply and ends the tool loop. This saves the model call that would phrase the answer. Add a response MODEL_CONTEXT so later turns know what was said.
• Use FLOW steps with REASONING: false for fixed steps such as setting values, calling a tool, or responding. They run deterministically, without a reasoning call. A GATHER step still makes a small extraction call to read the customer’s answer.
• Make one tool do one job end to end. Three sequential lookups cost three model rounds. One tool that does all three costs one. The model can also call independent tools in parallel in a single round.
• Turn on EXECUTION.inline_gather: true. Field extraction then happens in the same call as the reply. If the agent’s single knowledge tool also sets kb_fast_path: on, inline gather turns off the automatic knowledge-base search for that agent. Measure both settings before you commit.
• Validate inputs before calling. process: before: require: rejects bad inputs without a failed API call and a retry round.
• Use confirm only on irreversible actions. Each confirmation costs a full extra turn with the customer.
• Choose the multi-intent completion signal. With the default completion_signal: delivered_response, a small check runs after each queued request’s reply. With task_signal, the agent signals completion itself.

AGENT: Order_Status
GOAL: "Tell customers where their order is"
BEHAVIOR:
  role: assistant
EXECUTION:
  inline_gather: true
GATHER:
  order_id:
    prompt: "The order number, for example A-1031"
    type: string
    required: true
TOOLS:
  get_order_status(order_id: string) -> {status: string, eta: string, carrier: string}
    description: "Current status and delivery estimate for one order"
    effect: read_only
    MODEL_CONTEXT:
      FIELDS: [status, eta, carrier]
    on_result:
      do:
        - RESPOND: "Order {{order_id}} is {{result.status}} with {{result.carrier}}, arriving {{result.eta}}."
          MODEL_CONTEXT: "Told the customer order {{order_id}} is {{result.status}}, ETA {{result.eta}}."

Smaller, stable prompts and payloads

Prompt and context

• Explicit visibility. Declare BEHAVIOR: role: so ## Current Context holds only declared and granted values. Avoid {{context}} in production prompts.
• Small memory. The platform sends session memory in full on every call. Store IDs and short strings, not objects or lists. Set MODEL_VISIBLE: false on values the model does not need.
• Tight context budgets. On latency-sensitive agents, lower prompt_context.total_max_chars (voice default: 16,000) and auto_tool_result_value_max_chars (voice default: 2,000). See Context size, compaction, and budgets.
• Short history. Keep prior_turns: compact. Put a response MODEL_CONTEXT on every rich reply. Use history: last_n on handoffs in long conversations.
• Short instructions. The platform sends GOAL, PERSONA, LIMITATIONS, and search_instructions on every call. Move rarely needed guidance into a tool description or a RECALL INSTRUCTION that loads only when relevant.
• A stable start to the prompt. Identity, rules, and the tool list come first. Values that change each turn come later, in memory and ## Current Context. A {{variable}} in GOAL or PERSONA changes the start of the prompt each time the value changes. To avoid this, use an explicit-mode agent and declare the changing values in memory. They then stay in the memory sections and ## Current Context, and the stable start does not change.

Tools

• Fewer tools per agent. The platform sends every tool’s description and schema on every call. The model also chooses faster and better from a short list. Split large tool sets across specialists.
• Short descriptions and parameters. Use one or two sentences per tool. Hide parameters that the model does not fill with default_source: runtime and visibility_model: false.
• Small results. Add MODEL_CONTEXT: FIELDS to every tool that returns more than a few fields. This avoids the summarization call and keeps each tool round small. Better still, return compact payloads from the API itself: filter, paginate, and drop fields the agent never uses.
• Structured, not summarized. On chat agents where speed matters, prefer tool_results.strategy: structured with flat essential_fields. Summarizing costs an extra call. The voice low-latency profile sets structured by default.
• Tight timeouts. Set timeout on the project tool to what the API needs, not the 30-second default. Use transport retry only on idempotent reads.

Knowledge base

• Token budget. Set max_search_tokens on every KB tool. Use 2,000 to 4,000 for voice and more for long-form chat.
• Fewer, better chunks. Lower Top K and raise the similarity threshold.
• Automatic first search. Set kb_fast_path: on on a single-KB agent’s knowledge tool. The platform then searches before the model runs, which saves a model round. It is off by default. The agent is not eligible if it has other business tools, handoff or delegation tools, or GATHER fields collected with inline gather. See How the platform searches.
• Selective extras. Use search_instructions and retrieval_profile: object-aware only where they improve answers. Each one adds planning time or extra parallel searches with reranking.

AGENT: Policy_Answers
GOAL: "Answer HR policy questions from the handbook"
BEHAVIOR:
  role: assistant
EXECUTION:
  compaction:
    tool_results:
      strategy: structured
      keep_recent: 1
    prior_turns:
      strategy: compact
      assistant_preview_chars: 160
    prompt_context:
      total_max_chars: 12000
      auto_tool_result_value_max_chars: 2000
TOOLS:
  search_kb_hr_policies(query: string) -> object
    type: searchai
    kb_name: "HR Policies"
    compaction:
      max_search_tokens: 3000

Checks, side calls, models, and voice

Guardrails and checks

• Order rules by cost. Put check (CEL) rules first. Use provider rules for moderation. Use llm_check only for meaning patterns that other rules cannot catch. Each judge is an extra model call at its checkpoint.
• Streaming and output rules. An agent with output guardrails or response constraints holds a streamed reply until the checks pass. This applies in every stream-safety mode except Terminate forward and Retract capable. Nothing shows until the full reply is generated and checked. If you need both output rules and fast streaming, set the project’s stream-safety mode to Terminate forward. Text then flows, and the platform stops it on a violation.
• Reask costs a full regeneration. On latency-critical agents, use block with a clear message instead.
• Timeouts. Lower the policy’s Model and LLM timeouts for voice. Choose the fail mode on purpose.
• PII. Model-based detection waits up to its time budget on each scan (200 ms by default). A pattern-only rule skips that wait if pattern detectors cover the data.

Model tiers

Voice

• Keep the low-latency profile on. The platform adds it to a project when the project gets its first voice channel. It sets fast model tiers, inline gather, tight compaction, and voice fillers. It does not change model reasoning unless you enable its reasoning defaults. Make sure the workspace has a fast-tier model configured.
• Fillers hide waiting; they do not remove it. voiceDelayMs (300 ms in the profile) and status tags (piggybackEnabled) keep the caller informed while tools run.
• Short spoken replies. Use CONVERSATION: speaking: max_sentences and short instructions.
• Realtime voice. On Gemini Live, the platform handles tool calls without blocking the voice model only when the channel’s pending-tool fillers are on. Without them, a long tool call blocks the voice model. Amazon Nova Sonic has its own pending-tool filler setting. Keep tool results small for realtime voice with compaction: provider_fields.
• Buffered stream safety on voice. Buffered and Gated delay the first audio until the checks finish.

Measuring before and after

• Operate → Sessions → a session → Performance → LLM & Tools lists every model and tool call in a turn with its latency, tokens, and cost. Count the sequential model calls.
• Traces tab → an LLM step → Summary shows input tokens. Compare turn 1 with turn 20 to see growth.
• Insights → Analytics Explorer → Generations shows time to first token and latency across sessions.
• Change one thing at a time (topology, tool payloads, or tiers). Then re-run the same test conversation in a new session.

Speed checklist

• At most two routing hops, with deterministic WHEN where state allows it.
• Status-style answers come straight from the tool (on_result: do: RESPOND).
• Every tool with more than a few fields has MODEL_CONTEXT. No tool-result summaries on hot paths.
• Fewer than about 10 tools per agent, with one- or two-sentence descriptions.
• max_search_tokens is set on every knowledge base.
• Session memory holds IDs and short strings only.
• Routing and summarization use the cheapest tier that passes your evaluations.
• Output rules use check or providers, or the stream-safety mode keeps the reply streaming.

Design guidance

• Make visibility explicit. Production agents should declare BEHAVIOR: role: so ## Current Context holds only declared and granted values. Allow-list mode with {{context}} admits everything in the session.
• Shape tool results at the source. Give every tool that returns more than a few fields a MODEL_CONTEXT. Save anything that later turns need into session values with on_result, because history keeps only a placeholder.
• Keep memory lean. The platform shows session memory in full on every turn and does not size-limit it. Hold large objects in tool results or persistent memory, not in declared session memory.
• Hand off on purpose. For long conversations, prefer pass plus a templated summary, with history: last_n or summary_only, over the full-history default. Give delegates everything they need in INPUT and PURPOSE.
• Give the agent the date. The main system prompt has no current date or time. Only GATHER date extraction receives today’s date. Supply a date explicitly when an agent reasons about relative dates.
• Keep secrets out of prompt text. Put credentials in auth profiles, mark personal fields SENSITIVE, and use SECURE_INPUT for card data.
• Put stable text first. GOAL, PERSONA, LIMITATIONS, and the tool list form the stable start of the prompt. Values that change each turn come later, in ## Current Context. A stable start helps provider-side prompt caching, which shows as Cache Read tokens on the session Overview.
• Write guardrails for one message at a time. A check rule sees only the current message. Model-judge checks also receive up to five recent messages as context, and provider checks can receive recent messages too. Do not write rules that depend on earlier turns. Choose the fail mode on purpose.

Pre-ship checklist

• Each agent declares BEHAVIOR: role:, or you checked its ## Current Context in a trace.
• No {{placeholder}} shows unfilled in the system prompt of a test trace.
• No {{config.*}} or {{env.*}} appears in GOAL, PERSONA, LIMITATIONS, SYSTEM_PROMPT, or RESPOND.
• Every tool with a large or nested result has MODEL_CONTEXT, tested against a real response.
• Rich responses carry a response MODEL_CONTEXT.
• Handoff CONTEXT: keys are lowercase, and history uses a supported form.
• Delegates receive everything they need through INPUT or PURPOSE.
• Sensitive GATHER fields are marked SENSITIVE with a PII_TYPE.
• Guardrail rules set kind and action and use abl.-prefixed functions. The policy fail mode is chosen on purpose.
• Memory is tested with an identified user. Knowledge-base retrieval is tested with a real customer identity.
• Voice agents: the fast model tier has a model assigned. Custom-prompt voice agents state their own spoken-style rules.
• A long test conversation (20+ turns) is checked for prompt size in the trace.
• Final tests ran in a new conversation, on the version that you will deploy.

• Context size, compaction, and budgets — the size budgets this page refers to.
• How a turn is built — visibility modes and what reaches the model each turn.
• Test agents — running and reading a test conversation.
• Sessions — reading a trace’s Performance and Traces tabs.