Evaluators

An evaluator scores an agent's response against the example's expected response. Solace Agent Mesh provides two evaluator types: heuristic evaluators, which score with fixed rules, and LLM-as-a-Judge evaluators, which ask a large language model (LLM) to score the response. A third type, the decision-model evaluator, is available in Early Access. For more information, see Decision-Model Evaluators (Early Access).

Heuristic Evaluators (System-Seeded)

Heuristic evaluators are rule-based scorers. The Platform service ships these evaluators pre-registered. You cannot add custom heuristic evaluators. They appear in the evaluator list alongside the seeded LLM-as-a-Judge ones. In a declarative-config experiment you reference them by name (resolved to an ID at apply time); when you build an experiment through the Agent Mesh UI or the evaluations API directly, you reference them by ID. The sam config apply command does not manage heuristic evaluators. You do not write kind: evaluator for them.

heuristicType	What it scores	Needs expected_response?
valid_json	Whether the response parses as valid JSON.	No
json_diff	Structural diff between expected and actual JSON.	Yes
levenshtein	Edit-distance similarity between expected and actual strings.	Yes
rouge	Word-overlap F1 between expected and actual strings.	Yes

Pick valid_json when the agent must return structured output. Pick json_diff for strict-structure agents. Pick levenshtein or rouge for natural-language responses where exact match is the wrong bar.

LLM-as-a-Judge Evaluators

An LLM-as-a-Judge evaluator is a prompt template that asks an LLM to score the response on a labeled scale. The Platform service ships seven seeded LLM-as-a-Judge evaluators that you can use without authoring any YAML: LLM Judge (general-purpose), Factuality, Closed QA, Summary, SQL, Translation, and Security. The seeded set appears in the evaluator list alongside the heuristic ones and can be attached directly to an experiment. Operators can also author their own LLM-as-a-Judge evaluators as declarative config:

# evaluators/helpfulness.yaml
kind: evaluator
name: helpfulness
description: Score response helpfulness on a 1-5 scale.
spec:
  type: llm_judge
  model: openai/gpt-4o-mini
  promptTemplate: |
    Score the response below on helpfulness.

    Prompt:    {{ Prompt }}
    Expected:  {{ Expected Response }}
    Response:  {{ Response }}

    Reply with exactly one of: excellent, good, adequate, poor, unacceptable.
  choiceScores:
    excellent: 1.0
    good: 0.75
    adequate: 0.5
    poor: 0.25
    unacceptable: 0.0

The template has constraints. {{ Response }} is required because the LLM must see the agent's output to score it. {{ Prompt }} and {{ Expected Response }} are optional. choiceScores must have 2 to 26 entries. Every label the judge can emit needs a numeric score for the pass-rate computation.

Scores in choiceScores are 0.0 to 1.0 floats in the YAML, but the Agent Mesh UI displays them as percentages (0 to 100). The two forms are equivalent: a YAML value of 0.5 shows in the UI as 50%. Scores at or above 0.5 (50%) are marked passed; scores below are marked failed.

LLM-as-a-Judge evaluators need the judge model's credentials wired the same way as any other model. For more information, see Managing Secrets.

What the Judge Sees

Beyond the {{ ... }} template variables, every LLM-as-a-Judge call is automatically handed the agent's full execution context for the example. You do not template these in. The Platform service appends them for you:

• The execution transcript. The agent's assistant text is interleaved, in execution order, with every tool it called (the tool name and the parameter values it passed) and that tool's result. Very large or binary results are elided to a short placeholder. Tool definitions the agent merely had available are excluded. A tool counts as "used" only when it was actually invoked.

• Produced artifacts. Text artifacts (Markdown, JSON, CSV, source code) are inlined. Binary artifacts (images, PDFs, Office documents) appear as filename-only references.

This lets a promptTemplate criterion verify behavior from the trace rather than infer it from the wording of the final answer. Criteria like "the agent must call date_math rather than compute the date internally" or "the agent must route to the documents skill" become directly checkable. This resolves the long-standing ambiguity where an agent that called the right tool and then explained its result looked identical to one that hand-computed the answer. The same trace is available out-of-band via the taskEvents endpoint when you want to inspect it yourself.

An example's expected file is a grading reference stored with the example; the Platform service never delivers it to the agent under evaluation, so it cannot appear in the transcript or among the produced artifacts. For more information about attaching files to examples, see Datasets.

Decision-Model Evaluators (Early Access)

This feature is in the Early Access stage and under active development. Configuration schemas and behavior are subject to change. We recommend that you do not use this feature in production environments.

A decision-model evaluator asks a decision model one typed question about each response and turns the answer into a score. A decision model is a structured model from TypeSafe or OpenAI that answers typed questions and reports how confident it is in each answer. On this page, a case is one response that the evaluator scores: one run of one dataset example. Use a decision-model evaluator when your rubric is a classification, such as whether the agent answered the question or whether the answer is grounded in its sources. Compared with an LLM-as-a-Judge evaluator, a decision model answers faster and costs less per case. When Solace repeated runs in testing, the decision model returned the same answer each time. For open-ended grading, use an LLM-as-a-Judge evaluator.

Like every other evaluator, a decision-model evaluator produces a score from 0.0 to 1.0. You can attach it to an experiment alongside heuristic and LLM-as-a-Judge evaluators, and its scores appear in run comparisons. A score at or above 0.5 passes.

Prerequisites

Before you create a decision-model evaluator, complete the following prerequisites:

• Confirm that decision-model evaluators are enabled. They are enabled by default, but an administrator can disable them by setting the SAM_FEATURE_EVAL_DECISION_EVALUATOR environment variable to false on the Platform service. While they are disabled, the evaluations API rejects new and edited decision-model evaluators, and a run that includes a decision-model evaluator does not start. A run that was waiting to start when an administrator disabled decision-model evaluators scores the evaluator's cases as errored.

• Register a decision model. The example on this page uses a decision model with the alias eval-judge. The Add Model form offers decision models only while workflow decision nodes or decision-model evaluators are enabled. You can also define the decision model in declarative configuration. For more information, see Adding a Decision Model (Early Access) and Define a Decision Model (Early Access).

A decision-model evaluator sends the fields that its reads list names to the decision model's provider (TypeSafe or OpenAI), or to a pass-through service such as OpenRouter that forwards the call. Nothing leaves your deployment until an administrator registers a decision model and an evaluator names it. Review the provider's data-processing terms before you name a decision model in an evaluator, and add transcript or artifacts to reads only when the question depends on them.

Creating a Decision-Model Evaluator in the Agent Mesh UI

While decision-model evaluators are enabled, the Type field offers LLM as a Judge and Decision Model as a Judge. Select Decision Model as a Judge to create a decision-model evaluator. While they are disabled, the form shows Type as LLM as a Judge and does not let you change it. The form then creates an LLM-as-a-Judge evaluator.

To create a decision-model evaluator in the Agent Mesh UI, perform these steps:

1. Navigate to Evaluations > Experiment Lab > Evaluators, and click Create Evaluator.

2. Enter a Name and a Description.

3. Under Type, select Decision Model as a Judge. The form replaces the Model, Criteria, and Scores fields of an LLM-as-a-Judge evaluator with the fields of a decision-model evaluator.

4. Select a Decision Model from the registered decision models. If none is registered, the field links to the Models page. For more information, see Adding a Decision Model (Early Access).

5. Select a Question Type (Choice, Score, or Boolean) and enter the Question.

6. Define the answers for the question type:

   • For a Choice question, add from 2 to 26 Options. Each option has a name, an optional description, and a score from 0% to 100%.

   • For a Score question, add from 2 to 10 Levels, lowest first. Each level has a name and an optional description. The lowest level scores 0%, the highest scores 100%, and the levels between are evenly spaced.

   • For a Boolean question, optionally describe what true and false mean under Answers. Describe both answers, or neither.

7. For a Choice or Score question, select a Scoring method (Chosen answer or Weighted by probability). A Boolean question does not offer this field. For more information, see Answer Scoring.

8. Under Reads, select the case fields that the decision model reads. Prompt, Response, and Expected response are selected by default, and you cannot clear Response. The other two checkboxes are Transcript (the agent's tool calls and results) and Artifacts (files given to the agent, the expected file, and files it produced). Each field that you select is sent to the decision model's provider.

9. (Optional) To send some cases to an LLM-as-a-Judge evaluator, select that evaluator under Escalate To. The list offers only LLM-as-a-Judge evaluators, and the default is No escalation. When you select an evaluator, the Minimum Confidence field appears:

   • Enter a percentage greater than 0 and at most 100. For a Boolean question, the percentage must be greater than 50.

   • If you leave the field blank, the evaluator never escalates for low or missing confidence. It escalates a case only when the decision model's provider fails with an error other than a rejected or denied API key or a configuration error, or when Agent Mesh temporarily cannot read the decision model's settings.

   For more information, see Escalating to an LLM-as-a-Judge Evaluator.

10. Click Create. The button is available after you enter a Name, a Description, a Decision Model, and a Question.

For more information about each setting, see Defining a Decision-Model Evaluator, which describes the same settings by their YAML field names. To read the scores after a run, see Reading Decision-Model Scores in the Agent Mesh UI.

Defining a Decision-Model Evaluator

The following evaluator asks whether the agent answered every part of the user's question. When the decision model answers with less than 80% confidence, the evaluator escalates the case to an LLM-as-a-Judge evaluator named answer-quality-judge.

# evaluators/answers-the-question.yaml
kind: evaluator
name: answers-the-question
description: Did the agent answer every part of the user's question?
spec:
  type: decision
  model: eval-judge
  question:
    type: choice
    instructions: Does the response answer every part of the user's question?
    criteria:
      - key: fully
        description: Answers every part of the question
      - key: partially
        description: Answers some parts and misses others
      - key: not_at_all
        description: Does not answer the question
  choiceScores:
    fully: 1.0
    partially: 0.5
    not_at_all: 0.0
  scoring: chosen
  reads: [prompt, response, expected_response]
  minConfidence: 0.8
  escalateToEvaluatorId: answer-quality-judge

The following table describes the fields that a decision-model evaluator uses.

Field	Required	Description
type	Yes	The value must be decision. You cannot change an evaluator's type after you create it.
model	Yes	The alias of a registered decision model. You cannot name a chat model.
question	Yes	The question to ask. It has a type, instructions, and criteria, described in the next table.
choiceScores	For a choice question	The score, from 0.0 to 1.0, of each option. The keys must match the question's criteria keys. A boolean or score question does not take choiceScores.
scoring	No	How an answer becomes a score: chosen (the default) or expected. For more information, see Answer Scoring.
reads	No	The fields of each case that the decision model reads: prompt, response, expected_response, transcript, and artifacts. The default is prompt, response, and expected_response. The list must include response. The transcript value sends the agent's tool calls and their results. The artifacts value sends the files given to the agent, the example's expected file, and the files the agent produced. Text files include their content, and other files include only their name and type.
minConfidence	No	A confidence greater than 0 and at most 1. For a boolean question, the value must be greater than 0.5. The confidence of a boolean answer is the higher of its true and false probabilities, so it is never below 0.5. When the decision model answers a case with lower confidence, the escalateToEvaluatorId evaluator scores that case instead. This field requires escalateToEvaluatorId.
escalateToEvaluatorId	No	The name of an LLM-as-a-Judge evaluator that scores the cases this evaluator escalates. A decision-model evaluator cannot escalate to another decision-model evaluator.

A decision-model evaluator does not use promptTemplate. The question's instructions and criteria replace it.

The following table describes the three question types.

Question type	criteria	Answer
boolean	Optional. Describe both true and false, or neither. In YAML, quote the keys ("true" and "false") so that YAML parses them as text, not as boolean values.	The probability that the answer is true.
choice	From two to 26 options, each with a key and an optional description.	One option, with a probability for each option.
score	From two to 10 levels, lowest first, each with a key and an optional description.	A level, with a probability for each level.

When you start a run, Agent Mesh checks the decision model of each decision-model evaluator in the experiment. If the model is no longer registered, or is now a chat model, the run does not start, and the error names the evaluator and the model.

Answer Scoring

Every answer becomes a score from 0.0 to 1.0. The scoring field selects the method for choice and score questions:

• With chosen, the evaluator scores only the answer that the decision model chose. For a choice question, the score is the chosen option's choiceScores value. For a score question, the score is the chosen level's position, where the lowest level scores 0.0 and the highest scores 1.0. An LLM-as-a-Judge evaluator scores the same way, so you can compare the scores of the two evaluator types directly.

• With expected, the evaluator weights every option or level by the probability that the decision model gave it. The resulting scores vary less from run to run, which helps when you compare runs on a small dataset. To allow for rounding, the probabilities must add up to at least 1 minus 0.005 for each option, but the required total is never less than 0.97. For example, the probabilities of a four-option answer must add up to at least 0.98, and the probabilities of a 10-option answer must add up to at least 0.97. If they add up to less, the evaluator scores that answer with the chosen method instead, and the score's reasoning records the fallback.

A boolean question ignores scoring. Its score is always the probability that the answer is true, and its confidence is the probability of the more likely answer.

Escalating to an LLM-as-a-Judge Evaluator

A decision-model evaluator with escalateToEvaluatorId escalates a case to that LLM-as-a-Judge evaluator in the following situations:

• The decision model's confidence is below minConfidence, or the evaluator sets minConfidence and the decision model reports no confidence.

• The decision model's provider fails with a temporary or input error. For example, the provider rate-limits the request, times out, cannot be reached, returns an invalid response, refuses the request, or rejects a case that is too large.

• Agent Mesh temporarily cannot read the decision model's settings, for example, because the database does not respond. Agent Mesh tries again for the next case.

The LLM-as-a-Judge evaluator scores the case exactly as it does on its own, with its own prompt, choiceScores, and model. Its score becomes the decision-model evaluator's score for that case, and the result records the decision model's answer and the reason for the escalation. A run uses the LLM-as-a-Judge evaluator's settings from when the run started, so editing that evaluator later does not change the run. You cannot delete an LLM-as-a-Judge evaluator while a decision-model evaluator names it in escalateToEvaluatorId.

A case never escalates when the provider rejects the decision model's API key (HTTP 401) or denies the key access (HTTP 403), or when the decision model is misconfigured, because escalating would hide a problem with the decision model's setup. After an HTTP 401, Agent Mesh also stops calling that decision model for the rest of the run and scores each remaining case that uses it as errored. An HTTP 403 scores only the case that received it as errored, and later cases still call the decision model. Without escalateToEvaluatorId, a provider failure of any kind also scores the case as errored.

A run still finishes when cases score as errored, but it ends with warnings. Agent Mesh leaves errored cases out of the evaluator's average, so an average computed from a few successful cases can look healthy. Check the run's warnings before you rely on a decision-model evaluator's average.

In a declarative configuration repository, name the escalation target by its evaluator name. When the manifest declares both evaluators, sam config apply creates the LLM-as-a-Judge evaluator first. If the manifest leaves out an LLM-as-a-Judge evaluator that a decision-model evaluator escalates to, sam config plan reports a warning, and sam config apply --prune fails before it changes anything. To resolve the conflict, add the LLM-as-a-Judge evaluator to the manifest or change the escalation target of the decision-model evaluator.

Sizing Cases to the Decision Model

Decision models differ in how much text they read in one call. The evaluator limits the size of each call to its decision model, using the first of the following settings that the model defines:

1. The max_state_bytes (or maxStateBytes) entry in the decision model's modelParams, in bytes. The value must be a positive integer. If the value is 0, null, or empty, the evaluator uses the next setting in this list without logging a warning. The evaluator ignores any other value and logs a warning that names the setting.

2. The decision model's maxInputTokens.

3. The provider's published limit. The OpenAI Decisions API reads up to 1,000,000 tokens. TypeSafe publishes no limit, so a TypeSafe model uses the default limit of 128 KiB.

If max_state_bytes or maxInputTokens exceeds the provider's published limit, the evaluator caps the setting at that limit, because the provider rejects any case larger than its limit. The evaluator picks the provider limit from the decision model's provider field, not from its API base URL. For example, a provider: typesafe model that calls GPT-6 Luna Decisions through OpenRouter uses the 128 KiB default, so set its maxInputTokens.

The Agent Mesh UI does not show Max Input Tokens or Advanced Settings for a decision model. To set maxInputTokens or modelParams, define the model in declarative configuration. For more information, see Define a Decision Model (Early Access).

A case larger than the limit escalates, or scores as errored when the evaluator has no escalateToEvaluatorId. A case most often exceeds the limit when reads includes transcript.

Decision-Model Results

Each score from a decision-model evaluator records which judge decided it, the decision model's answer and confidence, the probability of each option, whether the case escalated and why, and the token usage of each judge. A score's reasoning summarizes the answer, for example fully · 93% confidence (jev-1.13.0). If the decision model reports no confidence, an evaluator with minConfidence escalates the case and records the reason as no_confidence rather than low_confidence. Otherwise, the reasoning shows confidence not reported. In both cases, the run's mean confidence excludes the case. For each decision-model evaluator, a finished run also records the share of cases that escalated, the decision model's mean confidence, and the total judge token usage.

Reading Decision-Model Scores in the Agent Mesh UI

On an experiment report, select an example to open its details panel, and then expand a trial. The Judge Reasoning section for each model has a row for each decision-model evaluator. The row shows a summary of how the score was decided. Expand the row to see the probability that the decision model gave each option or level. To see the same section without expanding a trial, click Open Full Details in the panel. The summary has one of the following forms:

• Decided by jev-1.13.0 · 93% when the decision model decided the score. The name is the decision model version that the provider served, and the percentage is its confidence. If the decision model reports no confidence, the summary shows confidence not reported in place of the percentage.

• Escalated · low confidence when the evaluator escalated the case to the LLM-as-a-Judge evaluator. The text after the middle dot (·) gives the reason: low confidence, no confidence, provider error, or model lookup failed. The reason no confidence applies when the evaluator escalates and sets a minimum confidence, and the decision model reports no confidence for the case. Without a minimum confidence, the case keeps the decision model's answer and shows confidence not reported instead. For the situations that escalate a case, see Escalating to an LLM-as-a-Judge Evaluator.

• Decision failed · timeout when the call to the decision model's provider failed and the evaluator did not escalate the case. The text after the middle dot is the failure category.

On the report, the tile for each decision-model evaluator shows a line under the evaluator's name with the share of cases that escalated and the decision model's mean confidence, for example Escalated 12% · Mean confidence 87%. The mean leaves out cases without a reported confidence.