Reading Results
After a run terminates, you can retrieve per-run and per-result artifacts:
| Endpoint | What it returns |
|---|---|
GET /api/v2/platform/evaluations/runs/{id}/results
|
A PaginatedResponse envelope ({data: [...], meta: {pagination: {pageNumber, count, pageSize, nextPage, totalPages}}}). The data array carries one row per example × trial, including per-evaluator scores and pass flags. A row also carries failureKind, which classifies why a trial produced no answer: it is auth_required when the trial reached a tool that the run's system user held no credential for, and null when the trial did not hit a classified failure of this kind. A workflow trial's row carries more fields, described after this table. |
GET /api/v2/platform/evaluations/runs/{id}/export?format=csv
|
The same rows as CSV. Header columns: example_id, trial_index, input_prompt, expected_response, agent_response, duration_seconds, error_message, plus one score_<evaluator> and one passed_<evaluator> column per evaluator (all score_* columns first, then all passed_* columns). The input_artifact and expected_artifact columns follow error_message; each holds the file recorded for the trial as filename@version, or is empty for an example without that file. The export carries no column for failureKind; read that field from the JSON endpoint. |
GET /api/v2/platform/evaluations/runResults/{id}/taskEvents
|
The per-task STIM event log in one response, not in pages: {data: {events: [...]}}. The events array is the event sequence in the order the agent loop emitted it for live tasks. Use it when you need to see which tool calls and which LLM turns produced a failing score. |
GET /api/v2/platform/evaluations/runResults/{id}/inputArtifact/{filename}
|
The input file the Platform service delivered for this trial, when the example carried one. Add ?version=N to pin a version; without it the Platform service resolves the version recorded on the result. The endpoint returns a not-found error after you remove or replace the example's file. |
GET /api/v2/platform/evaluations/runResults/{id}/expectedArtifact/{filename}
|
The expected file recorded for this trial, when the example carried one. Version resolution and not-found behavior match inputArtifact. The Platform service never sends this file to the agent; the file is the reference that evaluators grade against. |
For a workflow trial, a row in the /results response differs as follows:
-
The
failureKindfield can hold the other values in Trials Ending Early (Early Access). -
The
targetType,tokenBreakdown, andcostfields describe the trial's target and what it used. For more information, see Tokens and Cost per Node (Early Access).
On an agent trial's row, targetType is agent, and tokenBreakdown and cost are null.
All list endpoints in the evaluations API use the PaginatedResponse envelope, even when the list isn't paged, so a client never has to branch on whether a response is a bare array.
The Platform service computes pass rates across every score that carries a passed: true|false verdict. Scores without a verdict (typically intermediate LLM-as-a-Judge scores that don't map to the choice set) do not influence the rate.
The version recorded in input_artifact and expected_artifact is the one captured when the run started, so an old run stays auditable after the example changes. For more information about attaching files, see Datasets.
For visual inspection of an individual result, the per-example detail page in the Solace Agent Mesh UI renders the taskEvents trace as an activity diagram. The diagram shows the message flow between the orchestrator, the target agent, and any peer agents that participated. Open an example from the experiment report grid and select the Show Activity icon in the per-model header within the trial card to reveal the side panel.
Examples Blocked on Tool Authorization
An agent under evaluation can reach a tool that requires a user login. Because a run carries no signed-in person, the trial stops there: it produces no answer, it is recorded with failureKind set to auth_required, and it is left unscored rather than graded as a wrong answer. Its errorMessage begins Authentication required: and names the tool and the credential involved. When the agent under test reached the tool by delegating to another agent, the message instead names that peer agent and does not carry the Authentication required: prefix, so do not filter on the prefix alone.
A blocked trial counts as an error when the Platform service derives the run status, so a run with some blocked examples finishes as completed_with_warnings and a run with nothing but blocked examples finishes as failed. The experiment report in the Agent Mesh UI heads the affected run with a banner counting the blocked examples and linking to the panel that resolves them.
A blocked example does not mean the agent regressed, and its absent scores are not zeroes. They are missing measurements. Exclude blocked examples before you compare pass rates across runs, or the comparison reports a regression that did not happen.
The sam eval run CLI fails a run with a blocked example by default, because the run finishes as completed_with_warnings. When you pass --allow-warnings, the CLI instead counts each blocked trial as a failure for every evaluator that gives a pass/fail verdict, so the pass rate is checked against every trial in the run, not only the ones that were scored.
To give the run's system user the credentials it needs, see Connecting Tools for an Experiment. To diagnose the failure, see Troubleshooting Authorization Failures.
Trials Ending Early (Early Access)
This feature is in the Early Access stage and under active development. Configuration schemas and behavior are subject to change. We recommend that you do not use this feature in production environments.
A workflow trial can end before the workflow completes. When a trial ends early, Agent Mesh does the following:
-
It records the reason in the trial's
failureKindanderrorMessage. -
It writes no scores for the trial and leaves it out of the report's scores and pass rate.
-
It counts the trial as an error when it derives the run status, so a run with some of these trials finishes as
completed_with_warnings, and a run in which every trial ends early finishes asfailed. For more information, see Run Status Values.
With --allow-warnings, the sam eval run threshold check counts a trial that ended early as failed.
In the Agent Mesh UI, the trial's score cells read ended early. The example page shows what stopped the run, such as Unable to complete run with the node and the tool that failed, and the workflow's own message under Details.
The following table lists the reasons and the title that the example page shows for each.
failureKind
|
Title on the example page | What happened |
|---|---|---|
node_failed
|
Unable to complete run | A node failed and the workflow stopped. The message names the node and the tool that failed. |
paused_for_input
|
Run stopped at approval step | A node asked a person for input. A run cannot answer, so Agent Mesh fails the node immediately, and the workflow stops on that failure. |
timeout
|
Unable to complete run: Timed out | The workflow ran past its own timeout, the run ran past its time limit, or the trial sent nothing back for longer than the trial time budget (EVAL_EXAMPLE_BUDGET_SECONDS) while other trials waited for a free worker. For more information, see Operational Notes. |
cancelled
|
Unable to complete run: Cancelled | Someone canceled the run, or the workflow ended as canceled. |
auth_required
|
Unable to complete run: Authorization required | A tool node, or the agent behind an agent node, needed a tool login that the experiment's system user does not hold. |
invalid_input
|
Unable to complete run: Input not accepted | The example's input does not fit the workflow's input schema, the example has an input file, or the workflow refused the input. Agent Mesh skips the trial without sending it when the input does not fit. |
rejected
|
Unable to complete run: Request refused | The workflow refused the request outright, for example because the experiment's system user cannot invoke it. |
A trial that ends early is not a wrong answer, and its missing scores are not zeroes. When you compare runs, use the failureKind value in the /results response to find these trials and exclude them. The CSV export has no failureKind column. Exclude trials blocked on tool authorization the same way. For more information, see Examples Blocked on Tool Authorization.
A workflow can complete after a node failed, when the workflow lets the run continue after a failure. For such a trial, Agent Mesh does the following:
-
It grades the trial and counts its scores.
-
It sets
failureKindtopartial_failure, leaveserrorMessageempty, and counts the trial as a warning on the run.
The example page shows Run completed with a failed step. A node that asked a person for input fails the same way, so a workflow that continues after that failure also produces a partial_failure trial. If the failed node was a tool node that needed a login, Agent Mesh does not grade the trial, and it ends early as auth_required.
Tokens and Cost per Node (Early Access)
This feature is in the Early Access stage and under active development. Configuration schemas and behavior are subject to change. We recommend that you do not use this feature in production environments.
A workflow trial reports what it used, by node and by model, in the tokenBreakdown field of a result. Each token count has the fields inputTokens, cachedInputTokens, outputTokens, reasoningTokens, and totalTokens. The cachedInputTokens count is part of inputTokens, and reasoningTokens is part of outputTokens, so the counts overlap.
The tokenBreakdown field has the following parts:
-
The
byNodelist has an entry for each node that called a model, in the order in which each node first called a model. Each entry has the node'snodeId, themodelsthat it used, and atokensobject with the node's token count. -
The
unattributedobject is the token count of the calls that Agent Mesh cannot credit to a node, such as the calls of a decision node. Unlike abyNodeorbyModelentry, it holds the counts directly, not in atokensobject. -
The
byModellist has an entry for each model across the whole trial, sorted by model name. Its counts include the unattributed calls. Each entry has atokensobject and themodelname, which isnullfor calls that named no model. -
The
incompleteflag istruewhen the counts might leave calls out, for example when:-
a node failed or was retried
-
a map, loop, or sub-workflow node ran
-
on a deployment that tracks model spend, a tool node made model calls of its own
When the flag is
true, the example page notes that some model calls were not counted. -
-
The
traceIdfield is the ID under which Agent Mesh records the trial's model spend, ornullwhen the trial has none.
The tokenBreakdown field is null in cases such as these:
-
an agent trial
-
a workflow trial that Agent Mesh skipped without sending
-
a workflow trial that timed out or was canceled before the workflow responded
When the trial called no model, the tokenBreakdown field has empty lists and zero counts.
The trial's tokenUsage field is the sum of the counted calls when all of the following are true:
-
Agent Mesh captured the workflow's final response.
-
Agent Mesh counted at least one call.
-
The workflow reported no more tokens than Agent Mesh counted.
Otherwise, tokenUsage holds the token count that the workflow reported, which is often null. The counts cover the calls that the workflow's nodes made. They do not cover the calls that the judge makes to score the trial.
The cost field is a string that holds the cost of those calls in the currency of your model pricing, with 10 decimal places, such as "0.0142000000". It is null when the cost is unknown. Agent Mesh reads the cost from its record of model spend when you request the result, because that record can arrive after the trial finishes. Agent Mesh shows a cost only when it can account for every call, so the cost is unknown in these cases:
-
The deployment does not track model spend.
-
Pricing is not set for a model that the trial called.
-
The record of model spend does not yet hold every call that the trial made.
-
The trial timed out or was canceled, because its nodes might still be calling models.
-
The token counts were already incomplete when the trial finished.
-
The trial ran more than 90 days ago. Agent Mesh keeps its record of model spend for 90 days, so the cost of an older trial becomes
null.
When the deployment tracks model spend, a trial that Agent Mesh skipped without sending, with the Input not accepted outcome, costs "0.0000000000". Agent Mesh reports no cost for an agent trial.
The example page shows the trial's total tokens and cost, and a table with a row for each node that called a model. When some calls belong to no node, the table also has an Other steps, such as decisions row, which shows the unattributed counts. To fail a continuous integration (CI) run whose trials' known costs exceed a ceiling, see --max-cost in Running an Evaluation.
Consistency Across Trials
When an experiment runs each example more than once, a mean score hides whether an example passes every time. An example that passes two of three trials looks the same as one that scores 0.67 on every trial. The Agent Mesh UI therefore reports how consistently each evaluator passes each example, counting only the examples that have a pass or fail verdict on at least two trials. It shows consistency in two places:
-
Each evaluator tile on the report shows a line such as "1 of 3 examples passed all runs. 2 examples partially failed." An example partially failed when it passed some trials and failed others.
-
A score cell whose trials disagree shows a line such as "2/3 passed".
The sam eval run command prints its own consistency summary, which counts every example. For more information, see Consistency Across Trials.
Comparing Experiments
Two operator workflows answer "did this change regress the agent?":
-
Same experiment, two runs. Trigger the nightly experiment before and after the change. The Platform service run-comparison view (under the experiment detail page in the Agent Mesh UI) lines the runs up side-by-side. The CSV export from the CLI feeds the same comparison into a spreadsheet.
-
Two experiments, same data set. Author a second experiment that targets the same data set and the same evaluators but a different agent configuration (a different model alias or a different system prompt). Trigger both and compare the resulting runs. This approach is useful when the change you are evaluating is a YAML-level change to the agent itself, not a binary upgrade.
The Platform service groups runs by their owning experiment. The Agent Mesh UI exposes per-run drill-down and per-example comparison. Authoring a separate dashboard against the export endpoints (CSV piped into your aggregator of choice) is a valid alternative when your team already lives in Grafana, Datadog, or Looker.
Reports Page
To open the Reports page, navigate to Evaluations > Reports. The page has two parts:
-
A watchlist of the agents that you follow. For each agent, the watchlist shows up to three charts, one for each of the up to three experiments that target the agent and have the most reports. Each chart plots the experiment's primary evaluator score over its last 10 reports. To choose the agents, click Edit Watchlist.
-
The Experiment Report History list, with a row for each report that shows its name, the target that it evaluated, its status, its duration, and when it completed.
Workflows on the Reports Page (Early Access)
This feature is in the Early Access stage and under active development. Configuration schemas and behavior are subject to change. We recommend that you do not use this feature in production environments.
When workflow evaluations are turned on, the list of reports on the Reports page labels its second column Target instead of Agent and shows an agent or workflow icon beside each target. Beside the All Targets filter, a type filter (All Types, Agents, or Workflows) narrows the list to agent or workflow reports. The title of an experiment and of a report shows the same icon.
You can also follow a workflow on your watchlist. In Edit Watchlist, the list shows a workflow with "(workflow)" after its name, and each chart on the watchlist shows the agent or workflow icon beside its name. A workflow gets the same charts as an agent. Because the nodes of a workflow can use different models, the chart does not filter by model.
Tying Evaluation Results Into an Upgrade Gate
The most operationally useful place to put evaluation is as a pre-upgrade dry-run. The shape:
-
Restore production session-store and entrypoint-store backups in staging. For more information, see Managing Backups and Data Retention.
-
Roll the staging deployment to the new binary version.
-
Trigger the production-traffic-derived experiment against the staging deployment.
-
Compare pass rates against the previous run from the same experiment on the old binary.
-
Only start the production roll if the comparison meets your gating criterion (for example, per-evaluator pass rate within two percentage points).
Running this as a pre-upgrade check is the procedural counterpart. The non-zero exit from the CLI on --threshold failure makes this straightforward to script in a CI job. The upgrade job only proceeds when sam eval run exits 0.