Experiments

An experiment binds a data set to a target agent and a list of evaluators. Triggering an experiment produces one or more runs (one per declared model, when the experiment overrides the agent's default model). A trial is one test of one example in a run, and runsPerExample sets how many trials a run makes of each example. In Early Access, an experiment can target a workflow instead of an agent. For more information, see Evaluating Workflows (Early Access).

# nightly_support.yaml
kind: experiment
name: nightly_support
description: Nightly check that the support agent has not regressed.
spec:
  datasetId: customer_support
  targetAgent: SupportAgent
  evaluatorIds:
    - helpfulness
    - rouge
  primaryEvaluatorId: helpfulness
  runsPerExample: 1
  maxWorkers: 4

The sam config apply command resolves cross-references (datasetId, evaluatorIds, primaryEvaluatorId) by name, not by Platform UUID. The sam config plan command fails with an error naming the missing resource when a reference cannot be resolved. The Platform service applies these limits:

  • An experiment takes from one to five evaluators in evaluatorIds.

  • The runsPerExample value must be between 1 and 10.

  • The maxWorkers value must be between 1 and 20. It limits how many trials run at the same time, so a run with more trials than maxWorkers starts the rest as earlier trials finish.

Apply the manifest the same way as any other declarative-config tree:

sam config apply --url "${PLATFORM_URL}" --auth-env SAM_PLATFORM_TOKEN

The System User an Experiment Runs As

An evaluation run carries no signed-in person, so it authorizes as a system user: a principal Solace Agent Mesh owns, which holds role-based access control (RBAC) scopes like any other principal. An experiment that names none runs as the built-in system:eval, which can invoke any agent or workflow and call any tool. For the wider machine-identity model, see Machine Entrypoints and System Users.

To run an experiment under a narrower principal, name one in the runAs field. The Agent Mesh UI does not expose this field: set it in declarative config, or through the Platform service API. A bare name is namespaced for you, so runAs: reporting-evals resolves to the principal system:reporting-evals, and an already-prefixed value is kept as it is:

# nightly_support.yaml
kind: experiment
name: nightly_support
spec:
  datasetId: customer_support
  targetAgent: SupportAgent
  evaluatorIds:
    - helpfulness
  runAs: reporting-evals

Declare the system user and its role in the same configuration tree, so that sam config plan shows all three resources together:

# nightly_support_invoke.yaml
kind: rbacRole
name: nightly_support_invoke
spec:
  scopes:
    - "agent:SupportAgent:invoke"
# reporting-evals.yaml
kind: systemUser
name: reporting-evals
spec:
  displayName: "Nightly support evaluations"
  roleNames: [nightly_support_invoke]

Four rules govern the field:

  • The manifest is the desired state. Removing runAs from an experiment reverts its runs to system:eval, so declare the field explicitly on every experiment that needs a scoped principal.

  • Each run records the principal it used. The value is captured when the run is triggered, so an old run stays auditable after the experiment is edited to point somewhere else.

  • A principal that holds no scopes fails the whole run. The error names the principal. Assign it a role and trigger the run again; the Platform service picks the grant up without a restart.

  • system:eval cannot be narrowed. Declaring a systemUser under a built-in name is rejected, because the built-in roles union with anything you grant rather than replacing them. Scoping an experiment down means declaring a system user of your own and naming that.

runAs cannot name system:default or system:channel, the built-in principals that entrypoints run as, because an experiment naming one would share a tool credential with live traffic. For more information, see How the Credential Is Stored.

Pointing an experiment at a system user grants its runs that principal's reach, and no second authorization check applies. A role that holds evaluation:_:create or evaluation:_:update can therefore reach further than the scopes its own holder was granted. For more information, see RBAC Reference.

Connecting Tools for an Experiment

An agent whose tools require a user login cannot complete those calls during an unattended run: the run's system user holds no credential to present, and no person is present to authorize one. The Connect Tools panel on the experiment detail page in the Agent Mesh UI closes that gap ahead of time by storing a credential against the experiment's system user that every run of that experiment then uses.

The panel lists the target agent and the agents it delegates to, with one row for each credential they need:

  • A credential a person can authorize from the browser shows an Authorize button. Only OAuth 2.0 credentials qualify.

  • A credential the agent's own configuration provides shows Set in configuration and is not authorized from this panel. Basic and bearer credentials are always configuration-provided.

  • When the panel cannot determine the full set of agents the experiment delegates to, it offers Add Agent so that you can name one yourself.

For a workflow experiment (Early Access), the panel lists these groups:

  • the credentials that the workflow's tool nodes use, under a heading such as Required by tool steps in OrderTriage

  • each agent that the workflow's agent nodes call, under a heading that names the node and the agent, such as Required by the classify step, which calls the OrderClassifier agent

When the panel cannot read all of a workflow's nodes, for example because the workflow runs from YAML configuration, it offers Add Agent.

A tool node cannot ask for a login while a run is under way. If no credential is stored, the trial ends early with failureKind set to auth_required, and the report shows a banner that counts the examples blocked on tool authorization.

Authorizing signs in on behalf of the system user rather than on your own behalf, and the credential that results is one person's grant at the remote system.

The panel reports which principal the runs use but does not change it, and no other screen sets runAs either. To run against a different system user, edit the experiment's runAs in declarative config and apply it again.

Reading the panel requires the evaluation:_:read scope. Authorizing or disconnecting a credential requires evaluation:_:invoke, and adding or removing an agent requires evaluation:_:create or evaluation:_:delete.

Connect Tools uses the same delegated-access machinery as per-user tool authorization, and inherits its prerequisites: single sign-on (SSO), a SQL session store, the Trust Manager, and a callback registered at your identity provider (IdP). For those prerequisites and for how long a stored credential lasts, see Secure User-Delegated Tool Access.

When a run reaches a tool that its system user holds no credential for, the trial stops without producing a score. For more information, see Examples Blocked on Tool Authorization. To diagnose a run that fails on authorization, see Troubleshooting Authorization Failures.

How the Credential Is Stored

Connect Tools stores the credential the same way per-user tool authorization does, with one difference: it is keyed by the system user rather than by the person who authorized it. The prerequisites in Before You Begin, the registered callback, the encrypted session-store slot, and the 30-day lifetime all apply unchanged, with these consequences specific to a shared principal:

  • The credential is one person's grant, held under a principal that others use. An evaluation system user is the one case where the shared-account pattern that per-user credentials exist to replace is the better choice, because no person is present to re-authorize an unattended run. Solace recommends authorizing with a shared service account rather than an individual's account, so that any teammate can restore the access and so that it does not follow one person out of the organization.

  • A credential is shared by everything running as that principal. Agent Mesh stores it per agent, per principal, and per credential key, so every experiment that runs as the same system user on the same agent uses the same credential — as does an entrypoint that names that same system user in its run_as. The rejection of system:default and system:channel does not extend to a custom principal, so avoid reusing one across a live entrypoint and an experiment.

  • Disconnecting removes the stored credential; it does not revoke anything at the remote system. Agent Mesh stops presenting the credential, and every experiment running as that principal on that agent stops presenting it too, but the underlying grant remains valid at the remote system until it expires or you revoke it there.

Running an Evaluation

After an experiment exists on the Platform service, trigger it from a CI job or an operator shell.

  1. Set the auth token. The CLI reads the bearer token from SAM_AUTH_TOKEN or SAM_PLATFORM_TOKEN (in that order), or from the cached OAuth login (sam auth login). With --manifest, the CLI uses the manifest's auth block in place of the cached login, for example the environment variable that a bearer_token block names. The SAM_AUTH_TOKEN and SAM_PLATFORM_TOKEN environment variables still take precedence over the manifest when set. There is no --auth-env flag on sam eval run.

    export SAM_PLATFORM_TOKEN="${SAM_PLATFORM_TOKEN}"
  2. Trigger one or more experiments by name. sam eval run accepts a variadic list of experiment names and triggers them concurrently, prefixing each line of streamed output with the experiment name:

    sam eval run nightly_support nightly_orders \
      --url "${PLATFORM_URL}" \
      --threshold 0.9 \
      --timeout 30m

The CLI resolves each experiment by name, posts to the trigger endpoint, polls each run until it reaches a terminal status, and prints a per-evaluator summary. It exits non-zero in these cases:

  • Any run failed.

  • Any pass rate fell below --threshold.

  • A run completed with warnings, unless you pass --allow-warnings.

  • With --max-cost, the known costs of a workflow run's trials together exceeded the --max-cost value.

When an experiment runs each example more than once, the summary also shows how consistently each example passes. For more information, see Consistency Across Trials.

While the run is in flight, the CLI prints one progress line per poll tick:

  status=running progress=12/50

Experiment runs scale with the data set size, the number of evaluators (especially LLM-as-a-Judge evaluators, which call out to a model per example), and runsPerExample. A large nightly run can take significant time to finish, which is normal.

When the run reaches a terminal status, the summary appears:

Run 01972c40-d3c8-7c8b-a6c7-3f8c1f0c8f63 - status: completed

Per-evaluator results:
  helpfulness: 47 pass / 3 fail / 50 total
  rouge:       42 pass / 8 fail / 50 total

Overall pass rate: 89/100 (89.0%)

Useful flag overrides:

Flag Default Effect
--url (see below) Platform URL or bare hostname. Overrides --target and the address from --manifest.
--target (see below) Cached target name written by sam auth login.
--manifest / -m (see below) Manifest file that supplies the target: the address from target.url (or target.name) and the credentials from the auth block.
--threshold 1.0 Minimum pass rate (0.0–1.0). The CLI exits non-zero when the rate falls below this.
--max-cost (none) Early Access. The CLI exits non-zero when the known costs of a workflow run's trials together exceed this decimal amount, in the currency of your model pricing, such as 2.50. For more information about how the CLI counts the cost, see Running an Evaluation.
--allow-warnings false Check a run that finishes as completed_with_warnings against --threshold and --max-cost instead of failing it. Each trial that errors or produces no score, such as one that ends early or is blocked on tool authorization, counts as a failure for every evaluator that gives a pass/fail verdict. The CLI still exits non-zero when the run has no pass/fail scores. Cannot be combined with --watch=false.
--timeout 30m Overall wait limit while polling.
--poll-every 2s Poll interval against the run-status endpoint.
--watch true When false, the CLI fires the trigger, prints the run ID, and exits 0 immediately.
--cancel-on-interrupt true When the operator sends Ctrl-C, the CLI posts to the cancel endpoint before exiting.
--format <text\|json> text Output format. The json value emits a machine-readable JSON summary on stdout instead of the human table.
--output-dir / -o (none) Directory to download each run's per-example trace into (execution data + task events) under <dir>/<experiment>/<runID>/. When unset, traces are not dumped. The summary still prints to stdout either way.
--insecure false Allow sending the bearer token over plain http:// to the Platform service. The sam CLI still verifies TLS certificates. Development only.

The --max-cost check works as follows:

  • The CLI exits non-zero when the known costs of the run's trials together exceed the value.

  • The CLI does not count a trial whose cost is unknown, and reports how many such trials the run had.

  • The CLI reads the costs when the run finishes, so it does not count spend recorded later.

  • The cost does not include the judge's calls.

  • The CLI accepts the flag only for a workflow experiment with at most one model, and only with --watch. If you pass it in any other case, the CLI exits with an error before it triggers any experiment.

The CLI selects a target with the precedence --url > --target > --manifest > SAM_WEBUI_URL > the single cached login. At least one of these must resolve a target, or the CLI exits with an error. When you pass --url or --target with --manifest, the flag replaces only the address. Credentials still follow the manifest's auth block unless SAM_AUTH_TOKEN or SAM_PLATFORM_TOKEN is set. If the manifest declares auth: {type: oauth}, the CLI stops with an error rather than send that sam auth login token to an address other than the one you signed in to. For more information about how the CLI resolves a manifest's target, see Using the Same Manifest with Other Commands.

Consistency Across Trials

The runsPerExample value sets how many trials a run makes of each example. When runsPerExample is greater than 1, the summary also shows how consistently each example passes across its trials, for each evaluator that gives a pass/fail verdict. For example, the summary of a run of 16 examples with runsPerExample: 3 includes:

Consistency across trials:
  helpfulness: pass^3 14/16 examples, 1 inconsistent, 1 failed every trial
  rouge: pass^3 12/16 examples, 3 inconsistent, 1 failed every trial

The summary has one line for each evaluator that gave a pass/fail verdict on any trial of the run, sorted by evaluator name. Each line covers every example in the run, and the 3 in pass^3 is the number of trials the run makes of each example. In the helpfulness line, pass^3 14/16 means that 14 of the 16 examples passed on all three of their trials. Of the other two examples, one passed on some trials and failed on others, and one failed on every trial. When the run makes only one trial of each example, or no evaluator gives a pass/fail verdict, the summary leaves out the whole block and the JSON output has no consistency field.

A trial passes only when the evaluator passed it and the agent did not report an error. Every other trial counts as failed, including:

  • The evaluator could not score the trial, for example because the judge model's provider returned an error.

  • No evaluator scored the trial, for example because it was blocked on tool authorization. For more information, see Examples Blocked on Tool Authorization.

  • The agent reported an error on the trial, even if the evaluator scored it.

  • The trial's result never arrived, including every trial of an example that returned no result.

For a run that finishes with warnings, --allow-warnings makes the --threshold check count the same trials as failures. A run that fails or is canceled partway through shows each trial it did not finish as failed.

With --format json, the same counts appear under consistency as a list with one entry per evaluator, in the same order. Each entry carries evaluatorId, evaluatorName (the name, or the ID when the name is unknown), examples, passedEvery, failedEvery, inconsistent, and trialsPerExample.

Listing Available Experiments

Use sam eval list to discover the experiment names registered on the Platform before running:

sam eval list --url "${PLATFORM_URL}"

By default, the CLI prints a human-readable table with the columns NAME, DATASET, TARGET, TYPE, and LATEST STATUS. The TYPE column reads agent or workflow, and TARGET holds the name of that agent or workflow. Use --format json to emit a machine-readable list, which is useful when scripting sam eval run from a discovery step. Each entry has a targetType field, and the target's name is in targetAgent or, for a workflow experiment, targetWorkflow. sam eval list shares the same target-selection flags (--url, --target, --manifest, --insecure) as sam eval run.

Run Status Values

The Platform service emits the run status field as one of:

Status Meaning
pending Trigger accepted, run not yet started.
running Execution in progress.
completed Every example evaluated, no fatal issues.
completed_with_warnings Finished but the Platform service recorded non-fatal warnings, such as a trial that errored or a workflow trial that ended early. The CLI exits non-zero unless you pass --allow-warnings and the run clears --threshold and any --max-cost that you pass.
failed The Platform service marked the run failed. The CLI exits non-zero.
cancelled An operator canceled the run before it finished. The CLI exits non-zero.

The terminal statuses are completed, completed_with_warnings, failed, and cancelled. Only a completed run, or a completed_with_warnings run when you pass --allow-warnings, reaches the threshold check.

Diagram: Run Status Values

Troubleshooting Authorization Failures

An evaluation run carries no signed-in person: it authorizes as a system user, and it reaches tools using whatever credentials that principal holds. Two independent conditions can therefore stop a run, and each produces a different symptom. Either the principal holds no RBAC scopes, in which case nothing runs at all, or the principal holds no credential for a tool an agent uses, in which case individual examples stop partway. The fixes do not overlap, so establish which condition applies before acting.

Symptoms

  • A triggered run finishes as failed with no example evaluated, and sam eval run exits non-zero without printing a per-evaluator summary. The run's error names the principal and reports that it holds no RBAC scopes.

  • The run finishes as completed_with_warnings or failed, individual results carry failureKind set to auth_required, and those examples carry no scores at all.

  • The error on such a result begins Authentication required: and names a tool and a credential, or names a peer agent that required the authorization.

  • The experiment report in the Agent Mesh UI heads the run with a banner counting the examples blocked on tool authorization.

  • A run that succeeded previously fails now with no configuration change, because a stored credential lapsed at the remote system.

Diagnostic steps

  1. Read the runAs value recorded on the run, not the value on the experiment. The principal is recorded when the run is triggered, so an experiment edited since then reports a different value from the run that failed.

  2. Separate the two failure classes by counting results. Zero results means the principal was refused before any example ran, and the problem is RBAC scopes. Some results present, some carrying auth_required, means the principal ran but lacked a tool credential.

  3. For the scopes case, check that the principal named in runAs exists and has a role assigned. A principal that was never provisioned, or one whose role was revoked, resolves to no scopes. See Machine Entrypoints and System Users.

  4. For the credential case, read the error to find which agent and which credential were involved. When the error names a peer agent, the missing credential is downstream of the agent under test: that agent delegated the work, and the credential belongs to the agent it delegated to.

  5. Separate a credential that was never connected from one that lapsed. A run that previously succeeded points to expiry, because credentials are refreshed on use rather than on a schedule and can be rejected by the remote system before anything renews them.

  6. When a missing credential is reached through an externally proxied agent, Agent Mesh records a generic tool error instead of classifying the result as auth_required. Treat an unexplained tool-authorization error from a proxied agent as this same failure.

Resolution

  • The principal holds no scopes — assign it a role carrying the invoke scopes the experiment needs, then trigger the run again. Agent Mesh picks the grant up without a restart. See Enabling Role-Based Access Control (RBAC).

  • The principal holds no credential — authorize the tool from the Connect Tools panel on the experiment detail page, then trigger the run again. See Connecting Tools for an Experiment.

  • A credential lapsed — authorize it again from the same panel. Disconnecting first is unnecessary; re-authorizing replaces the stored value.

Prevention

  • Declare the experiment's system user and its role in the same configuration tree as the experiment, so sam config plan shows a missing principal before a scheduled run finds it.

  • Authorize tools with an account at the remote system that your team shares, rather than an individual's. For the reasons, see How the Credential Is Stored.

  • Re-authorize before a long or scheduled run rather than after it fails. A credential can read as present and still be rejected on first use.

  • Trigger a scheduled experiment once by hand after changing its runAs, its system user's roles, or any tool an agent under evaluation uses.

For what a blocked example means when you read the results rather than the run, see Examples Blocked on Tool Authorization.