Evaluating Workflows (Early Access)

This feature is in the Early Access stage and under active development. Configuration schemas and behavior are subject to change. We recommend that you do not use this feature in production environments.

An experiment can evaluate a workflow instead of an agent. For each example, Solace Agent Mesh runs the workflow with the example's input, grades what the workflow returns in each trial, and records what each node did and used. For more information about trials, see Experiments. The Agent Mesh UI calls a workflow's nodes steps, so its labels, such as Run stopped at approval step, use "step" where this documentation uses "node". Use a workflow experiment to check that a change to a workflow, to the agent behind one of its nodes, or to a model has not broken the answers or the route that the workflow takes to reach them.

A workflow experiment uses the same data sets, evaluators, experiments, and runs as an agent experiment. This page covers what is different. For more information about workflows, see Creating Workflows (Early Access).

Before You Begin

Complete these tasks before you evaluate a workflow:

  • Confirm that workflow evaluations are turned on. They are on by default, and an administrator can turn them off for a deployment. To check, navigate to Evaluations > Experiment Lab > Experiments, click Create Experiment, and confirm that Target offers Workflow. For more information about what changes while they are off, see Effects of Turning Off Workflow Evaluations.

  • Deploy the workflow. You can save an experiment for a workflow that you created in Agent Mesh but have not deployed, but the Agent Mesh UI does not start a run for it. The Platform service API and sam eval run accept the run, but Agent Mesh then fails it with an error message that begins Agent unavailable.

  • Make sure that the system user that the experiment runs as can invoke the workflow. By default, an experiment runs as the built-in system user system:eval, which can invoke any workflow.

  • If the workflow declares an input schema, make sure that every example's input fits it. Agent Mesh checks the examples when you save the experiment. For more information, see Checks Against the Workflow's Input Schema.

When role-based access control (RBAC) is in use, Agent Mesh checks that the experiment's system user can invoke the workflow. It checks when you save the experiment and again when you run it. A system user that you define must hold one of the following scopes:

  • workflow:<id>:invoke for a workflow that you created in Agent Mesh

  • workflow:<name>:invoke for a workflow defined in YAML

  • workflow:*:invoke for any workflow

For more information, see The System User an Experiment Runs As and RBAC Reference.

Creating a Workflow Experiment

To create a workflow experiment in the Agent Mesh UI, perform these steps:

  1. Navigate to Evaluations > Experiment Lab > Experiments, and click Create Experiment.

  2. Enter a Name and a Description.

  3. For Target, select Workflow, which is labeled EARLY ACCESS, and then select the workflow under Workflow.

    The list shows the workflows that you created in Agent Mesh and the workflows that run from YAML configuration. It tags a workflow that is not running Not Deployed.

  4. Select a Dataset, and select from one to five evaluators under Evaluator.

  5. Click Create or Create & Run.

    If an example's input does not fit the workflow's input schema, Agent Mesh does not save the experiment. The form lists each example that does not fit under the Dataset field, with its row in the data set editor, the field, and the value that it sent. Fix the examples and click Create again. Agent Mesh checks the data set again each time, so a fix that you make in another browser tab applies without refreshing the page.

You cannot change a workflow experiment's target in the Agent Mesh UI after you create it. To evaluate a different workflow, create a new experiment.

You can also name the workflow in the Platform service API or in declarative configuration. An experiment names exactly one target, so set targetWorkflow in place of targetAgent:

# order_triage_regression.yaml
kind: experiment
name: order_triage_regression
description: Checks that the order triage workflow still routes and answers correctly.
spec:
  datasetId: order_triage_cases
  targetWorkflow: OrderTriage
  evaluatorIds:
    - Workflow Path
    - Output Facts
  primaryEvaluatorId: Output Facts
  runsPerExample: 3
  maxWorkers: 4

Make sure that your manifest lists the experiment and the data set that it references by name under resources. You do not declare Workflow Path or Output Facts in the manifest, because they are built-in evaluators. A workflow that runs only from YAML configuration is not stored on the Platform service, so an experiment that you declare in a manifest cannot target it. Target it in the Agent Mesh UI or the Platform service API instead.

The sam config plan command fails with an error that names the workflow when targetWorkflow matches no workflow on the Platform service and none that the same plan declares. For a workflow experiment, the Platform service API returns targetType with the value workflow, and targetWorkflow with the workflow's name. For an agent experiment, targetType is agent. For more information about the other fields, see Experiments.

A workflow experiment has no model field: each agent node runs on its own model. To run the workflow's agent nodes on a different model, for example to try a new model before you change provider, set models on the experiment in declarative configuration or the Platform service API. Each entry names a model configuration by its ID, and the experiment starts one run for each entry, up to three. In each run, the entry's model replaces the model of each agent node, including the agent nodes of a nested sub-workflow. Tool nodes, decision nodes, and the models that the evaluators use do not change.

The following experiment starts two runs, one on each model configuration:

# order_triage_models.yaml
kind: experiment
name: order_triage_models
description: Compares the order triage workflow on two models.
spec:
  datasetId: order_triage_cases
  targetWorkflow: OrderTriage
  evaluatorIds:
    - Output Facts
  models:
    - modelConfigId: 0190f5e7-7a8b-7c9d-8e0f-1a2b3c4d5e6f
    - modelConfigId: 0190f5e8-1b2c-7d3e-9f4a-5b6c7d8e9f0a

To find a model configuration's ID, list the model configurations with the Platform service API at /api/v2/platform/models.

Make sure that your manifest lists the experiment and the data set that it references by name under resources.

Run a workflow experiment the same way as any other: click Run Experiment in the Agent Mesh UI, or use sam eval run. For more information, see Running an Evaluation.

Grading a Workflow Trial

For a workflow trial, the evaluators read these inputs:

  • Response — The workflow's output as JSON on one line with its keys in alphabetical order, so that equal outputs read the same. If the workflow returns no output, the response is the workflow's final message.

  • Prompt — The example's prompt text, the example's inputData as JSON, or both. When the example has both, the evaluators read the prompt with the input after it. For more information about what the workflow receives, see Structured Input for Workflow Examples (Early Access).

  • Trajectory — What each node did, in the order that the nodes ran. An LLM-as-a-Judge evaluator reads it, and so does a decision-model evaluator that reads transcript.

For each node, the trajectory shows:

  • the node's type

  • how the node ended: succeeded, failed with its error, skipped with the reason (such as a branch that the workflow did not take), or stopped because it asked a person for input, which a run cannot give

  • the input that the node received and the output that it returned, each truncated to 1,500 characters, with credentials such as access tokens, passwords, and keys replaced by <REDACTED>

  • the tools that the node called

  • for a switch or decision node, the branch that it selected, and for a decision node, its answers

  • for a loop node, how many iterations completed and why the loop stopped

For a very long run, the judge sees only the most recent part of the trajectory. For more information about node types, see Creating Workflows (Early Access).

To grade the output, the route, or one node, select from the evaluators in Evaluators for Workflows (Early Access).

Trials Ending Early

A trial can end before the workflow completes, for example because a node fails or the workflow runs past its timeout. Agent Mesh records the reason, leaves the trial out of the report's scores and pass rate, and counts the trial as an error in the run status. For more information about each reason, about the run status, and about finding and excluding these trials, see Trials Ending Early (Early Access).

When a workflow completes after a node fails, because the workflow lets the run continue after a failure, Agent Mesh still grades the trial. It flags the trial as a partial failure so that a lost branch does not read as a clean pass.

A node that asks a person for input cannot get an answer during a run, so Agent Mesh fails the node immediately instead of waiting until the run reaches its time limit. For more information about the time limit, see Known Limits of Workflow Evaluations. If the workflow stops on that failure, the trial ends as Run stopped at approval step. If the workflow continues after failures, Agent Mesh grades the trial as a partial failure.

Tokens and Cost

A workflow trial reports its tokens by node and by model, and what it cost. For more information, see Tokens and Cost per Node (Early Access).

Known Limits of Workflow Evaluations

Be aware of the following limits while workflow evaluations are in the Early Access stage:

  • A workflow example cannot attach an input file. Agent Mesh rejects the experiment when you save it, and a file added later ends the trial with failureKind set to invalid_input. Put the input in inputData instead.

  • A node that asks a person for input fails immediately. A run cannot pause and wait for an answer.

  • A tool node cannot ask for a login during a run. Store the credential ahead of time with Connect Tools, or the trial ends early with failureKind set to auth_required. For more information, see Connecting Tools for an Experiment.

  • The EVAL_EXAMPLE_BUDGET_SECONDS environment variable sets the trial time budget (600 seconds, or 10 minutes, by default). A run's time limit is this value multiplied by its number of trials, and Agent Mesh stops a run that exceeds this limit. The trials not yet finished end early with failureKind set to timeout. For more information, see Operational Notes.

  • While other trials wait for a free worker, Agent Mesh ends a trial that sends nothing back for longer than the trial time budget, with failureKind set to timeout, so that the trial does not hold its worker.

  • Agent Mesh shows a trial's cost only when it can account for every model call that the trial made. The cost never includes the judge's own calls. For more information about when the cost is unknown, see Tokens and Cost per Node (Early Access).

  • Agent Mesh redacts credentials in a node's input and output, and truncates them to 4,000 characters for the {{Node Input: <node id>}} and {{Node Output: <node id>}} variables, and to 1,500 characters in the trajectory.

  • A model that you set in the experiment's models field applies only to the workflow's agent nodes.

Effects of Turning Off Workflow Evaluations

While workflow evaluations are off, these differences apply:

  • The Agent Mesh UI offers only agents as an experiment's target.

  • The Platform service rejects an experiment that names a workflow.

  • An existing workflow experiment cannot run.

  • The Platform service rejects examples that carry inputData, expectedPath, or expectedFacts, including in an import and in sam config apply. For more information about these fields, see Structured Input for Workflow Examples (Early Access).

  • The Platform service does not check the {{Node Input: <node id>}} and {{Node Output: <node id>}} variables in an LLM-as-a-Judge template. For more information about these variables, see Judging One Node.

Next Steps