Datasets
A data set is a named collection of evaluation examples. Each example carries a prompt and an optional expected response. An example for a workflow experiment can carry structured input in place of the prompt. For more information, see Structured Input for Workflow Examples (Early Access).
Author a data set as a declarative-config resource:
# datasets/customer_support.yaml
kind: dataset
name: customer_support
description: Production traffic samples covering refunds, returns, and order lookups.
spec: {}
The data set YAML carries only the header. You add examples through one of three paths: the Solace Agent Mesh UI, the bulk-import endpoint, or an examples_file: reference under spec: that points to a CSV sibling and is applied at sam config apply time. Splitting the header from the example rows lets you decide whether to keep example content in version control (with examples_file:) or curate it through the UI without round-tripping every change through Git.
# datasets/customer_support.yaml - with examples in version control kind: dataset name: customer_support description: Production traffic samples covering refunds, returns, and order lookups. spec: examples_file: customer_support.csv
When a data set sets examples_file:, sam config plan reports an edit to the CSV file as an update to the data set, and sam config apply makes the stored examples match the file.
sam config apply deletes every stored example that the examples_file: CSV file does not list, even without --prune. The next apply deletes examples that you add to the data set through the Agent Mesh UI or bulk import. For more information about --prune, see Deleting Resources.
The Platform service can also generate examples for you through the orchestrator agent. PUT /api/v2/platform/evaluations/datasets/generate accepts a target agent and the number of examples to generate, and the orchestrator drafts examples derived from the target agent's metadata. The request returns 202 Accepted and a Location header. Poll that URL until data.status changes from inProgress to succeeded or failed. When it succeeds, data.resourceId is the ID of the new data set. The Platform service creates the data set only when the generation succeeds. The EVAL_DATASET_GEN_AGENT environment variable (default Orchestrator) selects which agent receives the generation request, and EVAL_DATASET_GEN_TIMEOUT (default 180 seconds) caps how long the Platform service waits for a response.
The Agent Mesh UI exposes the same generation surface without requiring you to call the API directly. On Evaluations > Experiment Lab > Datasets, the Create New Dataset menu includes a Generate with AI option that drafts a whole data set for a target agent. On an existing data set, the example list has an inline Generate with AI action that adds more examples without duplicating the ones already there.
Attaching Files to an Example
Each example can carry up to two optional files alongside its prompt, one per role:
-
The
inputfile is the material the agent works on: a configuration to review, a PDF to summarize, or an image to interpret. During the run, the Platform service delivers it to the target agent as a file part, the same way a chat attachment reaches an agent, so the agent can load it with its artifact tools. -
The
expectedfile is the reference that evaluators grade against, such as the document that a correct answer must match. The Platform service never sends it to the agent under evaluation; delivering the grading reference to the agent would invalidate every score in the run.
An example that carries files still requires a prompt telling the agent what to do. A workflow experiment does not accept an example with an input file. For more information, see Known Limits of Workflow Evaluations.
Input files require a namespace-scoped artifact service on the Platform service. Under any other artifact_scope value the stored file reference does not resolve for the agent, and every example that carries an input file fails its trial with an error naming the cause. Text-only examples in the same data set still run, and the run finishes with warnings. For more information about artifact scope, see Artifacts.
Attaching a file is a separate request from creating the example. Upload it to an example that already exists. Each role has its own endpoint: inputArtifact for the input file and expectedArtifact for the expected file.
curl -fsS -X PUT \
-H "Authorization: Bearer ${SAM_PLATFORM_TOKEN}" \
-F "file=@values.yaml" \
-F "description=Helm values under review" \
"${PLATFORM_URL}/api/v2/platform/evaluations/examples/${EXAMPLE_ID}/inputArtifact"
A DELETE on the same path removes that role's file. A GET on /api/v2/platform/evaluations/examples/{id}/inputArtifact/{filename} or /api/v2/platform/evaluations/examples/{id}/expectedArtifact/{filename} downloads it. In the Agent Mesh UI, the example list on a data set includes Input File and Expected File columns; each offers attach, replace, remove, and download actions.
The expected file must be text, such as Markdown, JSON, YAML, CSV, or source code, because evaluators read the reference as text. The Platform service rejects a binary expected file with a 422 Unprocessable Entity response. An input file can be any media type that the allowlist permits.
Removing or replacing an example's file destroys the stored bytes. A run that already used the old file keeps its record of the filename and version it received, and the run drill-down reports the file as no longer available instead of serving the replacement. Keep your own copy of any file you need to reproduce a historical run.
Three environment variables on the Platform service cap what an example can carry:
| Variable | Default | What it does |
|---|---|---|
EVAL_EXAMPLE_ARTIFACT_MAX_BYTES
|
10000000
|
Rejects an upload larger than this many bytes. The default is 10 MB. |
EVAL_DATASET_ARTIFACT_MAX_TOTAL_BYTES
|
5000000000
|
Rejects an upload that pushes one data set past this total; the total is the combined size of both roles' files. The default is 5 GB. Replacing a file credits the bytes it frees. |
EVAL_EXAMPLE_ARTIFACT_ALLOWED_MIME
|
empty | Comma-separated allowlist of media types that applies to both roles. An empty value accepts any type, matching what chat accepts. The Platform service trusts the media type an upload declares unless that type is generic, so the allowlist guides cooperating authors rather than enforcing what a file actually contains. |
CSV Import / Export Format
The data set bulk-import and export endpoints use CSV with this header row:
sequence_number,prompt,expected_response
The expected_response column is optional. Rows with a blank expected_response are scored by LLM-as-a-Judge evaluators; heuristic evaluators that require an expected response skip those rows.
CSV and JSON bulk import cannot carry example files, and export omits them. Attach files per example after importing. The same applies to sam config pull. It writes prompts, expected responses, and the input_data, expected_path, and expected_facts columns to the examples CSV, and it warns for each data set whose examples carry files that the command cannot export.
Export a data set from the Platform service:
curl -fsS \
-H "Authorization: Bearer ${SAM_PLATFORM_TOKEN}" \
"${PLATFORM_URL}/api/v2/platform/evaluations/datasets/${DATASET_ID}/examples/export?format=csv" \
> customer_support.csv
Import is the symmetric PUT .../examples/import against the same parent path.
Structured Input for Workflow Examples (Early Access)
This feature is in the Early Access stage and under active development. Configuration schemas and behavior are subject to change. We recommend that you do not use this feature in production environments.
A workflow takes structured input, not only text. An example for a workflow experiment can therefore carry inputData, a JSON object that Agent Mesh sends to the workflow as its input. Each example must have a prompt, inputData, or both.
An agent never receives inputData. Agent Mesh does not run an agent experiment whose data set has an example without a prompt, so use input-only examples only in a data set that a workflow experiment evaluates.
Agent Mesh sends the workflow one of two inputs:
-
With
inputData, Agent Mesh sends only the object to the workflow, as it is, and does not send a prompt that the example also has. -
With only a prompt, Agent Mesh sends
{"text": "<the prompt>"}, which is the input that a workflow reads from a text message. When the workflow declares an input schema and the prompt is itself a JSON object, Agent Mesh sends the object instead if both of the following are true:-
Every top-level field of the object is a field that the schema declares.
-
The object passes the schema.
If the object fails either condition, Agent Mesh sends
{"text": "<the prompt>"}when that input passes the schema. Otherwise it sends the object, and the schema check reports what does not fit. -
When an example has both, the evaluators read the prompt with the input after it.
An inputData value is at most 256 KiB as JSON. Agent Mesh rejects a value that is not a JSON object.
A workflow example can also carry two fields that the workflow evaluators check: expectedPath and expectedFacts. For more information about these fields, see Evaluators for Workflows (Early Access).
Checks Against the Workflow's Input Schema
If the workflow declares an input schema, Agent Mesh checks every example's input against it when you save the experiment and again when each trial starts.
When you save the experiment, Agent Mesh checks all the examples in the data set and collects every problem into one error. The error names each example by its sequence number and the field that does not fit, such as examples[2].inputData.
The Agent Mesh UI lists the problems under the Dataset field. For each problem, it gives the example's row in the data set editor, the field in the example's input data, and the value that the example sent.
Agent Mesh checks the data set again when you change an existing experiment's data set, target, or system user.
When a trial starts, Agent Mesh checks the example again before it sends the example to the workflow, because you can edit an example after you save the experiment. Agent Mesh does not send an example that no longer fits. Instead, it does the following:
-
It skips the trial and records the reason, with
failureKindset toinvalid_input. -
It counts the trial as an error in the run status, so the run finishes as
completed_with_warnings, or asfailedwhen Agent Mesh skips every trial. For more information, see Run Status Values.
If the workflow declares no input schema, Agent Mesh does not check the input against a schema. Agent Mesh still rejects an example that has an input file.
Adding Examples
When workflow evaluations are turned on, an example in a data set in the Agent Mesh UI has three more fields:
-
Input data (JSON): what the workflow receives for
inputData -
Expected steps (JSON): steps that must run and steps that must be skipped for
expectedPath -
Expected values (JSON): values the result must and must not contain for
expectedFacts
A parse error in one of these fields stays on screen until you fix it. A # column numbers the examples.
In a data set whose examples carry input data, the input comes first and the prompt sits under it as Evaluator instructions (optional). The workflow never receives that text, and an LLM-as-a-Judge evaluator reads it as context for what the example tests. Reports name an example by its evaluator instructions when it has them.
To create an example with the Platform service API, send inputData in the request body:
curl -fsS -X POST \
-H "Authorization: Bearer ${SAM_PLATFORM_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"inputData": {"orderId": "O-5001", "action": "classify"}, "expectedFacts": {"required": ["purchase"]}}' \
"${PLATFORM_URL}/api/v2/platform/evaluations/datasets/${DATASET_ID}/examples"
In the API, expectedPath holds lists rather than the maps that the Agent Mesh UI, CSV files, JSON files, and examples_file use. Each node entry has nodeId and expectation, and each tool entry has toolName and expectation, as the following request body shows:
{
"inputData": {"orderId": "O-5001"},
"expectedPath": {
"nodes": [{"nodeId": "approve", "expectation": "succeeded"}],
"tools": [{"toolName": "issue_refund", "expectation": "not_called"}]
}
}
For more information about the map form, see Checking the Route with Workflow Path.
To clear a field on an existing example, send null for it in a PATCH request. The Platform service rejects an update that would leave an example with neither a prompt nor inputData.
Importing Workflow Examples
A CSV file can carry the following columns in addition to the ones in CSV Import / Export Format. Each holds a JSON object in a single cell, so double each quotation mark inside the cell. The following table describes the columns.
| Column | Holds |
|---|---|
input_data
|
The example's inputData. With this column, prompt can be blank. |
expected_path
|
The example's expectedPath. |
expected_facts
|
The example's expectedFacts. |
The following file has one workflow example. The prompt and the expected response are blank:
sequence_number,prompt,expected_response,input_data,expected_path,expected_facts
1,,,"{""orderId"":""O-5001"",""action"":""classify""}","{""nodes"":{""classify"":""succeeded""}}","{""required"":[""purchase""]}"
Import and export handle these columns as follows:
-
Agent Mesh rejects the whole import when a cell is not a JSON object, and names the row.
-
Export adds each column only when at least one example in the data set has a value for it.
-
JSON import and export use
inputData,expectedPath, andexpectedFactsas keys.
To keep the examples in version control, set examples_file: as shown at the start of this page, and use the same columns. The sam config plan command checks the input_data, expected_path, and expected_facts columns and requires a prompt in each row that has no input_data. It reports a bad value before you apply anything. The sam config pull command writes the columns back.