Managing API Keys

An API key lets a pipeline call Solace Agent Mesh without a person signing in. Each key belongs to a system user and acts as that system user, not as the administrator who created it. A key carries its own list of scopes. When you create a key, you choose them from the scopes its system user already holds. For more information about system users, see System Users.

Use API keys for automation such as a continuous integration (CI) job that runs sam config plan, sam config apply, or sam eval run. People continue to sign in through single sign-on (SSO). For more information, see Enabling Single Sign-On (SSO).

API keys are on by default, but they work only on a deployment that runs the default_rbac authorization service, as described in the following prerequisites. To turn them off, set SAM_FEATURE_MACHINE_TOKENS=false on the deployment. For more information about feature flags, see Environment Variables.

Prerequisites

  • The deployment runs the default_rbac authorization service. On a deployment that does not run default_rbac, Agent Mesh refuses to create keys, and an existing key stops working if you switch the deployment away from default_rbac. For more information, see Enabling Role-Based Access Control (RBAC).

  • SSO is configured. Creating, renaming, and deleting keys requires a super administrator who has signed in through the browser or sam auth login.

  • A custom system user exists and holds roles that cover every scope the key requires. A key cannot own scopes its system user does not hold. The built-in system users default, channel, and eval cannot own keys.

Key Scopes

When you create a key, you choose which of its system user's scopes it carries. Agent Mesh refuses any scope that the system user's roles do not already grant, so several keys on one system user can each hold a different subset.

For example, a system user ci-deployer might hold these scopes through its roles:

Principal Scopes
System user ci-deployer agent_builder:*:read, agent_builder:*:create, agent_builder:*:update, agent_builder:*:deploy
Key deploy-pipeline agent_builder:*:create, agent_builder:*:update, agent_builder:*:deploy
Key agent-reader agent_builder:*:read

The scope list of a key never changes after you create it. To give a key more access, create a new key with the new scopes.

On every request, Agent Mesh checks each of the key's scopes against the system user's current roles and keeps only the scopes that are still covered:

  • If the system user loses a role, each key scope that the role covered stops applying. The scope applies again if you restore the role.

  • If the system user gains a role, its keys gain no scopes.

  • Agent Mesh keeps or drops each scope whole. If you replace agent_builder:*:read with a scope for one agent in the system user's roles, the agent-reader key loses read access entirely rather than narrowing to that agent. A key can select only scopes that its system user's roles still cover.

  • If none of a key's scopes are still covered, the key stops authenticating.

Creating the First Key

A super administrator signs in once per deployment to create the first key. After that, a pipeline that holds a key runs without anyone signing in.

  1. Sign in to the deployment as a super administrator:

    sam auth login https://agent-mesh.example.com
  2. Declare the system user the key belongs to, and give it a role that covers the scopes the key requires. Use the rbacRole and systemUser kinds, list them in a manifest, then run sam config apply. For the files and manifest to use, and for how a manifest that declares rbacRoles or systemUsers treats the roles and system users it does not list, see Create the Pipeline's API Key.

  3. Create the key:

    sam api-key create --user ci-deployer --label deploy-pipeline \
      --scopes 'agent_builder:*:create,agent_builder:*:update,agent_builder:*:deploy' \
      --expires-in 90d

    The --user flag accepts the system user's name, its system:-prefixed name (such as system:ci-deployer), or its ID. The command prints the key alone on standard output and prints its details and a warning on standard error, so a script can capture only the key by running the same command as KEY=$(sam api-key create ...).

  4. Store the key in your pipeline's secret store.

Agent Mesh shows a key exactly once, when you create it. It stores only a one-way hash, so you cannot recover a lost key. Create a replacement instead.

Every key requires an explicit expiry choice: pass either --expires-in with a duration such as 90d or 720h, or --never-expires. Agent Mesh applies no default and no maximum. A key that never expires keeps working until you delete it, so Solace recommends setting an expiry wherever your rotation process allows one.

Each system user can hold up to 10 active keys. Expired keys stay listed until you delete them and do not count toward the limit.

Using a Key

Send the key as a bearer token in the Authorization header. Every key starts with samk_. For example, to check that a key authenticates and can read agents, run:

curl -H "Authorization: Bearer ${SAM_PLATFORM_TOKEN}" \
  https://agent-mesh.example.com/api/v2/platform/agents

Agent Mesh accepts a key only in that header. It refuses a key sent in a cookie or in a ?token= query parameter, because those locations end up in browser history, proxy logs, and Referer headers.

Keys work only on these requests:

  • The Platform service requests that the sam config and sam eval commands make.

  • The key inventory reads.

Agent Mesh refuses a key on every other request, including task submission, sessions, projects, prompts, scheduled tasks, and the notification service:

  • Where the Web UI Entrypoint runs in the same process as the Platform service, the response is 403 Forbidden.

  • Where it does not, the response is 401 Unauthorized.

The Model Context Protocol (MCP) Entrypoint does not accept keys. It refuses a key without forwarding it to your identity provider.

For the sam CLI, set the key in SAM_PLATFORM_TOKEN. The sam config and sam eval commands read it:

export SAM_PLATFORM_TOKEN="$KEY"
sam config apply

The sam eval commands also read SAM_AUTH_TOKEN and prefer it over SAM_PLATFORM_TOKEN. The sam config commands do not read SAM_AUTH_TOKEN.

Applying the Full Configuration From a Pipeline

To let a pipeline apply every configuration kind, including roles, grants, and claim mappings, give the system user a role that holds *:*:* and create a key that selects *:*:*. That key passes the same super administrator check that a person does. For more information about a pipeline that plans, applies, and evaluates the configuration, see A Complete Pipeline.

A key that holds *:*:* has full control of the deployment, including the claim mappings that decide what each person who signs in can do. Store it as you would an administrator's password, and review the changes it made after you delete it, because those changes outlive the key.

Actions No Key Can Perform

No key can create, rename, or delete a key, however broad its scopes. Key management stays with a signed-in super administrator, so a leaked key cannot issue its own replacement directly. The sam api-key command refuses such a request with a message that tells you to unset SAM_AUTH_TOKEN and SAM_PLATFORM_TOKEN and sign in. A key that holds *:*:* can still change roles and claim mappings, including making a person a super administrator, so treat it as an administrator credential.

Reading the key inventory requires *:*:*.

Experiments That Run as Another System User

A key that holds evaluation:*:create can set an experiment's runAs to another system user. That experiment's runs then have that system user's access, not the key's scopes. A user account that holds the scope can do the same. Grant evaluation scopes to a key's system user only if you would trust the key with the access of any system user an experiment can name. For more information, see The System User an Experiment Runs As.

Listing and Renaming Keys

List all keys, filter the list by system user and status, or show one key by its ID, which sam api-key list shows in its ID column:

sam api-key list
sam api-key list --user ci-deployer --status active
sam api-key get "$KEY_ID" --format json

The list shows each key's label, owner, display prefix, status, expiry, and when it was last used. The sam api-key get command, and either command run with --format json, also shows each key's scopes. The display prefix is the first eight characters after samk_, enough to match a key in your secret store against the inventory without revealing it.

Agent Mesh updates the last-used time only for a request that authenticated successfully, and at most once every 5 minutes for each key on each Platform service replica, so the time can trail the key's most recent use by up to 5 minutes. Treat the last-used time as a guide to whether a key is in use, not as an audit trail.

To change a key's label, run sam api-key update <id> --label <new-label>. The label is the only part of a key you can change.

Rotating a Key

A system user can hold several keys at once, so you can rotate a key without a break in service:

  1. Create the replacement key with the same scopes.

  2. Deploy the replacement key to the pipeline.

  3. Verify that the pipeline works with the replacement, for example by checking that the inventory shows a last-used time for the replacement.

  4. Delete the original key.

Revoking a Key

Delete the key:

sam api-key delete "$KEY_ID"

The sam api-key delete command asks for confirmation in a terminal. Pass --force to delete without confirmation. A non-interactive session requires --force.

Deletion takes effect on the next request that presents the key, on every replica, because Agent Mesh checks every key against its database on every request. Deleting a system user deletes all of its keys.

Delete the key itself to revoke it. Removing a role from the key's system user is not an instant way to revoke a key. On a deployment with more than one Platform service replica, the other replicas receive the role change through the event broker. Until a replica receives that event, it can keep granting the removed scope.

An evaluation run that a key already started keeps running after you delete the key. Cancel the run separately if you need to stop it.

Auditing Key Use

Agent Mesh records an action taken with a key as an action by its system user, and the key's ID identifies which credential was used. The key ID appears in the credentialID field of the rbac audit line that the Platform service logs for each change to roles, role assignments, claim mappings, system users, and keys. If two pipelines share one key, their actions are indistinguishable, so give each pipeline its own key. For more information, see Managing Audit and Compliance.

Agent Mesh never writes the key or its hash to its logs. Your pipeline can still leak a key, for example by echoing it into a build log, so keep the key masked there.

Troubleshooting

Response Meaning
401 Unauthorized The key is unknown, expired, or deleted; its system user no longer qualifies to own it; none of its scopes are still covered; or the deployment does not accept keys, including any Web UI Entrypoint that does not run in the same process as the Platform service. Every one of these returns the same response, so check the inventory and the deployment's configuration.
403 Forbidden The key authenticated, but its scopes do not permit the request. Check the key's scopes and its system user's roles. If the message is "API keys can only call the Platform configuration API", no key can make the request. For more information, see Using a Key.
503 Service Unavailable Agent Mesh could not check the key against its database. Signed-in users are not affected.
409 Conflict when creating a key The system user already holds 10 active keys. Delete a key it no longer needs, or wait for one to expire.
422 Unprocessable Entity with "The owning system user does not hold every requested scope" shortly after sam config apply On a deployment with more than one Platform service replica, the replica that served the request has not yet picked up the role change. Retry the request.
sam api-key reports "API keys are not enabled on this deployment" The feature flag is off on this deployment, or --url does not point at the deployment's Platform service.

Next Steps

After a pipeline authenticates with a key, review what it changed in the audit records. For more information, see Managing Audit and Compliance. To choose the scopes for your next key, see Common Scopes.