Promoting Changes Between Environments

Start with one manifest. When you add staging and production, add a manifest for each environment and a pipeline. When several teams share the environments, see Scaling to Several Teams.

Before You Begin

You need the following items:

  • A desktop instance and the sam CLI.

  • For staging and production: one Agent Mesh deployment for each environment, with role-based access control (RBAC) enforced and single sign-on (SSO) configured, a CI/CD system, and a super administrator who can create API keys.

Start with One Manifest

Keep the manifest and the resource files in one Git repository:

orders/
├── manifest.yaml
├── agents/
│   └── orders-intake.yaml
└── workflows/
    └── orders-status-lookup.yaml
# manifest.yaml
kind: manifest
name: orders
target:
  name: desktop
resources:
  agents:
    - orders-intake
  workflows:
    - orders-status-lookup

To preview your changes and apply them to the desktop instance, run the following commands:

sam config plan -m manifest.yaml
sam config apply -m manifest.yaml

Commit each change to Git. If you create a resource in the Agent Mesh UI, export it with sam config pull, then commit the exported file. For more information, see Exporting and Migrating.

Add Staging and Production

Use one Agent Mesh deployment for each environment, and let only a pipeline apply changes to staging and production. The following diagram shows how a change moves to production:

Diagram: Add Staging and Production

Make a Manifest for Each Environment

  1. Move the manifest into a manifests directory, then make a copy of it for each environment:

    mkdir manifests
    git mv manifest.yaml manifests/local.yaml
    cp manifests/local.yaml manifests/staging.yaml
    cp manifests/local.yaml manifests/prod.yaml
  2. In each copy, set the target and the values that differ in that environment. Keep the same resource list in each copy:

    # staging.yaml
    kind: manifest
    name: orders-staging
    target:
      url: https://staging.agent-mesh.example.com
    variables:
      orders_api_base: https://orders.staging.example.com
    resources:
      agents:
        - orders-intake
      workflows:
        - orders-status-lookup

The resource files are the same in all environments. Only the manifests differ. Keep all the manifests on one branch. Avoid a Git branch for each environment: the branches drift apart, and a merge can promote changes that you did not test. For more information about variables and secrets, see Secrets and Variables.

Give the Pipeline an API Key

The pipeline authenticates with an API key that belongs to a system user. Do not give the pipeline the credentials of a person. To create the role, the system user, and the API key, see Create the Pipeline's API Key and Managing API Keys.

With staging and production, also do the following:

  • Create a separate API key on each Agent Mesh deployment, and store each key in its own CI/CD environment, for example staging and prod.

  • Keep the production API key in a protected CI/CD environment that only the release-tag jobs and the scheduled plan can read.

  • Give the pull-request job the staging API key, never the production API key.

  • Review the API keys, roles, and system users each month. Make sure that each one has only the scopes that it needs.

Run a Plan for Each Pull Request

Start from the pipeline in A Complete Pipeline. That pipeline applies one manifest. For staging and production, change it to run the following commands.

Event Command
Pull request sam config plan -m manifests/staging.yaml
Merge to the main branch sam config apply -m manifests/staging.yaml, then sam eval run <experiment> -m manifests/staging.yaml for each experiment
Release tag sam config apply -m manifests/prod.yaml, after a check that the pipeline applied the same commit to staging
Schedule sam config plan -m manifests/prod.yaml --detailed-exit-code, on a checkout of the latest release tag, because production runs that commit

Keep the other flags of that pipeline, such as --no-interactive, and remove --url "$SAM_URL", because each manifest sets its own target.url.

  • Run only one sam config apply at a time on each Agent Mesh deployment. Use the concurrency control of your CI/CD system.

  • Agent Mesh does not record which commit an apply used, so the release-tag job must check that the pipeline applied its commit to staging. For example, configure the pipeline to record the commit ID as a CI/CD build artifact after each successful staging apply, and make the release-tag job fail when its commit ID is not in that record.

  • Do not move or delete a release tag. Each tag records a commit that the pipeline applied to production, and the scheduled plan compares production with the latest tag.

  • To roll back, revert the commit on the main branch. The pipeline applies the reverted files to staging, then create a new release tag to apply them to production. A revert does not delete resources that the reverted commit added. For more information, see Remove Resources.

The pull-request plan job runs code that is not merged yet, including the build script of each toolset that has source code. Run it only for branches in your repository.

The scheduled plan finds changes that someone made outside Git to the resources that the manifest lists. Exit status 2 means that changes are pending. A resource that someone adds outside Git counts as pending only if you pass --prune to the plan. For more information about flags, exit codes, and JSON output, see Exit Codes and JSON Output.

Remove Resources

By default, sam config apply does not delete a resource that you remove from the manifest. To delete the resource, use one of the following methods:

  • Delete the resource yourself after you apply the manifest, in the Agent Mesh UI or with sam api. The sam api delete call takes the resource's ID, not its name, and -m selects the environment. To list the IDs, run sam api /api/v2/platform/agents --paginate --jq '.data[] | {id, name}' -m manifests/staging.yaml. Then delete the resource, for example: sam api -X DELETE /api/v2/platform/agents/<id> -m manifests/staging.yaml. Your account needs the delete scope for that kind.

  • Use --prune, but only when this repository is the only source of changes to the deployment and the manifest lists every resource of its kinds on the deployment, including the built-in agents. Before you prune, run sam config plan --prune and review each delete line. In a pipeline, sam config apply --prune also requires --force.

The --prune flag deletes each resource of the declared kinds that the manifest does not list, including resources that you created in the Agent Mesh UI and the built-in agents.

Delete a resource before you delete the resources that it uses. For example, delete an agent before you delete the model that it uses.

When you delete a model, Agent Mesh undeploys the agents and workflows that use it. If you add the model again, they stay undeployed, and a later apply does not deploy them unless their files change. An apply with --prune stops before it changes anything when it would delete a model that a listed agent, workflow, or evaluator still uses, and the error names that resource. For more information, see Configuring Models with the CLI.

Troubleshooting

Symptom Cause Solution
The plan fails with a 403 error. When the manifest declares agents, the plan reads every skill, connector, and toolset, and the API key lacks the read scope for one of these kinds. Add the read scope for skills, connectors, and toolsets to the API key.
The apply fails for an agent with an error about its model configuration. The agent file specifies a model alias that the Platform service rejects. sam config plan does not check model aliases. Correct the model alias in the agent file.
A resource that you removed from Git is still on the deployment. sam config apply does not delete resources by default. See Remove Resources.

What Next?

When a second team starts to use the same environments, see Scaling to Several Teams. For a checklist of all the practices, see Best Practices for Configuration as Code.

Related Topics