Configuring Notifications (Early Access)

This feature is in the Early Access stage and under active development. Configuration schemas and behavior are subject to change. We recommend that you do not use this feature in production environments.

The notification service is on by default. To turn it off in a production environment, set SAM_FEATURE_NOTIFY_SERVICE=false as described in Enabling and Disabling Notifications.

The Solace Agent Mesh notification service delivers notifications to each user's in-app inbox and, after the user enables them, to direct messages in Slack or Microsoft Teams. This page covers the operator configuration: turning the service on or off, its database and retention, delivery through entrypoints, webhooks, and connectors, and the role-based access control (RBAC) scopes. For more information about what users see, see Receiving Notifications (Early Access).

Notification Service Overview

The notification service runs inside the Entrypoint Executor that hosts the Platform service. It serves its REST API under /api/v1/notify on the same origin as the Agent Mesh UI, behind the same authentication. When all of Agent Mesh runs as a single process on one machine, as in the desktop application or a local sam run --embedded, the notification service runs in that same process.

Agent Mesh components send notification requests over the event broker. The notification service stores each notification in its database, which backs the in-app inbox, and delivers it to the other channels that the user turned on:

  • The in-app inbox is the Notifications item in the navigation bar and the Notifications page of the Agent Mesh UI. Every notification reaches it unless the user turned its notification type off.

  • Slack and Microsoft Teams deliver a direct message from the Agent Mesh application through a running Slack Entrypoint or Teams Entrypoint, after the user enables the application in Settings.

  • Webhooks and connectors deliver to per-user destinations that you configure in YAML. For more information, see Sending Notifications to a Webhook or Connector.

The following Agent Mesh features send notifications:

  • Scheduled tasks notify the task's owner, and its creator when that is a different user, each time a run completes, fails, or times out, unless the task's own notification settings turn that off. For more information, see Scheduling Tasks (Experimental).

  • Agents and workflows send notifications with the notify_user built-in tool. For more information, see Letting Agents Send Notifications.

  • An agent that waits for a tool approval, or for an answer to the ask_user_question tool, also sends the request to the notification service, so that the user sees it in the inbox and can answer it in the chat it came from, in any browser. Tool approvals also reach Slack and Microsoft Teams.

  • Sharing a chat with another user notifies that user.

  • Budgets notify their users and budget administrators when spending crosses a budget's alert threshold or reaches its limit. For more information, see Managing Model Spend with Budgets.

For more information about how Agent Mesh components communicate over the event broker, see The Event Mesh Communication Layer.

Prerequisites

Enabling and Disabling Notifications

The notification service is on by default. To turn it off, set the following environment variable on every Agent Mesh process, and then restart the processes:

export SAM_FEATURE_NOTIFY_SERVICE=false

When the service is off:

  • The Entrypoint Executor does not open the notification database or subscribe to notification requests.

  • The Agent Mesh UI hides the Notifications item in the navigation bar, the Notifications page, and the Notifications section of Settings.

  • Slack Entrypoints and Teams Entrypoints do not deliver notifications.

The Agent Mesh UI shows notifications only when the Entrypoint Executor that serves it also runs the notification service. An Entrypoint Executor that does not host the Platform service reports notifications as off to its Agent Mesh UI, even when SAM_FEATURE_NOTIFY_SERVICE is on.

To verify that the service is running, call its health endpoint, which requires no authentication:

curl https://agent-mesh.example.com/api/v1/notify/health

A running service answers with HTTP status 200.

Choosing the Notification Database

The notification service stores the inbox, each user's notification settings, the Slack and Microsoft Teams channels that users enable, and pending input requests. It supports PostgreSQL and SQLite, and runs its own schema migrations at startup, tracked separately from the Platform service's migrations.

The Entrypoint Executor that hosts the Platform service reads the notify block of the Platform service or Web UI Entrypoint application configuration:

# Platform service application configuration
notify:
  database_url: ${NOTIFY_DATABASE_URL:- ${PLATFORM_DATABASE_URL}}
  retention_days: ${NOTIFY_RETENTION_DAYS, 30}

With this configuration, the service uses NOTIFY_DATABASE_URL when you set it, and the Platform service's database otherwise.

The ${NAME:- ${OTHER}} form falls back to another variable, and the ${NAME, 30} form falls back to a literal value. For more information about the variables, see Environment Variables.

When the configuration has no notify block, or its database_url is empty, the service also uses the Platform service's database, including the Platform service's built-in SQLite default when no database URL is configured for it.

In the desktop application and with sam run --embedded, the service stores its data in a SQLite file named sam-notify.db in the directory that SAM_DATA_DIR names. A notify.database_url in the Web UI Entrypoint or Platform service application configuration overrides the file. Where the file goes depends on SAM_DATA_DIR:

  • Both sam run --embedded and the desktop application set SAM_DATA_DIR to the Agent Mesh data directory.

  • With sam run --embedded --system-env and no SAM_DATA_DIR, the file goes in the operating system's temporary directory, which the operating system can clear, so set SAM_DATA_DIR in that case.

Setting How Long Notifications Are Kept

Every hour, one Agent Mesh instance deletes read notifications that are older than the retention period, counted from when the notification service received each one. Agent Mesh never deletes unread notifications automatically. The same job removes input requests that were answered, canceled, failed, or expired, and input requests that are still pending, once they are older than the retention period.

To change the retention period, set notify.retention_days, or NOTIFY_RETENTION_DAYS with the shipped configuration. The default is 30 days. A value of 0 or less turns the cleanup off, so read notifications stay until users delete them.

Delivering to Slack and Microsoft Teams

A Slack Entrypoint or Teams Entrypoint delivers notifications automatically while the notification service is on, and requires no notification-specific configuration. Each entrypoint announces over the event broker that it delivers notifications, and the Notifications section of Settings lists Slack or Teams for users to enable while such an entrypoint is running. Delivery always goes to a direct message between the Agent Mesh application and the user, never to a Slack or Microsoft Teams channel. On a Solace event broker, notifications sent while the entrypoint is stopped wait in a durable queue and arrive after it starts again.

How the entrypoint reaches a user depends on the application:

  • The Slack Entrypoint looks up the Slack account whose email matches the user's Agent Mesh login email, and opens a direct message with it. The user's login identity must carry a single plain email address.

  • The Teams Entrypoint identifies the user's Microsoft Teams account from a message that the user sends to the Agent Mesh application, and delivers to a private one-to-one chat with the user. A user who has never messaged the application cannot enable Microsoft Teams.

For Slack, the Bot Token must have the following scopes. The Slack Entrypoint setup already grants them.

Scope Used For
users:read and users:read.email Finding the user's Slack account by email
im:write Opening the direct message
chat:write Posting the notification
files:write Attaching a scheduled task's files

For more information about setting up the entrypoints, see Slack Entrypoints and Microsoft Teams Entrypoints.

Sending Notifications to a Webhook or Connector

You can give a user extra delivery destinations: an HTTP webhook, or a deployed connector tool such as a Slack connector's send-message tool. List them per user under notify.directory. Users then see each destination as an extra tab in the Notifications section of Settings and choose which notification types it receives. Users cannot add or remove these destinations.

# Platform service application configuration
notify:
  database_url: ${NOTIFY_DATABASE_URL:- ${PLATFORM_DATABASE_URL}}
  directory:
    - user_id: alice@example.com
      channels:
        - type: webhook
          config:
            url: https://hooks.example.com/notifications
            webhook_type: slack
            label: Team alerts

Each directory entry has a user_id and a list of channels, each with a type and a config. Note the following about the directory:

  • The user_id is the user's login ID: the user ID that your identity provider reports when the user logs in, for example alice@example.com.

  • The type is webhook or connector. A channel with another type, such as slack, teams, or email, delivers nothing.

  • To send notifications by email, use a connector channel with an email connector tool.

  • The Entrypoint Executor reads the directory at startup, so restart it after you change the directory.

  • An entry without a user_id, a channels value that is not a list, or a channel without a type stops the Entrypoint Executor from starting.

The user_id must exactly match the user's login ID. An entry whose user_id matches no user delivers nothing and reports no error.

Webhook Channel Reference

A webhook channel sends an HTTP request with a JSON body.

Field Type Required Default Description
url String Yes None The http or https URL to send to. The notification service rejects a URL that resolves to a private, loopback, or link-local address.
method String No POST The HTTP method.
headers Map of string to string No None Extra request headers. The notification service rejects Authorization, Cookie, Host, Proxy-Authorization, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, and X-Real-IP.
webhook_type String No generic The body format. generic sends a JSON object with title, message, and the notification's other fields; slack sends a Slack Block Kit message; teams sends a Microsoft Teams card.
label String No The URL's host The name that users see in parentheses on the destination's tab, for example Webhook (Team alerts).

A webhook request times out after 30 seconds.

The destination's tab appears in Settings even when the notification service rejects its URL or headers, and the webhook then receives nothing. Check the URL and headers against the preceding table when you add a webhook.

Connector Channel Reference

A connector channel calls a deployed connector tool, for example a Slack connector's send-message tool, with the notification text.

Field Type Required Default Description
tool_name String Yes None The full name of the connector tool to call, for example slack_send_message_990e8400.
str_binary String Yes None The connector type, for example slack.
args Map No None Fixed arguments passed on every call, for example the target channel.
text_arg String No text The argument that receives the notification text.
label String No The connector type, capitalized The name that users see in parentheses on the destination's tab, for example Connector (Team Slack).

A connector call times out after 30 seconds.

The following example sends each notification through a Slack connector's send-message tool to one Slack channel. The channel ID is a placeholder:

# Platform service application configuration
notify:
  directory:
    - user_id: alice@example.com
      channels:
        - type: connector
          config:
            tool_name: slack_send_message_990e8400
            str_binary: slack
            args:
              channel: C0123456789
            text_arg: text
            label: Team Slack

Letting Agents Send Notifications

The notify_user built-in tool lets an agent or a workflow tool node send a notification. To give an agent the tool, add the notification_tools group to the agent's runtime configuration. The group does not appear in the toolset selection of the Agent Mesh UI.

# Agent runtime configuration
tools:
  - tool_type: builtin-group
    group_name: notification_tools

The tool requires the notify:_:send scope. Its notifications have the Messages from agents and workflows notification type and link back to the chat that the agent ran in. For more information about the tool's parameters, see Built-In Tools.

The notify_user tool can send a notification to any user ID that the agent passes in recipient_user_id. The notify:_:send_others scope does not restrict it; that scope applies only to the REST API. Give the notification_tools group only to agents that you trust to address other users.

Controlling Access with RBAC Scopes

Every user can read and manage their own notifications without a scope. The following scopes control access beyond that.

Scope Grants
notify:_:send Using the notify_user tool from an agent or workflow.
notify:_:send_others Sending a notification to another user through the REST API, managing another user's notifications and notification settings, and answering another user's input requests.
notify:_:read_others Reading another user's notifications and notification settings, and marking their notifications read.

No user can enable or disable Slack or Microsoft Teams on behalf of another user. API keys cannot call the notification service. For more information about granting scopes, see Enabling Role-Based Access Control (RBAC) and the RBAC Reference.

Troubleshooting

Use the following entries to diagnose notification problems on your deployment.

The Agent Mesh UI Has No Notifications Item

Users see no Notifications item in the navigation bar. The notification service is off, or the Entrypoint Executor that serves the Agent Mesh UI does not host the Platform service.

To resolve this issue:

  1. Check that SAM_FEATURE_NOTIFY_SERVICE is not set to false on the Entrypoint Executor.

  2. Check that the Entrypoint Executor that serves the Agent Mesh UI also hosts the Platform service.

  3. Call /api/v1/notify/health and check that it answers with HTTP status 200.

Users Cannot Enable Slack

Clicking Enable on the Slack card shows an error that Agent Mesh is not allowed to look up Slack users by email, or that it found the account but cannot send a direct message. The Slack application's Bot Token lacks a scope, or Slack rejected the token.

To resolve this issue:

  1. On the Slack application's OAuth & Permissions page, add users:read.email for the lookup error, or im:write for the direct-message error.

  2. Reinstall the Slack application to the workspace, so that the new scopes take effect, and check that the Slack Entrypoint uses the current Bot Token.

  3. Ask the user to click Enable again.

Users Cannot Enable Slack Because Their Login Has No Email Address

Clicking Enable on the Slack card shows an error that the user's login has no single email address to look them up with. The identity provider supplies a username, or an address with a display name, in place of a plain email address.

To resolve this issue:

  1. Configure your identity provider to report the user's plain email address, for example alice@example.com, as the login email.

  2. Ask the user to log out of Agent Mesh and log in again.

  3. Ask the user to click Enable again.

Slack or Teams Is Not Listed in Settings

The External Application Notifications part of the Notifications section of Settings shows no Slack or Teams card, or says that no external applications are available. No Slack Entrypoint or Teams Entrypoint is running, or the notification service has not yet learned that the entrypoint is running.

To resolve this issue:

  1. Check that the Slack Entrypoint or Teams Entrypoint is running and connected to the same event broker namespace as the Entrypoint Executor.

  2. Wait about a minute after the entrypoint or the Entrypoint Executor restarts, because the entrypoint reports that it is running about once a minute.

  3. Refresh the Notifications section of Settings and check that the card appears.

A Webhook or Connector Destination Receives Nothing

The destination's tab appears in Settings, but the destination receives no notifications.

To resolve this issue:

  1. For a webhook, check that the URL resolves to a public address and that headers contains no header listed in Webhook Channel Reference.

  2. For a connector, check that tool_name names a deployed connector tool and that str_binary is its connector type.

  3. Check that the user did not turn off the notification types for the destination's tab.

  4. Restart the Entrypoint Executor to load the changed configuration.

What Next?

You can now run the notification service with the database, retention, and delivery channels that your deployment needs. For more information about what users see, see Receiving Notifications (Early Access).