Solace PubSub+ Troubleshooting

This topic describes how operators and developers can troubleshoot instances of Solace PubSub+ services.

A Solace PubSub+ Service Instance is backed by a Message VPN on one or more PubSub+ Software Event Brokers, depending on the plan you’ve chosen.

You can discover the Solace PubSub+ Software Event Broker backing a service instance from the Solace PubSub+ Credentials which become available by binding an app to the instance or creating a service key for the instance.

Operator Troubleshooting

The operator will have access to the logs of all the components of Solace PubSub+ for VMware Tanzu:

• Service Broker Logs
• Service Broker Agent Logs
• Solace PubSub+ Event Broker Logs

An installation of Solace PubSub+ for VMware Tanzu may be configured to use System Logging as a means of gathering all logs.

Accessing Logs

The following locations in Solace PubSub+ BOSH VMs will hold logs of interest that can help in troubleshooting issues:

• All VM job logs
  • /var/vcap/sys/log
• All Solace PubSub+ Event Broker Logs
  • /var/vcap/store/containers/pubsub/volumes/jail/logs

The service broker logs are collected in the management VM under /var/vcap/sys/log/broker-logs.

When the Management app is deployed in High Availability mode, the service broker logs are captured on both the primary and backup VMS so it does not matter from which you get the logs.

This means that you can use the bosh logs command to download the service broker logs along with the other logs from the management VM.

You can also watch service broker logs as follows:

1. Set your API endpoint to the Cloud Controller of your deployment.

   $ cf api api.YOUR-SYSTEM-DOMAIN
   Setting api endpoint to api.YOUR-SYSTEM-DOMAIN...
   OK
   API endpoint:  <span>https:</span>//api.YOUR-SYSTEM-DOMAIN (API version: 2.82.0)
   Not logged in. Use 'cf login' to log in.
   

2. Log in to your deployment and select an org and a space.

    $ cf login
    API endpoint: <span>https:</span>//api.YOUR-SYSTEM-DOMAIN
    Email> user<span>@</span>example.com
    Password>
    ```

1. Target the `solace` org and `solace-broker` space.

    ```
    $ cf target -o solace -s solace-broker
    api endpoint:   https://api.YOUR-SYSTEM-DOMAIN
    api version:    2.82.0
    user:           admin
    org:            solace
    space:          solace-broker
    ```

1. Discover the Solace PubSub+ Service Broker App name.

    ```
    $ cf apps
    Getting apps in org solace /space solace-broker as user@example.com...
    OK

    name                           requested state   instances   memory   disk   urls
    solace-broker-1.2.0            started           1/1         1G       512M   solace-broker.YOUR-SYSTEM-DOMAIN

    OK
    ```

    <p class="note"><strong>Note:</strong> Take note of the application name. In this case 'solace-broker-1.2.0'.</p>

3. To watch the logs of Solace PubSub+ Service Broker:

    ```
    $ cf logs solace-broker-1.2.0 | tee saved_solace-broker-1.2.0.txt
    Retrieving logs for app solace-broker-1.2.0 in org solace /space solace-broker as user@example.com...
    ....
    ```

### <a id='operator_diag'></a> Additional Diagnostics

In addition to regular logging, Solace PubSub+ Event Brokers tools can help gather diagnostics logs on demand.

For example, having identified a problem with a given Solace service instance, the operator can access the backing Solace PubSub+ Event Brokers to examine logs and gather diagnostics.

The backing Solace PubSub+ Event Brokers for a given service instance can be identified from the IPs used in the bindings or service keys, or mentioned in the service broker logs or message when an exception occurs.

The operator should look for the VM in the Solace PubSub+ bosh deployment with the matching IP.

Get additional diagnostics from "EnterpriseLarge/0" VM.

$ bosh ssh EnterpriseLarge/0 # sudo -i # /var/vcap/jobs/broker_agent/bin/gather_diagnostics.sh “`

Then you can look at the gathered logs under /var/vcap/store/diag.

In the event that a process fails and creates a core dump, the core file can be found under /var/vcap/store/cores. Core files are compressed using gzip.

Installation Issues

If you get the error:

  Colocated job 'solace-bosh-dns-aliases' is already added to the instance group 'compute'

Then do the following:

• Log into your bosh environment
• Execute the command bosh configs
• Look for runtime-configs with ‘solace’ in their name
• Delete the solace runtime-configs using the bosh delete-config command
• Redeploy

Upgrade Issues

The following issues may arise while upgrading a tile:

Action Failed get_task: result: 1 of 1 drain scripts failed. Failed Jobs: containers.

Action Failed get_task: result: 1 of 1 post-start scripts failed. Failed Jobs: containers.

Process 'mariadb_ctrl' Execution failed

Note: For Shared service plans, every service on a broker must be marked for upgrade before the upgrade can happen. This prevents one tenant from upgrading a router before the other tenants are ready.

Note: If some Solace PubSub+ Event Brokers have been upgraded and others have not, it is not possible to create new bindings or service keys with the non-upgraded services. This is because features such as authentication schemes might have changed with the upgrade, and would not be compatible with the services created before the upgrade.

Developer Troubleshooting

A developer using Solace PubSub+ for VMware Tanzu may encounter errors when using Cloud Foundry Command-Line Interface (cf CLI) to perform basic operations on a Solace PubSub+ for VMware Tanzu service instance.

In general, most errors are about these types:

• Reaching limits (plan limits, inventory fully used)
• Communication problems between the event broker and the VM inventory it manages
• Unexpected health state, such as a degraded, high-availability setup

While this list it not complete, it provides representative samples with explanations and possible resolutions. Some of the resolutions will require operator intervention.

cf create-service solace-pubsub enterprise-large

Server error, status code: 502, error code: 10001, message: Service broker error:
com.solace.cloudfoundry.servicebroker.exception.SolaceServiceException: No matching Solace Message VPNs available.

cf update-service my-large-instance -c '{"some_option": "some_value" }'

Server error, status code: 502, error code: 10001, message: Service broker error: Unrecognized parameter key some_option

cf update-service my-large-instance -c '{ "mqtt_tcp_route_enabled" : "false" }'

Server error, status code: 502, error code: 10001, message: Service broker error: The parameter
mqtt_tcp_route_enabled is invalid given the current configuration. It requires [ TCP Routes Enabled ]

cf update-service my-large-instance -c '{ "ldapGroupAdminReadOnly" : "cn=username1,ou=groups,dc=solace,dc=com" }'

Server error, status code: 502, error code: 10001, message: Service broker error: The parameter ldapGroupAdminReadOnly
is invalid given the current configuration. It requires [ LDAP Enabled, Management Access set to LDAP ]

cf delete-service -f my-large-instance

cf service my-large-instance

Service instance: my-large-instance
Service: solace-pubsub
Bound apps:
Tags:
Plan: enterprise-large
Description: Solace PubSub+ Event Broker for real-time, multi-protocol data distribution
Documentation url: http://docs.solace.com
Dashboard: https://enterprise-large-0.YOUR-SYSTEM.DOMAIN/#/msg-vpns/djAwNQ==?token=
YWJj.eyJhY2Nlc3NfdG9rZW4iOiJ2MDA1LWlaksjdlasdjas09dasdlkansdlakslZmRmOWFlNjM3ZGYwMDcifQ%3D%3D.eHl6

Last Operation
Status: delete failed
Message: com.solace.cloudfoundry.servicebroker.exception.SolaceServiceException:
Unable to delete Service, the associated Message VPN v001 on 10.244.0.3 is not currently available
Started: 2020-01-00T00:00:00Z
Updated: 2020-01-00T00:00:00Z

cf bind-service my-app my-ha-instance

MessageRouterException: Primary VMR 10.233.0.3 HA Group Status is degraded v001 for messageVpn v001

cf unbind-service my-app my-service-instance

Server error, status code: 502, error code: 10001, message: Service instance test-shared: Service broker error:
Operation canceled. Orphaned Endpoints Policy was violated.
Endpoints owned by v002.cu000001 must first be deleted:
        Queues: [ someQ ]
        Topics Endpoints: [ someTPE ]

cf update-service my-app -c '{"vpnOptions:"{...}}'
cf bind-service my-app my-service-instance # if LDAP is not the authentication scheme
cf unbind-service my-app my-service-instance # if LDAP is not the authentication scheme

The operation [...] is not supported on message-vpn v001 with its current configuration: replication admin-state [enabled]
and config-state [standby]. Management operations may only be run on the 'active' replication group

Dashboard Troubleshooting

403: Forbidden

Advanced Asset Management

In cases where a service or event broker can no longer be managed by regular managenent operations, the instance in question can be purged. This is done by accessing the Service Dashboard as an administrator at the solace-pubsub-broker-2.x.x app’s registered route. When a service is purged, its associated Cloud Foundry service and bindings are not deleted, and the underlying resources may or may not be freed. When an event broker is purged, the underlying deployment (either on demand or operator allocated) may not be deleted resulting in orphaned VMs requiring additional cleanup operations. Services or event brokers should only be purged as a last resort.

Service broker error: Will not bind service instance because the VMR is not expected: VMR 10.0.4.11
Expected Edition:'Solace PubSub+ Enterprise Evaluation Edition' Actual:'Solace PubSub+ Standard'