Discovery

Discovery is a tool to discover events, schemas, and application interactions, running over your event brokers. You can use it to discover, import, and then visualize your event-driven architecture (EDA), including all associated applications, events, and schemas and their relationships from event messaging clusters such as Apache Kafka, Confluent, or Amazon MSK. You can run the Runtime Discovery multiple times, and on each run, it will discover the data, compare it to what is in Designer, allow the user to enrich it, and then import it into Event Portal's data model. Once imported, it will be available in Catalog and Designer to visualize and understand your event-driven architecture. The Discovery functionality includes a component called the Event Discovery Agent, which gathers runtime data from the event brokers.

On this page, we will dive into the detailed features and functionality of Discovery. We will walk you through some of the common activities you will perform when using Discovery. If you are getting started, take a look at Discover and Audit Your Kafka Runtime Data Flows. Additionally, we also have a guided Discovery codelabs available at our Solace Developer Codelabs portal.

Prerequisites

  • Ensure that you have the correct user role and permission in Event Portal. At the minimum, you will need Event Portal Editor permission. For instructions, refer to Managing Accounts, Roles, and Permissions.
  • The following event broker versions are currently supported:
    • Apache Kafka versions 2.2, 2.3, 2.4, 2.5
    • Confluent versions 5.3, 5.4, 5.5
    • Amazon MSK version 2.2

Installing the Offline Discovery Agent

You can install the Offline Discovery Agent as a Docker container on Linux, Mac, and Windows operating systems. Alternatively, you can download an executable file and install it locally in Windows or Linux operating systems. The Discovery Agent requires JDK/JRE 11 to run; OpenJDK 11 is bundled with the agent installation package.

Install the Offline Discovery Agent as a Docker Container

Follow the instructions below to create the Offline Discovery Agent as a Docker container in your local environment.

Mac OS and Linux

  1. Copy and paste the following commands into a terminal window. The Discovery Agent may take a few minutes to initialize.
    echo 'cMopz4m+GV60hBb8DysZna8uMP4tM84P' | docker login --username discovery-preview --password cMopz4m+GV60hBb8DysZna8uMP4tM84P solaceclouddev.azurecr.io
    	docker pull solaceclouddev.azurecr.io/maas-event-discovery-agent:latest
    	docker run \
    	--env SPRING_APPLICATION_NAME=maas-event-discovery-agent-offline \
    	--env EVENT_DISCOVERY_OFFLINE=true \
    	--env MAAS_VMR_ENABLED=false \
    	--env MAAS_HEARTBEATS_ENABLED=false \
    	--env MAAS_RESTPROXY_GATEWAY=false \
    	--name discovery_agent -p 8120:8120 \
    	-d solaceclouddev.azurecr.io/maas-event-discovery-agent:latest
  2. Go to http://localhost:8120 to start running a scan. Refer to Running a Discovery Scanfor more information

Windows

  1. Copy and paste the following commands into a terminal window. The Discovery Agent may take a few minutes to initialize.
    echo 'cMopz4m+GV60hBb8DysZna8uMP4tM84P' | docker login --username discovery-preview --password cMopz4m+GV60hBb8DysZna8uMP4tM84P solaceclouddev.azurecr.io
    	docker pull solaceclouddev.azurecr.io/maas-event-discovery-agent:latest
    	docker run `
    	--env SPRING_APPLICATION_NAME=maas-event-discovery-agent-offline `
    	--env EVENT_DISCOVERY_OFFLINE=true `
    	--env MAAS_VMR_ENABLED=false `
    	--env MAAS_HEARTBEATS_ENABLED=false `
    	--env MAAS_RESTPROXY_GATEWAY=false `
    	--name discovery_agent -p 8120:8120 `
    	-d solaceclouddev.azurecr.io/maas-event-discovery-agent:latest
  2. Go to http://localhost:8120 to start running a scan. Refer to Running a Discovery Scanfor more information

Install the Offline Discovery Agent via a Binary Executable

You can install the Offline Discovery Agent locally in Windows and Linux. For local installation, download the Discovery runtime archive from the console, and execute bin/event-discovery-agent for Linux or bin/run.bat for windows.

To download and install the Offline Discovery Agent, do the following:

  1. Log in to your PubSub+ Cloud account and select Discovery.
  2. Click the How do I run a Discovery scan?.
  3. On the pop-up dialog, click the download.
  4. When the download is complete, extract the archive and execute bin/event-discovery-agent for Linux or bin/run.bat script for Windows.
  5. Go to http://localhost:8120 to start running a scan. Refer to Running a Discovery Scanfor more information

Running a Discovery Scan

Once the Offline Discovery Agent is installed, you can configure and run a scan from your browser. The agent captures and uploads various objects and their relationships within an event-driven-architecture. To learn more about what information is captured in the Discovery scan, refer to Information Discovery Agent Captures and Uploads.

In the following examples, we will show you how to run Discovery scan on Kafka clusters and PubSub+ event brokers.

Kafka Runtime Discovery

To configure and run a Discovery scan on a Kafka Cluster, do the following:

  1. Go to http://localhost:8120.
  2. Select the Apache Kafka, Confluent, or Amazon MSK to perform a run-time discovery. In this example, we have selected Apache Kafka.
  3. Complete the input fields as shown in the example below:

    Discovery Name: The name that will be displayed on the list of discovered files in the Event Portal

    Host: Location where your event broker is hosted.

    Authentication: Authentication type. The following authentication methods are supported:

    • SASL Plain
    • SASL GSSAPI (Kerberos)
    • SASL SCRAM 256
    • SASL SCRAM 512
    • SSL

    Connector Authentication: Optional information, if you have a connector to authenticate.

    Topics Subscription: Options to scan specific topics or multiple topics.

    Schema Registry Authentication: For Confluent Run-time Discovery, you have the option to configure the Schema Registry Authentication.

  4. Click Start Scan. Once the scan is completed, you can upload the Discovery file to PubSub+ Cloud.

  5. To upload the file directly, click Upload to Solace Cloud.To manually upload the Discovery file, see Manually Add a Discovery File to PubSub+ Cloud.

PubSub+ Runtime Discovery (Preview)

You can execute a runtime Discovery scan on event broker service, software event broker, and appliance. The following event stream related data are discovered: Topics, Clients, Queues, Durable Topic Endpoints (DTEs), and Subscriptions. Once the scan is complete, a file in JSON format will be generated and available for download.

The following event broker versions are currently supported: 8.11.0.1033, 9.1.1.12 , 9.3.1.17, 9.5.0.25, 9.6.0.34.

At this Preview stage, the Run-time Discovery Agent can only run a scan and generate a JSON Discovery file with discovered objects and relationships. Uploading and staging the Discovery file in Event Portal is not yet supported. If you are a Solace customer, you can send us the sample Discovery files from the Agent output, which will help us develop and test as we implement the functionality in the next phases.

To configure and execute a runtime Discovery scan on a PubSub+ event broker, do the following:

  1. Go to http://localhost:8120.
  2. Select Solace PubSub+ Runtime Discovery.
  3. Complete the input fields as shown in the example below.

    Discovery Name: The name that will be displayed on the list of discovered files in the Event Portal.

    Client Username: The client username assigned to the Message VPN.

    Client Password: Password for the given client username.

    SEMP username: SEMP username.

    SEMP password: SEMP password.

    Client Protocol: TCP or TCPS protocol.

    Client Host: IP address or hostname of the event broker to connect to.

    SEMP Host: IP address or hostname of the event broker to connect to.

    Messaging Port: The port number the agent will use when connecting to the event broker.

    Message VPN: Message VPN name.

    SEMP Port: Port number to access the event broker. The following SEMP ports are supports:

    • Event broker service: 943.
    • Software event broker: 8080 and 1943. For software event broker versions before the 9.4.0 release, use port 943.
    • Appliance: 80.

    SEMP Scheme: SEMP URI scheme.

    Topics Subscriptions: Topics and subscriptions that you want to scan and discover.

    Scan Duration: Duration of the scan, which depends on the number of objects discovered.

  4. Click Start Scan.
  5. Once the scan is completed, you can upload the Discovery file to PubSub+ Cloud or download it.

Manually Add a Discovery File to PubSub+ Cloud

Once the scan is completed, you can upload the Discovery file to the PubSub+ Cloud directly through the Offlline Discovery Agent or by downloading and adding it manually. In this topic, we will show you how to add the Discovery file manually.

To manually add a Discovery file to PubSub+ Cloud, perform the following steps:

  1. Log in to your PubSub+ Cloud account and select Discovery on the left navigation bar.
  2. Click the Add Discovery link located on the top right.
  3. Locate your Discovery file and click open.
  4. The new Discovery file will be added to the list of discoveries in the PubSub+ Cloud. You can now import the file into Event Portal, compare it with existing Application Domains, or archive the file.

In the next few topics, we will walk you through a few things you can do with your newly uploaded Discovery file.

PubSub+ Topic Metrics Discovery (Preview)

You can run a Topic Metrics Discovery scan on event broker service, software event broker, and appliance. Once the scan is complete, a file in JSON format will be generated and available for download.

The following event broker versions are currently supported: 8.11.0.1033, 9.1.1.12 , 9.3.1.17, 9.5.0.25, 9.6.0.34.

At this Preview stage, the PubSub+ Topic Metrics Discovery displays the topic structure and associated metrics in graphical formats. This is to provide our users with a preview of future event metrics capabilities.

To configure and run a Topic Metrics Discovery scan on a PubSub+ event broker, do the following:

  1. Go to http://localhost:8120.
  2. Select Solace PubSub+ Topic Metrics Discovery to perform a run-time discovery.
  3. Complete the input fields as shown in the example below.

    Client Username: The client username assigned to the Message VPN.

    Client Password: Password for the given client username.

    Message VPN: Message VPN name.

    Secure SMF Host: Secure SMF hostname.

    Topics Subscriptions: Topics and subscriptions that you want to scan and discover.

    Scan Duration: Duration of the scan, which depends on the number of objects discovered.

  4. Click Start Scan, and then click Continue on the dialog that appears.
  5. Once the scan is completed, you can download the Discovery file as well as visualize the discovered topic hierarchy.

Staging and Committing Discovery Scans into Event Portal

Staging and committing the discovered data into Event Portal is a multi-step process.

First, you have to stage the Discovery file for import; the system will trigger an underlying comparison with Designer. The results of running the comparison will return the following:

  • Match: A match represents objects and relations that are found to be an exact match to the corresponding item in Designer. A match percentage is provided that signifies how closely the Discovery file matches the associated Application Domains that exist in Designer.
  • Discrepancy: A discrepancy represents objects and relationships identified in the Discovery that do not have an exact match in Designer. The audit algorithm will use the attributes that are being added to the Designer to determine if resources in Designer match the discovered objects. Discrepancies are highlighted with a yellow dot.

After the comparison is completed, the Discovery file's content will be moved to the Staging Siew area. The Staging View is where you can resolve discrepancies, assign objects to different Application Domains, and map undiscoverable inter-object relationships, before importing them into the Designer and Catalog. The discrepancies must be resolved before committing your event-driven architecture (EDA) into Event Portal.

In this tutorial, we will walk you through the basic steps of staging and importing your EDA into Event Portal.

  1. Log in to your PubSub+ Cloud account, and select Discovery on the left navigation bar.
  2. Select a Discovery from the Runtime Discoveries list. On the right panel, click Stage for Import or Continue Staging. You can also select Assign to Workspace (or change ) to trigger an audit against an Application Domain that already exists in the Designer. The Discovery data will be compared with the objects in Designer and the discovered objects and relationships will be moved to the staging view environment.
  3. Click the expand arrow () located at the bottom of the page to review the objects in list view. In the above example, we can see some discrepancies and warnings for "To Do" items—highlighted by yellow dots—that need action before the objects are imported.Additional information about the discovered objects, including actions required, will be displayed on the right panel.
  4. Each object must be assigned to an Application Domain before being imported or committed to Designer. If you have events moving across two or more Application Domains, you can create multiple Application Domains and assign the objects accordingly. To create an Application Domain, select the Click Here link or the + Create button on the top right corner in the discovery view.
  5. Once you've created an Application Domain, you can assign the objects to that domain. In the table, click the check box to select an object or multiple objects. On the right panel, select Set Application Domain if the objects are not assigned to an Application Domain. If the objects are already associated with an Application Domain, click Change to change the Application Domain.
  6. In the dialog that's displayed, you can choose an existing Application Domain or create a new one to associate the objects to. Then, click Set Application Domain. ou can review the changes using the side panel on the table, or see the discovery view changes.

    At this point, you can commit your EDA to Event Portal or perform additional tasks to stage the EDA before committing to the Designer. For more information, refer to Managing Objects in the Staging View.

  7. Once the discrepancies/warnings are resolved, and all the objects are in the application domain, you will notice blue dots, which means that those objects are ready to be committed into Event Portal. Click Commit to Event Portal.
  8. In the dialog that appears, you must assign a Workspace, if you haven't already done so. In the input field, write the name of the Workspace and click Commit. To learn more, refer to Workspaceor check the Managing Workspace tutorial.
  9. The newly discovered EDA will be available in Designer.

You can now extend your architecture by creating additional events, applications, and schemas. Take a look at Designerto learn about adding events, applications, and schemas to your newly discovered application domain.

Managing Objects in the Staging View

Once your Discovery scans are staged, you can perform many additional tasks to scale your event-driven architecture (EDA). You can resolve discrepancies and perform tasks such as creating and assigning Application Domains, sorting and filtering objects, setting the event flow as published or subscribed, creating or removing object associations, or deleting objects. In the example below, we will walk you through some of the common tasks you can perform in the staging view area before committing the discovered data into Event Portal.

Assigning Application Domain

You can assign Application Domain to consumer groups, topics, connectors, and schemas. Objects can be assigned to an Application Domain individually, or you can perform a bulk operation to assign an Application Domain to multiple objects simultaneously. Furthermore, you can assign the Application Domain in the table or discovery view. In the discovery view, you can reassign objects from one Application Domain to another.

To assign Application Domain using the table, do the following:

  1. Click the navigation drawer () located at the bottom of the page to review the objects in list view.
  2. Click the check box to select an object or multiple objects that you want to assign an Application Domain.
  3. Once the objects are selected, on the right panel, you will see the option to Set Application Domain if they haven't already been assigned one. If the objects are already associated to an Application Domain, the option to Change will be available.

  4. Click on Set Application Domain or Change.
  5. In the dialog that's displayed, choose an existing Application Domain or create a new one to associate the objects to.
  6. Click Set Application Domain. You can review the changes on the side panel, or see the changes in the discovery view.

To reassign Application Domain using the discovery view, do the following:

  1. In the discovery view, select an object by clicking it; to select multiple objects, press and hold the Shift key on your keyboard and then click the desired objects. Selected objects will be highlighted in green
  2. After you've selected single or multiple objects, drag those objects into an Application Domain. In the example below, we've selected multiple objects, including applications, consumer groups, connectors, and events that are currently assigned to AppDomain1, which were then dragged into AppDomain-2. The objects are now reassigned to AppDomain-2.

Setting Events as Published or Subscribed

Within an Application Domain, you can associate an event to multiple applications or connectors, and set the event flow as published or subscribed. You can perform this action in discovery view or the table. Note that the events must be assigned to an Application Domain before performing this action.

To associate events to application or connectors in the discovery view, perform the following steps:

  1. Hover your cursor over an event, a plus icon () to add an event path will display.
  2. Click the icon and drag it into an application or connector of your choice.
  3. In the dialog that appears, select is Published by or is Consumed by. You can also set a bi-directional event flow so that the application can publish and consume the same event.
  4. Click Set. Notice the event path that connects the event to the application and arrows that indicate the association type (published or subscribed).

To associate events to application or connectors using the table, perform the following steps:

  1. Click the navigation drawer () located at the bottom of the page to review the objects in list view.
  2. Select the Topics tab to view the list of available events. At this point, you can use the filter to quickly find a topic or list of topics.
  3. Select an event or multiple events by clicking the check box.
  4. On the right panel, click the plus icon beside Associate Applications.
  5. On the dialog that appears, select Produced by or Consume by and then select Application or Connector. Use the dropdown menu to select a specific application or connector.
  6. Click Set. The new event flow will be visible in the discovery view.

Setting Cross-domain Flow of Events

Sometimes events are produced by an application in one domain and are consumed by applications in other domains. For example, you may have an application in Application Domain A that is or should be subscribing to or publishing to an event in Application Domain B. To achieve this, you can set a cross-domain flow of events between the two Application Domains. To do so, follow the steps below:

  1. Select an Application Domain to set a cross-domain flow of events from that domain to another Application Domain. In this example, we will set a cross-domain flow of events AppDomain1 to AppDomain-2.
  2. Inside the Application Domain, hover your cursor over an event; a plus icon () to add an event path will display.
  3. Click on the icon and drag it to another Application Domain. For this example, we will drag the event from AppDomain1 to ApplDomain-2.
  4. On the dialog that appears, select Is Published by or is Consumed by.
  5. Under Publisher Type, select Application or Connector, depending on whether you want the event to be published or consumed by an Application or a Connector. You can also set a bi-drectional flow of events by repeating the steps.
  6. Click Set. The second Application Domain will be visible inside the first Application Domain from where the cross-domain flow was set. Notice the event path that connects the event to the second Application Domain and arrows that indicate the association type (published or subscribed).

Removing Object Associations

You can undo event to application, consumer group, and connector associations. Similarly you can also undo consumer group to application association. It is useful to remove associations if you made a mistake or want redo the association with additional setup.

To remove event to application, consumer group, and connector association, do the following:

  1. Right-click on the event path and then click Remove Associations.
  2. On the dialog that appears, click the check box beside the object.
  3. Click Remove.

To remove association between consumer group to application, delete the application to remove the association. Once the application is deleted, the consumer group will be available in the Application Domain.

Sorting and Filtering Objects

You can filter and sort objects based on whether the objects are New, Match, or by using the Display Name option to find objects in the available list. This enables you to work effectively and efficiently when you have a large lists of objects. You can quickly find a specific list of objects, view the details, or perform specific actions.

Filtering Objects

You can filter objects based on whether the objects are New, Match, or by filtering the objects using the Display Name.

To filter objects, do the following:

  1. Click the navigation drawer () located at the bottom of the page to review the objects in list view. You will see additional information about the discovered objects, including actions required.
  2. Beside the filter icon ():
    • click New to filter object that are new to the Workspace
    • click Match to filter objects that are a match
    • Use the Display Name field to filter objects using their display name. Your newly uploaded data may have numerous objects, which could make it difficult to scrub through individually. A quick filter using the object's display name narrows down the list and helps you quickly find the list of objects that meets the filtered criteria. In this example below, we have filtered topics to display only driver.ride, which will include all topics related to driver rides.
  3. You can now review the filtered objects and perform individual or bulk operations.

Sorting Objects

Objects in the tables in the Staging View area are listed in alphabetical order by their Display Name. You can sort objects in the tables in ascending or descending order on any column. By default, the list is sorted in an ascending order; notice the upward arrow. Click the upward arrow to sort the list in a descending order. If the list is not sorted, you will notice a grey upward and downward arrow.

Performing Bulk Operations

You can select multiple objects in the table and perform actions in bulk. This feature makes it easy to efficiently work with large discovery files by classifying and categorizing them quickly. You can perform the following operations in bulk: Assign Application Domain to Multiple Objects, Designate Events as Published or Subscribed, or Delete Multiple Objects.

To perform a bulk operation, do the following

  1. Click the navigation drawer () located at the bottom of the page to review the objects in list view. You will see additional information about the discovered objects, including actions required.
  2. In the table, select the multiple objects you want to use. You can choose multiple consumer groups, connectors, topics, or schemas.

  3. On the right panel, options will be displayed to do the following:
    • Set or change Application Domain
    • Delete objects
    • Set or remove association between consumer group and application
    • Associate events to an application or connector
  4. Based on the object you are working with, you can select the available option and follow the prompt.
  5. Once the action is performed, the changes will be visible in the table and also on the discovery view. For schemas, updates will be visible only in the table.

Deleting Objects

You can delete Application Domains, applications, consumer groups, connectors, topics, and schemas individually or in bulk. You can perform this action in the discovery view or the table.

To delete objects using the discovery view, do the following:

  1. Select an object or press and hold the Shift key to select multiple objects.
  2. Right-click on the selected object and click Delete. Updates will be visible in the discovery view and the table.

To delete objects using the table, do the following:

  1. Click the checkbox to select one or multiple objects on the table
  2. Click the delete icon () located on the right pane.
  3. On the dialog that appears, click the Delete button. Updates will be visible in the discovery view and the table.

Managing Workspace

After adding a Discovery file, a Workspace must be assigned before being imported into the Event Portal's data model. You can assign Workspace while staging the discovered data, which will trigger an audit against the assigned Workspace, or after staging the data. While assigning a Workspace, you can start an compare with an Application Domain in an existing Workspace or an Application Domain in a new Workspace. If the Application Domain(s) already exists in the Designer, the system does a comparison to check whether the objects exist or not, and identifies which ones are new.

For an overview information, refer to Workspace.

Assigning Discovery to an Existing Workspace

In this tutorial, we will use a Workspace to compare two Discovery files in an existing Application Domain in the Designer. Note that Workspace can also have multiple Application Domains.

  1. In the example below, we have a Discoveryfile called Acme Rideshare that has consumer groups, topics, and schemas.

  2. After we resolved the discrepancies / warnings, the file is staged and imported, and is 100% in-sync with the Event Portal. Notice that we created a new group called Acme Rideshare Group before importing it.

    The application domain is now available in the Designer.

  3. We will now add another version of the Discovery file (taken at a later time for instance) and compare it with Acme Rideshare Workspace that already exits in the Designer. To do so:
    1. Click on the Discovery and then click Compare with Designer.
    2. On the dialog that appears, select the Existing Workspace radio button.
    3. Select a previously created Workspace from the list ( For example, Acme Rideshare Version 1 in this case) that you want to compare the new discovery against.
    4. Click Change Workspace.
    5. As you can see in the image below, both the Discoveries are now assigned the same Acme Rideshare Workspace. After running the synchronization audit, the sync percentage on the new Acme Rideshare Discovery file is 94% because it has new items discovered that were not already in the Designer. Once all the items that need attention are addressed, the staged discovery can be imported into the Designer, by clicking the Commit to Event Portal button. After import, the synchronization status of that discovery should be 100%.

Assigning Discovery to a New Workspace

In this tutorial, we change the Workspace and compare the previously uploaded Acme Rideshare Discoveryfile with an application domain in a New Workspace.

To do so, follow the steps below.

  1. Select a Discovery file and click Change located on the right hand panel. .
  2. In the pop-up dialog, do the following:
    1. Select the New Workspace radio button.
    2. Add a name in the Name the Group field.
    3. Select an Application Domain to compare against. You can do a comparison with multiple application domains at the same time.
    4. Click Change Workspace. By changing the Worspace, the system compares the Discovery file with the selected Application Domain (s)
  3. In this example, the Acme Rideshare Discoveryfile is compared with the default application domain that exists in the Designer. The synchronization percentage is showing a 7% match, which is expected as we are comparing two different application domains—the Discovery audit is smart enough to understand these discrepancies.

Archiving a Discovery File

You have the option to archive a Discovery file or even delete the file permanently. You may want to use the archive option if you have already imported the file into Event Portal and to reduce clutter.

To archive a Discovery file, follow the steps below:

  1. Click the ellipsis (three dots) on the Discovery file you want to archive.
  2. Click Archive Discovery. The state of the file will change to archived.

Note that if a comparison was previously triggered on the file, then all objects and association are deleted from the database.

Once the file is archived, you have the following options:

Activate Discovery: If you activate discovery, the audit will automatically run to populate the staging area with all of the discovered objects and associations.

Delete Discovery: In this case, the Discovery file will be permanently deleted from Event Portal Runtime Discoveries list.

Additional Resources