Discover and Import PubSub+ Data Flows
This tutorial will teach you how to scan and import a PubSub+ Discovery into Designer. The following topics are covered in this tutorial:
- Install the Discovery Agent (Optional)
- Run A Discovery Scan (Optional)
- Add a Discovery Sample
- After you've completed importing events and client delivery endpoints (CDE) from your Discovery, you can further Design and Model Your EDA.
Before you begin
- To complete these tutorials, you must have a PubSub+ Cloud account. If you haven't already, complete the Accessing PubSub+ Cloud tutorial.
- Ensure that you have the correct user role and permission in Event Portal. At minimum, you will need the Event Portal Manager permission. You will also need the Cluster Manager Viewer permission to get the connection information of the event broker service you want perform the Discovery. For more information, refer to Managing Users, Roles, and Permissions. Note that the Administrator, Event Portal Manager, and Event Portal Viewer user roles will all count towards the Event Portal's maximum user limit.
You can install the Discovery Agent as a Docker container on Linux, Mac, and Windows operating systems. Alternatively, you can download an executable file and install it locally in Windows or Linux operating systems. To learn how install the Discovery agent, refer to Installing the Runtime Discovery Agent
In this tutorial, we will show you have to install it using Docker.
You can execute a runtime Discovery scan on event broker service, software event broker, and appliance. The following event stream related data are discovered: Topics, Clients, Queues, Durable Topic Endpoints (DTEs), and Subscriptions. Once the scan is complete, a file in JSON format will be generated and available for download.
RuntimeDiscovery has been tested against broker versions 184.108.40.2063, 220.127.116.11 , 18.104.22.168, 22.214.171.124, 126.96.36.199 so far, but works on all supported PubSub+event broker versions.
There are sample Discoveries in the console that you can use to understand and learn the Discovery process.
To use a sample Discovery, perform the following steps:
- There are a few ways to access the samples:
- If you are a new user with a trial account, on the Topology screen, click the sample you want to try.
- Click Actions (...) located on the top right and select Use Sample Discovery File > Use Sample and follow the prompt.
Click the Learn & Help icon and select Event Portal Sample Files and follow the prompt.
- The selected Discovery sample will be available on the list. You can now import the sample Discovery into Designer.
After you have a Discovery file either directly uploaded into PubSub+ Cloud or manually added into Discovery, you're ready to import the discovered data to Designer so that you can visualize your event-driven architecture (EDA).
As part of importing a Discovery to Designer, you must associate the discovered data with an application domain. To do this, the first thing you need to do is to assign the Discovery to a Logical Event Mesh (LEM). A Logical Event Mesh represents the event mesh over which associated events flow within an EDA. The following steps show you how to import the Discovery and create a LEM.
- Log in to your PubSub+ Cloud account, if you haven't already.
- Select the Discovery card or icon on the navigation bar.
- On the Discovery: Runtime Discoveries page, select the Discovery that you want to import to Designer. A panel will expand on the right where you can assign the Discovery to a LEM or view the Discovery details.
- Select Import to Designer located under Assign to Logical Event Mesh.
On the dialog that appears, you can import the Discovery to an existing LEM or create a new LEM. Assuming this is your first Discovery, you must first create a LEM. Select Create New and do the following:
- Give a Name to the LEM. This name must be unique in the account.
- Optionally, write the Description.
- In the Level Delimiter "
\" will be set by default for PubSub+ discoveries.
- Click Continue Import.
After you create a LEM, the discovered data will be visible on the screen. In the example below, the Discovery file contains the discovered topics and CDEs. Note that if you use your own Discovery file, it might not have any CDEs. As part of importing a Discovery into Designer, you must associate the discovered data to an application domain in the EDA. To do so, you must following tasks:
When importing a Discovery into Designer you must associate topics to events. You can traverse the Topic Tree that consists of a number of topic level and compress some of the levels with variables that are substituted at run time. Once you have identified the topics you want to use, you can turn them into events and add them to an application domain.
- On the Add discovered items to Event Portal page, select the Topics tab, and expand each topic to drill down the hierarchy of the topic tree. At any topic level, you can optionally create variable to make that level dynamic.
If we drill down further into one of the topics, we can see a long event with a number of topic levels. While we could create this event as it is, it is more useful to use variables at a topic level to build a dynamic topic address for events. We can do that by compressing individual topic levels into variables, which will be replaced at runtime with the application's literal levels. For detailed information about creating variables, refer to Creating Variables for a Topic Level.
Select the check box at the topic level beside the name of the event instance you want to create a variable.
(Optionally) If you want to create a variable to represent all the event instances at that level, click Select All.
Click Create Variable.
- In the Create Variable dialog, enter a unique name for the variable in the Name field.
Optionally, you can create an enum set that can contain a list of pre-defined values. To learn more about enum, refer to step 7 in Creating Variables for a Topic Level
- Once we have created all the variables we need, we can create a new event for that data. To do so, click the Add Events () beside the topic for which you want to create an event.
- In the dialog that appears, do the following:
- Type a name for the event in Name field.
- Assign an application domain. You can perform one of the following actions:
- Select Existing Application Domain and then choose an existing domain from the drop-down list.
- Select New Application Domain and type a name to create an Application Domain.
Click Create Event.
On the right-most panel, you'll see the event you added to the application domain and in the menu, you'll see an event.
Client Delivery Endpoints (CDEs) are model extensions that represent client endpoints such as event queue, durable topic endpoint, and direct client endpoint. These endpoints attract messages based on topic subscriptions. These represent the data that you're interested in. To learn more about CDEs, refer to Client Delivery Endpoints.
In the example below, we have a few CDEs in the Discovery file. We can now add new applications and map the CDEs to those applications.
- On the Add discovered items to Event Portal, click on the Client Delivery Endpoints tab.
- Optionally, filter on the Name, type of Client Delivery Endpoint, or Client Username using a combination of the first and second dropdown lists.
- If you have created an application domain with an application within it, proceed to the next step, otherwise the complete the following steps as required.
- If you don't have an application domain, click Add Domain in the Import Tool panel.
- In the dialog that appears, provide the Name, Topic Domain and Description for the application domain, and click Save.
- If you don't have an Application, you can expand an application domain in the Import Tool panel and then click Add Application.
- In the Create Applcation dialog, type a name for the application in the Name field; optionally type a description in the Description field.
- Select at least one CDE using the check box beside it . You can select multiple CDEs to add to an application at the same time.
- Click Add to Application.
- In the Place in Application dialog:
- Select an application domain from the first list.
- Select an aplication from the second list.
- Click Add Application.
After you've added events and Client Delivery Endpoints to your EDA, click Return to Designer on the bottom-right of the page to further design and model your EDA.
For more information about using Designer, check out the Design and Model Your Event-Driven Architecture tutorial.
- Learn Discovery's advanced features and functionality.
- Refer to Design and Model Your Event-Driven Architecture or Browse Event Streams and Visualize their Relationshipsto get started with Designer and Catalog.