Discover and Import Kafka Runtime Data Flows

This tutorial will teach you how to scan and import a Kafka Discovery into Designer. We will use a sample Discovery to walk you through the steps, but you can also use your own Discovery file.

The following topics are covered in this tutorial:

  1. Install the Discovery Agent (Optional)
  2. Run A Discovery Scan(Optional)
  3. Add a Discovery Sample
  4. Import Discovery to Designer and complete the following to visualize your event-driven architecture (EDA):

  5. After you've completed the above steps, you can further Design and Model Your EDA.

Before you begin

  • To complete these tutorials, you must have a PubSub+ Cloud account . If you haven't already, complete the Accessing PubSub+ Cloud tutorial.
  • Ensure that you have the correct user role and permission in Event Portal. At minimum, you will need the Event Portal Manager permission. For more information, refer to Managing Users, Roles, and Permissions. Note that the Administrator, Event Portal Manager, and Event Portal Viewer user roles will all count towards the Event Portal's maximum user limit.
  • The following event broker versions are currently supported:
    • Apache Kafka versions 2.2, 2.3, 2.4, 2.5
    • Confluent versions 5.3, 5.4, 5.5
    • Amazon MSK version 2.2

Install the Discovery Agent

You can skip this step if you are using a sample Discovery available in PubSub+ Cloud.

You can install the Discovery Agent as a Docker container on Linux, Mac, and Windows operating systems. Alternatively, you can download an executable file and install it locally in Windows or Linux operating systems. To learn how install the Discovery agent, refer to Installing the Runtime Discovery Agent

Run A Discovery Scan

This step is optional if you choose to use a sample Discovery available in PubSub+ Cloud.

Once the Discovery Agent is installed, you can configure and run a scan from your browser. To learn how to run a Discovery scan and upload the discovered data, refer to Kafka Runtime Discoveryor Uploading Discovery to PubSub+ Cloud.

Add a Discovery Sample

There are sample discoveries in the console that you can use to understand and learn the Discovery process.

To use a sample Discovery, perform the following steps:

  1. There are a few ways to access the samples:
    • If you are a new user with a trial account, on the Topology screen, click the sample you want to try.
    • Click Actions (...) located on the top right and select Use Sample Discovery File > Use Sample and follow the prompt.
    • Click the Learn & Help icon and select Event Portal Sample Files and follow the prompt.

  2. The selected Discovery sample will be available on the list. You can now import the sample Discovery into Designer.

Import Discovery to Designer

After you have a Discovery file either directly uploaded into PubSub+ Cloud or manually added into Discovery, you're ready to import it to Designer so that you can visualize your event-driven architecture (EDA).

As part of importing a Discovery to Designer, you must associate the discovered data with an application domain. To do this, the first thing you need to do is to assign the Discovery to a Logical Event Mesh (LEM). A Logical Event Mesh represents the event mesh over which associated events flow within an EDA. The following steps show you how to import the Discovery and create a LEM.

  1. Select the Discovery you want to import and click Import to Designer A panel will expand on the right where you can assign the Discovery to a LEM or view additional details.
  2. Select Import to Designer located under Assign to Logical Event Mesh.
  3. On the dialog that appears, you can import the Discovery to an existing LEM or create a new LEM. Assuming this is your first Discovery, you must first create a LEM. Select Create New and do the following:
    1. Give a Name to the LEM. This name must be unique in the account.
    2. Optionally, write the Description.
    3. In the Level Delimiter drop-down list, select the delimiter that matches your Discovery. You have the option to select a dot (.), hyphen (-), or an underscore (_).
  4. Click Continue Import. In the next few topics, we will show you have to manage the discovered data. The example below shows the list of data that were discovered in the sample. It has a simple topic hierarchy with seven topics, connecters, consumer groups, and, schemas.

    After you create a LEM, the discovered data will be visible on the screen .As part of importing a Discovery into Designer, you must associate the discovered data to an application domain in the EDA. To do so, you must following tasks:

Create an Event Type

From the list of discovered topics, you must create an event type that you can add to the application domain of your choice. In this example, the sample we are using has only on one topic level. However, there may be cases where you may need to traverse the topic tree that consists of multiple topic levels. In that case, you can compress some of the levels with variables that are substituted with specific values during run time. Once you have identified the topics you want to use, you can turn them into events and add them to an application domain. For detailed information on topic levels, refer to Managing Events.

To create an event, do the following:

  1. On the Add discovered items to Event Portal page, select the Topics tab. Since we are using a sample Discovery, which has a single topic level, we don't need to create any variables.
  2. Click the create event icon () beside the topic for which you want to create an event.
  3. In the dialog that appears, do the following:
    1. Type a name for the event in Name field.
    2. The Value Schema and Key Schema fields will be populated with the discovered values. If those fields are empty, you can configure it after importing it to Designer.
    3. Assign an application domain. You can perform one of the following actions:
      • Select Existing Application Domain and then choose an existing domain from the drop-down list.
      • Select New Application Domain and type a name to create an application domain.

Since we did not previously have an application domain, we created one and assigned the events to that domain. In the example below, we have a new application domain with seven associated events.

Import Connectors as Applications

In Event Portal, a Kafka Connector is an application type you select to configure associated published and/or subscribed events and a set of Kafka-native attributes. When importing a Kafka Discovery, the connectors must be imported as applications. To learn more about how Kafka connectors are supported in Event Portal, refer to Solace to Kafka Connectors.

Each discovered connector is marked with a yellow status indicator on the left-side of the name of the object. The yellow status indicators shows that a connector is not associated with an application domain. Once the connector is associated with an application domain, the yellow status indicator is removed.


To import a connector as an application, do the following:

  1. On the Add discovered items to Event Portal, click on the Connectors tab.
  2. (optional) If you already have an existing application domain, proceed to the next step, otherwise complete the following steps as required:
    1. Click Add Domain in the Import Tool panel.
    2. In the dialog that appears, provide the Name, Topic Domain and Description for the application domain.
    3. Click Save. Your new application domain will be visible in the Import Tool panel.
  3. Select at least one connector using the check box beside it. You can select multiple connectors to import as application at the same time.
  4. Click Import as Application.
  5. In the dialog that appears:
    1. Select an application domain.
    2. Click Import.

The connectors will be imported as applications inside the associated application domain. After the connectors are added to the application domain, the yellow status will disappear.

Map Consumer Groups to Applications

For Kafka discoveries, you must map the consumer groups to applications through a consumer group type Client Delivery Endpoints (CDEs). CDEs are Event Portal data model extensions that model how subscribing applications attract the appropriate event types. CDE model Kafka's consumer groups in Event Portal.

Each discovered consumer group is marked with a yellow status indicator on the left-side of the name of the object. The yellow status indicators means that consumer group is not mapped to an application’s client delivery endpoint (CDE). Once the consumer groups are mapped to respective applications, the yellow status indicator is removed.


To map a consumer group to a CDE in an application, do the following:

  1. On the Add Architecture from Discovery File, click on the Consumer Groups tab.
  2. (Optional) Filter on the Name of the consumer group.
  3. (Optional) If you have created an application domain with an application within it, proceed to the next step, otherwise complete the following steps as required.
    • If you don't have a domain, click Add Domain in the Import Tool panel.
      • In the dialog that appears, provide the Name, Topic Domain and Description for the application domain, and click Save.
    • If you don't have an Application, you can expand an Application Domain in the Import Tool panel and then click Add Application.
      • In the dialog that appears, type a name for the application in the Name field; optionally type a description in the Description field.
  4. Select at least one consumer group using the check box beside it . You can associate multiple consumers to an Application at the same time.
  5. Click Add to Application.
  6. In the dialog that appears:
    1. Select Convert to individual applications or Merge into one application. If you have ten consumer groups and you want to create ten applications, select the first option. If you want all the consumer groups to be part of one application, select the second option. Sometimes many consumer groups are discovered that belong to the same application; in that case, you can use the option to merge those consumer groups into one application. However, this will depend on what is discovered and how you want to map the discovered data.
    2. Select the Application Domain.
    3. (Optional) Use Consumer Group Name as Application Name is selected by default to ensure the applications names will be the same as consumer group names To give a new name to the application, deselect the checkbox and write your preferred name.
    4. Click Import to Designer.

The consumer groups will be imported as applications inside the associated application domain. After the consumer groups are added to the application domain, notice that the yellow status is removed.

Add Schemas to Application Domain

For Kafka Discovery, you need to add schemas to an Application Domain, so that you can view the schema and it's relationship in Designer and Catalog. Note that you cannot import primitive type schemas from Discovery, but you can create them in Designer for an event type.

To add schemas to an application domain, do the following:

  1. On the Add Architecture from Discovery File, click on the Schemas tab.
  2. Optionally, filter on the Name, type of Schema using a combination of the first and second dropdown lists.
  3. (OptionaI) f you have an existing application domain, proceed to the next step, otherwise the complete the following steps as required.
    1. Click Add Domain in the Import Tool panel.
    2. In the dialog that appears, provide the Name, Topic Domain and Description for the application domain.
    3. Click Save. Your new application domain will be visible in the Import Tool panel.
  4. Select at least one schema using the check box beside it. You can select multiple schemas add to an application domain.
  5. Click Add to Application Domain.

  6. In the dialog that appears do one of the following:

    • Select Existing Application Domain, choose an a domain from the drop-down list, and click Add.
    • Select New Application Domain, give a Name to the domain, and click Add.

After a schema has been added, the yellow status beside the schema disappears, and on the Import Tool panel, the schema number will increment under the associated application domain.

Design and Model Your EDA

After you've added the discovered events, schemas, connectors, and consumer group, click Return to Designer. The imported Discovery will be available in Designer, where you can further evolve the design and model of your EDA.


Double-click the application domain to view the discovered applications, events, and schemas.

For more information about using Designer, check out the Design and Model Your Event-Driven Architecture tutorial.

Next Steps