The Kafka connector allows connection to Apache’s Kafka message bus on a real-time streaming basis. Specifically the connector allows Panopticon Designer (Desktop) to subscribe to XML, JSON, Text or FIX based messages that are published on topics. The data format itself is arbitrary, and consequently, the connection includes the message definition.
When creating a new data table, select Kafka from the Connect to data dialog.
The Kafka Connection dialog is displayed:
Enter the connection details including:
List of host/port pairs of Kafka servers used to bootstrap connections to a Kafka cluster.
By default, the value is localhost:9092. However, this can be overridden by specifying another bootstrap server in the External Settings text box (as specified in step 3).
Schema Registry Host
Where the Schema Registry is located. This can be in a different location from the Kafka cluster.
Schema Registry Port
The port number of the schema registry which provides the serving layer for the metadata. Default is 8081.
To support authentication (i.e., username and password), expand External Settings to specify the system settings for the Designer and Server.
If there is no need to connect to the Panopticon Visualization Servers and publish workbooks, enter the External Settings in the Designer text box only.
If the bootstrap server is not secure, then there is no need to authenticate and you may leave the External Settings blank.
Below is an example of system settings for an SASL authentication in the Designer text box:
Below is an example of system settings for an SASL authentication in the Server text box:
org.apache.kafka.common.security.plain.PlainLoginModule required \
Click Fetch Topics to populate the Topic drop-down list. Initially, the first topic in the list is displayed in the Topic drop-down box.
NOTE: Ensure that the ability to ping is enabled in the ZooKeeper Host. Otherwise, if ping is disabled, the Fetch Topics button will not be able to populate the list of topics and you need to manually enter the topic names.
Click the drop-down list to search and select the desired topic.
For the unprotected broker endpoint:
For the secure broker endpoint:
If the selected topic is available or currently running in the Schema Registry Host and Port, the schema is displayed. This populates the list of columns, with the data type found from inspecting the first ‘n’ rows of the files.
The dialog may change, depending on the type of selected topic.
For Avro topics:
Note that the Update Schema button becomes available.
For non-Avro topics:
Select the corresponding steps to proceed in retrieving the data table using the Apache Kafka connector: