Kafka Receiver

Apache Kafka is an open-source distributed event-streaming data platform. The Kafka Receiver enables a STEP platform integrated with Apache Kafka to utilize a Kafka queue. For more information about Apache Kafka, search the web.

Prerequisites

Before setting up Kafka Receiver, read the Considerations for Setting up Kafka Receiver or Delivery (here) topic in this documentation.
Prior to configuration, clicking the Server dropdown parameter displays the required server name. Provide a selection for the dropdown parameter via the sharedconfig.properties file on the STEP application server using the case-sensitive Kafka.Server configuration property. If connecting to a cluster, use a comma-separated list.

The following example shows two Kafka server configurations where the first server is a cluster:
```
Kafka.Server.1=mybroker1a:9094,mybroker1b:9094
```
```
Kafka.Server.2=mybroker2a:9094
```
Prior to configuration, clicking the Topic dropdown parameter displays the available topics. Provide a selection for the dropdown parameter via the sharedconfig.properties file on the STEP application server using the case-sensitive Kafka.Server configuration property. For example:
```
Kafka.Topic.1 = sample4
```
For multiple Kafka server configurations, add 'Kafka.Server.2,' etc
Prior to configuration, clicking the Keystore Location dropdown parameter displays the required property name. Provide a selection for the dropdown parameter via the sharedconfig.properties file on the STEP application server using the case-sensitive Kafka.SSLKeyStoreLocation property. For example:
```
Kafka.SSLKeyStoreLocation.1=[/[path]/key_store.jks].
```
Prior to configuration, clicking the Truststore Location dropdown parameter displays the required property name. Provide a selection for the dropdown parameter via the sharedconfig.properties file on the STEP application server using the case-sensitive Kafka.SSLTrustStoreLocation property. For example:
```
 Kafka.SSLTrustStoreLocation.1=[/[path]/trust_store.jks].
```

While STEPXML is recommended for the messaging format, other formats supported for integration with STEP will also work. The descriptions below have been written for STEPXML.

Configuration

The Kafka Receiver option is available in an inbound integration endpoint (IIEP).

For the Receiver, choose Kafka Receiver.
For the Server, select the server(s) where the Kafka broker instances used by the endpoint are running.
In Topic, select the topic used by this endpoint.
For Max Events To Read Per Poll, enter the maximum number of events to read out of Kafka per invocation.

Note: Setting this too high can require allocating additional memory in the system.

In Decompress Message Content, optionally select a decompression option:

None - message content will remain compressed.
LZ4 - lossless data compression algorithm. Search the web for more information.

In Keystore Location, if SSL encryption is required, select an SSL encrypted connection to Kafka. Otherwise, leave this parameter blank.
In Keystore Password, enter the password for keystore if required.
In Keystore PrivateKey Password, enter the password of the private key in the keystore file, if needed.
In Truststore Location, if SSL encryption is required, select an SSL encrypted connection to the Kafka. Otherwise, leave this parameter blank.
In Truststore Password, enter the password for truststore if required.
In Topic Offset, disable the IIEP to edit the value. The Kafka offset tracks the sequential order that messages are received by Kafka topics. Enter the offset from which STEP should read messages. For more information, refer to Kafka Topic Offset on the web.
Click the Next button to continue with the IIEP - Configure Endpoint step (here), or the Finish button to close the wizard.

Execution Report

The Kafka Receiver execution report includes the following entries:

When the Kafka broker is running but no messages are retrieved, the number of events, the Topic, and the Topic Offset are reported.
When the Kafka broker is not running, the IIEP fails, and a minimum number of entries are reported since no messages are being processed.