Developer Guide - T-Systems · 1.3.1 Configuring Kafka Clients in Java This section describes how to add Kafka clients in Maven, and use the clients to access Kafka instances and

Distributed Message Service

Developer Guide

Date 2020-02-20

Contents

1 Kafka Premium Developer Guide........................................................................................ 11.1 Overview.................................................................................................................................................................................... 11.2 Collecting Connection Information................................................................................................................................... 21.3 Java...............................................................................................................................................................................................31.3.1 Configuring Kafka Clients in Java...................................................................................................................................31.3.2 Setting Up the Java Development Environment.....................................................................................................101.4 Python....................................................................................................................................................................................... 151.5 Recommendations on Client Usage............................................................................................................................... 17

2 DMS Kafka Developer Guide.............................................................................................. 192.1 Overview.................................................................................................................................................................................. 192.2 Preparing the Environment............................................................................................................................................... 192.3 Creating a Project................................................................................................................................................................. 242.4 Configuring Parameters...................................................................................................................................................... 262.5 Running the Sample Project..............................................................................................................................................272.6 Compiling the Sample Project Code...............................................................................................................................292.7 Code of the Sample Project...............................................................................................................................................302.8 Using the Enhanced Java SDK.......................................................................................................................................... 302.9 Recommended Parameter Settings for Kafka Clients.............................................................................................. 32

3 Change History...................................................................................................................... 36

Distributed Message ServiceDeveloper Guide Contents

2020-02-20 ii

1 Kafka Premium Developer Guide

1.1 OverviewKafka premium instances are compatible with Apache Kafka and can be accessedusing open-source Kafka clients. To access an instance in SASL mode, you wouldalso need certificates.

This document describes how to collect instance connection information, such asthe instance connection address, certificate required for SASL connection, andinformation required for public access. It also provides examples of accessing aninstance in Java and Python.

The examples only demonstrate how to invoke Kafka APIs for producing andconsuming messages. For more information about the APIs provided by Kafka,visit the official Kafka website.

Client Network Environment

A client can access a Kafka instance in any of the following three modes:

1. Within a Virtual Private Network (VPC)

If the client runs an Elastic Cloud Server (ECS) and is in the same region andVPC as the Kafka instance, the client can access the instance using an IPaddress within a subnet in the VPC.

2. Using a VPC peering connection

If the client runs an ECS and is in the same region but not the same VPC asthe Kafka instance, the client can access the instance using an IP addresswithin a subnet in the VPC after a VPC peering connection has beenestablished.

3. Over public networks

If the client is not in the same network environment or region as the Kafkainstance, the client can access the instance using a public network IP address.

For public access, modify the inbound rules of the security group configuredfor the Kafka instance, allowing access over port 9095.

Distributed Message ServiceDeveloper Guide 1 Kafka Premium Developer Guide

2020-02-20 1

https://cwiki.apache.org/confluence/display/KAFKA/Clients

https://obs.eu-de.otc.t-systems.com/dms-demo/cert.zip

http://kafka.apache.org/documentation/#api

The three modes differ only in the connection address for the client to access the instance.This document takes intra-VPC access as an example to describe how to set up thedevelopment environment.If the connection times out or fails, check the network connectivity. You can use telnet totest the connection address and port of the instance.

1.2 Collecting Connection Information

Obtaining Kafka Instance Information1. Instance connection address and port

Obtain the connection addresses and port numbers on the Basic Informationtab page. Configure all the three addresses listed on the page for the client.For public access, you can use the connection addresses listed in the PublicAccess section.

Figure 1-1 Viewing the connection addresses and ports of brokers of a Kafkainstance

2. Topic nameObtain the topic name from the Topic Management tab page of the instanceas shown in the following figure.

Figure 1-2 Viewing the topic name

3. SASL connection informationIf SASL access is enabled for the instance, the username, password, and SSLcertificate are required. The username and password are set during instancecreation.Obtain the username on the Basic Information tab page. If the password islost, you can reset Kafka password.


2020-02-20 2



Figure 1-3 Resetting Kafka password

Figure 1-4 Viewing the username used for SASL access

1.3 Java

1.3.1 Configuring Kafka Clients in JavaThis section describes how to add Kafka clients in Maven, and use the clients toaccess Kafka instances and produce and consume messages. To check how thedemo project runs in IDEA, see Setting Up the Java Development Environment.

The Kafka instance connection addresses, topic name, and user information usedin the following examples are obtained in Collecting Connection Information.

Adding Kafka Clients in Maven//Kafka premium instances support Kafka 2.3.1. Use the same version for clients. <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>2.3.1</version> </dependency>

Preparing Kafka Configuration FilesThe following describes example producer and consumer configuration files. IfSASL is not enabled for the Kafka instance, comment out lines regarding SASL. IfSASL has been enabled, set SASL configurations for encrypted access.

● Producer configuration file (the dms.sdk.producer.properties file in the demoproject)The information in bold is specific to different Kafka instances and must bemodified. Other parameters can also be added.#The topic name is in the code for producing or consuming messages.########################Obtain the information about Kafka instance brokers on the console. For example: bootstrap.servers=100.xxx.xxx.87:9095,100.xxx.xxx.69:9095,100.xxx.xxx.155:9095


2020-02-20 3

bootstrap.servers=ip1:port1,ip2:port2,ip3:port3#Producer acknowledgementacks=all#Method of turning the key into byteskey.serializer=org.apache.kafka.common.serialization.StringSerializer#Method of turning the value into bytesvalue.serializer=org.apache.kafka.common.serialization.StringSerializer#Memory available to the producer for bufferingbuffer.memory=33554432#Number of retriesretries=0########################Comment out the following parameters if SASL access is not enabled.########################Configure the JAAS username and password on the console.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="username" \ password="password";#SASL mechanismsasl.mechanism=PLAIN#Encryption protocol. Currently, only SASL_SSL is supported.security.protocol=SASL_SSL#Location of ssl.truststoressl.truststore.location=E:\\temp\\client.truststore.jks#Password of ssl.truststoressl.truststore.password=dms@kafka# Disable certificate domain name verification.ssl.endpoint.identification.algorithm=

● Consumer configuration file (the dms.sdk.consumer.properties file in thedemo project)The information in bold is specific to different Kafka instances and must bemodified. Other parameters can also be added.#The topic name is in the code for producing or consuming messages.########################Obtain the information about Kafka instance brokers on the console. For example: bootstrap.servers=100.xxx.xxx.87:9095,100.xxx.xxx.69:9095,100.xxx.xxx.155:9095bootstrap.servers=ip1:port1,ip2:port2,ip3:port3#Unique string to identify the group of consumer processes to which the consumer belongs. Configuring the same group.id for different processes indicates that the processes belong to the same consumer group.group.id=1#Method of turning the key into byteskey.deserializer=org.apache.kafka.common.serialization.StringDeserializer#Method of turning the value into bytesvalue.deserializer=org.apache.kafka.common.serialization.StringDeserializer#Offset reset policyauto.offset.reset=earliest########################Comment out the following parameters if SASL access is not enabled.########################Configure the JAAS username and password on the console.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="username" \ password="password";#SASL mechanismsasl.mechanism=PLAIN#Encryption protocol. Currently, only SASL_SSL is supported.security.protocol=SASL_SSL#Location of ssl.truststoressl.truststore.location=E:\\temp\\client.truststore.jks#Password of ssl.truststoressl.truststore.password=dms@kafka# Disable certificate domain name verification.ssl.endpoint.identification.algorithm=


2020-02-20 4

Producing Messages● Test code

package com.dms.producer;

import org.apache.kafka.clients.producer.Callback;import org.apache.kafka.clients.producer.RecordMetadata;import org.junit.Test;

public class DmsProducerTest { @Test public void testProducer() throws Exception { DmsProducer<String, String> producer = new DmsProducer<String, String>();

try { for (int i = 0; i < 10; i++){ String data = "The msg is " + i; //Enter the name of the topic you created. There are multiple APIs for producing messages. For details, see the Kafka official website or the following code. producer.produce("topic-0", data, new Callback() { public void onCompletion(RecordMetadata metadata, Exception exception) { if (exception != null) { exception.printStackTrace(); return; } System.out.println("produce msg completed"); } }); System.out.println("produce msg:" + data); } }catch (Exception e) { // TODO: Exception handling e.printStackTrace(); }finally { producer.close(); }

}}

● Message production codepackage com.dms.producer;

import java.io.BufferedInputStream;import java.io.FileInputStream;import java.io.IOException;import java.io.InputStream;import java.net.URL;import java.util.ArrayList;import java.util.Enumeration;import java.util.List;import java.util.Properties;

import org.apache.kafka.clients.producer.Callback;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.Producer;import org.apache.kafka.clients.producer.ProducerRecord;

public class DmsProducer<K, V> { //Add the producer configurations that have been specified earlier. public static final String CONFIG_PRODUCER_FILE_NAME = "dms.sdk.producer.properties";

private Producer<K, V> producer;

DmsProducer(String path)


2020-02-20 5

{ Properties props = new Properties(); try { InputStream in = new BufferedInputStream(new FileInputStream(path)); props.load(in); }catch (IOException e) { e.printStackTrace(); return; } producer = new KafkaProducer<K,V>(props); } DmsProducer() { Properties props = new Properties(); try { props = loadFromClasspath(CONFIG_PRODUCER_FILE_NAME); }catch (IOException e) { e.printStackTrace(); return; } producer = new KafkaProducer<K,V>(props); }

/** * Producing messages * * @param topic Topic * @param partition partition * @param key Message key * @param data Message data */ public void produce(String topic, Integer partition, K key, V data) { produce(topic, partition, key, data, null, (Callback)null); }

/** * Producing messages * * @param topic Topic * @param partition partition * @param key Message key * @param data Message data * @param timestamp timestamp */ public void produce(String topic, Integer partition, K key, V data, Long timestamp) { produce(topic, partition, key, data, timestamp, (Callback)null); } /** * Producing messages * * @param topic Topic * @param partition partition * @param key Message key * @param data Message data * @param callback callback */ public void produce(String topic, Integer partition, K key, V data, Callback callback) { produce(topic, partition, key, data, null, callback); }

public void produce(String topic, V data) { produce(topic, null, null, data, null, (Callback)null); }


2020-02-20 6

/** * Producing messages * * @param topic Topic * @param partition partition * @param key Message key * @param data Message data * @param timestamp timestamp * @param callback callback */ public void produce(String topic, Integer partition, K key, V data, Long timestamp, Callback callback) { ProducerRecord<K, V> kafkaRecord = timestamp == null ? new ProducerRecord<K, V>(topic, partition, key, data) : new ProducerRecord<K, V>(topic, partition, timestamp, key, data); produce(kafkaRecord, callback); }

public void produce(ProducerRecord<K, V> kafkaRecord) { produce(kafkaRecord, (Callback)null); }

public void produce(ProducerRecord<K, V> kafkaRecord, Callback callback) { producer.send(kafkaRecord, callback); }

public void close() { producer.close(); }

/** * get classloader from thread context if no classloader found in thread * context return the classloader which has loaded this class * * @return classloader */ public static ClassLoader getCurrentClassLoader() { ClassLoader classLoader = Thread.currentThread() .getContextClassLoader(); if (classLoader == null) { classLoader = DmsProducer.class.getClassLoader(); } return classLoader; }

/** * Load configuration information from classpath. * * @param configFileName Configuration file name * @return Configuration information * @throws IOException */ public static Properties loadFromClasspath(String configFileName) throws IOException { ClassLoader classLoader = getCurrentClassLoader(); Properties config = new Properties();

List<URL> properties = new ArrayList<URL>(); Enumeration<URL> propertyResources = classLoader .getResources(configFileName); while (propertyResources.hasMoreElements()) {


2020-02-20 7

properties.add(propertyResources.nextElement()); }

for (URL url : properties) { InputStream is = null; try { is = url.openStream(); config.load(is); } finally { if (is != null) { is.close(); is = null; } } }

return config; }}

Consuming Messages● Test code

package com.dms.consumer;

import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.junit.Test;import java.util.Arrays;

public class DmsConsumerTest { @Test public void testConsumer() throws Exception { DmsConsumer consumer = new DmsConsumer(); consumer.consume(Arrays.asList("topic-0")); try { for (int i = 0; i < 10; i++){ ConsumerRecords<Object, Object> records = consumer.poll(1000); System.out.println("the numbers of topic:" + records.count()); for (ConsumerRecord<Object, Object> record : records) { System.out.println(record.toString()); } } }catch (Exception e) { // TODO: Exception handling e.printStackTrace(); }finally { consumer.close(); } }}

● Message consumption codepackage com.dms.consumer;

import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import java.io.BufferedInputStream;import java.io.FileInputStream;import java.io.IOException;import java.io.InputStream;import java.net.URL;


2020-02-20 8

import java.util.*;

public class DmsConsumer {

public static final String CONFIG_CONSUMER_FILE_NAME = "dms.sdk.consumer.properties";

private KafkaConsumer<Object, Object> consumer;

DmsConsumer(String path) { Properties props = new Properties(); try { InputStream in = new BufferedInputStream(new FileInputStream(path)); props.load(in); }catch (IOException e) { e.printStackTrace(); return; } consumer = new KafkaConsumer<Object, Object>(props); }

DmsConsumer() { Properties props = new Properties(); try { props = loadFromClasspath(CONFIG_CONSUMER_FILE_NAME); }catch (IOException e) { e.printStackTrace(); return; } consumer = new KafkaConsumer<Object, Object>(props); } public void consume(List topics) { consumer.subscribe(topics); }

public ConsumerRecords<Object, Object> poll(long timeout) { return consumer.poll(timeout); }

public void close() { consumer.close(); }

/** * get classloader from thread context if no classloader found in thread * context return the classloader which has loaded this class * * @return classloader */ public static ClassLoader getCurrentClassLoader() { ClassLoader classLoader = Thread.currentThread() .getContextClassLoader(); if (classLoader == null) { classLoader = DmsConsumer.class.getClassLoader(); } return classLoader; }

/** * Load configuration information from classpath. *


2020-02-20 9

* @param configFileName Configuration file name * @return Configuration information * @throws IOException */ public static Properties loadFromClasspath(String configFileName) throws IOException { ClassLoader classLoader = getCurrentClassLoader(); Properties config = new Properties();

List<URL> properties = new ArrayList<URL>(); Enumeration<URL> propertyResources = classLoader .getResources(configFileName); while (propertyResources.hasMoreElements()) { properties.add(propertyResources.nextElement()); }

for (URL url : properties) { InputStream is = null; try { is = url.openStream(); config.load(is); } finally { if (is != null) { is.close(); is = null; } } }

return config; }}

1.3.2 Setting Up the Java Development EnvironmentWith the information collected in Collecting Connection Information and thenetwork environment prepared for Kafka clients, you can proceed to configuringKafka clients. This section describes how to configure Kafka clients to produce andconsume messages.

Preparing Tools

Table 1-1 Required tools

Tool Required Version How to Obtain

Apache Maven 3.0.3 or later http://maven.apache.org/download.cgi

Java Development Kit(JDK) set with Javaenvironment variables

1.8.111 or later https://www.oracle.com/technetwork/java/javase/downloads/index.html


2020-02-20 10

http://maven.apache.org/download.cgi



https://www.oracle.com/technetwork/java/javase/downloads/index.html





Tool Required Version How to Obtain

IntelliJ IDEA https://www.jetbrains.com/idea/

Procedure

Step 1 Download the demo package.

Decompress the package to obtain the following files.

Table 1-2 Files in the demo package

File Directory Description

DmsConsumer.java .\src\main\java\com\dms\consumer

API for consuming messages

DmsProducer.java .\src\main\java\com\dms\producer

API for producing messages

dms.sdk.consumer.properties

.\src\main\resources Configuration information forconsuming messages

dms.sdk.producer.properties

.\src\main\resources Configuration information forproducing messages

client.truststore.jks .\src\main\resources SSL certificate, used for SASLconnection

DmsConsumerTest.java

.\src\test\java\com\dms\consumer

Test code of consumingmessages

DmsProducerTest.java

.\src\test\java\com\dms\producer

Test code of producingmessages

pom.xml .\ Maven configuration file,containing the Kafka clientdependencies

Step 2 In IntelliJ IDEA, import the demo project.

The demo project is a Java project built in Maven. Therefore, you need the JDKand the Maven plugin in IDEA.

1. Select Import Project.


2020-02-20 11

https://www.jetbrains.com/idea/



https://dms-demo.obs.eu-de.otc.t-systems.com/Kafka-sdk-java.zip

2. Select Maven.

3. Select the JDK.

4. You can select other options or retain the default settings. Then, click Finish.The demo project has been imported.


2020-02-20 12

Step 3 Configure Maven.

Choose Files > Settings, set Maven home directory correctly, and select therequired settings.xml file.

Step 4 Specify Kafka configurations.


2020-02-20 13

The following is a configuration example for producing messages. Replace theinformation in bold with the actual values.

#The information in bold is specific to different Kafka instances and must be modified. Other parameters can also be added.#The topic name is in the code for producing or consuming messages.########################Obtain the information about Kafka instance brokers on the console. For example: bootstrap.servers=100.xxx.xxx.87:9095,100.xxx.xxx.69:9095,100.xxx.xxx.155:9095bootstrap.servers=ip1:port1,ip2:port2,ip3:port3#Producer acknowledgementacks=all#Method of turning the key into byteskey.serializer=org.apache.kafka.common.serialization.StringSerializer#Method of turning the value into bytesvalue.serializer=org.apache.kafka.common.serialization.StringSerializer#Memory available to the producer for bufferingbuffer.memory=33554432#Number of retriesretries=0########################Comment out the following parameters if SASL access is not enabled.########################Configure the JAAS username and password on the console.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="username" \ password="password";#SASL mechanismsasl.mechanism=PLAIN#Encryption protocol. Currently, only SASL_SSL is supported.security.protocol=SASL_SSL#Location of ssl.truststoressl.truststore.location=E:\\temp\\client.truststore.jks#Password of ssl.truststoressl.truststore.password=dms@kafka

Step 5 In the down left corner of IDEA, click Terminal. In terminal, run the mvn testcommand to see how the demo project goes.

Figure 1-5 Opening terminal in IDEA

The following information is displayed for the producer:

------------------------------------------------------- T E S T S-------------------------------------------------------Running com.dms.producer.DmsProducerTestproduce msg:The msg is 0produce msg:The msg is 1produce msg:The msg is 2produce msg:The msg is 3produce msg:The msg is 4produce msg:The msg is 5produce msg:The msg is 6


2020-02-20 14

produce msg:The msg is 7produce msg:The msg is 8produce msg:The msg is 9Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 138.877 sec

The following information is displayed for the consumer:

------------------------------------------------------- T E S T S-------------------------------------------------------Running com.dms.consumer.DmsConsumerTestthe numbers of topic:0the numbers of topic:0the numbers of topic:6ConsumerRecord(topic = topic-0, partition = 2, offset = 0, CreateTime = 1557059377179, serialized key size = -1, serialized value size = 12, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = The msg is 2)ConsumerRecord(topic = topic-0, partition = 2, offset = 1, CreateTime = 1557059377195, serialized key size = -1, serialized value size = 12, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = The msg is 5)

----End

1.4 PythonThis section describes how to access a Kafka premium instance using a Kafkaclient in Python, including installing the client, and producing and consumingmessages.

Before getting started, ensure that you have collected the information listed inCollecting Connection Information.

Preparing Tools● Python

Generally, Python is pre-installed in the system. Enter python in a CLI. If thefollowing information is displayed, Python has already been installed.[root@ecs-heru bin]# pythonPython 2.7.5 (default, Oct 30 2018, 23:45:53) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2Type "help", "copyright", "credits" or "license" for more information.>>>

If the Python has not been installed, run the following command to install it:yum install python

● Kafka clients in PythonRun the following command to install Kafka clients:pip install kafka-python

Producing Messages

Replace the following information in bold with the actual values.

● Connection with SASLfrom kafka import KafkaProducerimport ssl##Connection informationconf = {


2020-02-20 15

'bootstrap_servers': ["ip1:port1,ip2:port2,ip3:port3"], 'topic_name': 'topic_name', 'sasl_plain_username': 'username', 'sasl_plain_password': 'password'}

context = ssl.create_default_context()context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)context.verify_mode = ssl.CERT_REQUIRED##Certificatecontext.load_verify_locations("phy_ca.crt")

print (start producer)producer = KafkaProducer(bootstrap_servers=conf['bootstrap_servers'], sasl_mechanism="PLAIN", ssl_context=context, security_protocol='SASL_SSL', sasl_plain_username=conf['sasl_plain_username'], sasl_plain_password=conf['sasl_plain_password'])

data = "hello kafka!"producer.send(conf['topic_name'], data)producer.close()print (end producer)

● Connection without SASLfrom kafka import KafkaProducer

conf = { 'bootstrap_servers': 'ip1:port1,ip2:port2,ip3:port3', 'topic_name': 'topic-name',}

print (start producer)producer = KafkaProducer(bootstrap_servers=conf['bootstrap_servers'])

data = "hello kafka!"producer.send(conf['topic_name'], data)producer.close()print (end producer)

Consuming Messages● Connection with SASL

from kafka import KafkaProducerimport ssl##Connection informationconf = { 'bootstrap_servers': ["ip1:port1,ip2:port2,ip3:port3"], 'topic_name': 'topic_name', 'sasl_plain_username': 'username', 'sasl_plain_password': 'password', 'consumer_id': 'consumer_id'}

context = ssl.create_default_context()context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)context.verify_mode = ssl.CERT_REQUIRED##Certificatecontext.load_verify_locations("phy_ca.crt")

print (start consumer)consumer = KafkaConsumer(conf['topic_name'], bootstrap_servers=conf['bootstrap_servers'], group_id=conf['consumer_id'], sasl_mechanism="PLAIN", ssl_context=context, security_protocol='SASL_SSL', sasl_plain_username=conf['sasl_plain_username'],


2020-02-20 16

sasl_plain_password=conf['sasl_plain_password'])

for message in consumer: print("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,message.offset, message.key,message.value))

print (end consumer)

● Connection without SASLReplace the information in bold with the actual values.from kafka import KafkaConsumer

conf = { 'bootstrap_servers': 'ip1:port1,ip2:port2,ip3:port3', 'topic_name': 'topic-name', 'consumer_id': 'consumer-id'}

print (start consumer)consumer = KafkaConsumer(conf['topic_name'], bootstrap_servers=conf['bootstrap_servers'], group_id=conf['consumer_id'])

for message in consumer: print("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,message.offset, message.key,message.value))

print (end consumer)

1.5 Recommendations on Client Usage● Producer parameters

– Allow for retries in cases where messages fail to be sent.For example, allow for three retries by setting the value of retries to 3.

– The producer cannot block on callback functions. Otherwise, messagesmay fail to be sent.For messages that need to be send immediately, set linger.ms to 0.Ensure that the producer has sufficient JVM memory to avoid blockages.

● Consumer parameters– Ensure that the owner thread does not exit abnormally. Otherwise, the

client may fail to initiate consumption requests and the consumption willbe blocked.

– Use long polling to consume messages and do not close the consumerconnection immediately after the consumption is completed. Otherwise,rebalancing will take place frequently, blocking consumption.

– Ensure that the consumer polls at regular intervals (for example, every200 ms) for it to keep sending heartbeats to the server. If the consumerstops sending heartbeats for long enough, the consumer session will timeout and the consumer will be considered to have stopped. This will alsoblock consumption.

– Always close the consumer before exiting. Otherwise, consumers in thesame group may block the session.timeout.ms time.

– Set the timeout time for the consumer session to a reasonable value. Forexample, set session.timeout.ms to 30000 so that the timeout time is30s.


2020-02-20 17

– The number of consumers cannot be greater than the number ofpartitions in the topic. Otherwise, some consumers may fail to poll formessages.

– Commit messages after they have been processed. Otherwise, themessages may fail to be processed and cannot be polled for again.

– Ensure that there is a maximum limit on the size of messages bufferedlocally to avoid an out-of-memory (OOM) situation.

– Kafka supports the exactly-once delivery. Therefore, ensure theidempotency of processing messages for services.


2020-02-20 18

2 DMS Kafka Developer Guide

2.1 Overview

NO TICE

This document describes how to connect to Distributed Message Service (DMS)Kafka (non-premium) through clients.

DMS Kafka API

DMS Kafka supports open-source Kafka application programming interfaces(APIs). Third-party applications can implement open-source Kafka servicecapabilities by directly using a Kafka client to call DMS.

Usage Restrictions

Generally, DMS Kafka can process thousands of messages per second. If moremessages need to be processed per second, submit a service ticket or contact thecustomer service.

The recommended Kafka client version is 0.10.2.1 or higher.

If the Kafka SDK is used to produce messages, the maximum size of a singlemessage is 10 MB. If the DMS console is used to produce messages, the maximumsize of a single message is 512 KB.

2.2 Preparing the Environment

Helpful Links● Download DMS Kafka SDK

● Download Sample Project

Distributed Message ServiceDeveloper Guide 2 DMS Kafka Developer Guide

2020-02-20 19

https://obs.eu-de.otc.t-systems.com/dms-demo/dms.kafka.sasl.zip

https://obs.eu-de.otc.t-systems.com/dms-demo/KafkaDemo.zip

To create a new project, use the downloaded SDK. To write code based on the sampleproject, use the SDK included in the project.

Preparing ToolsEclipse: Download Eclipse 3.6.0 or later from the Eclipse official website.

JDK: Download Java Development Kit 1.8.111 or later from the Oracle officialwebsite.

Apache Maven: Download Apache Maven 3.0.3 or later from the Maven officialwebsite.

Obtaining a Topic ID and Consumer Group IDBefore accessing DMS using the SDK, create a Kafka queue and consumer groupon the DMS console, and obtain the topic ID and consumer group ID.

Step 1 Log in to the management console.

Step 2 Choose Service List > Application > Distributed Message Service to launch theDMS console.

Step 3 In the navigation pane, choose Queue Manager.

Step 4 On the Queue Manager page, click Create Queue.

Step 5 Specify queue parameters.

Table 2-1 Parameter description

Parameter Description

Name Name of the queue you want to create. The name must beunique.When you create a queue, a default queue name is generated,which you can change if required. A queue name is 1 to 64characters long. Only letters, digits, underscores (_), andhyphens (-) are allowed.The queue name cannot be modified after creation of thequeue.

Type Select Kafka queue.

Mode Select either High throughput or High reliability.Default value: High throughput.High throughput: All message replicas are flushed to diskasynchronously. Select this mode when high message deliveryperformance is required.High reliability: All message replicas are flushed to disksynchronously. Select this mode when high message deliveryreliability is required.


2020-02-20 20

http://www.eclipse.org/downloads/

http://www.oracle.com/technetwork/java/javase/downloads/index.html




Parameter Description

MessageRetentionPeriod (h)

This parameter is available only for Kafka queues.The number of hours for which messages will be preserved in aKafka queue. Messages older than that period will be deleted.Deleted messages are not retrievable to consumer groups.Value range: integers from 1 to 72Default value: 72

Tags When creating a queue, you can add tags to identify the queue.You can classify and search for queues by tags.● Tags of the same queue cannot have the same key.● You can customize tags or use tags predefined by Tag

Management Service (TMS).● You can add a maximum of 10 tag keys to a queue.

Description(optional)

The description consists of a maximum of 160 characters andcannot contain angle brackets (< and >).

Figure 2-1 Creating a Kafka queue

Step 6 Click OK.

Step 7 Click the name of the queue. On the displayed queue details page, obtain theKafka topic ID.


2020-02-20 21

Figure 2-2 Obtaining the Kafka topic ID

Step 8 Click Create Consumer Group. The Create Consumer Group dialog box isdisplayed.

Step 9 Enter a consumer group name.

A default queue name is generated, which you can change if required. A consumergroup name consists of 1 to 32 characters. Only letters, digits, underscores (_), andhyphens (-) are allowed. Consumer group names must be unique within the samequeue.

Step 10 Click OK. Obtain the ID of the consumer group in the consumer group list.

Figure 2-3 Obtaining the consumer group ID

----End

Obtaining a Project IDWhen calling APIs, you need to specify project_id in API requests. Obtain a projectID by performing the following procedure.

Step 1 Sign up and log in to the management console.

Step 2 Click the username and choose My Credentials from the drop-down list.


2020-02-20 22

Step 3 On the My Credentials page, view project IDs in the project list.

Figure 2-4 Obtaining a project ID

----End

Obtaining an AK/SK

Step 1 Sign up and log in to the management console.

Step 2 Click the username and choose My Credentials from the drop-down list.

Step 3 On the My Credentials page, click the Access Keys tab.

Step 4 Click Create Access Keys.

Step 5 Enter the password for login.

Step 6 Enter the verification code received by email or SMS message.

Step 7 Click OK.

Keep the key secure and do not disclose it to any unauthorized people.

Step 8 Download the credentials.csv file containing your AK and SK to a local computer.

----End

Obtaining Region and Endpoint InformationObtain the region and endpoint from Regions and Endpoints.


2020-02-20 23

https://docs.otc.t-systems.com/en-us/endpoint/index.html

Summary of Environment Information

Table 2-2 Required environment information

Category Information Example

ECS EIP x.x.x.x

Username name

Password password

DMS Queue name my-kafka-queue

Queue ID 4df89da6-ede4-4072-93e0-28dc6e866299

Queue type Kafka

Kafka topic k-bd67aaead60940d688b872c31bdc653b-4df89da6-ede4-4072-93e0-28dc6e866299

Consumergroup name

my-consumer-group

Consumergroup ID

g-7ec0caac-01fb-4f91-a4f2-0a9dd48f8af7

AK/SK AK VAODAIIJGPUAYTJRRL**

SK ZHN49c6bpwDiQvPqKJ5CxutJxqc04Glt9xSzxY**

Project Region eu-de

Project name eu-de

Project ID bd67aaead60940d688b872c31bdc653b

Region andendpoint

Region eu-de

Endpoint dms-kafka.eu-de.otc.t-systems.com:37000

DNS DNS server IPaddress

172.16.16.65

2.3 Creating a ProjectThis section uses the Maven project kafkademo as an example to describe how tocreate a project.

Procedure

Step 1 Download the demo package.

1. Log in to the DMS console.


2020-02-20 24

2. In the navigation pane, choose Using APIs.3. Choose Kafka APIs.4. Click Download Sample Code to download KafkaDemo.zip.

Step 2 Click Download SDK to download the DMS Kafka SASL package.

Decompress the following directories from the package:

● client.truststore.jks: client certificate● dms.kafka.sasl.client-1.0.0.jar: DMS Kafka SASL package● dms_kafka_client_jaas.conf: client configuration file

You can also decompress the SDK package from \KafkaDemo\dist\libs\dms.kafka.sasl.client-1.0.0.jar.

Step 3 On Eclipse (the recommended version is 4.6 or later), create a Maven project. Theproject name kafkademo is used as an example.

Figure 2-5 Creating a Maven project

Step 4 Click Finish.

Step 5 Import the DMS Kafka SASL package.

1. Right-click the new project kafkademo, and create a libs folder.2. Copy dms.kafka.sasl.client-1.0.0.jar to libs.3. Add the following information to the pom.xml file to import

dms.kafka.sasl.client-1.0.0.jar into the Maven repository:<dependency> <groupId>dms</groupId> <artifactId>kafka.sasl.client</artifactId> <version>1.0.0</version> <scope>system</scope> <systemPath>${project.basedir}/libs/dms.kafka.sasl.client-1.0.0.jar</systemPath> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.10.2.1</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.7</version> </dependency> <dependency> <groupId>org.slf4j</groupId>


2020-02-20 25

<artifactId>slf4j-log4j12</artifactId> <version>1.7.7</version> </dependency> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.17</version> </dependency>

4. Save the pom.xml file.

----End

2.4 Configuring Parameters

Procedure

Step 1 (Optional) Configure a private DNS server.

You need to use an ECS to connect to DMS Kafka. You do not need to configure aprivate DNS server for a newly created ECS. If an existing ECS is used, you need toconfigure the DNS server IP address.

The DNS server IP address in the eu-de region is 172.16.16.65

Step 2 Configure access_key, secret_key, and project_id in the dms_kafka_client_jaas.conf file.

The three parameters are used to authenticate DMS Kafka API requests.

KafkaClient { com.dms.kafka.sasl.client.KafkaLoginModule required access_key="your ak" secret_key="your sk" project_id="projectID";};

Replace them with the actual access_key, secret_key, and project_id of youraccount.

To access the queues authorized by other tenants, set target_project_id to theproject ID of the authorizing tenant.

Step 3 Configure SASL access to start upon process using either of the following methods.In both methods, replace /path with the actual path name.

1. Method 1: Configure the following JVM parameter to add the location ofSASL configuration file:-Djava.security.auth.login.config=/path/kafka_client_jaas.conf

2. Method 2: Add the following information to project code so that SASL accesscan start before Kafka Producer and Consumer start:System.setProperty("java.security.auth.login.config", "/path/kafka_client_jaas.conf");

Step 4 Add the following information to the consumer.properties file:connections.max.idle.ms=30000

Step 5 Configure key parameters in the consumer.properties/producer.properties file.


2020-02-20 26

Table 2-3 Key parameters in the consumer.properties/producer.properties file

Parameter Description Setting

bootstrap.servers IP address ordomain name ofthe DMS server

dms-kafka.eu-de.otc.t-systems.com:37000

ssl.truststore.location

Path in which theclient certificateclient.truststore.jks is located

/path/client.truststore.jks, where /path must be replaced with the actualpath name

ssl.truststore.password

Client certificatepassword

-

security.protocol Security protocol SASL_SSL

sasl.mechanism Service name DMS (Note: All letters in the enteredservice name must be capitalized.)

For details about other Kafka parameters, visit the official Kafka website.

Step 6 Enable Kafka debug logging by modifying the log4j.properties file.log4j.rootLogger=DEBUG, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppenderlog4j.appender.stdout.layout=org.apache.log4j.PatternLayoutlog4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%nlog4j.logger.org.apache.kafka.clients=DEBUGlog4j.logger.kafka=INFO, stdoutlog4j.additivity.kafka=falselog4j.logger.org.apache.kafka=DEBUG, stdoutlog4j.additivity.org.apache.kafka=false

Step 7 Write code. For details about APIs, visit the official Kafka website.

----End

2.5 Running the Sample ProjectThe following describes how to access DMS Kafka queues to produce andconsume messages in Java.

Procedure

Step 1 Log in to the ECS.

You can run the sample project on an ECS with an IP address in the 192 network segment.

Step 2 Install JDK or Java runtime environment (JRE). Add the following settings ofenvironment variables JAVA_HOME and PATH to the ~/.bash_profile:export JAVA_HOME=/opt/java/jdk1.8.0_151export PATH=$JAVA_HOME/bin:$PATH


2020-02-20 27

https://kafka.apache.org/0100/documentation.html

https://kafka.apache.org/0100/documentation.html

Run the source .bash_profile command for the modification to take effect.

Use Oracle JDK instead of ECS's default JDK (for example, OpenJDK), because ECS's defaultJDK may not be suitable for the sample project. To obtain Oracle JDK, download JavaDevelopment Kit 1.8.111 or a later version from https://www.oracle.com/technetwork/java/javase/downloads/index.html.

Step 3 Add the DNS server IP address to the /etc/resolv.conf file as the root user.

Add the following content to the first line in the file:

nameserver 172.16.16.65

Step 4 Run the following command to download the code package of the sample projectKafkaDemo.zip.$ wget https://obs.eu-de.otc.t-systems.com/dms-demo/KafkaDemo.zip

Step 5 Run the following command to decompress KafkaDemo.zip.$ unzip KafkaDemo.zip

Step 6 Run the following command to navigate to the KafkaDemo/dist directory, whichcontains pre-compiled binary files and executable scripts.$ cd KafkaDemo/dist

Step 7 Edit the config/dms_kafka_client_jaas.conf file and configure access_key,secret_key, and project_id.$ vim config/dms_kafka_client_jaas.conf

The values in bold are examples. Replace them with actual values.

KafkaClient { com.dms.kafka.sasl.client.KafkaLoginModule required access_key="******************" secret_key="******************" project_id="bd67aaead60940d688b872c31bdc653b";};

Step 8 Edit the config/producer.properties file and configure topic andbootstrap.servers.$ vim config/producer.properties


topic=k-bd67aaead60940d688b872c31bdc653b-4df89da6-ede4-4072-93e0-28dc6e866299 bootstrap.servers=dms-kafka.eu-de.otc.t-systems.com:37000ssl.truststore.password=************acks=allretries=1batch.size=16384buffer.memory=33554432key.serializer=org.apache.kafka.common.serialization.StringSerializervalue.serializer=org.apache.kafka.common.serialization.StringSerializersecurity.protocol=SASL_SSLsasl.mechanism=DMS

The parameter topic can be set to a queue name or a Kafka topic name. For moreinformation, see Table 2-2.

Step 9 Edit the config/consumer.properties file and configure topic, bootstrap.servers,and group.id.


2020-02-20 28



$ vim config/consumer.properties


topic=k-bd67aaead60940d688b872c31bdc653b-4df89da6-ede4-4072-93e0-28dc6e866299bootstrap.servers=dms-kafka.eu-de.otc.t-systems.com:37000group.id=g-7ec0caac-01fb-4f91-a4f2-0a9dd48f8af7ssl.truststore.password=************security.protocol=SASL_SSLsasl.mechanism=DMSkey.deserializer=org.apache.kafka.common.serialization.StringDeserializervalue.deserializer=org.apache.kafka.common.serialization.StringDeserializerauto.offset.reset=earliestenable.auto.commit=false

The parameter topic can be set to a queue name or a Kafka topic name. For moreinformation, see Table 2-2.

Step 10 Run the sample project to produce messages:$ bash produce.sh

After the command is run, 10 messages are automatically sent to the Kafkaqueue.

Step 11 Run the sample project to consume messages:$ bash consume.sh

----End

2.6 Compiling the Sample Project CodeStep 1 Download and decompress the sample project code KafkaDemo.zip.

Step 2 Import the sample project code.

1. On Eclipse, choose File > Import.

2. In the Select an import source area, select Exiting Projects Into Workspace.

3. Select the directory to which the sample project code KafkaDemo isdecompressed.

Step 3 Choose Project > Build Project to build the project.

Step 4 Export the new JAR file.

1. Right-click the sample project KafkaDemo and choose Export from theshortcut menu.

2. Choose Java > JAR file. Enter the path and name of the JAR file to begenerated.

Step 5 Replace the dms.kafka.demo.jar file in the KafkaDemo/dist/libs directory withthe new JAR file. Run the newly built project by following the procedure inRunning the Sample Project.

----End


2020-02-20 29

https://obs.eu-de.otc.t-systems.com/dms-demo/KafkaDemo.zip

2.7 Code of the Sample Project

Producer

DMS Kafka APIs are compatible with native open-source Kafka clients. Comparedwith native Kafka service code, the sample project code additionally contains aclient certificate and simple authentication and security layer (SASL)configuration, which are used for identity authentication and securecommunication. To realize smooth migration of producer applications, you onlyneed to import the client certificate and SASL configuration before creating theKafka Producer without modifying any other Kafka service code.

Code pertaining to client certificate and SASL:

Properties producerConfig = Config.getProducerConfig(); producerConfig.put("ssl.truststore.location", Config.getTrustStorePath()); System.setProperty("java.security.auth.login.config", Config.getSaslConfig());

The code for creating a producer and sending messages does not need to bemodified.

Consumer

DMS Kafka APIs are compatible with native open-source Kafka clients. Comparedwith native Kafka service code, the sample project code additionally contains aclient certificate and SASL configuration, which are used for identityauthentication and secure communication. To realize smooth migration ofconsumer applications, you only need to import the client certificate and SASLconfiguration before creating the Kafka Consumer without modifying any otherKafka service code.

Code pertaining to client certificate and SASL:

Properties consumerConfig = Config.getConsumerConfig(); consumerConfig.put("ssl.truststore.location", Config.getTrustStorePath()); System.setProperty("java.security.auth.login.config", Config.getSaslConfig());

The code for creating a consumer and consuming messages does not need to bemodified.

2.8 Using the Enhanced Java SDKThe enhanced Kafka Java SDK is optimized based on the open-source Kafka0.10.2.1 client, greatly improving system performance.

Procedure

Step 1 Download the open-source Kafka 0.10.2.1 client package.

Step 2 Decompress the following from the open-source client package:


2020-02-20 30

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.1/kafka_2.11-0.10.2.1.tgz

Step 3 Download the enhanced Kafka 0.10.2.1 SDK package.

Step 4 Decompress the following from the SDK package:

Step 5 Copy all JAR packages in the dms_kafka_0.10.2.1-client/ directory to the libsfolder in the directory where the open-source Kafka 0.10.2.1 client package isdecompressed, and overwrite the kafka-clients-0.10.2.1.jar package with thesame name.

Step 6 Perform the same configuration as other open-source clients.

----End


2020-02-20 31

2.9 Recommended Parameter Settings for Kafka Clients

Table 2-4 Producer parameters

Parameter

Default Value

RecommendedValue

Description

acks 1 all (ifhighreliabilitymode isselected)1 (if highthroughput modeisselected)

Indicates the number of acknowledgmentsthe producer requires the server to returnbefore considering a request complete. Thiscontrols the durability of records that aresent. The value of this parameter can be anyof the following:0: The producer will not wait for anyacknowledgment from the server at all. Therecord will be immediately added to thesocket buffer and considered sent. Noguarantee can be made that the server hasreceived the record, and the retriesconfiguration will not take effect (as theclient generally does not know of anyfailures). The offset given back for eachrecord will always be set to –1.1: The leader will write the record to its locallog but will respond without waiting untilreceiving full acknowledgement from allfollowers. If the leader fails immediately afteracknowledging the record but before thefollowers have replicated it, the record will belost.all: The leader will wait for the full set ofreplicas to acknowledge the record. This isthe strongest available guarantee that therecord will not be lost as long as there is atleast one replica.

retries 0 0 Setting this parameter to a value greaterthan zero will cause the client to resend anyrecord that failed to be sent due to apotentially transient error. Note that thisretry is no different than if the client resentthe record upon receiving the error. Allowingretries will potentially change the ordering ofrecords because if two batches are sent tothe same partition, and the first fails and isretried but the second succeeds, then therecords in the second batch may appear first.


2020-02-20 32

Parameter

Default Value

RecommendedValue

Description

request.timeout.ms

30000 120000 Indicates the maximum amount of time theclient will wait for the response of a request.If the response is not received before thetimeout elapses, the client will throw aTimeout exception.

block.on.buffer.full

TRUE Defaultvalue

When buffer memory is exhausted, theproducer must stop receiving new messagerecords or throw an exception. By default,this parameter is set to TRUE. However, insome cases, non-blocking usage is desiredand it is better to throw an exceptionimmediately. Setting this parameter to FALSEwill cause the producer to instead throw"BufferExhaustedException" when buffermemory is exhausted.

batch.size 16384 262144 The producer will attempt to batch recordstogether into fewer requests whenevermultiple records are being sent to the samepartition. This helps improve performance ofboth the client and the server. This parametercontrols the default batch size in bytes.No attempt will be made to batch recordslarger than this size.Requests sent to brokers will contain multiplebatches, one for each partition with dataavailable to be sent.A smaller batch size will make batching lesscommon and may reduce throughput (abatch size of zero will disable batchingentirely). A larger batch size may use morememory as a buffer of the specified batchsize will always be allocated in anticipationof additional records.


2020-02-20 33

Parameter

Default Value

RecommendedValue

Description

buffer.memory

33554432

536870912

The total bytes of memory the producer canuse to buffer records waiting to be sent tothe server. If records are sent faster than theycan be delivered to the broker, the producerwill stop sending records or throw a"block.on.buffer.full" exception.This setting should correspond roughly to thetotal memory the producer will use, but isnot a rigid bound since not all memory theproducer uses is used for buffering. Someadditional memory will be used forcompression (if compression is enabled) aswell as for maintaining in-flight requests.

Table 2-5 Consumer parameters

Parameter DefaultValue

RecommendedValue

Description

auto.commit.enable

TRUE FALSE If this parameter is set to TRUE, the offset ofmessages already fetched by the consumer willbe periodically committed to ZooKeeper. Thiscommitted offset will be used when the processfails as the position from which the newconsumer will begin.Constraints: If this parameter is set to FALSE, toavoid message loss, an offset must becommitted to ZooKeeper after the messages aresuccessfully consumed.

auto.offset.reset

latest earliest Indicates what to do when there is no initialoffset in ZooKeeper or if the current offset hasbeen deleted. Options:earliest: The offset is automatically reset to thesmallest offset.latest: The offset is automatically reset to thelargest offset.none: The system throws an exception to theconsumer if no offset is available.anything else: The system throws an exceptionto the consumer.


2020-02-20 34

Parameter DefaultValue

RecommendedValue

Description

connections.max.idle.ms

600000

30000 Indicates the timeout interval for an idleconnection. The server closes the idle connectionafter this period of time ends.Setting thisparameter to 30000 can reduce the serverresponse failures when the network condition ispoor.


2020-02-20 35

3 Change History

Date Description

2020-02-20 Modified the following content:● Updated Kafka premium compatibility from Kafka 2.3.0

to 2.3.1.

2019-11-08 Added the following content:● Added description about Kafka premium instances in

Kafka Premium Developer Guide.

2018-02-03 This issue is the second official release.● Revised description in the previous issue.

2018-01-19 This issue is the first official release.

Distributed Message ServiceDeveloper Guide 3 Change History

2020-02-20 36

Documents

Developer Guide - T-Systems · 1.3.1 Configuring Kafka Clients in Java This section describes how to add Kafka clients in Maven, and use the clients to access Kafka instances and