Kafka

Question 1

CLI create a topic in Kafka

Answer 1

kafka-topics.sh –zookeeper 127.0.0.1:2181 –topic my_topic –create –partitions 3 –replication-factor 2

1) must reference zookeeper
2) must define partitions & replication factor

Question 2

What limits your replication factor?

Answer 2

Broker number. You cannot have a replication factor higher than the number of brokers you have set.

Question 3

CLI list Kafka topics

Answer 3

kafka-topics.sh –zookeeper 127.0.0.1:2181 –list

Question 4

CLI get info about a topic

Answer 4

kafka-topics.sh –zookeeper 127.0.0.1:2181 –topic my_topic –describe

Question 5

CLI delete a topic

Answer 5

kafka-topics.sh –zookeeper 127.0.0.1:2181 –topic my_topic –delete
DON’T DO THIS ON WINDOWS

Question 6

CLI create a producer

Answer 6

kafka-console-producer.sh –broker-list 127.0.0.1:9092 –topic my_topic (–producer-property acks=all)

script, specify broker and where to find them, topic (optional properties)

Question 7

CLI get out of producer

Answer 7

ctrl + C

Question 8

what happens when you produce to a topic that doesn’t exist?

Answer 8

Kafka creates the topic, sends the message when a leader becomes available.
Topic is created with default values (partitions 1, replication factor 1)

Question 9

change default values for topics

Answer 9

edit the server.properties file (in text editor):

num,partitions=1 (change to custom setting)

Question 10

CLI create consumer

Answer 10

kafka-console-consumer.sh –bootstrap-server 127.0.0.1:9092 –topic my_topic
(begins getting messages produced after this is run. For all, add “–from-beginning”)

Question 11

CLI set consumer group

Answer 11

Every consumer is part of a group; Id is the name of our application.
kafka-console-consumer.sh –bootstrap-server 127.0.0.1:9092 –topic my_topic –group my-application

Question 12

Why consumer groups?

Answer 12

Every consumer is part of a group; Id is the name of our application.
Members of a group share the load, do load balancing. You can have as many consumers in a group as you have partitions. If one consumer goes down, the others will automatically rebalance the load.

Question 13

CLI consumer groups: 1) list groups, 2) get info about a group

Answer 13

1) kafka-consumer-groups.sh –bootstrap-server 127.0.0.1:9092 –list
2) –describe –group my-application

Question 14

CLI reset offset in a consumer group

Answer 14

Tell kafka where the group should start reading. (see options with kafka-consumer-groups.sh) (Shift forward or backwards with num or -num.)

• specify group, where to rest offsets, topic
• execute

kafka-consumer-groups.sh –bootstrap-server 127.0.0.1:9092 –group my-application –reset-offsets –to-earliest –execute –topic my_topic

Question 15

What dependencies does Kafka need in Java POM?

Answer 15

1) Kafka-clients (not just kafka)

2) slf4j-simple (for logging) and remove scope

Question 16

Java: define the steps to create a producer

Answer 16

1) set producer properties (Properties p = new Properties();
2) create producer
3) create producer record (the thing to send)
4) send record

Question 17

Java: How to set up kafka properties?

Answer 17

1) Create new properties (Prop p = new Prop();
2) Go to Kafka documentation, producer configs
3) 3 props are required:
p.setProperty(“bootstrap.servers”, “127.0.0.1:9092”); (you can make the address:port an instance variable)
p.setProperty(“key.serializer”, StringSerializer.class.getName());
same for value

4) to make sure you are getting these right, replace the string in the first place in () with ProducerConfig.BOOTSTRAP_SERVERS_CONFIG. There will be an enum for all you configuration properties

Question 18

Java: set up producer

Answer 18

1) KafkaProducer kp = new KP(properties); where objects in <> are KEY, and VALUE, whatever type you want, and properties are those created before in same class.

Question 19

Java: send a message

Answer 19

1) ProducerRecord pr = new PR(“some-topic”, “some-value”);
2) producer.send(record);
3) producer.flush(); //pushes out the record (or if you want to close, producer.close() which will flush and close)