Kafka Flashcards
CLI create a topic in Kafka
kafka-topics.sh –zookeeper 127.0.0.1:2181 –topic my_topic –create –partitions 3 –replication-factor 2
1) must reference zookeeper
2) must define partitions & replication factor
What limits your replication factor?
Broker number. You cannot have a replication factor higher than the number of brokers you have set.
CLI list Kafka topics
kafka-topics.sh –zookeeper 127.0.0.1:2181 –list
CLI get info about a topic
kafka-topics.sh –zookeeper 127.0.0.1:2181 –topic my_topic –describe
CLI delete a topic
kafka-topics.sh –zookeeper 127.0.0.1:2181 –topic my_topic –delete
DON’T DO THIS ON WINDOWS
CLI create a producer
kafka-console-producer.sh –broker-list 127.0.0.1:9092 –topic my_topic (–producer-property acks=all)
script, specify broker and where to find them, topic (optional properties)
CLI get out of producer
ctrl + C
what happens when you produce to a topic that doesn’t exist?
Kafka creates the topic, sends the message when a leader becomes available.
Topic is created with default values (partitions 1, replication factor 1)
change default values for topics
edit the server.properties file (in text editor):
num,partitions=1 (change to custom setting)
CLI create consumer
kafka-console-consumer.sh –bootstrap-server 127.0.0.1:9092 –topic my_topic
(begins getting messages produced after this is run. For all, add “–from-beginning”)
CLI set consumer group
Every consumer is part of a group; Id is the name of our application.
kafka-console-consumer.sh –bootstrap-server 127.0.0.1:9092 –topic my_topic –group my-application
Why consumer groups?
Every consumer is part of a group; Id is the name of our application.
Members of a group share the load, do load balancing. You can have as many consumers in a group as you have partitions. If one consumer goes down, the others will automatically rebalance the load.
CLI consumer groups: 1) list groups, 2) get info about a group
1) kafka-consumer-groups.sh –bootstrap-server 127.0.0.1:9092 –list
2) –describe –group my-application
CLI reset offset in a consumer group
Tell kafka where the group should start reading. (see options with kafka-consumer-groups.sh) (Shift forward or backwards with num or -num.)
- specify group, where to rest offsets, topic
- execute
kafka-consumer-groups.sh –bootstrap-server 127.0.0.1:9092 –group my-application –reset-offsets –to-earliest –execute –topic my_topic
What dependencies does Kafka need in Java POM?
1) Kafka-clients (not just kafka)
2) slf4j-simple (for logging) and remove scope
Java: define the steps to create a producer
1) set producer properties (Properties p = new Properties();
2) create producer
3) create producer record (the thing to send)
4) send record
Java: How to set up kafka properties?
1) Create new properties (Prop p = new Prop();
2) Go to Kafka documentation, producer configs
3) 3 props are required:
p.setProperty(“bootstrap.servers”, “127.0.0.1:9092”); (you can make the address:port an instance variable)
p.setProperty(“key.serializer”, StringSerializer.class.getName());
same for value
4) to make sure you are getting these right, replace the string in the first place in () with ProducerConfig.BOOTSTRAP_SERVERS_CONFIG. There will be an enum for all you configuration properties
Java: set up producer
1) KafkaProducer kp = new KP(properties); where objects in <> are KEY, and VALUE, whatever type you want, and properties are those created before in same class.
Java: send a message
1) ProducerRecord pr = new PR(“some-topic”, “some-value”);
2) producer.send(record);
3) producer.flush(); //pushes out the record (or if you want to close, producer.close() which will flush and close)
