Distributed Transport and Streaming

Question 1

What is Kafka

Answer 1

Kafka is a Distributed event based data streaming platform

Question 2

What is Sqoop and what is is used for

Answer 2

Sqoop is and ingestion application for importing/exporting structured data

Question 3

What is flume and what is its use case

Answer 3

Flume is and event-based ingestion service used for streaming data. It was originally designed for importing logs to HDFS

Question 4

What are the four main components of Flume

Answer 4

Sources: Data origin
Sink: Data destination
Channel: Acts as a bridge between source and sink
Agent: Independent daemon process which receives and forwards events to sinks and/or other agents

Question 5

Explain the 4 main components of Kafka

Answer 5

Producers: Publishes data to a topic in Lafka
Topics: A particular stream of data with a replication factor and a partition size
Consumers: Subscribes to a topic in Kafka
Consumer group: Collection of consumers reading from the same topic

Question 6

How does Kafka ensure high availability

Answer 6

Kafka is built up as a cluster of multiple instances called brokers, that can distribute reads and writes. Data is also replicated across different brokers ensuring access to data if one broker fails. Replicates are divided into leaders and followers

Question 7

What is zookeeper

Answer 7

Zookeeper is a Distributed Coordination Service for Distributed Applications. E.g. it helps keep services as Kafka alive

Question 8

What is ksqldb

Answer 8

Specialized database optimized for stream processing which exposes an SQL-Like interface to handle the data in kafka.
ksqldb runs as its own fault tolerant cluster

Question 9

What is kafka connect

Answer 9

Kafka Connect is a framework for connecting external systems to Kafka. Kafka connect can be used as a replacement for sqoop and flume.

Question 10

Explain the 6 main parts of Kafka Connect

Answer 10

Connectors: A connector connects systems to kafka. There are both sources and sinks
Tasks: The implementation of how data is copied to or from Kafka
Workers: A process that executes connectors and tasks
Transforms: Simple logic to alter each message produce or sent by a connector
Dead Letter Queue: How Connect handles Connector Errors