Kafka utilizes Zoookeeper for storing metadata information about the brokers, topics, and partititions.

Producer

Sending a Message to Kafka

Sending a Message Asynchronously

Configuring Producers

acks =1, all

Chapter 4

Kafka Consumers: Reading Data from Kafka

Kafka consumers are typically part of a consumer group.

Rebalance

Creating a Kafka Consumer

The Poll Loop

Thread Safety

One consumer per thread is the rule.

It is useful to wrap the consumer logic in its own object and then use Java’s ExecutorService to start multiple threads each with its own consumer.

Configuring Consumers

Commits and Offsets

Consumer: How Do We Exit

Chapter 5:Kafka Internals

Cluster Membership

Every broker has a unique identifier that is either set in the broker configuration file or automatically generated. Every time a broker process starts, it registers itself with its ID in Zookeeper by creating an ephemeral node.

To summarize, Kafka uses Zookeeper’s ephemeral node feature to elect a controller and to notify the controller when nodes join and leave the cluster. The controller is responsible for electing leaders among the partitions and replicas whenever it notices nodes join and leave the cluster. The controller uses the epoch number to prevent a “split brain” scenario where two nodes believe each is the current controller.

Replication

Request Processing

Fetch Requests

Physical Storage

The partitions will be stored in the log.dirs.

Partition Allocation

File Management

Indexes