Kafka replication

kafka_replication_detailed_design_v2.pdf

kafka Detailed Replication Design V3

Apache Kafka中Follower如何从Leader fetch消息

Kafka深度解析，众人推荐，精彩好文！

Kafka 的集群复制设计

Kafka的Log存储解析

KIP-1 - Remove support of request.required.acks

0.8.0 Producer Example

Hands-free Kafka Replication: A lesson in operational simplicity

Kafka Replication

Intra-cluster Replication in Apache Kafka

The current leader of a partition further maintains 3 sets: AR, ISR, CUR and RAR, which correspond to the set of replicas that are assigned to the partition, in-sync with the leader, catching up with the leader, and being reassigned to other brokers. Normally, ISR ⊆ AR and AR = ISR + CUR.

Replica

Replica { // a replica of a partition

broker_id : int

partition : Partition

isLocal : Boolean // is this replica local to this broker

log : Log // local log associated with this replica

hw : long // offset of the last committed message

leo : long // log end offset

}

Partition

Partition { //a partition in a topic

    topic : string

    partition_id : int

    leader: Replica // the leader replica of this partition

    commitQ: Queue // produce requests pending commit at the leader

    AR: Set[Replica] // replicas assigned to this partition

    ISR: Set[Replica] // In-sync replica set, maintained at the leader

    CUR: Set[Replica] // Catch-up replica set, maintained at the leader

    RAR: Set[Replica] // Reassigned replica set, maintained at the leader

}