标签：log partition Kafka note records leader partitions
- The Kafka cluster stores streams of records in categories called topics.
- Each record consists of a key, a value, and a timestamp.
- Kafka has four core APIs: Producer(publish a stream of records to one or more Kafka topics), Consumer(subscribe to one or more topics and process the stream of records produced to them), Streams(act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams), Connector(allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems)
- In Kafka the communication between the clients and the servers is done with TCP
- A topic is a category or feed name to which records are published, a topic can have zero, one, or many consumers that subscribe to the data written to it.
- For each topic, the Kafka cluster maintains a partitioned log, like this
Anatomy of Topic
partiotion 0: | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | ...
partiotion 1: | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | ...
partiotion 2: | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 12 | ...
..........................................................
Old ------------------------------>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> New- Each partition is an ordered, immutable sequence of records that is continually appended to—a structured commit log. The records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition.
- The Kafka cluster durably persists all published records—whether or not they have been consumed—using a configurable retention period, after which it will be discarded to free up space.
- The only metadata retained on a per-consumer basis is the position of that consumer in the log.
- This offset is controlled by the consume, so that the consumer can consume records in any order it likes, normally is linearly.
- Kafka consumers are very cheap, for example, you can use our command line tools to "tail" the contents of any topic without changing what is consumed by any existing consumers.
- The partitions in the log serve several purposes. First, they allow the log to scale beyond a size that will fit on a single server, a topic may have many partitions so it can handle an arbitrary amount of data. Second they act as the unit of parallelism—more on that in a bit.
- The partitions of the log are distributed over the servers in the Kafka cluster with each server handling data and requests for a share of the partitions, each partition is replicated across a configurable number of servers for fault tolerance.
- Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster.
- The producer is responsible for choosing which record to assign to which partition within the topic. This can be done in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the record).
标签：log,partition,Kafka,note,records,leader,partitions 来源： https://www.cnblogs.com/seliote/p/11768839.html
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。