What is Apache Kafka Consumer?
Apache Kafka consumer is used to ingesting messages from a topic within a Kafka Cluster. It consumes messages within a partition of a topic in a sequential manner. The consumer tracks the messages being taken/read from the topic and when asked for a message it is required to supply a sequential ID and the messages from that ID onwards are read sequentially thereafter. Kafka uses the pull method from the consumer to consume a message which creates a high throughput of message consumption in Kafka. The throughput of message consumption is left to individual consumers, and because of this Kafka is ready to integrate with a variety of consumers having different capabilities.
Old messages are consumed by consumers if required because these messages are stored in the Memory of Kafka. In a consumer group, a message is certainly consumed at least once.
What are Apache Kafka Consumer Groups?
Multiple hosts arrange a consumer group that allows accessing a particular topic. Consumer groups assure that a message is only read by one consumer in the group. As shown in the following figure, partitions are assigned to each consumer in a group and these are trusted to read the messages in a topic.
The following figure shows the working of a consumer group in action in a Kafka deployment.
To understand a consumer group we have just taken two brokers with one topic replicated with the two brokers, and in each broker, the topic has two partitions. We have two consumer groups, group A having two consumers and group B having 4 consumers. now both groups will read messages from broker 0 and broker 1.