The consumer is the one who consumes data from a certain partition from the Kafka cluster. It knows from which broker to read or consume data on a specific topic.

Consumer Groups, on the other hand, consist of a group of more than one consumer and each customer reads or consumes data from the assigned partitions. If the number of consumers will exceed the number of partitions, the new consumers will go into the inactive state until an existing customer unsubscribes.    

Consider this scenario with two consumers -

Let us assume there are two groups of consumers- Consumer Group A and Consumer Group B. If the consumers from Consumer Group A are reading data from different partitions, it implies that they are reading it parallel under Topic - T and therefore will remain in an active state.

Now, if one of the consumers of Group B reads data from Partition - 1 under Topic - T, then also the consumer remains in an active state as it belongs to Group B. 

Consider another scenario with three consumers -

There are three consumers with two partitions available. Two consumers - Consumer A and Consumer B are in an active state with consumer A reading data from Partition 0 and consumer B reading from Partition 1. Consumer C will remain in an inactive state unless consumer A or consumer B leaves. 

Now let us see examples of how to send and receive messages through Java client.

Here’s how you can create a sample group with two consumers -

>>java -cp “/path/to/kafka/kafka_2.11-0.9.0.0/libs/*":. 

ConsumerGroup <topic-name> my-group

>>java -cp "/home/bala/Workspace/kafka/kafka_2.11-0.9.0.0/libs/*":. 

ConsumerGroup <topic-name> my-group

Now send messages using Producer CLI -

Test consumer group 01

Test consumer group 02

It will give the output of the first and the second process respectively as -

Subscribed to topic Hello-kafka

offset = 3, key = null, value = Test consumer group 01

And 

Subscribed to topic Hello-kafka

offset = 3, key = null, value = Test consumer group 02