Kafka Consumer – CLI Tutorial

In Apache Kafka, a consumer is a client application that reads data from a Kafka cluster. The consumer subscribes to one or more topics and reads the messages that are published on those topics. The consumer is responsible for tracking its position in each topic so that it can read new messages as they are produced to the topic. The consumer can also store the messages that it reads, either for temporary use or for long-term storage. Kafka consumers can be written in any language, as long as they have access to the Kafka client library for that language. The Kafka client library provides APIs that allow the consumer to connect to a Kafka cluster, subscribe to topics, and read messages from the cluster.

How to Consume Data in a Kafka Topic using the CLI?

To consume data from a Kafka topic using the Kafka console consumer, you will need to provide the following mandatory parameters:

• The Kafka hostname and port: You can use the –bootstrap-server option to specify the hostname and port of your Kafka cluster (e.g., localhost:9092).
• The topic name is: You can use the –topic option to specify the name of the topic that you want to consume messages from.

Optionally, you can use the –-from-beginning option to start consuming messages from the beginning of the topic’s message history. Without this option, the consumer will only read new messages that are produced on the topic after they have started.

For example, the following command will start the Kafka console consumer and read messages from the my-topic topic on a Kafka cluster running on localhost:9092:

Kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic my-topic

This will launch the consumer and display any new messages sent to the my-topic topic as they arrive. To consume only the future messages of a Kafka topic, you can start the Kafka console consumer and specify the –bootstrap-server and –topic options as shown in your example:

Example:

kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic my-topic

This will start the consumer and display any new messages that are produced to the my_topic topic after the consumer has started. The consumer will not display any historical messages that were produced on the topic before it started.

Keep in mind that you will need to have followed the Kafka Producer CLI tutorial and created and sent data to the my_topic topic before you can consume messages from it.

To consume all historical and future messages in a Kafka topic, you can use the –from-beginning option when starting the Kafka console consumer.

For example, the following command will start the Kafka console consumer and read all messages from the beginning of the my_topic topic’s message history, as well as any new messages that are produced for the topic after the consumer has started:

Kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic my_topic –from-beginning

This will start the consumer and display all messages that have been produced for the my_topic topic, including historical messages and any new messages that are produced after the consumer has started.

The console consumer will remain open until you exit it and will continuously display any new messages that are produced for the topic(s) that you have specified. It does not show the key or partition information for each message by default, but you can use additional options to customize the output.

If you are not seeing any output from the console consumer but you know that your Kafka topic has data in it, you can use the –from-beginning option to start consuming messages from the beginning of the topic’s message history.

Output:

Gotchas

• When using the Kafka-console-consumer.sh command, it’s important to note that the key and metadata information will not be displayed by default, but can be shown by using specific options. Unless the –from-beginning option is specified, only new messages will be read.
• If the specified topic does not exist, it will be created with default settings. 
• The command also allows for consuming multiple topics at once, either through a comma-separated list or a pattern.
• If a consumer group id is not specified, the command will generate one automatically. 
• Also, remember that the order of messages is determined at the partition level and not the topic level.
• If you’re running a Kafka-console-consumer in a terminal window, and you want to stop consuming messages and exit the command, you can use CTRL-C to stop the process.
• You can use the –property option to set various consumer configuration properties, such as the maximum number of messages to fetch in a single poll, or the amount of time to wait for messages if the consumer is idle.
• The –from-beginning option will make the consumer start consuming messages from the earliest offset in the topic, rather than only consuming new messages. This can be useful for replaying historical data, or for debugging purposes.
• By default, the Kafka-console-consumer will display each message as a single line of text. If you want to display the key and metadata information, you can use the –property print.key=true and –property print.value=true options.
• Remember that Kafka is a distributed system, and data is stored in partitions across multiple broker nodes. So, If you want to consume data from multiple topics at the same time, make sure that the topics have the same number of partitions and replication factors.
• If you want to consume the data in order, you will need to make sure that the partitioning strategy ensures that the keys are grouped together in the same partition.
• If you want to consume data from a specific group of the consumer group, specify –consumer-property group.id= along with the group ID.
• Be mindful of the retention policy of the topic, if the data is getting expired and deleted then you will not be able to consume it.

Extra Important options you can set (advanced)

These are some additional options that you can use with the Kafka console consumer to customize its behavior. Here is a brief explanation of each option:

1. –from-beginning: This option allows you to start consuming messages from the beginning of the topic’s message history, rather than only reading new messages that are produced after the consumer has started.
2. –formatter: This option allows you to specify a custom format for displaying the messages that are consumed by the console consumer. For example, you can use the Kafka tools. DefaultMessageFormatter is formatted to display the key for each message.
3. –consumer-property: This option allows you to pass in any consumer property as a key-value pair. For example, you can use this option to set the allow.auto.create.topics property, which controls whether the consumer is allowed to automatically create non-existent topics when it starts.
4. –group: This option allows you to specify the consumer group that the console consumer belongs to. By default, the console consumer will use a random consumer group ID, but you can use this option to specify a different group ID.
5. –max-messages: This option allows you to specify the maximum number of messages that the console consumer should read before exiting.
6. –partition: This option allows you to specify a specific partition to consume from. If you do not specify a partition, the console consumer will consume from all partitions in the topic.

How to consume a Kafka Topic and show both the key and value using the Kafka Console Consumer CLI?

To consume a Kafka topic and display both the key and value using the Kafka-console-consumer.sh command, you can use the –property option to set the print.key and print.value properties to true.

Here’s an example of how you can use the Kafka-console-consumer.sh command to consume messages from a topic named “my-topic” and display both the key and value of each message:

Example:

$ kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic my-topic –formatter kafka.tools.DefaultMessageFormatter –property print.timestamp=true –property print.key=true –property print.value=true –from-beginning

This command tells the consumer to connect to a Kafka cluster running on localhost at port 9092, consumes messages from the topic “my-topic“, and sets the print.key and print.value properties to true, which causes the consumer to display both the key and value of each message.

Output:

Conclusion:

In conclusion, the Kafka console consumer is a command line tool that allows you to consume data from a Kafka topic by connecting to a Kafka cluster, subscribing to a topic, and reading messages from the cluster. To use the Kafka console consumer, you need to provide the hostname and port of your Kafka cluster, and the name of the topic you want to consume messages. You can also specify the –from-beginning option to start consuming messages from the beginning of the topic’s message history. It’s important to note that the key and metadata information will not be displayed by default, and the consumer will only read new messages unless the –from-beginning option is specified.