Apache Kafka – Consumer Seek and Assign using Java
Last Updated :
18 Mar, 2023
Kafka Consumer is used to reading data from a topic and remember a topic again identified by its name. So the consumers are smart enough and will know which broker to read from and which partitions to read from. And in case of broker failures, the consumers know how to recover and this is again a good property of Apache Kafka. Now data for the consumers is going to be read in order within each partition. In this article, we are going to discuss the step-by-step implementation of Apache Kafka Consumer Seek and Assign using Java. assign() and seek() are mainly used to reply to data or fetch a specific message.
Step-by-Step Implementation
Step 1: Create a New Apache Kafka Project in IntelliJ
To create a new Apache Kafka Project in IntelliJ using Java and Maven please refer to How to Create an Apache Kafka Project in IntelliJ using Java and Maven.
Step 2: Install and Run Apache Kafka
To Install and Run Apache Kafka in your local system please refer to How to Install and Run Apache Kafka.
Step 3: Create a Consumer
First, we have to create Consumer Properties. And to create Consumer Properties refer to the below code snippet
Create Consumer Properties:
String bootstrapServer = "127.0.0.1:9092";
// Create Consumer Properties
Properties properties = new Properties();
properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServer);
properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
Create Kafka Consumer:
KafkaConsumer<String, String> consumer =
new KafkaConsumer<>(properties);
Kafka Consumer assign():
String topics = "gfg_topic";
TopicPartition partitionToReadFrom = new TopicPartition(topics, 0);
long offSetToReadFrom = 3L;
consumer.assign(List.of(partitionToReadFrom));
Kafka Consumer seek():
consumer.seek(partitionToReadFrom, offSetToReadFrom);
Poll the data:
var messagesToRead = 3;
var continueReading = true;
var messagesReadSoFar = 0;
// Poll the data
while (continueReading) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
messagesReadSoFar++;
logger.info("Key: " + record.key() +
" Value: " + record.value() +
" Partition: " + record.partition() +
" Offset: " + record.offset()
);
if (messagesReadSoFar >= messagesToRead) {
continueReading = false;
break;
}
}
}
Below is the complete code. Comments are added inside the code to understand the code in more detail.
Java
package org.kafkademo.basics;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Duration;
import java.util.List;
import java.util.Properties;
public class KafkaConsumerWithAssignSeekDemo {
public static void main(String[] args) {
Logger logger = LoggerFactory.getLogger(KafkaProducerWithKeyDemo. class );
String bootstrapServer = "127.0.0.1:9092" ;
String topics = "gfg_topic" ;
Properties properties = new Properties();
properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServer);
properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer. class .getName());
properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer. class .getName());
properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest" );
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
TopicPartition partitionToReadFrom = new TopicPartition(topics, 0 );
long offSetToReadFrom = 3L;
consumer.assign(List.of(partitionToReadFrom));
consumer.seek(partitionToReadFrom, offSetToReadFrom);
var messagesToRead = 3 ;
var continueReading = true ;
var messagesReadSoFar = 0 ;
while (continueReading) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis( 100 ));
for (ConsumerRecord<String, String> record : records) {
messagesReadSoFar++;
logger.info( "Key: " + record.key() +
" Value: " + record.value() +
" Partition: " + record.partition() +
" Offset: " + record.offset()
);
if (messagesReadSoFar >= messagesToRead) {
continueReading = false ;
break ;
}
}
}
}
}
|
Step 4: Run the Application
Now run the application and below is the output.
"C:\Users\Amiya Rout\.jdks\corretto-11.0.15\bin\java.exe" "
[main] INFO org.apache.kafka.clients.consumer.ConsumerConfig - ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = earliest
bootstrap.servers = [127.0.0.1:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = consumer-null-1
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = null
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.8.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: ebb1d6e21cc92130
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1674842186190
[main] INFO org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-null-1, groupId=null] Subscribed to partition(s): gfg_topic-0
[main] INFO org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-null-1, groupId=null] Seeking to offset 3 for partition gfg_topic-0
[main] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-null-1, groupId=null] Cluster ID: orhF-HNsR465cORhmU3pTg
[main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_3 Value: hello_geeksforgeeks 3 Partition: 0 Offset: 3
[main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_4 Value: hello_geeksforgeeks 4 Partition: 0 Offset: 4
[main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_5 Value: hello_geeksforgeeks 5 Partition: 0 Offset: 5
Process finished with exit code 0
And you can see in the output screen, we have consumed only 3 messages and it started from offset 3. Refer to the below image.
Share your thoughts in the comments
Please Login to comment...