Spring Boot – Start/Stop a Kafka Listener Dynamically

When a Spring Boot application starts, Kafka Listener's default behavior is to begin listening for a certain topic. However, there are situations when we don't want to start it right away once our application launches.

To start or stop a Kafka Listener dynamically we need major three approaches i.e. start/stop when Kafka messages need to be processed, use of @KafkaListener annotation, use of kafkaListenerEndpointRegistry

In this article, we will go through how to dynamically start or stop a Kafka listener.

Different Approaches to Start/Stop a Kafka Listener

Approach 1:

Start an application, when there is a need for the Kafka messages to be processed.
Stop an application after successful processing.

Approach 2: While registering Kafka Listener we can set the below id property.

@KafkaListener(id = "id-1", groupId = "group-1", topics = "Message-topic", 
                   containerFactory = "messageListenerFactory", autoStartup = "false")
    public void consumeMessage(Message message)

Approach 3: Autowire KafkaListenerEndpointRegistry bean for controlling the start or stop of the Kafka Listener.

@Autowired
    KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;

To start:

    public boolean startListener(String listenerId) {
        MessageListenerContainer listenerContainer = 
          kafkaListenerEndpointRegistry.getListenerContainer(listenerId);
        assert listenerContainer != null : false;
        listenerContainer.start();

To stop:

public boolean stopListener(String listenerId) {
        MessageListenerContainer listenerContainer = 
          kafkaListenerEndpointRegistry.getListenerContainer(listenerId);
        assert listenerContainer != null : false;
        listenerContainer.stop();
        logger.info("{} Kafka Listener Stopped.", listenerId);

We will be implementing the above syntactical approaches as an example below.

Implementation of start or stop a specific Kafka Listener

Make a class whose objects will be consumed by the Kafka listener.

File: Message.java

Java

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Message {
    private String message;
}

Configure the consumer that will be used by Kafka Listener.

File: KakfaConsumerConfig.java

Java

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.support.serializer.JsonDeserializer;

import java.util.HashMap;
import java.util.Map;

@Configuration
public class KafkaConsumerConfig {

    private String kafkaUrl = "localhost:9092";

    @Bean
    public ConsumerFactory<String, Message> messageConsumerFactory() {
        JsonDeserializer<Message> deserializer = 
             new JsonDeserializer<>(Message.class, false);
        deserializer.setRemoveTypeHeaders(false);
        deserializer.addTrustedPackages("*");
        deserializer.setUseTypeMapperForKey(true);

        Map<String, Object> config = new HashMap<>();
        config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaUrl);
        config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, 
                   StringDeserializer.class);
        config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, 
                   JsonDeserializer.class);
        config.put(ConsumerConfig.GROUP_ID_CONFIG, "group-1");
        return new DefaultKafkaConsumerFactory<>(config, new StringDeserializer(), 
                                                 deserializer);
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Message> 
      messageListenerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, Message> containerFactory = 
          new ConcurrentKafkaListenerContainerFactory();
        containerFactory.setConsumerFactory(messageConsumerFactory());
        return containerFactory;
    }
}

Create a Kafka listener with the necessary parameters.

id: The container's unique identifier for this listener. If none is specified, an auto-generated ID is used.
groupId: Override the group.id property for the consumer factory with this value for this listener only.
topics: The topics of this listener. The entries can be "topic names," "property placeholder keys," or "expressions." The topic name must be resolved from an expression. This uses group management, and Kafka will assign partitions to group members.
containerFactory: The beans name of the KafkaListenerContainerFactory that will be used to create the message listener container that will serve this endpoint.
autoStartup: Set to true or false to override the container factory's default setting. By default, the value is set to true, and because of this, it'll start consuming messages as soon as our application starts.

File: KafkaMessageListener.java

Java

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.KafkaListener;

@Configuration
public class KafkaMessageListener {

    Logger logger = LoggerFactory.getLogger(KafkaMessageListener.class);

    @KafkaListener(id = "id-1", groupId = "group-1", topics = "Message-topic", 
                   containerFactory = "messageListenerFactory", autoStartup = "false")
    public void consumeMessage(Message message) {
        logger.info("Message received : -> {}", message);
    }
}

The KafkaListenerEndpointRegistry class can be used to get a Kafka Listener Container by listenerId. Here we have used @KafkaListener annotation to stating the bean method as a listener for the Kafka Listener Container. The Kafka Listener can now be started or stopped using this container.

File: KafkaListenerAutomation.java

Java

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.config.KafkaListenerEndpointRegistry;
import org.springframework.kafka.listener.MessageListenerContainer;
import org.springframework.stereotype.Component;

@Component
public class KafkaListenerAutomation {

    private final Logger logger = LoggerFactory.getLogger(KafkaListenerAutomation.class);

    @Autowired
    KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;

    public boolean startListener(String listenerId) {
        MessageListenerContainer listenerContainer = 
           kafkaListenerEndpointRegistry.getListenerContainer(listenerId);
        assert listenerContainer != null : false;
        listenerContainer.start();
        logger.info("{} Kafka Listener Started", listenerId);
        return true;
    }

    public boolean stopListener(String listenerId) {
        MessageListenerContainer listenerContainer = 
          kafkaListenerEndpointRegistry.getListenerContainer(listenerId);
        assert listenerContainer != null : false;
        listenerContainer.stop();
        logger.info("{} Kafka Listener Stopped.", listenerId);
        return true;
    }
}

Using API endpoints, we can start or stop a specific Kafka listener by providing the listenerID.

File: StartOrStopListenerController.java

Java

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class StartOrStopListenerController {

    @Autowired
    KafkaListenerAutomation kafkaListenerAutomation;

    @GetMapping("/start")
    public void start(@RequestParam("id") String listenerId) {
        kafkaListenerAutomation.startListener(listenerId);
    }

    @GetMapping("/stop")
    public void stop(@RequestParam("id") String listenerId) {
        kafkaListenerAutomation.stopListener(listenerId);
    }
}

Output:

1. Kafka Listener started:

2. Kafka Listener received message:

3. Kafka Listener stopped:

Conclusion

The app should ideally be started when the Kafka messages need to be processed and stopped as soon as that process is complete. It's a good practice to constrain your Kafka listener to utilize it efficiently.

Article Tags :

Java

Technical Scripter

Apache Kafka Java

Java-Spring

Java-Spring-Boot

Technical Scripter 2022