How to Create an Apache Kafka Project in IntelliJ using Java and Maven?

Apache Kafka allows you to decouple your data streams and systems. So the idea is that the source systems will be responsible for sending their data into Apache Kafka. Then any target systems that want to get access to this data feed this data stream will have to query and read from Apache Kafka to get the stream of data from these 3 systems and so by having this decoupling we are putting the responsibility of receiving and sending the data all on Apache Kafka. It is a publish-subscribe messaging system. A messaging system lets you send messages between processes, applications, and servers. Apache Kafka is software where topics (A topic might be a category) can be defined and further processed. 

In this article, we are going to discuss the step-by-step implementation of how to Create an Apache Kafka Project in IntelliJ using Java and Maven.

Step-by-Step Implementation

Step 1: Open IntelliJ and go to File > New > Project as shown in the below image.

Step 2: After Step 1 a pop-up will be shown like this. Here you have to choose Language as Java, and Build System as Maven. You may also provide your GroupId and ArtifactId also. And at last click on Create. Refer to the below image.

Note: It is recommended that the JDK version must be 8 or greater than 8.

Step 3: Now to use Kafka in our project we have to add its dependency. So go to Maven Repository and search for Apache Kafka Client. Then from the list of versions choose your version. In this article, we have chosen the 2.8.0 version. And copy the dependency. Refer to the below image.

Step 4: Now come back to your IntelliJ project and inside the pom.xml file paste the dependency like this.

XML

<dependencies>

        <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.8.0</version>
        </dependency>

</dependencies>

Refer to the below image.

Below is the complete code for the pom.xml file.

XML

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.kafkademo</groupId>
    <artifactId>kafka-demo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>

        <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.8.0</version>
        </dependency>

    </dependencies>

</project>

And you are done. Now go to the src > main > java and create your first Apache Kafka Producer like this and use all the Classes, Methods, and Interfaces provided by the Kafka library.

Java

package org.kafkademo.basics;

public class KafkaProducerDemo {
    public static void main(String[] args) {

      // ........ Write your code here ......

    }
}