Configuring Multiple Spring Batch Jobs in a Spring Boot Application

Spring Batch serves as a robust framework within the Spring ecosystem, specifically tailored for managing batch processing tasks efficiently. It’s designed to tackle big data jobs efficiently and comes with handy tools to make batch application development a breeze. In the context of a Spring Boot application, setting up multiple Spring Batch jobs involves creating configurations for each job and ensuring a clear and organized structure for the batch processing logic.

This includes defining the details of each job, specifying the steps involved, and establishing the necessary components to execute the jobs seamlessly. The aim is to facilitate a systematic and modular approach to batch processing within the overall Spring Boot framework.

Example of Configuring Multiple Spring Batch Jobs

Let’s relate Spring Batch to managing some backend processes for the GeeksforGeeks website:

Scenario: Daily Content Update for GeeksforGeeks

• Coding Challenges Update (Job 1):
  Objective: Update the set of daily coding challenges on the platform.
  How Spring Batch Helps: It organizes and executes the process of fetching, validating, and updating coding challenges from different sources. Each day, this job ensures a fresh set of challenges is available for users.
• Interview and Exam Prep Content Update (Job 2):
  Objective: Keep the interview and exam preparation sections up to date with the latest content.
  How Spring Batch Helps: This job manages the extraction and processing of content from various contributors and ensures that the interview and exam preparation sections are always enriched with relevant material.
• Articles and Practice Questions Update (Job 3):
  Objective: Regularly update the articles, quizzes, and practice questions available on the website.
  How Spring Batch Helps: It takes care of organizing the retrieval and publication of articles, quizzes, and practice questions. This job guarantees that users have access to the most recent and valuable resources.
• User Engagement Metrics Update (Job 4):
  Objective: Analyze and update user engagement metrics for the website.
  How Spring Batch Helps: This job involves processing user interaction data, updating engagement metrics, and generating reports. Spring Batch ensures this task is done efficiently and reliably, providing insights into how users are interacting with the content.

In this scenario, Spring Batch acts as the reliable manager, orchestrating these backend processes as separate jobs. Each job has a specific responsibility, and Spring Batch ensures that these tasks are carried out seamlessly, maintaining the freshness and relevance of GeeksforGeeks content for its users.

Setting up Multiple Spring Batch Jobs in Spring Boot: A Step-by-Step Guide

Step 1: Create a Spring Boot Project

Create a new Spring Boot project using your preferred IDE or Spring Initializr.

Step 2: Add Spring Batch Dependencies

In your project’s pom.xml file, include the necessary Spring Batch dependencies:

For Maven Project:consider below pom.xml file

XML

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!-- Parent project information for managing versions -->
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.6.1</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <!-- Project information -->
    <groupId>com.example.springbatchmultiplejobsconfig</groupId>
    <artifactId>SpringBatchMultipleJobsConfiguration</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>SpringBatchMultipleJobsConfiguration</name>
    <description>RESTful API for Spring Batch Multiple Jobs Configuration</description>

    <!-- Java version -->
    <properties>
        <java.version>8</java.version>
    </properties>

    <!-- Project dependencies -->
    <dependencies>
        <!-- Spring Boot Starter for Batch Processing -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-batch</artifactId>
        </dependency>

        <!-- Spring Boot Starter for Web -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <!-- H2 Database for runtime use -->
        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
            <scope>runtime</scope>
        </dependency>

        <!-- Lombok for reducing boilerplate code (optional) -->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>

        <!-- Spring Boot Starter for testing -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <!-- Spring Batch Testing Support -->
        <dependency>
            <groupId>org.springframework.batch</groupId>
            <artifactId>spring-batch-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <!-- Maven build configuration -->
    <build>
        <plugins>
            <!-- Spring Boot Maven Plugin for packaging and running the application -->
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>
                            <groupId>org.projectlombok</groupId>
                            <artifactId>lombok</artifactId>
                        </exclude>
                    </excludes>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

For Gradle Project:
implementation 'org.springframework.boot:spring-boot-starter-batch'

Below is the project directory structure:

Step 3: Create Entity and Define Job Configurations

Create a configuration class for each job. Below is the job configurations for above discussed example:
Job 1: Coding Challenges Update

Java

// CodingChallenge.java

package com.example.multibatchspringbootapp.entity;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

import lombok.Data;

@Entity
@Data
public class CodingChallenge {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String title;
    private String description;
    private String difficulty;
    private String source;
}

Java

// CodingChallengesJobConfig.java
package com.example.multibatchspringbootapp.batch;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import com.example.multibatchspringbootapp.entity.CodingChallenge;

/**
 * @author rahul.chauhan
 */
@Configuration
@EnableBatchProcessing
public class CodingChallengesJobConfig {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final ItemReader<CodingChallenge> codingChallengesItemReader;
    private final ItemWriter<CodingChallenge> codingChallengesItemWriter;

    public CodingChallengesJobConfig(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory,
            ItemReader<CodingChallenge> codingChallengesItemReader,
            ItemWriter<CodingChallenge> codingChallengesItemWriter) {
        this.jobBuilderFactory = jobBuilderFactory;
        this.stepBuilderFactory = stepBuilderFactory;
        this.codingChallengesItemReader = codingChallengesItemReader;
        this.codingChallengesItemWriter = codingChallengesItemWriter;
    }

    @Bean
    public Job codingChallengesJob() {
        return jobBuilderFactory.get("codingChallengesJob").incrementer(new RunIdIncrementer())
                .start(codingChallengesStep()).build();
    }

    @Bean
    public Step codingChallengesStep() {
        return stepBuilderFactory.get("codingChallengesStep").<CodingChallenge, CodingChallenge>chunk(10)
                .reader(codingChallengesItemReader).writer(codingChallengesItemWriter).build();
    }
}

Job 2: Interview and Exam Prep Content Update

Java

// InterviewExamContent.java
package com.example.multibatchspringbootapp.entity;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

import lombok.Data;

@Data
@Entity
public class InterviewExamContent {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String title;
    private String content;
    private String contributor;
    private String category;

}

Java

// InterviewExamContentJobConfig.java
package com.example.multibatchspringbootapp.batch;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import com.example.multibatchspringbootapp.entity.InterviewExamContent;

/**
 * @author rahul.chauhan
 */
@Configuration
@EnableBatchProcessing
public class InterviewExamContentJobConfig {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final ItemReader<InterviewExamContent> interviewExamContentItemReader;
    private final ItemWriter<InterviewExamContent> interviewExamContentItemWriter;

    public InterviewExamContentJobConfig(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory,
            ItemReader<InterviewExamContent> interviewExamContentItemReader,
            ItemWriter<InterviewExamContent> interviewExamContentItemWriter) {
        this.jobBuilderFactory = jobBuilderFactory;
        this.stepBuilderFactory = stepBuilderFactory;
        this.interviewExamContentItemReader = interviewExamContentItemReader;
        this.interviewExamContentItemWriter = interviewExamContentItemWriter;
    }

    @Bean
    public Job interviewExamContentJob() {
        return jobBuilderFactory.get("interviewExamContentJob").incrementer(new RunIdIncrementer())
                .start(interviewExamContentStep()).build();
    }

    @Bean
    public Step interviewExamContentStep() {
        return stepBuilderFactory.get("interviewExamContentStep").<InterviewExamContent, InterviewExamContent>chunk(10)
                .reader(interviewExamContentItemReader).writer(interviewExamContentItemWriter).build();
    }
}

Job 3: Articles and Practice Questions Update

Java

// ArticlesPracticeQuestions.java
package com.example.multibatchspringbootapp.entity;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

import lombok.Data;

@Data
@Entity
public class ArticlesPracticeQuestions {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String title;
    private String content;
    private String category;
    private String author;

}

Java

// ArticlesPracticeQuestionsJobConfig.java
package com.example.multibatchspringbootapp.batch;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import com.example.multibatchspringbootapp.entity.ArticlesPracticeQuestions;

/**
 * @author rahul.chauhan
 */
@Configuration
@EnableBatchProcessing
public class ArticlesPracticeQuestionsJobConfig {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final ItemReader<ArticlesPracticeQuestions> articlesPracticeQuestionsItemReader;
    private final ItemWriter<ArticlesPracticeQuestions> articlesPracticeQuestionsItemWriter;

    public ArticlesPracticeQuestionsJobConfig(JobBuilderFactory jobBuilderFactory,
            StepBuilderFactory stepBuilderFactory,
            ItemReader<ArticlesPracticeQuestions> articlesPracticeQuestionsItemReader,
            ItemWriter<ArticlesPracticeQuestions> articlesPracticeQuestionsItemWriter) {
        this.jobBuilderFactory = jobBuilderFactory;
        this.stepBuilderFactory = stepBuilderFactory;
        this.articlesPracticeQuestionsItemReader = articlesPracticeQuestionsItemReader;
        this.articlesPracticeQuestionsItemWriter = articlesPracticeQuestionsItemWriter;
    }

    @Bean
    public Job articlesPracticeQuestionsJob() {
        return jobBuilderFactory.get("articlesPracticeQuestionsJob").incrementer(new RunIdIncrementer())
                .start(articlesPracticeQuestionsStep()).build();
    }

    @Bean
    public Step articlesPracticeQuestionsStep() {
        return stepBuilderFactory.get("articlesPracticeQuestionsStep")
                .<ArticlesPracticeQuestions, ArticlesPracticeQuestions>chunk(10)
                .reader(articlesPracticeQuestionsItemReader).writer(articlesPracticeQuestionsItemWriter).build();
    }
}

Job 4: User Engagement Metrics Update

Java

// UserEngagementMetricsJobConfig.java
package com.example.multibatchspringbootapp.batch;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * @author rahul.chauhan
 */
@Configuration
@EnableBatchProcessing
public class UserEngagementMetricsJobConfig {

    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final ItemReader<UserEngagementMetrics> userEngagementMetricsItemReader;
    private final ItemWriter<UserEngagementMetrics> userEngagementMetricsItemWriter;

    public UserEngagementMetricsJobConfig(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory,
            ItemReader<UserEngagementMetrics> userEngagementMetricsItemReader,
            ItemWriter<UserEngagementMetrics> userEngagementMetricsItemWriter) {
        this.jobBuilderFactory = jobBuilderFactory;
        this.stepBuilderFactory = stepBuilderFactory;
        this.userEngagementMetricsItemReader = userEngagementMetricsItemReader;
        this.userEngagementMetricsItemWriter = userEngagementMetricsItemWriter;
    }

    @Bean
    public Job userEngagementMetricsJob() {
        return jobBuilderFactory.get("userEngagementMetricsJob").incrementer(new RunIdIncrementer())
                .start(userEngagementMetricsStep()).build();
    }

    @Bean
    public Step userEngagementMetricsStep() {
        return stepBuilderFactory.get("userEngagementMetricsStep")
                .<UserEngagementMetrics, UserEngagementMetrics>chunk(10).reader(userEngagementMetricsItemReader)
                .writer(userEngagementMetricsItemWriter).build();
    }
}

Java

// UserEngagementMetrics.java
package com.example.multibatchspringbootapp.controller;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

import lombok.Data;

@Data
@Entity
public class UserEngagementMetrics {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private String userId;
    private String interactionType;
    private String contentId;
    private String timestamp;

}

Step 4: Create a Common Batch Configuration

Create a common batch configuration class that provides common beans such as JobLauncher, JobRepository, DataSource, etc.

Java

package com.example.multibatchspringbootapp.batch;

import javax.batch.api.chunk.ItemReader;
import javax.batch.api.chunk.ItemWriter;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
 * @author rahul.chauhan
 */
@Configuration
@EnableBatchProcessing
public class CommonBatchConfig {

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    // ItemReaders and ItemWriters for each job
    @Autowired
    private ItemReader<String> codingChallengesReader;
    @Autowired
    private ItemWriter<String> codingChallengesWriter;

    @Autowired
    private ItemReader<String> interviewExamContentReader;
    @Autowired
    private ItemWriter<String> interviewExamContentWriter;

    @Autowired
    private ItemReader<String> articlesPracticeQuestionsReader;
    @Autowired
    private ItemWriter<String> articlesPracticeQuestionsWriter;

    @Autowired
    private ItemReader<String> userEngagementMetricsReader;
    @Autowired
    private ItemWriter<String> userEngagementMetricsWriter;

    // Step configuration for Coding Challenges job
    @Bean
    public Step codingChallengesStep() {
        return stepBuilderFactory.get("codingChallengesStep")
                .<String, String>chunk(10)
                .reader(codingChallengesReader)
                .writer(codingChallengesWriter)
                .build();
    }

    // Job configuration for Coding Challenges job
    @Bean
    public Job codingChallengesJob() {
        return jobBuilderFactory.get("codingChallengesJob")
                .incrementer(new RunIdIncrementer())
                .start(codingChallengesStep())
                .build();
    }

    // Step configuration for Interview Exam Content job
    @Bean
    public Step interviewExamContentStep() {
        return stepBuilderFactory.get("interviewExamContentStep")
                .<String, String>chunk(10)
                .reader(interviewExamContentReader)
                .writer(interviewExamContentWriter)
                .build();
    }

    // Job configuration for Interview Exam Content job
    @Bean
    public Job interviewExamContentJob() {
        return jobBuilderFactory.get("interviewExamContentJob")
                .incrementer(new RunIdIncrementer())
                .start(interviewExamContentStep())
                .build();
    }

    // Step configuration for Articles Practice Questions job
    @Bean
    public Step articlesPracticeQuestionsStep() {
        return stepBuilderFactory.get("articlesPracticeQuestionsStep")
                .<String, String>chunk(10)
                .reader(articlesPracticeQuestionsReader)
                .writer(articlesPracticeQuestionsWriter)
                .build();
    }

    // Job configuration for Articles Practice Questions job
    @Bean
    public Job articlesPracticeQuestionsJob() {
        return jobBuilderFactory.get("articlesPracticeQuestionsJob")
                .incrementer(new RunIdIncrementer())
                .start(articlesPracticeQuestionsStep())
                .build();
    }

    // Step configuration for User Engagement Metrics job
    @Bean
    public Step userEngagementMetricsStep() {
        return stepBuilderFactory.get("userEngagementMetricsStep")
                .<String, String>chunk(10)
                .reader(userEngagementMetricsReader)
                .writer(userEngagementMetricsWriter)
                .build();
    }

    // Job configuration for User Engagement Metrics job
    @Bean
    public Job userEngagementMetricsJob() {
        return jobBuilderFactory.get("userEngagementMetricsJob")
                .incrementer(new RunIdIncrementer())
                .start(userEngagementMetricsStep())
                .build();
    }
}

Step 5: Define a batch controller to handle all the defined jobs

A batch controller that provides endpoints for triggering each of the defined batch jobs. Each endpoint corresponds to a specific batch job, and you can trigger these endpoints as needed, either manually or through some external mechanism.

Java

@RestController
@RequestMapping("/batch")
public class BatchController {

    @Autowired
    private JobLauncher jobLauncher;

    @Autowired
    private Job codingChallengesJob;

    @Autowired
    private Job interviewExamContentJob;

    @Autowired
    private Job articlesPracticeQuestionsJob;

    @Autowired
    private Job userEngagementMetricsJob;

    @PostMapping("/runCodingChallengesJob")
    public ResponseEntity<String> runCodingChallengesJob() {
        return runJob(codingChallengesJob);
    }

    @PostMapping("/runInterviewExamContentJob")
    public ResponseEntity<String> runInterviewExamContentJob() {
        return runJob(interviewExamContentJob);
    }

    @PostMapping("/runArticlesPracticeQuestionsJob")
    public ResponseEntity<String> runArticlesPracticeQuestionsJob() {
        return runJob(articlesPracticeQuestionsJob);
    }

    @PostMapping("/runUserEngagementMetricsJob")
    public ResponseEntity<String> runUserEngagementMetricsJob() {
        return runJob(userEngagementMetricsJob);
    }

    private ResponseEntity<String> runJob(Job job) {
        try {
            JobParameters jobParameters = new JobParametersBuilder()
                    .addString("jobName", job.getName())
                    .toJobParameters();

            JobExecution jobExecution = jobLauncher.run(job, jobParameters);

            return ResponseEntity.ok("Batch Job " + job.getName()
                                     + " started with JobExecutionId: " + jobExecution.getId());
        } catch (Exception e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                                    .body("Failed to start Batch Job: " + e.getMessage());
        }
    }
}

Explanation of the above Program:

In summary, this controller allows you to trigger different batch jobs by hitting the corresponding endpoints. It simplifies the initiation of batch processes in your Spring Boot application.

Annotation / Keyword	Description
@RestController	This class handles HTTP requests and produces HTTP responses.
@RequestMapping(“/batch”)	All endpoints in this controller will start with “/batch”.
@Autowired	Spring injects dependencies like JobLauncher and various Job instances into the controller.
@PostMapping(“/runCodingChallengesJob”) jobs.	When a POST request is made to “/batch/runCodingChallengesJob”, it triggers the runCodingChallengesJob method. Similar endpoints exist for other
runJob method	A helper method to start a specific job. It creates job parameters, launches the job, and returns a response indicating success or failure.

Step 5: Run the Application

Start your Spring Boot application. You can use the embedded server provided by Spring Boot or package your application as a deployable artifact.

Step 6: Trigger Batch Jobs

Use tools like Postman, curl, or create a simple UI to trigger the batch jobs by making HTTP POST requests to the corresponding endpoints defined in your BatchController.

Objective: Efficiently process large data volumes using Spring Batch in Spring Boot.

Steps:

• Define distinct batch jobs (e.g., coding challenges update, content refresh).
• Create job configuration classes for each job.
• Develop a common batch configuration (BatchConfig) for shared components.
• Implement a BatchController to expose job-triggering endpoints.
• Integrate the controller into the Spring Boot application.
• Write tests for jobs and the controller.
• Optionally, ensure security and error-handling measures.
• Deploy and maintain the application.
• Outcome: Simplifies batch processing organization and execution in a Spring Boot application.

Conclusion

In conclusion, configuring multiple Spring Batch jobs in a Spring Boot app is like assigning different workers to handle various data tasks. It’s a smart way to keep things organized and easily manageable. The combo of Spring Batch and Spring Boot makes data processing smooth and straightforward.