Spring Boot Batch Processing Using Spring Data JPA to CSV File

The Spring Batch is a framework in the Spring Boot ecosystem It can provide a lot of functionalities for Batch processing. The Spring Batch framework simplifies the batch development of applications by providing reliable components and other patterns for common batch processing concerns.

Mostly, batch processing is used to read and write data in bulk amounts of data, This Batch processing can be able to handle transactions of data and process the data in the form of chunks, and the other one manages the job execution also.

In this article, we will learn the required terminology of Spring Batch processing using Spring Data JPA to CSV File.

Key Terminologies:

• Spring Boot Integration: The Spring Boot Batch is used to simplify the configuration and deployment of batch applications. One more thing is The Spring Boot provides auto-configuration this feature simplifies development when stepping up the components for Spring Boot batch components.
• Job: A Job is defined as an overall process to be executed means from starting of the batch application to the end of the execution of the Spring Boot Batch Application the Job contains one or more steps for handling the batch processing.
• ItemReader: The ItemReader is mostly used for reading data from resources like CSV files, Flat files and from Databases, and other data sources. It plays an important role in Batch applications for reading data.
• ItemProcessor: The ItemProcessor is used to process read data before sending to writer. At this moment It can perform lot of functions like Data Filtering, transformation and other things.
• ItemWriter: The ItemWriter is used for writes the processed data to a destination such as a database and Other File types. In our case, our destination is a CSV File.
• Chunk: The Chunk is defined as means It reads data in the form of chunks, processes the data and It writes results into chunks. It can be able to handle large datasets in efficient ways.
• Job Repository: The Spring Batch is uses Job Repository for store metadata about Spring Batch Applications. Means metadata about Job, Step and Job execution and other things.
• Listeners: Spring Batch supports listeners that allow you to hook into the batch processing of Batch life cycle. you can use listeners to perform actions before or after a step or a job.

Required Tools and Technologies

Below, the tools and Technologies we have used for this Spring Boot Batch Processing using Spring Data JPA to CSV File and also, we need to know How to create a Spring Boot Project with STS.

• Spring Tool Suite
• MySQL Workbench
• Spring Boot version 2.6.3
• Java 17

Project Structure:

This structure is only for learning purpose but in real time The Folder Structure is dependents on the Project. And we can observe there is no CSV File in that Folder Structure.

Add dependencies

Below we have provided the required dependencies for this project. Every dependency is used for unique purpose. For your reference the dependencies listed below.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-batch'
    implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'mysql:mysql-connector-java:8.0.23'
    compileOnly 'org.projectlombok:lombok'
    developmentOnly 'org.springframework.boot:spring-boot-devtools'
    annotationProcessor 'org.projectlombok:lombok'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testImplementation 'org.springframework.batch:spring-batch-test'
}

Note: We have already some data in the table.

Below we have the books table which has some data inside it.

Main Class

In this class, we have used this annotation @EnableBatchProcessing. This Annotation is used for Batch related functionality in the Spring Boot. Then only we can call different functions belongs to Batch Processing.

Java

package com.batch.app;

import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
@EnableBatchProcessing
public class BatchApplication {

    public static void main(String[] args)
    {
        SpringApplication.run(BatchApplication.class, args);
    }
}

Book Entity Class

Now, we will create one Entity class in the project Folder named as Book.

• In this class, we have defined some variables like id, author, name and price these are all attributes of Book.
• By using lombok dependency, we have created setters and getters methods for those attributes.

Below we have provided that code for better understanding.

Java

package com.batch.app;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@AllArgsConstructor
@NoArgsConstructor
@Entity
@Table(name = "book")
public class Book {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private int id;
    private String author;
    private String name;
    private String price;
}

BookEntityRepository

Here, we have created one interface named BookEntityRepository which is extends to JpaRepository.

• In this interface we have used @Repository and @EnableJpaRepositories.
• These annotations can enable the Database related functions in the background.
• This JpaRepository is take two different inputs as arguments named targeted entity class and the unique id Datatype.

Java

package com.batch.app;

import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.stereotype.Repository;

@Repository
@EnableJpaRepositories
public interface BookEntityRepository
    extends JpaRepository<Book, Integer> {
}

BookEntityItemProcessor

Now, we will create one class that is BookEntityItemProcessor which is used for processing the data based the business logic.

• In our case we just read data and write into CSV file that’s it.
• Here, we have used one built-in class that is ItemProcessor, it will take two arguments as input named Targeted entity class and other one is reference entity class.

Java

package com.batch.app;

import org.springframework.batch.item.ItemProcessor;
import org.springframework.stereotype.Component;

@Component
public class BookEntityItemProcessor
    implements ItemProcessor<Book, Book> {

    @Override
    public Book process(Book item) throws Exception
    {
        return item;
    }
}

BookEntityCsvWriter

Here, we have created one more Java class with named BookEntityCsvWriter for Handling getting the processed data and write into CSV file.

• The CSV file is dynamically created while data is available to write into file.
• The CSV file created with output.csv.
• This class is implemented to ItemWriter.
• This ItemWriter is take one argument as input that is Book.
• After that, we have created one constructor for reading data.
• Then we have written that into a CSV File.

Java

package com.batch.app;

import java.io.File;
import java.util.List;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor;
import org.springframework.batch.item.file.transform.DelimitedLineAggregator;
import org.springframework.core.io.FileSystemResource;

public class BookEntityCsvWriter
    implements ItemWriter<Book> {
    private static final String CSV_FILE = "output.csv";
    private FlatFileItemWriter<Book> writer;

    public BookEntityCsvWriter()
    {
        initializeCsvFile();
        this.writer = new FlatFileItemWriter<>();
        this.writer.setResource(
            new FileSystemResource(CSV_FILE));
        this.writer.setLineAggregator(
            new DelimitedLineAggregator<Book>() {
                {
                    setDelimiter(",");
                    setFieldExtractor(
                        new BeanWrapperFieldExtractor<
                            Book>() {
                            {
                                setNames(new String[] {
                                    "id", "author", "name",
                                    "price" });
                            }
                        });
                }
            });
    }

    private void initializeCsvFile()
    {
        File file = new File(CSV_FILE);
        if (!file.exists()) {
            try {
                file.createNewFile();
            }
            catch (Exception e) {
                throw new RuntimeException(
                    "Error creating CSV file", e);
            }
        }
    }

    public void write(List<? extends Book> items)
        throws Exception
    {

        writer.write(items);
    }
}

Batch Configuration

This is the required Java logic for handling entire Batch processing logic.

• In this class, we have used two different annotations those are @Configuration and @EnableBatchProcessing. This @Configuration is used for indicating that an object is a source of bean definitions.
• After this, we have used JobBuilderFactory and StepBuilderFactory.
• Here JobBuilderFactory is used for handling Job in the batching processing then the StepBuilderFactory is used for handling Steps in the Batch Programming.

Java

package com.batch.app;

import java.util.Properties;
import javax.persistence.EntityManagerFactory;
import javax.sql.DataSource;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.ItemProcessor;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.database.JpaPagingItemReader;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter;

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Autowired private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Bean
    public ItemReader<Book>
    reader(EntityManagerFactory entityManagerFactory)
    {
        JpaPagingItemReader<Book> reader
            = new JpaPagingItemReader<>();
        reader.setEntityManagerFactory(
            entityManagerFactory);
        reader.setQueryString(
            "SELECT b FROM Book b"); // Use the entity name
                                     // 'Book'
        reader.setPageSize(10);
        return reader;
    }

    @Bean public ItemProcessor<Book, Book> processor()
    {
        return new BookEntityItemProcessor();
    }

    @Bean public ItemWriter<Book> writer()
    {
        return new BookEntityCsvWriter();
    }

    @Bean public Job exportJob(Step exportStep)
    {
        return jobBuilderFactory.get("exportJob")
            .incrementer(new RunIdIncrementer())
            .flow(exportStep)
            .end()
            .build();
    }

    @Bean
    public Step
    exportStep(ItemReader<Book> reader,
               ItemProcessor<Book, Book> processor,
               ItemWriter<Book> writer)
    {
        return stepBuilderFactory.get("exportStep")
            .<Book, Book>chunk(10)
            .reader(reader)
            .processor(processor)
            .writer(writer)
            .build();
    }

    @Bean public EntityManagerFactory entityManagerFactory()
    {
        LocalContainerEntityManagerFactoryBean emf
            = new LocalContainerEntityManagerFactoryBean();
        emf.setDataSource(dataSource());
        emf.setPackagesToScan("com.batch.app");
        emf.setJpaVendorAdapter(
            new HibernateJpaVendorAdapter());
        emf.setJpaProperties(jpaProperties());
        emf.afterPropertiesSet();
        return emf.getObject();
    }

    @Bean public DataSource dataSource()
    {
        DriverManagerDataSource dataSource
            = new DriverManagerDataSource();
        dataSource.setDriverClassName(
            "com.mysql.cj.jdbc.Driver");
        dataSource.setUrl(
            "jdbc:mysql://localhost:3306/books");
        dataSource.setUsername("root");
        dataSource.setPassword("password");
        return dataSource;
    }

    @Bean public Properties jpaProperties()
    {
        Properties properties = new Properties();
        properties.setProperty(
            "hibernate.dialect",
            "org.hibernate.dialect.MySQLDialect");
        return properties;
    }
}

Output:

After running this project as Spring Boot Application, one CSV file is created. Then it will fetch data from database then write that data into that CSV File. Below we have provided the CSV file output.

CSV File Output: