Difference between Batch Processing and Stream Processing
Prerequisite – Types of Operating Systems
1. Batch Processing :
Batch processing refers to processing of high volume of data in batch within a specific time span. It processes large volume of data all at once. Batch processing is used when data size is known and finite. It takes little longer time to processes data. It requires dedicated staffs to handle issues. Batch processor processes processes data in multiple passes. When data is collected overtime and similar data batched/grouped together then in that case batch processing is used.
Challenges with Batch processing :
- Debugging of these system is difficult as it requires dedicated professional to fix the error.
- Software and training requires high expenses initially just to understand batch scheduling, triggering, notification etc.
2. Stream Processing :
Stream processing refers to processing of continuous stream of data immediately as it is produced. It analyzes streaming data in real time. Stream processing is used when the data size is unknown and infinite and continuous. It takes few seconds or milliseconds to process data. In stream processing data output rate is as fast as data input rate. Stream processor processes data in few passes. When data stream is continuous and requires immediate response then in that case stream processing is used.
Challenges with Stream processing :
- Data input rate and output rate sometimes creates a problem.
- Cope with huge amount of data and immediate response.
Difference between Batch Processing and Stream processing :
|S.No.||BATCH PROCESSING||STREAM PROCESSING|
|01.||Batch processing refers to processing of high volume of data in batch within a specific time span.||Stream processing refers to processing of continuous stream of data immediately as it is produced.|
|02.||Batch processing processes large volume of data all at once.||Stream processing analyzes streaming data in real time.|
|04.||In Batch processing data size is known and finite.||In Stream processing data size is unknown and infinite in advance.|
|05.||In Batch processing the data is processes in multiple passes.||In stream processing generally data is processed in few passes.|
|06.||Batch processor takes longer time to processes data.||Stream processor takes few seconds or milliseconds to process data.|
|07.||In batch processing the input graph is static.||In stream processing the input graph is dynamic.|
|08.||In this processing the data is analyzed on a snapshot.||In this processing the data is analyzed on continuous.|
|09.||In batch processing the response is provided after job completion.||In stream processing the response is provided immediately.|
|10.||Examples are distributed programming platforms like MapReduce, Spark, GraphX etc.||Examples are programming platforms like spark streaming and S4 (Simple Scalable Streaming System) etc.|
|11.||Batch processing is used in payroll and billing system, food processing system etc.||Stream processing is used in stock market, e-commerce transactions, social media etc.|