Data Stream Management System Architecture

A Database Management System (DBMS) is a software system that is designed to manage and organize data in a structured manner. It allows users to create, modify, and query a database, as well as manage the security and access controls for that database.

What is DSMS Architecture ?

DSMS stands for data stream management system. It is nothing but a software application just like DBMS (database management system) but it involves processing and management of continuously flowing data stream rather than static data like excel or pdf or other files. It generally use for deal data stream from with various sources which included sensors data, social media field, financial report etc.

Just like DBMS, DSMS also provides the wide range of operations likes storage, processing, analyzing, integration also helps to generate the visualization and report only use for data streams.

There are wide range of DSMS application available in the market among them Apache flint, Apache Kafka, Apache storm, Amazon kinesis etc. DSMS process 2 types of queries standard queries and ad hoc queries.

Data stream Management system architecture

DSMS consists of various layer which are dedicated to perform particular operation which are as follows:

1. Data source Layer

The first layer of DSMS is data source layer as it name suggest it is comprises of all the data sources which includes sensors, social media feeds, financial market, stock markets etc. In the layer capturing and parsing of data stream happens. Basically it is the collection layer which collects the data.

2. Data Ingestion Layer

You can consider this layer as bridge between data source layer and processing layer. The main purpose of this layer is to handle the flow of data i.e., data flow control, data buffering and data routing.

3. Processing Layer

This layer consider as heart of DSMS architecture it is functional layer of DSMS applications. It process the data streams in real time. To perform processing it is uses processing engines like Apache flink or Apache storm etc., The main function of this layer is to filter, transform, aggregate and enriching the data stream. This can be done by derive insights and detect patterns.

4. Storage Layer

Once data is process we need to store the processed data in any storage unit. Storage layer consist of various storage like NoSQL database, distributed database etc., It helps to ensure data durability and availability of data in case of system failure.

5. Querying Layer

As mentioned above it support 2 types of query ad hoc query and standard query. This layer provides the tools which can be used for querying and analyzing the stored data stream. It also have SQL like query languages or programming API. This queries can be question like how many entries are done? which type of data is inserted? etc.,

6. Visualization and Reporting Layer

This layer provides tools for perform visualization like charts, pie chart, histogram etc., On the basis of this visual representation it also helps to generate the report for analysis.

7. Integration Layer

This layer responsible for integrating DSMS application with traditional system, business intelligence tools, data warehouses, ML application, NLP applications. It helps to improve already present running applications.

The layers are responsible for working of DSMS applications. It provides scalable and fault tolerance application which can handle huge volume of streaming data. These layer can change according to the business requirements some may include all layer some may exclude layers.