In the beginning times of computers and Internet, the data used was not as much of as it is today, The data then could be so easily stored and managed by all the users and business enterprises on a single computer, because the data never exceeded to the extent of 19 exabytes but now in this era, the data has increased about 2.5 quintillions per day.
Most of the data is generated from social media sites like Facebook, Instagram, Twitter, etc, and the other sources can be e-business, e-commerce transactions, hospital, school, bank data, etc. This data is impossible to manage by traditional data storing techniques. So Big-Data came into existence for handling the data which is big and impure.
Big Data is the field of collecting the large data sets from various sources like social media, GPS, sensors etc and analyzing them systematically and extract useful patterns using some tools and techniques by enterprises. Before analyzing and determining the data, the data architecture must be designed by the architect.
Data architecture Design and Data Management :
Data architecture design is set of standards which are composed of certain policies, rules, models and standards which manages, what type of data is collected, from where it is collected, the arrangement of collected data, storing that data, utilizing and securing the data into the systems and data warehouses for further analysis.
Data is one of the essential pillars of enterprise architecture through which it succeeds in the execution of business strategy.
Data architecture design is important for creating a vision of interactions occurring between data systems, like for example if data architect wants to implement data integration, so it will need interaction between two systems and by using data architecture the visionary model of data interaction during the process can be achieved.
Data architecture also describes the type of data structures applied to manage data and it provides an easy way for data preprocessing. The data architecture is formed by dividing into three essential models and then are combined :
-
Conceptual model –
It is a business model which uses Entity Relationship (ER) model for relation between entities and their attributes. -
Logical model –
It is a model where problems are represented in the form of logic such as rows and column of data, classes, xml tags and other DBMS techniques. -
Physical model –
Physical models holds the database design like which type of database technology will be suitable for architecture.
A data architect is responsible for all the design, creation, manage, deployment of data architecture and defines how data is to be stored and retrieved, other decisions are made by internal bodies.
Factors that influence Data Architecture :
Few influences that can have an effect on data architecture are business policies, business requirements, Technology used, economics, and data processing needs.
-
Business requirements –
These include factors such as the expansion of business, the performance of the system access, data management, transaction management, making use of raw data by converting them into image files and records, and then storing in data warehouses. Data warehouses are the main aspects of storing transactions in business. -
Business policies –
The policies are rules that are useful for describing the way of processing data. These policies are made by internal organizational bodies and other government agencies. -
Technology in use –
This includes using the example of previously completed data architecture design and also using existing licensed software purchases, database technology. -
Business economics –
The economical factors such as business growth and loss, interest rates, loans, condition of the market, and the overall cost will also have an effect on design architecture. -
Data processing needs –
These include factors such as mining of the data, large continuous transactions, database management, and other data preprocessing needs.
Data Management :
- Data management is the process of managing tasks like extracting data, storing data, transferring data, processing data, and then securing data with low-cost consumption.
- Main motive of data management is to manage and safeguard the people’s and organization data in an optimal way so that they can easily create, access, delete, and update the data.
- Because data management is an essential process in each and every enterprise growth, without which the policies and decisions can’t be made for business advancement. The better the data management the better productivity in business.
- Large volumes of data like big data are harder to manage traditionally so there must be the utilization of optimal technologies and tools for data management such as Hadoop, Scala, Tableau, AWS, etc. Which can further used for big data analysis in achieving improvements in patterns.
- Data management can be achieved by training the employees necessarily and maintenance by DBA, data analyst, and data architects.