History of Data Warehousing

The data warehouse is a core repository that performs aggregation to collect and group data from various sources into a central integrated unit. The data from the warehouse can be retrieved and analyzed to generate reports or relations between the datasets of the database which enhances the growth of many industries. Data warehouse comes under Business Intelligence. The data warehouse is designed to provide real-time information. Storage of data has evolved from simple magnetic tapes to integrated data warehouses. This article will give an overview of the history of warehousing.

Early mechanisms to store data:

Early methods to hold data started with punched cards, paper tapes. Then the development of magnetic tapes took place. Though we can write and rewrite data in magnetic tapes, it is not a stable medium to hold data. Disk storage came into existence where you can store and access large amounts of data.

DBMS in disk storage:

Later DBMS (Database Management Systems) was integrated with disk storage to store the data directly on the disk itself. The main advantage of integrating DBMS is we can locate the data fast. The features include location and deletion of data, solving problems when two different data are mapped to the same location. The physical location can be extended when the data exceeds the storage limit.

Online Applications:

There came the advent of online applications after the usage of DBMS in disk storage. Online applications are the products of online processing which has its applications in the commercial industry. For e.g. Retail and sales processing, ticket reservation systems, Automated Teller processing, etc. Online applications play an important role in the current years due to their intertwined applications. But it has a drawback which the end-users of the application put forward. Since there is an enormous amount of data, the end-users find it difficult to retain the desired data. Even if they obtained it, they are not sure whether it is correct or accurate due to the constant escalation of data.

Fourth Generation Technologies (4GL) and Personal computers:

The motive of 4GL technology is to provide end-users the direct opportunity of accessing data, using the programming languages and system development without the interference of the IT department. The same happens with personal computers. So, the individuals can bring their own personalized systems into the business firm and can access the specific data accessible to them. This reduced the need for a centralized department of technology to provide the requested data to the users. Spreadsheets are a good example. But it has its drawbacks. The data retrieved may be incomplete, misleading, or wrong. It lacks finesse in the end result due to the lack of documentation and the existence of multiple versions of the same data.

Spider web environment:

The Spider web environment ended up as a dilemma to end-users, IT professionals due to its unfavorable nature and complexity. This environment is called spider web environment because there are many lines connecting which reminded the lines of a spider web. Though data can be retrieved, the efficiency and accuracy are very less. These severe drawbacks called the need for building information architecture centering data warehouse.

Evolution of Data warehouse environment:

As the corporation shifted from spider web to data warehouse environment, it created a major change in the usual techniques in which data is stored. Before the introduction of data warehouse, it was thought that a database must aid all the purposes of data. After the advent of data warehouse, it is evident that there are different types of databases that serve for different purposes. 

A data warehouse is a place where information is processed into bot integrated and granular forms of data and history. Though not all the warehouses are integrated, integrated data warehouses has its benefit of providing the enterprise view of a company. The granular data has the benefit of looking same data in different ways. A set of data can be looked in marketing way or it can be looked in finance way. The same data can be used to look in accounting way too. Data warehouses are used to store historical data of many years. 

Challenges of data warehouse:

• First is integration of data which is the most difficult and time consuming process as one need to touch the root of old legacies of corporates to derive useful integrated data. It is a painful step, but it is worthful.
• High volume of data created by data warehousing techniques which makes the process tedious. So, there comes a need to get rid of old data. But, for analyses of data they are so valuable and can’t be ignored.
• Data warehouses can’t be created all at once like other operational applications. It must be developed iteratively, like one step at a time.

Reasons for the development of Data Warehouse 2.0 Environment (DW 2.0):

The earlier techniques have evolved so much and ended as DW 2.0. We need to travel back and forth to understand the forces that initiated the architecture of DW 2.0. Some of them are given below.

• The end-user’s demand for new system or architecture.
• Financially economic
• Online processing techniques
• High storage capacity
• Need for integrated data
• The need to included unstructured data for analytics purpose in the data mixture.

Data Warehouse evolution (from business perspective):

• The output of the earlier techniques is in unrefined format. For e.g. It is a hectic process to read all those hexadecimal inputs just to find a small piece of information from the cryptic codes.
• Now, the end-users have become more futuristic. So, they demand the need for more sophisticated output and instantaneous source of output.
• For online processing techniques to be done, the data need to be integrated. Also, it needs historical data for analysis.
• First generation data warehouse came into existence due to the end-user’s thirst for corporate data.

Mutated forms of data warehouse:

Due to the appealing features of data warehouse, the business consultants have mutated the concept of data warehouse in accordance to their corporate needs. Some variations of the data warehouses are:

• The Active data warehouse: The online processing and updates are taken place in this warehouse. The major feature of this warehouse is that the transaction has a very high performance rate. The demerits of this mutated warehouse are the uprightness of the transaction is questioned, hefty statistical processing, large capacities are wasted which in turn increases the operational cost.
• The Federated data warehouse: In this type of approach, due to the high complexity in the integration of data, they skip this process. Technically, a warehouse doesn’t exist in this approach. The scheme behind that is to construct a data warehouse magically by merging the old legacies of the corporate to fetch and process data simultaneously. This approach seems attractive with less work but it is just a delusion rather than a solution itself.  It has numerous pitfalls like bad performance, limited history, absence of data integration, complexity, inherited granularity which provides poor performance to the end-user when he requests data from different level of granularity from the federal warehouse.
• The Star schema data warehouse: The outlook used in this data warehousing needs the construction of dimension tables and fact tables. It provides lot of benefits as a data warehouse but has its limitations. It is designed only for limited requirements and when the requirements change, the data warehouse becomes brittle. The level of granularity keeps changing due to multiple schema formation which questions the integrity of data. It cannot be extended more than a certain limit and it is designed only for one-audience type.
• Data Mart data warehouse: The consultants of the online application processing first build a data mart which gives the chance to know the sales of the product without any complications of building an actual data warehouse. The demerits include non- extensibility, high error occurrences, reconciliation of data is not possible and extract proliferation which makes extraction of legacy data difficult. Another fact about this approach is there is no way a data mart can be converted into a data warehouse. It’s like the core of each is different and they can’t be mutated to change into warehouse.