Methodologies for Implementing Real Time Data Warehouses

Authors

Abstract

A data warehouse (DW) provides information for analytical processing, decision making and data mining tools. This information is updated periodically from transactional systems. Traditional DW systems have static structures of their data schemas and relationships, and therefore are not prepared to support the dynamics of real-time live data processing. As the concept of real time enterprise evolves, the synchronism between transactional data and DW, statically implemented, has been reviewed. For these purposes, Real-Time Data Warehouses (RTDW) seem to be very promising. This paper presents methodological indications for implementing RTDW, in which transactional data sources are available through common standard database access, allowing to minimize the time needed to accomplish extraction, transformation and loading (ETL) processes of that data, as well as its loading into the DW. The main method presented consists on using structural replicas of all fact tables without primary keys or index files, adapting those replica’s data structures for housing, in real-time, all insert, edit and delete operations (data transactions) that occur within operational systems databases. This is accomplished using only append record operations towards those fact table replicas, allowing to minimize processing time, record locking and concurrency data access problems, both in transactional systems and the DW. Concurrently, this allows maintaining DW availability and keeping OLAP tools functioning properly, providing the most recent business data.

Keywords

Real-Time Data Warehousing, ETL

Subject

Data Warehousing

Conference

Data Gadgets 2004 - Bringing Up Emerging Solutions for Data Warehousing Systems, November 2004

PDF File