ATTENUATING THE EFFECT OF DATA ABNORMALITIES ON DATA WAREHOUSES

Anália Lourenço, Orlando Belo

2004

Abstract

Today’s informational entanglement makes it crucial to enforce adequate management systems. Data warehousing systems appeared with the specific mission of providing adequate contents for data analysis, ensuring gathering, processing and maintenance of all data elements thought valuable. Data analysis in general, data mining and on-line analytical processing facilities, in particular, can achieve better, sharper results, because data quality is finally taken into account. The available elements must be submitted to an intensive processing before being able to integrate them into the data warehouse. Each data warehousing system embraces extraction, transformation and loading processes which are in charge of all the processing concerning the data preparation towards its integration into the data warehouse. Usually, data is scoped at several stages, inspecting data and schema issues and filtering all those elements that do not comply with the established rules. This paper proposes an agent-based platform, which not only ensures the traditional data flow, but also tries to recover the filtered data when an data error occurs. It is intended to perform the process of error monitoring and control automatically. Bad data is processed and eventually repaired by the agents, integrating it again into the data warehouse’s regular flow. All data processing efforts are registered and afterwards mined in order to establish data error patterns. The obtained results will enrich the wrappers knowledge about abnormal situations’ resolution. Eventually, this evolving will enhance the data warehouse population process, enlarging the integrated volume of data and enriching its actual quality and consistency.

Paper Citation

in Harvard Style

Lourenço A. and Belo O. (2004). ATTENUATING THE EFFECT OF DATA ABNORMALITIES ON DATA WAREHOUSES . In Proceedings of the Sixth International Conference on Enterprise Information Systems - Volume 1: ICEIS, ISBN 972-8865-00-7, pages 411-415. DOI: 10.5220/0002629604110415

in Bibtex Style

@conference{iceis04,
author={Anália Lourenço and Orlando Belo},
title={ATTENUATING THE EFFECT OF DATA ABNORMALITIES ON DATA WAREHOUSES},
booktitle={Proceedings of the Sixth International Conference on Enterprise Information Systems - Volume 1: ICEIS,},
year={2004},
pages={411-415},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0002629604110415},
isbn={972-8865-00-7},
}

in EndNote Style

TY - CONF
JO - Proceedings of the Sixth International Conference on Enterprise Information Systems - Volume 1: ICEIS,
TI - ATTENUATING THE EFFECT OF DATA ABNORMALITIES ON DATA WAREHOUSES
SN - 972-8865-00-7
AU - Lourenço A.
AU - Belo O.
PY - 2004
SP - 411
EP - 415
DO - 10.5220/0002629604110415