DQ-MeeRKat: Automating Data Quality Monitoring with a Reference-Data-Profile-Annotated Knowledge Graph

Lisa Ehrlinger, Lisa Ehrlinger, Alexander Gindlhumer, Lisa-Marie Huber, Wolfram Wöß

2021

Abstract

High data quality (e.g., completeness, accuracy, non-redundancy) is essential to ensure the trustworthiness of AI applications. In such applications, huge amounts of data is integrated from different heterogeneous sources and complete, global domain knowledge is often not available. This scenario has a number of negative effects, in particular, it is difficult to monitor data quality centrally and manual data curation is not feasible. To overcome these problems, we developed DQ-MeeRKat, a data quality tool that implements a new method to automate data quality monitoring. DQ-MeeRKat uses a knowledge graph to represent a global, homogenized view of local data sources. This knowledge graph is annotated with reference data profiles, which serve as quasi-gold-standard to automatically verify the quality of modified data. We evaluated DQ-MeeRKat on six real-world data streams with qualitative feedback from the data owners. In contrast to existing data quality tools, DQ-MeeRKat does not require domain experts to define rules, but can be fully automated.

Paper Citation

in Harvard Style

Ehrlinger L., Gindlhumer A., Huber L. and Wöß W. (2021). DQ-MeeRKat: Automating Data Quality Monitoring with a Reference-Data-Profile-Annotated Knowledge Graph. In Proceedings of the 10th International Conference on Data Science, Technology and Applications - Volume 1: DATA, ISBN 978-989-758-521-0, pages 215-222. DOI: 10.5220/0010546202150222

in Bibtex Style

@conference{data21,
author={Lisa Ehrlinger and Alexander Gindlhumer and Lisa-Marie Huber and Wolfram Wöß},
title={DQ-MeeRKat: Automating Data Quality Monitoring with a Reference-Data-Profile-Annotated Knowledge Graph},
booktitle={Proceedings of the 10th International Conference on Data Science, Technology and Applications - Volume 1: DATA,},
year={2021},
pages={215-222},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010546202150222},
isbn={978-989-758-521-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Conference on Data Science, Technology and Applications - Volume 1: DATA,
TI - DQ-MeeRKat: Automating Data Quality Monitoring with a Reference-Data-Profile-Annotated Knowledge Graph
SN - 978-989-758-521-0
AU - Ehrlinger L.
AU - Gindlhumer A.
AU - Huber L.
AU - Wöß W.
PY - 2021
SP - 215
EP - 222
DO - 10.5220/0010546202150222