Look before You Leap! Designing a Human-centered AI System for Change Risk Assessment

Binay Gupta, Anirban Chatterjee, Subhadip Paul, Matha Harika, Lalitdutt Parsai, Kunal Banerjee, Vijay Agneeswaran

2022

Abstract

Reducing the number of failures in a production system is one of the most challenging problems in technology driven industries, such as, the online retail industry. To address this challenge, change management has emerged as a promising sub-field in operations that manages and reviews the changes to be deployed in production in a systematic manner. However, it is practically impossible to manually review a large number of changes on a daily basis and assess the risk associated with these. This warrants the development of an automated system to assess the risk associated with a large number of changes. There are a few commercial solutions available to address this problem but those solutions lack the ability to incorporate domain knowledge and continuous feedback from domain experts into the risk assessment process. As part of this work, we aim to bridge the gap between model-driven risk assessment of change requests and the assessment of domain experts by building a continuous feedback loop into the risk assessment process. Here we present our work to build an end-to-end machine learning system along with the discussion of some of the practical challenges we faced related to extreme skewness in class distribution, concept drift, estimation of the uncertainty associated with the model’s prediction and the overall scalability of the system.

Download


Paper Citation


in Harvard Style

Gupta B., Chatterjee A., Paul S., Harika M., Parsai L., Banerjee K. and Agneeswaran V. (2022). Look before You Leap! Designing a Human-centered AI System for Change Risk Assessment. In Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART, ISBN 978-989-758-547-0, pages 655-662. DOI: 10.5220/0010877500003116


in Bibtex Style

@conference{icaart22,
author={Binay Gupta and Anirban Chatterjee and Subhadip Paul and Matha Harika and Lalitdutt Parsai and Kunal Banerjee and Vijay Agneeswaran},
title={Look before You Leap! Designing a Human-centered AI System for Change Risk Assessment},
booktitle={Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,},
year={2022},
pages={655-662},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010877500003116},
isbn={978-989-758-547-0},
}


in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,
TI - Look before You Leap! Designing a Human-centered AI System for Change Risk Assessment
SN - 978-989-758-547-0
AU - Gupta B.
AU - Chatterjee A.
AU - Paul S.
AU - Harika M.
AU - Parsai L.
AU - Banerjee K.
AU - Agneeswaran V.
PY - 2022
SP - 655
EP - 662
DO - 10.5220/0010877500003116