Object-less Vision-language Model on Visual Question Classification for Blind People

Tung Le, Khoa Pho, Thong Bui, Thong Bui, Huy Tien Nguyen, Huy Tien Nguyen, Minh Le Nguyen

2022

Abstract

Despite the long-standing appearance of question types in the Visual Question Answering dataset, Visual Question Classification does not received enough public interest in research. Different from general text classification, a visual question requires an understanding of visual and textual features simultaneously. Together with the enthusiasm and novelty of Visual Question Classification, the most important and practical goal we concentrate on is to deal with the weakness of Object Detection on object-less images. We thus propose an Object-less Visual Question Classification model, OL–LXMERT, to generate virtual objects replacing the dependence of Object Detection in previous Vision-Language systems. Our architecture is effective and powerful enough to digest local and global features of images in understanding the relationship between multiple modalities. Through our experiments in our modified VizWiz-VQC 2020 dataset of blind people, our Object-less LXMERT achieves promising results in the brand-new multi-modal task. Furthermore, the detailed ablation studies show the strength and potential of our model in comparison to competitive approaches.

Paper Citation

in Harvard Style

Le T., Pho K., Bui T., Nguyen H. and Nguyen M. (2022). Object-less Vision-language Model on Visual Question Classification for Blind People. In Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART, ISBN 978-989-758-547-0, pages 180-187. DOI: 10.5220/0010797400003116

in Bibtex Style

@conference{icaart22,
author={Tung Le and Khoa Pho and Thong Bui and Huy Tien Nguyen and Minh Le Nguyen},
title={Object-less Vision-language Model on Visual Question Classification for Blind People},
booktitle={Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,},
year={2022},
pages={180-187},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010797400003116},
isbn={978-989-758-547-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,
TI - Object-less Vision-language Model on Visual Question Classification for Blind People
SN - 978-989-758-547-0
AU - Le T.
AU - Pho K.
AU - Bui T.
AU - Nguyen H.
AU - Nguyen M.
PY - 2022
SP - 180
EP - 187
DO - 10.5220/0010797400003116