Knowledge Commons of Institute of Automation,CAS
NExT-OOD: Overcoming Dual Multiple-Choice VQA Biases | |
Zhang Xi(张熙)1,3![]() ![]() | |
发表期刊 | IEEE Transactions on Pattern Analysis and Machine Intelligence
![]() |
2023 | |
页码 | 1913-1931 |
摘要 | In recent years, multiple-choice Visual Question Answering (VQA) has become topical and achieved remarkable progress. However, most pioneer multiple-choice VQA models are heavily driven by statistical correlations in datasets, which cannot perform well on multimodal understanding and suffer from poor generalization. In this paper, we identify two kinds of spurious correlations, i.e., a Vision-Answer bias (VA bias) and a Question-Answer bias (QA bias). To systematically and scientifically study these biases, we construct a new video question answering (videoQA) benchmark NExT-OOD in OOD setting and propose a graph-based cross-sample method for bias reduction. Specifically, the NExT-OOD is designed to quantify models’ generalizability and measure their reasoning ability comprehensively. It contains three sub-datasets including NExT-OOD-VA, NExT-OOD-QA, and NExT-OOD-VQA, which are designed for the VA bias, QA bias, and VA&QA bias, respectively. We evaluate several existing multiple-choice VQA models on our NExT-OOD, and illustrate that their performance degrades significantly compared with the results obtained on the original multiple-choice VQA dataset. Besides, to mitigate the VA bias and QA bias, we explicitly consider the cross-sample information and design a contrastive graph matching loss in our approach, which provides adequate debiasing guidance from the perspective of whole dataset, and encourages the model to focus on multimodal contents instead of spurious statistical regularities. Extensive experimental results illustrate that our method significantly outperforms other bias reduction strategies, demonstrating the effectiveness and generalizability of the proposed approach. The proposed dataset is available at https://zhangxi1997.github.io. |
七大方向——子方向分类 | 多模态智能 |
国重实验室规划方向分类 | 多模态协同认知 |
是否有论文关联数据集需要存交 | 否 |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/58524 |
专题 | 多模态人工智能系统全国重点实验室_多媒体计算 |
作者单位 | 1.State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences 2.Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin University of Technology 3.School of Artificial Intelligence, University of Chinese Academy of Sciences 4.Peng Cheng Laboratory |
第一作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | Zhang Xi,Feifei Zhang,Changsheng Xu. NExT-OOD: Overcoming Dual Multiple-Choice VQA Biases[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2023:1913-1931. |
APA | Zhang Xi,Feifei Zhang,&Changsheng Xu.(2023).NExT-OOD: Overcoming Dual Multiple-Choice VQA Biases.IEEE Transactions on Pattern Analysis and Machine Intelligence,1913-1931. |
MLA | Zhang Xi,et al."NExT-OOD: Overcoming Dual Multiple-Choice VQA Biases".IEEE Transactions on Pattern Analysis and Machine Intelligence (2023):1913-1931. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
NExT-OOD- Overcoming(4719KB) | 期刊论文 | 作者接受稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论