AnANet: Association and Alignment Network for Modeling Implicit Relevance in Cross-Modal Correlation Classification

doi:10.1109/TMM.2022.3229960

	AnANet: Association and Alignment Network for Modeling Implicit Relevance in Cross-Modal Correlation Classification
	Xu, Nan1,2 ; Wang, Junyan 2; Tian, Yuan1,3 ; Zhang, Ruike 1,3; Mao, Wenji1,3
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
	2023
卷号	25 页码:7867-7880
通讯作者	Mao, Wenji(wenji.mao@ia.ac.cn)
摘要	With the explosive increase of multimodal data, cross-modal correlation classification has become an important research topic and is in great demand in many cross-modal applications. A variety of classification schemes and predictive models have been built based on the existing cross-modal correlation categorization. However, these classification schemes typically follow the prior assumption that the paired cross-modal samples are strictly related, and thus pay great attention to the fine-grained relevant types of cross-modal correlation, ignoring the high volume of implicitly relevant data which are often wrongly classified into irrelevant types. Even more, previous predictive models fall short of reflecting the essence of cross-modal correlation according to their definitions, especially in the modeling of network structure. Thus in this paper, by comprehensively investigating the current image-text correlation classification research, we redefine a new classification scheme for cross-modal correlation based on the implicit and explicit relevance. To predict the types of image-text correlation based on our proposed definition, we further devise the Association and Alignment Network (namely AnANet) to model the implicit and explicit relevance, which captures both the implicit association of global discrepancy and commonality between image and text and explicit alignment of cross-modal local relevance. Experimental studies on our constructed new image-text correlation dataset verify the effectiveness of our proposed model.
关键词	Association and alignment network classification scheme cross-modal correlation implicit relevance
DOI	10.1109/TMM.2022.3229960
收录类别	SCI
语种	英语
资助项目	Ministry of Science and Technology of China[2020AAA0108405] ; National Natural Science Foundation of China[62206287] ; National Natural Science Foundation of China[11832001] ; National Natural Science Foundation of China[71621002]
项目资助者	Ministry of Science and Technology of China ; National Natural Science Foundation of China
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:001121212400020
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/55435
专题	多模态人工智能系统全国重点实验室
通讯作者	Mao, Wenji
作者单位	1.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China 2.Beijing Wenge Technol Co Ltd, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 101408, Peoples R China
第一作者单位	中国科学院自动化研究所
通讯作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Xu, Nan,Wang, Junyan,Tian, Yuan,et al. AnANet: Association and Alignment Network for Modeling Implicit Relevance in Cross-Modal Correlation Classification[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2023,25:7867-7880.
APA	Xu, Nan,Wang, Junyan,Tian, Yuan,Zhang, Ruike,&Mao, Wenji.(2023).AnANet: Association and Alignment Network for Modeling Implicit Relevance in Cross-Modal Correlation Classification.IEEE TRANSACTIONS ON MULTIMEDIA,25,7867-7880.
MLA	Xu, Nan,et al."AnANet: Association and Alignment Network for Modeling Implicit Relevance in Cross-Modal Correlation Classification".IEEE TRANSACTIONS ON MULTIMEDIA 25(2023):7867-7880.