Towards Human-Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework

doi:10.1109/TCSS.2022.3199819

CASIA OpenIR > 多模态人工智能系统全国重点实验室 > 互联网大数据与信息安全

	Towards Human-Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework
	Zhang, Xingwei 1,2; Zheng, Xiaolong1,2 ; Liu, Bin3 ; Wang, Xiao1,2 ; Mao, Wenji1,2 ; Zeng, Daniel Dajun1,2 ; Wang, Fei-Yue1,2
发表期刊	IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS
ISSN	2329-924X
	2022-08-29
页码	13
通讯作者	Zheng, Xiaolong(xiaolong.zheng@ia.ac.cn)
摘要	The multimodality nature of web data has necessitated complex multimodal information retrieval for a wide range of web applications. Deep neural networks (DNNs) have been widely employed to extract semantic features from raw samples to improve retrieval accuracy. In addition, hashing is widely used to improve computational and storage efficiency. As such, deep hashing frameworks have been applied for multimodal retrieval tasks. However, there is still a great recognitive gap between primate brain structure-inspired DNNs and humans. On computer vision tasks, well-crafted DNN models can be easily defeated by invisible small attacks, and this phenomenon indicates a large recognition gap between DNN models and humans. Recently, adversarial defense methods have been shown to improve the human-machine recognition alignment in several classification tasks. However, the robustness problem on the retrieval tasks, especially on the deep hashing-based multimodal retrieval models, is still not well studied. Therefore, in this article, we present an adversarially robust training mechanism to improve model robustness for the purpose of human-machine recognition alignment on retrieval tasks. Through extensive experimental results on several social multimodal retrieval benchmarks, we show that the robust training hashing framework proposed can mitigate the recognition gap on retrieval tasks. Our study highlights the necessity of robustness enhancement on deep hashing models.
关键词	Training Task analysis Semantics Perturbation methods Feature extraction Computational modeling Robustness Adversarial perturbation adversarially robust training deep hashing multimodal retrieval
DOI	10.1109/TCSS.2022.3199819
关键词[WOS]	NEURAL-NETWORKS ; MODELS
收录类别	SCI
语种	英语
资助项目	Ministry of Science and Technology of China[2020AAA0108401] ; Natural Science Foundation of China[72225011] ; Natural Science Foundation of China[71621002]
项目资助者	Ministry of Science and Technology of China ; Natural Science Foundation of China
WOS研究方向	Computer Science
WOS类目	Computer Science, Cybernetics ; Computer Science, Information Systems
WOS记录号	WOS:000849234000001
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：2[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/50057
专题	多模态人工智能系统全国重点实验室_互联网大数据与信息安全多模态人工智能系统全国重点实验室_平行智能技术与系统团队
通讯作者	Zheng, Xiaolong
作者单位	1.Chinese Acad Sci, State Key Lab Management & Control Complex Syst, Inst Automat, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 101408, Peoples R China 3.West Virginia Univ, Dept Management Informat Syst, Morgantown, WV 26506 USA
第一作者单位	中国科学院自动化研究所
通讯作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Zhang, Xingwei,Zheng, Xiaolong,Liu, Bin,et al. Towards Human-Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework[J]. IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS,2022:13.
APA	Zhang, Xingwei.,Zheng, Xiaolong.,Liu, Bin.,Wang, Xiao.,Mao, Wenji.,...&Wang, Fei-Yue.(2022).Towards Human-Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework.IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS,13.
MLA	Zhang, Xingwei,et al."Towards Human-Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework".IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS (2022):13.