Weakly Paired Multimodal Fusion for Object Recognition

doi:10.1109/TASE.2017.2692271

CASIA OpenIR > 多模态人工智能系统全国重点实验室 > 机器人理论与应用

	Weakly Paired Multimodal Fusion for Object Recognition
	Liu, Huaping 1,2,3; Wu, Yupei 1,2,3; Sun, Fuchun 1,2,3; Fang, Bin 1,2,3; Guo, Di 1,2,3
发表期刊	IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING
ISSN	1545-5955
	2018-04-01
卷号	15 期号:2 页码:784-795
通讯作者	Liu, Huaping(hpliu@tsinghua.edu.cn)
摘要	The ever-growing development of sensor technology has led to the use of multimodal sensors to develop robotics and automation systems. It is therefore highly expected to develop methodologies capable of integrating information from multimodal sensors with the goal of improving the performance of surveillance, diagnosis, prediction, and so on. However, real multimodal data often suffer from significant weak-pairing characteristics, i.e., the full pairing between data samples may not be known, while pairing of a group of samples from one modality to a group of samples in another modality is known. In this paper, we establish a novel projective dictionary learning framework for weakly paired multimodal data fusion. By introducing a latent pairing matrix, we realize the simultaneous dictionary learning and the pairing matrix estimation, and therefore improve the fusion effect. In addition, the kernelized version and the optimization algorithms are also addressed. Extensive experimental validations on some existing data sets are performed to show the advantages of the proposed method. Note to Practitioners-In many industrial environments, we usually use multiple heterogeneous sensors, which provide multimodal information. Such multimodal data usually lead to two technical challenges. First, different sensors may provide different patterns of data. Second, the full-pairing information between modalities may not be known. In this paper, we develop a unified model to tackle such problems. This model is based on a projective dictionary learning method, which efficiently produces the representation vector for the original data by an explicit form. In addition, the latent pairing relation between samples can be learned automatically and be used to improve the classification performance. Such a method can be flexibly used for multimodal fusion with full-pairing, partial-pairing and weak-pairing cases.
关键词	Intelligent robot system manipulation and grasping multimodal data projective dictionary learning weakly paired data
DOI	10.1109/TASE.2017.2692271
关键词[WOS]	CLASSIFICATION
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[U1613212] ; National Natural Science Foundation of China[61673238] ; National Natural Science Foundation of China[91420302] ; National Natural Science Foundation of China[61327809] ; National High-Tech Research and Development Plan[2015AA042306] ; National Science and Technology Pillar Program[2015BAK12B03] ; National Natural Science Foundation of China[U1613212] ; National Natural Science Foundation of China[61673238] ; National Natural Science Foundation of China[91420302] ; National Natural Science Foundation of China[61327809] ; National High-Tech Research and Development Plan[2015AA042306] ; National Science and Technology Pillar Program[2015BAK12B03]
项目资助者	National Natural Science Foundation of China ; National High-Tech Research and Development Plan ; National Science and Technology Pillar Program
WOS研究方向	Automation & Control Systems
WOS类目	Automation & Control Systems
WOS记录号	WOS:000429217900030
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：69[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/28251
专题	多模态人工智能系统全国重点实验室_机器人理论与应用
通讯作者	Liu, Huaping
作者单位	1.Tsinghua Univ, Dept Comp Sci & Technol, Beijing 100084, Peoples R China 2.Tsinghua Univ, Tsinghua Natl Lab Informat Sci & Technol, State Key Lab Intelligent Technol & Syst, Beijing 100084, Peoples R China 3.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100190, Peoples R China
第一作者单位	中国科学院自动化研究所
通讯作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Liu, Huaping,Wu, Yupei,Sun, Fuchun,et al. Weakly Paired Multimodal Fusion for Object Recognition[J]. IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING,2018,15(2):784-795.
APA	Liu, Huaping,Wu, Yupei,Sun, Fuchun,Fang, Bin,&Guo, Di.(2018).Weakly Paired Multimodal Fusion for Object Recognition.IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING,15(2),784-795.
MLA	Liu, Huaping,et al."Weakly Paired Multimodal Fusion for Object Recognition".IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING 15.2(2018):784-795.