中国科学院自动化研究所机构知识库

In this paper we address the problem of learning
multimodal word representations by integrating
textual, visual and auditory inputs.
Inspired by the re-constructive and associative
nature of human memory, we propose
a novel associative multichannel autoencoder
(AMA). Our model first learns the associations
between textual and perceptual modalities,
so as to predict the missing perceptual information
of concepts. Then the textual and
predicted perceptual representations are fused
through reconstructing their original and associated
embeddings. Using a gating mechanism
our model assigns different weights to each
modality according to the different concepts.
Results on six benchmark concept similarity
tests show that the proposed method significantly
outperforms strong unimodal baselines
and state-of-the-art multimodal models.