Knowledge Commons of Institute of Automation,CAS
PAN: Prototype-based Adaptive Network for Robust Cross-Modal Retrieval | |
Zhixiong Zeng1,2; Shuai Wang1,2; Nan Xu1,2; Wenji Mao1,2 | |
2021-07 | |
会议名称 | Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval |
会议日期 | 2021.7.11 |
会议地点 | Virtual Event |
摘要 | In practical applications of cross-modal retrieval, test queries of the retrieval system may vary greatly and come from unknown category. Meanwhile, due to the cost and difficulty of data collection as well as other issues, the available data for cross-modal retrieval are often imbalanced over different modalities. In this paper, we address two important issues to increase the robustness of cross-modal retrieval system for real-world applications: handling test queries from unknown category and modality-imbalanced training data. The first issue has not been addressed by existing methods and the second issue was not well addressed in the related research. To tackle the above issues, we take the advantage of prototype learning, and propose a prototype-based adaptive network (PAN) for robust cross-modal retrieval. Our method leverages a unified prototype to represent each semantic category across modalities, which provides discriminative information of different categories and takes unified prototypes as anchors to learn cross-modal representations adaptively. Moreover, we propose a novel prototype propagation strategy to reconstruct balanced representations which preserves the semantic consistency and modality heterogeneity. Experimental results on the benchmark datasets demonstrate the effectiveness of our method compared to the SOTA methods, and further robustness tests show the superiority of our method in solving the above issues. |
是否为代表性论文 | 是 |
七大方向——子方向分类 | 多模态智能 |
国重实验室规划方向分类 | 其他 |
是否有论文关联数据集需要存交 | 否 |
文献类型 | 会议论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/52011 |
专题 | 多模态人工智能系统全国重点实验室_互联网大数据与信息安全 |
通讯作者 | Wenji Mao |
作者单位 | 1.Institute of Automation, Chinese Academy of Sciences 2.School of Artifcial Intelligence, University of Chinese Academy of Sciences |
第一作者单位 | 中国科学院自动化研究所 |
通讯作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | Zhixiong Zeng,Shuai Wang,Nan Xu,et al. PAN: Prototype-based Adaptive Network for Robust Cross-Modal Retrieval[C],2021. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
PAN.pdf(1417KB) | 会议论文 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论