MsIFT: Multi-Source Image Fusion Transformer
Zhang, Xin1,2; Jiang, Hangzhi1,2; Xu, Nuo1,2; Ni, Lei3; Huo, Chunlei1,2; Pan, Chunhong1,2
Source PublicationREMOTE SENSING
2022-08-01
Volume14Issue:16Pages:19
Abstract

Multi-source image fusion is very important for improving image representation ability since its essence relies on the complementarity between multi-source information. However, feature-level image fusion methods based on the convolution neural network are impacted by the spatial misalignment between image pairs, which leads to the semantic bias in merging features and destroys the representation ability of the region-of-interests. In this paper, a novel multi-source image fusion transformer (MsIFT) is proposed. Due to the inherent global attention mechanism of the transformer, the MsIFT has non-local fusion receptive fields, and it is more robust to spatial misalignment. Furthermore, multiple classification-based downstream tasks (e.g., pixel-wise classification, image-wise classification and semantic segmentation) are unified in the proposed MsIFT framework, and the fusion module architecture is shared by different tasks. The MsIFT achieved state-of-the-art performances on the image-wise classification dataset VAIS, semantic segmentation dataset SpaceNet 6 and pixel-wise classification dataset GRSS-DFC-2013. The code and trained model are being released upon the publication of the work.

Keywordtransformer multi-source image fusion non-local
DOI10.3390/rs14164062
WOS KeywordSHIP CLASSIFICATION ; LIDAR
Indexed BySCI
Language英语
Funding ProjectNational Natural Science Foundation of China[62071466] ; Fund of National Key Laboratory of Science and Technology on Remote Sensing Information and Imagery Analysis, Beijing Research Institute of Uranium Geology[6142A010402] ; Guangxi Natural Science Foundation[2018GXNSFBA281086]
Funding OrganizationNational Natural Science Foundation of China ; Fund of National Key Laboratory of Science and Technology on Remote Sensing Information and Imagery Analysis, Beijing Research Institute of Uranium Geology ; Guangxi Natural Science Foundation
WOS Research AreaEnvironmental Sciences & Ecology ; Geology ; Remote Sensing ; Imaging Science & Photographic Technology
WOS SubjectEnvironmental Sciences ; Geosciences, Multidisciplinary ; Remote Sensing ; Imaging Science & Photographic Technology
WOS IDWOS:000845420000001
PublisherMDPI
Sub direction classification目标检测、跟踪与识别
planning direction of the national heavy laboratory视觉信息处理
Citation statistics
Cited Times:5[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/50033
Collection多模态人工智能系统全国重点实验室_先进时空数据分析与学习
Corresponding AuthorHuo, Chunlei
Affiliation1.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 101408, Peoples R China
3.Beijing Inst Remote Sensing, Beijing 100085, Peoples R China
First Author AffilicationChinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Corresponding Author AffilicationChinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Recommended Citation
GB/T 7714
Zhang, Xin,Jiang, Hangzhi,Xu, Nuo,et al. MsIFT: Multi-Source Image Fusion Transformer[J]. REMOTE SENSING,2022,14(16):19.
APA Zhang, Xin,Jiang, Hangzhi,Xu, Nuo,Ni, Lei,Huo, Chunlei,&Pan, Chunhong.(2022).MsIFT: Multi-Source Image Fusion Transformer.REMOTE SENSING,14(16),19.
MLA Zhang, Xin,et al."MsIFT: Multi-Source Image Fusion Transformer".REMOTE SENSING 14.16(2022):19.
Files in This Item: Download All
File Name/Size DocType Version Access License
remotesensing-14-040(4788KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Zhang, Xin]'s Articles
[Jiang, Hangzhi]'s Articles
[Xu, Nuo]'s Articles
Baidu academic
Similar articles in Baidu academic
[Zhang, Xin]'s Articles
[Jiang, Hangzhi]'s Articles
[Xu, Nuo]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Zhang, Xin]'s Articles
[Jiang, Hangzhi]'s Articles
[Xu, Nuo]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: remotesensing-14-04062-v2.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.