Knowledge Commons of Institute of Automation,CAS
Image De-occlusion via Event-enhanced Multi-modal Fusion Hybrid Network | |
Si-Qi Li1,2,3,4![]() | |
发表期刊 | Machine Intelligence Research
![]() |
ISSN | 2731-538X |
2022 | |
卷号 | 19期号:4页码:307-318 |
摘要 | Seeing through dense occlusions and reconstructing scene images is an important but challenging task. Traditional framebased image de-occlusion methods may lead to fatal errors when facing extremely dense occlusions due to the lack of valid information available from the limited input occluded frames. Event cameras are bio-inspired vision sensors that record the brightness changes at each pixel asynchronously with high temporal resolution. However, synthesizing images solely from event streams is ill-posed since only the brightness changes are recorded in the event stream, and the initial brightness is unknown. In this paper, we propose an event-enhanced multi-modal fusion hybrid network for image de-occlusion, which uses event streams to provide complete scene information and frames to provide color and texture information. An event stream encoder based on the spiking neural network (SNN) is proposed to encode and denoise the event stream efficiently. A comparison loss is proposed to generate clearer results. Experimental results on a largescale event-based and frame-based image de-occlusion dataset demonstrate that our proposed method achieves state-of-the-art performance. |
关键词 | Event camera multi-modal fusion image de-occlusion spiking neural network (SNN) image reconstruction |
DOI | 10.1007/s11633-022-1350-3 |
七大方向——子方向分类 | 其他 |
国重实验室规划方向分类 | 其他 |
是否有论文关联数据集需要存交 | 否 |
中文导读 | https://mp.weixin.qq.com/s/dLI7nr0c448SGrZZ0JqoHw |
视频解析 | https://www.bilibili.com/video/BV1gK411d7FP/ |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/55947 |
专题 | 学术期刊_Machine Intelligence Research |
作者单位 | 1.Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China 2.Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China 3.Beijing Laboratory of Brain and Cognitive Intelligence, Beijing Municipal Education Commission, Tsinghua University, Beijing 100084, China 4.Key Laboratory for Information System Security, School of Software, Tsinghua University, Beijing 100084, China 5.Department of Automation, Tsinghua University, Beijing 100084, China |
推荐引用方式 GB/T 7714 | Si-Qi Li,Yue Gao,Qiong-Hai Dai. Image De-occlusion via Event-enhanced Multi-modal Fusion Hybrid Network[J]. Machine Intelligence Research,2022,19(4):307-318. |
APA | Si-Qi Li,Yue Gao,&Qiong-Hai Dai.(2022).Image De-occlusion via Event-enhanced Multi-modal Fusion Hybrid Network.Machine Intelligence Research,19(4),307-318. |
MLA | Si-Qi Li,et al."Image De-occlusion via Event-enhanced Multi-modal Fusion Hybrid Network".Machine Intelligence Research 19.4(2022):307-318. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
MIR-2022-05-144.pdf(4505KB) | 期刊论文 | 出版稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论