Knowledge Commons of Institute of Automation,CAS
TextFormer: A Query-based End-to-end Text Spotter with Mixed Supervision | |
Yukun Zhai1; Xiaoqiang Zhang2; Xiameng Qin2; Sanyuan Zhao1; Xingping Dong1; Jianbing Shen1 | |
发表期刊 | Machine Intelligence Research
![]() |
ISSN | 2731-538X |
2024 | |
卷号 | 21期号:4页码:704-717 |
摘要 | End-to-end text spotting is a vital computer vision task that aims to integrate scene text detection and recognition into a unified framework. Typical methods heavily rely on region-of-interest (RoI) operations to extract local features and complex post-processing steps to produce final predictions. To address these limitations, we propose TextFormer, a query-based end-to-end text spotter with a transformer architecture. Specifically, using query embedding per text instance, TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multitask modeling. It allows for mutual training and optimization of classification, segmentation and recognition branches, resulting in deeper feature sharing without sacrificing flexibility or simplicity. Additionally, we design an adaptive global aggregation (AGG) module to transfer global features into sequential features for reading arbitrarilyshaped texts, which overcomes the suboptimization problem of RoI operations. Furthermore, potential corpus information is utilized from weak annotations to full labels through mixed supervision, further improving text detection and end-to-end text spotting results. Extensive experiments on various bilingual (i.e., English and Chinese) benchmarks demonstrate the superiority of our method. Especially on the TDA-ReCTS dataset, TextFormer surpasses the state-of-the-art method in terms of 1-NED by 13.2%. |
关键词 | End-to-end text spotting arbitrarily-shaped texts transformer mixed supervision multitask modeling |
DOI | 10.1007/s11633-023-1460-6 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/58568 |
专题 | 学术期刊_Machine Intelligence Research |
作者单位 | 1.School of Computer Science, Beijing Institute of Technology, Beijing 100081, China 2.Department of Computer Vision Technology, Baidu Inc., Beijing 100193, China |
推荐引用方式 GB/T 7714 | Yukun Zhai, Xiaoqiang Zhang, Xiameng Qin,et al. TextFormer: A Query-based End-to-end Text Spotter with Mixed Supervision[J]. Machine Intelligence Research,2024,21(4):704-717. |
APA | Yukun Zhai, Xiaoqiang Zhang, Xiameng Qin, Sanyuan Zhao,Xingping Dong,& Jianbing Shen.(2024).TextFormer: A Query-based End-to-end Text Spotter with Mixed Supervision.Machine Intelligence Research,21(4),704-717. |
MLA | Yukun Zhai,et al."TextFormer: A Query-based End-to-end Text Spotter with Mixed Supervision".Machine Intelligence Research 21.4(2024):704-717. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
MIR-2023-03-038.pdf(2312KB) | 期刊论文 | 出版稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论