中国科学院自动化研究所机构知识库

Multilingual handwritten text recognition is often accomplished in two cascaded steps: script identification and handwriting recognition. However, this scheme is not optimal due to error accumulation. To perform simultaneous script identification and handwriting recognition, in this paper, we propose a new framework named multilingual text recognition networks (MuLTReNets). Specifically, the system has four major modules: feature extractor, script identifier, handwriting recognizer and auto-weighter. The feature extractor integrates both spatial and temporal knowledge to encode text images into features shared by the script identifier and recognizer. The script identifier predicts script category from a variable-length sequence incorporating an auto-weighter for balancing different scripts, while the handwriting recognizer adopts long-short term memory (LSTM) and Connectionist Temporal Classification (CTC) to accomplish sequence decoding. Via multi-task learning, the proposed framework can benefit both two multilingual recognition schemes: unified recognition with merged alphabet (MuLTReNetV1) and cascaded script identification-single script recognition with joint training (MuLTReNetV2). We evaluated the performance of the proposed method on handwritten text databases of five languages, which are English, French, Kannada, Urdu, and Bangla. Experimental results demonstrate that our method performs superiorly for both script identification and handwriting recognition. The accuracy of script identification reaches 99.9%. While in handwriting recognition, the proposed system not only outperforms cascade systems but also surpasses systems particularly designed for specific scripts.