中国科学院自动化研究所机构知识库

Chinese-English code-switching refers to the phenomenon of language switching between Chinese and English in the process of communication. With the progress of globalization, Chinese-English code-switching expression has become a common way of communication. Especially in some scenarios, such as English teaching classrooms, academic conferences, multinational enterprise conferences, etc. It effectively reduces the cost of communication and promotes understanding between different cultures. With the increasing number of Chinese and English bilinguals, code-switching expressions are becoming more and more common, which has become a problem that cannot be ignored in the automatic speech recognition (ASR) system. Although the monolingual ASR system is practical, it is still unable to effectively handle code-switching speech. In view of the urgent practical needs and the poor speech recognition performance, this paper studies the Chinese-English code-switching ASR problem under the framework of end-to-end speech recognition.

Focusing on the goal of "improving the performance of Chinese-English code-switching ASR methods", we analyze the factors affecting the recognition performance from multiple perspectives. Based on a detailed examination of the existing research work, we carry out research from several aspects such as language information, language context information, and semantic information. The innovative algorithms of this study are not limited to Chinese-English code-switching scenarios, and can be extended to code-switching ASR systems in other languages, which has good generalization; at the same time, this research has broad application prospects and can promote many real-world scenarios. And the development of the code-switching ASR system has huge social and economic benefits. Specifically, this paper mainly completes three innovative works.

1. Propose an ASR method for unified modeling of language and speech. At present, there are many problems in the methods of using language information to assist code-switching ASR, such as complex model structure, high computational cost, and insignificant improvement of ASR performance by language information. This paper proposes a new language and speech joint modeling method. After adding language labels to the target text of the training data, the neural network transducer model is used to simultaneously learn the tasks of ASR and language identification. This method does not add additional language identification modules, nor does it increase the computational cost of training and inference. In addition, language identification and ASR tasks are highly coupled, and language information can effectively improve the performance of Chinese-English code-switching ASR tasks. In the recognition process, language information is used to guide the next decoding direction, further reducing the recognition error rate.

2. Propose an efficient multilingual language context modeling method. Aiming at the problems of complex multilingual context modeling and lack of training data in Chinese-English code-switching ASR task, a pronunciation and language decoupled method is proposed, which improves the language context modeling capability of the end-to-end model. The method decouples the speech-to-text recognition process into two parts, the speech-phoneme process, and the phoneme-text process. The speech-phoneme process uses the connectionist temporal classification (CTC) loss to train an acoustic encoder with phonemes as modeling units. The independent nature of the output units of the CTC reduces the mismatch between code-switching and monolingual speech, and effectively utilizes monolingual speech data. At the same time, phonemes are designed according to pronunciation rules, which can better model the acoustic distribution. The phoneme-text process uses plain text data to train the model, learns the mapping relationship between phoneme sequences and texts, and models pronunciation dictionary information and language context information at the same time. The two processes are independently trained. The former uses monolingual data and the latter uses plain text data, which effectively alleviates the difficulty of modeling multilingual context and greatly improves the performance of Chinese-English code-switching ASR.

3. Propose an effective code-switching semantic modeling method. According to the semantic characteristics of Chinese-English code-switching expressions, an effective semantic information modeling method is proposed, which significantly improves the performance of Chinese-English code-switching ASR. Specifically, due to the randomness of switching between Chinese and English, the same sentence may correspond to a variety of different Chinese-English code-switching expressions, but the semantics between these different expressions are consistent. Based on this feature, synonyms with multiple code-switching expressions are constructed from text, and semantic consistency loss is designed according to these synonyms to participate in model training. This semantic loss can be used to train both the end-to-end ASR model and the neural network language model. These two methods can jointly improve the performance of the Chinese-English code-switching ASR system.