毕业生知识库

The rapid development of internet technology has enriched the ways of information access in recent years. With the popularity of microblog, twitter and other social media, many short texts including tweets, film reviews, instant messages and headlines are emerging every day. It can greatly facilitate information management and storage by building an auto-classification system of short texts. Furthermore, tasks like public opinion analysis and real-time hot spot analysis may be built upon in order to make it easier for people to get and understand information.

Traditional text classification systems are designed to manage long texts like books, documents and news. And the key algorithms of those systems are based on term frequencies and vector space models. Ideal outcomes cannot be seen simply by applying long text categorization algorithm to short texts, the main reason of which is that, on the one hand, short texts contain limited information, so the term frequency-based algorithm is not suitable for short text categorization; on the other hand, the existing algorithms tend to measure the match between the topic and key words, rather than base on the general understanding of the text.

In recent years, research on deep learning has advanced rapidly and related methods have shown great advantages in fields of speech recognition, image processing and natural language processing, and a breakthrough has been made in core problems of these areas. This paper will focus on problems of traditional algorithms in short texts classification from the following three parts, leveraging the advantages of deep learning.

First, we improve the structure of single layer neural network. By comparing the existing neural network units like Long-Short-Term-Memory (LSTM) and Gated-Recurrent-Unit (GRU) we found the most efficient unit for short text classification tasks. At the same time, we improve the outputs of Recurrent-Neural-Network (RNN) and their variations. Traditionally only the last output is used to classify texts, while adopting the pooling method, usually used in Convolutional-Neural-Networks (CNN), we merge all the outputs of a RNN in order to get a better representation of a short text. Next, we work out two ways to fine tune the inputs and middle parameters of our model. One is to optimize the input of the neural network with pre-trained word vectors. The other is to optimize the middle parameters in the neural network by using the auto-encoder. Our experiments show that these two methods can help the model converge to the global optimal point, thus greatly improve the results of short text classification tasks. Finally, we introduce an improved multi-layer recurrent neural network method for short text classifications. The traditional multi-layer neural network building is simply by taking the output of the former layer as the later layer’s input, layer upon layer. In this paper, drawing on the gate idea in LSTM, we improve the connections between different layers. By doing so, we get a more representative expression of short texts. Our experiments show that the performance of the improved multi-layer neural network is better than that of the single layer neural network.