中村雅己,鹿野清宏
ニューラルネットによる英文単語列予測モデルの検討
Abstract:Using traditional statistical approaches, it is difficult to make an N-gram word prediction model
to construct an accurate word recognition system because of the increased demand for sample data
and parameters to memorize probabilities. To solve this problem, NETgrams, which are neural
networks for N-gram word category prediction in text, are proposed. NETgrams are constructed
by a trained Bigram network with two hidden layers. Each hidden layer learns the coarse-coded
Micro Features (MF1 or MF2) of the input or output word category. NETgrams can easily be
expanded from Bigram to N-gram networks without explosively increasing the number of free
parameters.
NETgrams are tested by training experiments with a Brown Corpus English Text Database. The
training method is the Back-Propagation algorithm. After training, the Trigram word category
prediction rates for test data show that the NETgrams are comparable to the statistical model and
compress information more than 130 times. Results of analyzing the hidden layer (Micro
Features) show that the word categories are classified into some linguistically significant groups.
We are now training the 4-gram networks and obtaining good results.
In addition, this paper proposes a new method to speed up the Back-Propagation algorithm, which
dynamically controls the training parameters, updating step size and momentum. This new
method can automatically determine better parameters and achieve a shorter training time.