We show an effective way of adding context information to shallow neural language models. We propose to use Subspace Multinomial Model (SMM) for context modeling and we add the extracted i-vectors in a computationally efficient way. By adding this information, we shrink the gap between shallow feed-forward network and an LSTM from 65 to 31 points of perplexity on the Wikitext-2 corpus (in the case of neural 5-gram model). Furthermore, we show that SMM i-vectors are suitable for domain adaptation and a very small amount of adaptation data (e.g. endmost 5% of a Wikipedia article) brings a substantial improvement. Our proposed changes are compatible
with most optimization techniques used for shallow feedforward LMs.
Added on December 19, 2018
Contributed by : Individual
Product Type : Research Paper
License Type : Freeware
System Requirement :
Author : Karel Benes,Santosh Kesiraju,Lukas Burget