next up previous contents
Next: Modeling the Document Space Up: Document Space Modelling Previous: Document Space Modelling

Mixture LM

A mixture LM, ${\cal M}$, is constructed as the weighted sum of component LMs $<\!\! {\cal M}_1,\cdots,{\cal M}_j,\cdots \!\!>$ derived from the partitioned corpus (either hand-labeled or automatic) [7]. Given a document, i.e., a sequence of words $<\!\! w_1, \cdots, w_i, \cdots \!\!>$, it is computed using the conventional trigram LMs by
 
$\displaystyle f(w_i\vert w_{i-2},w_{i-1};{\cal M}) = \sum_j c_j f(w_i\vert w_{i-2},w_{i-1};{\cal M}_j)$     (3.6)

where cj is a mixing factor such that $\sum_j c_j = 1$.

Mixing factors cj are tuned on-the-fly to the previously processed part of the document using the expectation-maximization (EM) type algorithm. Suppose n words, $<\!\! w_1,\cdots,w_n \!\!>$, have been processed from the beginning. Then, considering the likelihood function $f(w_1,\cdots,w_n\vert{\cal M})$ for the mixture LM, it is straightforward to derive incrementally adjusting formula for cj(n);

 
$\displaystyle c_j^{(n)} = \frac{1}{n} \sum_{i=1}^n \gamma_j(i)$     (3.7)

where $\gamma_j(i)$ is estimated by
 
$\displaystyle \gamma_j(i) = \frac{c_j^{(i-1)} f(w_i\vert w_{i-2},w_{i-1};{\cal M}_j)}
{\displaystyle \sum_k c_k^{(i-1)} f(w_i\vert w_{i-2},w_{i-1};{\cal M}_k)}$     (3.8)

with appropriate terminating condition. Note that a posterior mode may be used instead by combining some prior function at Equation (3.7).


next up previous contents
Next: Modeling the Document Space Up: Document Space Modelling Previous: Document Space Modelling
Christophe Ris
1998-11-10