Download master degree project

20
18
17,71
16
18,53
16,58
17,59
14,48
BLEU
14
12
10
8
6
4
2
0
Kacenka
only
Books 2 only CzEng 0.7
only
Kacenka +
Books 2
Kacenka +
CzEng 0.7
Figure 3.5 Training language model on various corpora combinations56
We can clearly see that eliminating the Kacenka corpus from the language model training data is
counterproductive. However, we can see that working with smaller Books 2 corpus still yields
better results than the CzEng 0.7 corpus. The genre of the corpus is more important than its size.
The last two columns convey a similar message: it is better to combine Kacenka with a fiction
corpus, not the multi-domain CzEng 0.7 (the BLEU in this last case still drops slightly).
3.1.3 Morphological pre-processing
As already indicated in section 2.3.2.2, we will test three scenarios of morphological preprocessing, each time using both the Libma library57 and the Prague Dependency Treebank58.
Both Libma and PDT are exploited through a Python interface59. The interface for Libma is more
sophisticated and allows us to set up several parameters. We will set the case sensitivity to 0 and
lemmatization level60 to 111. These quite restrictive settings should have the effect that we do
The work for the five scenarios can be found at
minerva1:/mnt/minerva1/nlp/projects/mt/work/moses_tests/ in the following directories:
work_kacenka_base, work_kacenka_lm_books2, work_kacenka_lm_czeng1,
work_kacenka_lm_books2_kacenka, and work_kacenka_lm_czeng1_kacenka
57 The pre-processing is done by two our scripts – morphology_ma.py and morphology_ma_env.py
58 The pre-processing is done by the morphology_pdt.py script.
59 Libma through minerva1:/mnt/minerva1/nlp/projects/mt/tools/ma/libma/pylibma/. PDT through
minerva1:/mnt/minerva1/nlp/local/lib/python2.5/site-packages/.
60 The SetLemmatization() method.
56
39