Download Jane: User's Manual

Chapter 6
Phrase training
For the phrase-based decoder, Jane implements the forced alignment phrase training
technique described in [Wuebker & Mauser+ 10].
6.1
Overview
The idea of forced alignment phrase training is to use a modified version of the translation decoder to force-align the training data. We apply the identical log-linear model
combination as in free translation, but restrict the decoder to produce the reference
translation for each source sentence. Rather than using the word alignment to heuristically extract possible phrases, we now have real phrase alignment information to work
with. From these we extract real phrase counts, either from an n-best list or via the
forward-backward algorithm (not implemented yet), and estimate phrase translation
probabilities as relative frequencies. To counteract over-fitting, we apply a leave-one-out
technique in the first iteration and cross-validation on further iterations.
The easiest way to invoke phrase training in Jane is via the extraction script. It takes
two config files as parameters: an extended version of the standard extraction config (cf.
Chapter 4), and a Jane decoder config (cf. Chapter 5).
6.2
Usage of the training script
The training script trainHierarchical.sh is capable of performing the full training
pipeline by itself (an SGE grid engine should be available for reasonable performance).
First it performs the standard extraction. If the option --phraseTraining is set, it
will perform forced alignment training. The first iteration is initialized with the extracted rule table. You can invoke the script with the config file extract.config with
trainHierarchical.sh --config extract.config --phraseTraining [options].
If the extraction is already finished, you can also directly call the phrase training
via trainHierarchical.sh [options] startPhraseTraining. In this case you should
make sure that the extraction-related options are identical to the ones used before. Many
of them are used again for the normalization of the trained rule table.
63