Download FreeLing User Manual

16
CHAPTER 3. ANALYSIS MODULES
! !
, ,
: :
...
Fat
Fc
Fd
... Fs
One special line may be included defining the tag that will be assigned to any other punctuation
symbol not found in the list. Any token containing no alphanumeric character is considered a
punctuation symbol. This special line has the format: <Other> tag.
E.g.
<Other> Fz
3.7
User Map Module
The user map module assigns Part-of-Speech tags to words matching a given regular expression. It
can be used to customize the behaviour of the analysis chain to specific applications, or to process
domain-specific special tokens. The API of the class is the following:
class RE_map {
public:
/// Constructor
RE_map(const std::wstring &);
/// analyze given sentence.
void analyze(sentence &);
/// analyze given sentences.
void analyze(std::list<sentence> &);
/// return analyzed copy of given sentence
sentence analyze(const sentence &);
/// return analyzed copy of given sentences
std::list<sentence> analyze(const std::list<sentence> &);
};
The constructor receives as parameter the name of a file containing a list of regular expressions,
and the list of pairs lemma-PoS tag to be assigned to each word matching the expression.
Note that this module will be applied afer the tokenizer, so, it will only annotate symbols
that have been separated at the tokenization step. So, customizing your application to recognize
certain special tokens will require modifying also the tokenizer configuration file.
Note also that if you introduce in this file PoS-tags which are not in the tagset known to the
tagger, it may not be able to properly disambiguate the tag sequence.
Note that this module sequentially checks each regular expression in the list against each word
in the text. Thus, it should be used for patterns (not for fixed strings, which can be included in
a dictionary file), and with moderation: using a very long list of expressions may severely slow
down your analysis chain.
3.7.1
User Map File
The format of the file containing the user map from regular expression to pairs lemma-PoS is one
regular expression per line, each line with the format: regex lemma1 tag1 lemma2 tag2 ....
The lemma may be any string literal, or $$ meaning that the string matching the regular
expression is to be used as a lemma.
E.g.:
@[a-z][0-9] $$ NP00000
<.*> XMLTAG Fz
hulabee hulaboo JJS hulaboo NNS