Download latest PDF - Read the Docs

educe Documentation, Release 0.1
educe.ptb.annotation.PTB_TO_TEXT = {“’‘”: ‘”’, ‘‘‘’: ‘”’, ‘-LSB-‘: ‘[’, ‘-RRB-‘: ‘)’, ‘-LCB-‘: ‘{‘, ‘-LRB-‘: ‘(‘, ‘-RSB-‘:
Straight substitutions you can use to replace some PTB-isms with their likely original text
class educe.ptb.annotation.TweakedToken(word, tag, tweaked_word=None, prefix=None)
Bases: educe.external.postag.RawToken
A token with word, part of speech, plus “tweaked word” (what the token should be treated as when aligning
with corpus), and offset (some tokens should skip parts of the text)
This intermediary class should only be used within the educe library itself. The context is that we sometimes
want to align PTB annotations (see educe.external.postag.generic_token_spans) against text which is almost but
not quite identical to the text that PTB annotations seem to represent. For example, the source text might have
sentences that end in abbreviations, like “He moved to the U.S.” and the PTB might annotation an extra full stop
after this for an end-of-sentence marker. To deal with these, we use wrapped tokens to allow for some manual
substitutions:
•you could “delete” a token by assigning it an empty tweaked word (it would then be assigned a zero-length
span)
•you could skip some part of the text by supplying a prefix (this expands the tweaked word, and introduces
an offset which you can subsequentnly use to adjust the detected token span)
•or you could just replace the token text outright
These tweaked tokens are only used to obtain a span within the text you are trying to align against; they can be
subsequently discarded.
educe.ptb.annotation.basic_category(label)
Get the basic syntactic category of a label.
This is done by truncating whatever comes after a (non-word-initial) occurrence of one of the label_annotation_introducing_characters().
educe.ptb.annotation.is_empty_category(postag)
True if postag is the empty category, i.e. -NONE- in the PTB.
educe.ptb.annotation.is_non_empty(tree)
Filter (return False for) nodes that cover a totally empty span.
educe.ptb.annotation.is_nonword_token(text)
True if the text appears to correspond to some kind of non-textual token, for example, *T*-1 for some kind of
trace. These seem to only appear with tokens tagged -NONE-.
educe.ptb.annotation.post_basic_category_index(label)
Get the index of the first char after the basic label.
This should never match the first char of the label ; if the first char is such a char, then a matched char is also not
used iff there is something in between, e.g. (-LRB- => -LRB-) but (–PU => -).
educe.ptb.annotation.prune_tree(tree, filter_func)
Prune a tree by applying filter_func recursively.
All children of filtered nodes are pruned as well. Nodes whose children have all been pruned are pruned too.
The filter function must be applicable to Tree but also non-Tree, as are leaves in an NLTK Tree.
educe.ptb.annotation.strip_subcategory(tree,
retain_TMP_subcategories=False,
tain_NPTMP_subcategories=False)
Transform tree to strip additional label annotation at each node
re-
educe.ptb.annotation.transform_tree(tree, transformer)
Transform a tree by applying a transformer at each level.
The tree is traversed depth-first, left-to-right, and the transformer is applied at each node.
58
Chapter 4. educe package