Download Heritrix User Manual
Transcript
Heritrix User Manual 6.2.2. Filters Filters are modules that take a CrawlURI and determine if it matches the criteria of the filter. If so it returns true, otherwise it returns false. Filters are used in a couple of different contexts in Heritrix. Their use in scopes has already been discussed in Section 6.1.1, “Crawl Scope” and the problems with using them that in Section 6.1.1.1, “Problems with the current Scopes”. Note A DecidingFilter was added in 1.4.0 to address problems with current filter model. DecideRules can be added into a DecidingFilter with the filter decision the result of all included DecideRule set processing. There are DecideRule equivalents for all Filter-types mentioned below. See Section 6.1.1.2, “DecidingScope” for more on the particulars of DecideRules and on the new Deciding model in general. Aside from scopes, filters are also used in processors. Filters applied to processors always filter URIs out. That is to say that any URI matching a filter on a processor will effectively skip over that processor. This can be useful to disable (for instance) link extraction on documents coming from a specific section of a given website. 6.2.2.1. Adding, removing and reordering filters The Submodules page of the configuration section of the WUI lists existing filters along with the option to remove, add, or move Filters up or down in the listing. Adding a new filters requires giving it a unique name (for that list), selecting the class type of the filter from a combobox and clicking the associated add button. After the filter is added, its custom settings, if any, will appear in the Settings page of the configuration UI. Since filters can in turn contain other filters (the OrFilter being the best example of this) these lists can become quite complex and at times confusing. 6.2.2.2. Provided filters The following is an overview of the most useful of the filters provided with Heritrix. 6.2.2.2.1. org.archive.crawler.filter.OrFilter Contains any number of filters and returns true if any of them returns true. A logical OR on its filters basically. 6.2.2.2.2. org.archive.crawler.filter.URIRegExpFilter Returns true if a URI matches the regular expression set for it. See Regular expressions for more about regular expressions in Heritrix. 6.2.2.2.3. org.archive.crawler.filter.ContentTypeRegExpFilter This filter runs a regular expression against the response Content-Type header. Returns true if content type matches the regular expression. ContentType regexp filter cannot be used until after fetcher processors have run. Only then is the Content-Type of the response known. A good place for this filter is the writer step in processing. See Regular expressions for more about regular expressions in Heritrix. 6.2.2.2.4. org.archive.crawler.filter.SurtPrefixFilter 21