Download as pdf

Appendix B - Managing Heritrix Harvest Templates (order.xml)
Contents
Mandatory elements in the NetarchiveSuite and their role
A. The QuotaEnforcer
B. The DeDuplicator
C. The "http-headers" element
D. The Archiver element
E. The ContentSize element
F. The Scope element
The anatomy of a decidingscope
The header
The defining deciderule
Standard harvest rules
Define general crawlertraps to be avoided
The HarvestTemplateApplication tool
Predefined harvest templates
Templates w/ DomainScope
Templates w/ HostScope
Templates w/ PathScope
The NetarchiveSuite software uses Heritrix 1.14.4 to harvest webpages. A harvest done by Heritrix is specified with a harvest template (invariably
named order.xml). A harvest template describes how much to harvest and from where. Furthermore a seedlist is always associated with a given
order.xml.
The standard harvest template used by NetarchiveSuite follow the order.xml standard of Heritrix 1.10+.
Our default harvest template can be seen here in full: default_orderxml.xml
If you intend to build your own templates, it is recommended to use this template as a baseline.
Mandatory elements in the NetarchiveSuite and their role
A number of elements in the order.xml are required in all NetarchiveSuite harvest templates:
A. The QuotaEnforcer
The QuotaEnforcer is used to restrict the number of bytes harvested from each domain in the seedlist.
<newObject name="QuotaEnforcer" class="org.archive.crawler.prefetch.QuotaEnforcer">
<boolean name="force-retire">false</boolean>
<boolean name="enabled">true</boolean>
<newObject name="QuotaEnforcer#decide-rules" class=
"org.archive.crawler.deciderules.DecideRuleSequence">
<map name= "rules">
</map>
</newObject>
<long name="server-max-fetch-successes">-1</long>
<long name="server-max-success-kb">-1</long>
<long name="server-max-fetch-responses">-1</long>
<long name="server-max-all-kb">-1</long>
<long name="host-max-fetch-successes">-1</long>
<long name="host-max-success-kb">-1</long>
<long name="host-max-fetch-responses">-1</long>
<long name="host-max-all-kb">-1</long>
<long name="group-max-fetch-successes">-1</long>
<long name="group-max-success-kb">-1</long>
<long name="group-max-fetch-responses">-1</long>
<long name="group-max-all-kb">-1</long>
<boolean name="use-sparse-range-filter">true</boolean>
</newObject>
B. The DeDuplicator
The DeDuplicator is a module authored by Kristinn Sigurdsson from the National Library of Iceland. It is part of the Write-processor chain. It
enables us to avoid saving duplicates in our storage. It does this by looking up the url of the potential duplicate object in the index associated with