Download Heritrix User Manual

Heritrix User Manual
each link or embed followed from the seed.
The character legend is as follows.
R
E
X
L
P
-
Redirect
Embed
Speculative embed (aggressive/Javascript link extract
Link
Prerequisite (as for DNS or robots.txt before another
The discovery path of seeds is an empty string.
Frontier
A Frontier is a pluggable module in Heritrix that maintains the internal state of the crawl. See Section 6.1.2, “Frontier”.
"Holding Jobs" vs. "Crawling
Jobs"
The mode Crawling Jobs generally means that the crawler will start
executing a job as soon as one is made available in the pending jobs
queue (as long as there is not a job already running).
If the crawler is in the Holding Jobs mode, jobs added to the
pending jobs queue will be held; they will not be started, even if
there are no jobs currently being run.
Host
A host can serve multiple domains or a single domain can be served
by multiple hosts. For our purposes so far, host == hostname in URI.
DNS is not considered; it is volatile and may be unavailable. So
when Heritrix gets the URIs...
http://www.example.com
http://search.example.com
http://201.199.7.15
...even if they all point to the 201.199.7.15 IP, they are 3 different
logical hosts (at the level of the URI/HTTP protocol).
Conformant HTTP proxies behave similarly, we think, even if they
know www.example.com == 201.199.7.15, they will not consider
them interchangeable.
This is not ideal for politeness where we'd want politeness rules to
apply to the physical host rather than the logical.
Link hop count
Number of link follow from the seed to the current URI. Seeds have
a link hop count of 0.
This number is equal to counting the 'L's in a URIs discovery path.
Pending URIs
Number of URIs that are awaiting detailed processing.
Number of discovered URIs that have not been inspected for scope
or duplicates. Depending on the implementation of the Frontier this
might always be zero. It may also be an adjusted number that tries to
account for duplicates by estimation.
Politeness
Politeness refers to attempts by the crawler software to limit load on
a site. Without politeness restrictions the crawler might otherwise
overwhelm smaller sites and even cause moderately sized sites to
slow down significantly.
54