Download Adaptive Revisiting with Heritrix

thread is ordered to wait until it becomes ready. Whenever the processing
of a URI is completed, all waiting threads are notified so they can check if
the top queue is now ready. Eventually a queue should become available.
The ARFrontier then requests the next URI from the queue. The amount
of time that the URI is overdue is then calculated and stored in the
CrawlURI. This value is logged later and is also optionally used by the
WaitEvaluator.
Once a URI is issued, the ARFrontier only keeps track of it by the state of
its queue, but does not otherwise concern itself with it. If a URI is never
returned back to the Frontier, via the finished() method, then the queue in
question, assuming a host valence of 1, will become permanently
snoozed, effectively blocking it from further progress. While this should
never happen, bugs in the processors or ToeThread can have this effect. If
this occurs, the operator can intervene and ‘kill’ the ToeThread, causing
the URI to be returned to the Frontier with an error code.
Eventually, the URI should be returned to the Frontier. It is here that the
ARFrontier departs from other Frontiers in its handling, although much
remains similar.
First the Frontier determines how the URI fared in the processing. There
are four possible outcomes:
1. URI was successfully processed
The web server was contacted, we may have received a HTTP error,
such as ‘404 URL not found.’ We still treat those as successes and will
revisit them.
2. URI encountered a retriable error
There are two flavors of this. Errors requiring immediate retrying, such
as missing preconditions (the preconditions are scheduled with higher
scheduling directive and should be processed ahead of any repeat
attempt) and those that should be retried after a delay. This assumes
that transient network errors caused the error and retrying the URI at a
later time may yield a different result.
3. URI was disregarded
Typically means that that robots.txt rules precluded a fetch attempt.
This state means that the URI should be regarded as being out of scope.
We will not be revisiting it.
63