Download Adaptive Revisiting with Heritrix

Once the CrawlController has set everything up it waits for instructions to
start the crawl. Once started it can be instructed to effect a pause,
resumption or a termination of the crawl. This is generally only used by
the UI. To enforce these orders it issues instructions to the relevant
components.
When writing modules for Heritrix interactions with the CrawlController
are rarely of any great significance. As it implements generic services,
any new modules are likely to utilize them in a similar manner to existing
code, i.e. accessing the logs and firing off events. It is just important that
they (especially frontiers) correctly implement the methods that the
CrawlController uses to control the crawl.
3.2 ToeThreads
Heritrix is multithreaded and uses a pool of worker threads to process the
discovered URIs. These worker threads are called ToeThreads. The
Heritrix FAQ [17] explains the logic behind this unusual name. "While
the mascots of web crawlers have usually been spider-related, I'd rather
think of Heritrix as a centipede or millipede: fast and many-segmented.
Anything that 'crawls' over many things at once would presumably have a
lot of feet and toes. Heritrix will often use many hundreds of worker
threads to "crawl", but 'WorkerThread' or 'CrawlThread' seem mundane.
So instead, we have 'ToeThreads'."
ToeThreads request URIs from the Frontier. Assuming one is available,
the Frontier will issue a URI. The ToeThread then applies a series of
processors to the URI. Once all configured processors have been applied,
the URI is returned to the Frontier and the ToeThread requests another
URI to process.
It is the Frontier’s responsibility to pass the ToeThreads signals to pause,
resume and terminate crawls. Typically, this is done when a ToeThread
requests a new URI. The frontier can then either throw an
EndedException or make the ToeThread wait until the crawl can be
resumed.
15