Download Incremental crawling with Heritrix

4
Kristinn Sigurðsson
readers needs to be consistent in its presentation of new material. As noted before,
this is not an absolute and further research is needed to establish just how much an
issue erratic change rates in web pages are.
Secondly, the file type of resources, or documents, significantly affects the
probable change rates. [2] While changing HTML or other simple text files is both
easy and straight forward (not to mention expected by the user in many cases and
frequently handled by content management systems), changing images is more
complex. In fact browsers and proxies will generally cache images, while HTML
pages are requested for each page view. Images and other multimedia files are usually
not generated on demand, unlike many HTML files, and when a new or changed
graphic is needed, it is customary to add rather than replace an image.
This allows us to predict different initial wait times between revisits before we
gather any further information that enables us to adapt to observed change rates.
HTML and other pure text documents can be assigned a relatively short initial wait,
while a longer initial wait time is assigned to image files. Higher bandwidth media
such as audio and video files are even less likely to change, due to their complex
nature and can thus be assigned even higher initial wait times. Formatted text files
such as Word and PDF documents are also less likely to change than pure text files.
There is a certain amount of connection between the size of a resource and its
change rate. As the size increases, its likely change rate goes down. This assumes that
image, audio, video and formatted files are generally larger than pure text documents.
While not always true, this generally holds.
We are therefore able to use this information not only to reduce the number of
visits we must make to a certain part of our scope, but to one of the most bandwidth
intensive part of our scope.
Another factor that might be taken into consideration is a document's location in a
web site’s hierarchy. Generally, the front page of a web site changes more often than
any given page in the archive (where changes may never occur). Assuming that the
initial seeds are the front pages of each site, it would be possible to use the depth from
seed as a factor in deciding the initial wait time.
Finally, one could conceivably do some form of contextual analysis on the
resource's content or metadata to evaluate the probable change rate. For example [4]
observes that documents without the last-modified information are about twice as
likely to change between two visits, as those that have it. There are undoubtedly many
other possible indicators that could be used, but covering them all is well outside the
scope of this article.
The primary goal in using incremental crawling is to to utilize bandwidth more
effectively and thus limit the stress our harvesting places on web servers. It is worth
noting however that while doing this we also achieve a secondary benefit in reduced
storage costs, since duplicate data need not be stored. That however brings us to
another problem; detecting change.
One of the key aspects discussed above, relies on being able to detect if a change
has occurred between the times when a resource is visited. Doing a simple, strict hash,
comparison does achieve this to a degree. It certainly detects any change and reduces
the odds that we discard a changed document to nearly zero. Unfortunately, it relies
on a bit by bit comparison and takes no notice of the actual content. Thus any change