Download Adaptive Revisiting with Heritrix

Transcript
While it would, potentially, be quite powerful to allow the user to set up
and use any database application that they are already using, the fact that
Heritrix would be writing and reading a tremendous amount of
information means that in practical terms the database would have to be
located on the same machine (or accessed via a high-speed network with
little other traffic at the very least) and do nothing else. Obviously, if the
database was also handling other applications, the load from Heritrix
could seriously degrade their performance.
Then there is the issue of performance. Additional to the above is the fact
that communications would have to be via a JDBC connection, rather than
native access. This would impose a performance penalty.
There is also an issue with extensibility. The fact is that CrawlURIs
include a lot of data. This data would has to be converted for storage in a
table. Since the CrawlURIs contain a hash map of data that may contain
any number of entries, this gets complicated and would probably require
that a serialized version of an object be stored in one field for restoration,
with other fields being used only for indexing. Since object serialization
in Java is bulky (a lot of information about the class is repeated for each
object), this is far from ideal. In fact, one of the main problems with the
custom written queues used by the HostQueuesFrontier was the extremely
bulky nature of the queues which stored serialized versions of the
CandidateURIs and CrawlURIs.
Finally, relational databases simply contain a great number of features
entirely superfluous to the task at hand. A more streamlined piece of
software would be preferable.
For all the above reasons, traditional object oriented databases were
excluded as solutions to the disk storage problem.
The desired features for a data storage solution were:
• Open source with a license what would allow us to distribute it with
Heritrix.
• Implemented in Java, like Heritrix.
• Capable of being embedded in Heritrix, rather than running as a
separate program.
56