Download Adaptive Revisiting with Heritrix
Transcript
While it would, potentially, be quite powerful to allow the user to set up and use any database application that they are already using, the fact that Heritrix would be writing and reading a tremendous amount of information means that in practical terms the database would have to be located on the same machine (or accessed via a high-speed network with little other traffic at the very least) and do nothing else. Obviously, if the database was also handling other applications, the load from Heritrix could seriously degrade their performance. Then there is the issue of performance. Additional to the above is the fact that communications would have to be via a JDBC connection, rather than native access. This would impose a performance penalty. There is also an issue with extensibility. The fact is that CrawlURIs include a lot of data. This data would has to be converted for storage in a table. Since the CrawlURIs contain a hash map of data that may contain any number of entries, this gets complicated and would probably require that a serialized version of an object be stored in one field for restoration, with other fields being used only for indexing. Since object serialization in Java is bulky (a lot of information about the class is repeated for each object), this is far from ideal. In fact, one of the main problems with the custom written queues used by the HostQueuesFrontier was the extremely bulky nature of the queues which stored serialized versions of the CandidateURIs and CrawlURIs. Finally, relational databases simply contain a great number of features entirely superfluous to the task at hand. A more streamlined piece of software would be preferable. For all the above reasons, traditional object oriented databases were excluded as solutions to the disk storage problem. The desired features for a data storage solution were: • Open source with a license what would allow us to distribute it with Heritrix. • Implemented in Java, like Heritrix. • Capable of being embedded in Heritrix, rather than running as a separate program. 56