Download Heritrix User Manual

Heritrix User Manual
If you do not have Java installed you can download Java from:
•
Sun -- java.sun.com [http://java.sun.com/]
•
IBM -- www.ibm.com/java [http://www.ibm.com/java]
2.1.1.2. Hardware
A default java heap is 256MB RAM, which is usually suitable for crawls that range over hundreds of
hosts. Assign more -- see Section 2.2.1.3, “JAVA_OPTS” for how -- of your available RAM to the heap
if you are crawling thousands of hosts or experience Java out-of-memory problems.
2.1.1.3. Linux
The Heritrix crawler has been built and tested primarily on Linux. It has seen some informal use on
Macintosh, Windows 2000 and Windows XP, but is not tested, packaged, nor supported on platforms
other than Linux at this time.
2.2. Running Heritrix
To run Heritrix, first do the following:
% export HERITRIX_HOME=/PATH/TO/BUILT/HERITRIX
...where $HERITRIX_HOME is the location of your untarred heritrix.?.?.?.tar.gz.
Next run:
% cd $HERITRIX_HOME
% chmod u+x $HERITRIX_HOME/bin/heritrix
% $HERITRIX_HOME/bin/heritrix --help
This should give you usage output like the following:
Usage: heritrix --help
Usage: heritrix --nowui ORDER.XML
Usage: heritrix [--port=#] [--run] [--bind=IP,IP...] --admin=LOGIN:PASSWORD \
[ORDER.XML]
Usage: heritrix [--port=#] --selftest[=TESTNAME]
Version: @VERSION@
Options:
-b,--bind
Comma-separated list of IP addresses or hostnames for web
server to listen on. Set to / to listen on all available
network interfaces. Default is 127.0.0.1.
-a,--admin
Login and password for web user interface administration.
Required (unless passed via the 'heritrix.cmdline.admin'
system property). Pass value of the form 'LOGIN:PASSWORD'.
-h,--help
Prints this message and exits.
-n,--nowui
Put heritrix into run mode and begin crawl using ORDER.XML. Do
not put up web user interface.
-p,--port
Port to run web user interface on. Default: 8080.
-r,--run
Put heritrix into run mode. If ORDER.XML begin crawl.
-s,--selftest
Run the integrated selftests. Pass test name to test it only
(Case sensitive: E.g. pass 'Charset' to run charset selftest).
Arguments:
ORDER.XML
Crawl order to run.
Launch the crawler with the UI enabled by doing the following:
3