Download SPAWC: A Scalable Parallel Web Crawler

Computer Science 9868
SPAWC: A Scalable Parallel Web Crawler
A.5
Jeffrey Shantz <jshantz(5-1)@csd.uwo.ca>
June 02, 2010
Diagnostics
As discussed in section 2.6, the coordinator and crawler processes periodically log status messages
to disk, allowing for more detailed information to be obtained than that presented on the Web dashboard. All logs are available in the SPAWC VM in the directory /home/cs9868/crawler/lib/logs.
Previously, these logs were viewable from the dashboard, but this functionality was removed as refreshing the logs consumed too many server resources.
40