Web Page Crawler Revealed - Information Camouflage bruceediger.com

Bruce Ediger says Apache logs show a centrally coordinated crawler operating from Contabo GmbH IP addresses repeatedly traversed every page in his Scraper Junkyard, following its binary-tree links. Across six crawls, it made about four requests per second, with no duplicate page requests; 7,690 IP addresses appeared across four series, and 2,964 took part in all four.