Adding compression support can be very simple -- if your spider is coded in Perl using LWP::UserAgent, then the
addition of a single line of code will enable compression support.
$ua->default_header('Accept-Encoding' => 'gzip');
and then you need to make sure that you always refer to 'decoded_content' when dealing with the response object.
For other languages, all
you need to do is to add
Accept-encoding: gzipto the HTTP request that you send, and then be prepared to deal with a 'content-encoding: gzip' in the response.
Happily, some of the large spiders do support compression -- the googlebot and Yahoo Slurp do (to name but two). Since I started prodding crawler implementors, a couple have implemented compression (one within hours), and another reported that it was a bug that it didn't work -- which would be fixed shortly.
Crawlers which do more than 5% of the total (uncompressed) crawling activity are marked in bold below.
| Crawler | Last IP used |
|---|---|
| ia_archiver" "charon.gladstonefamily.net | 54.173.35.129 |
| ia_archiver" "pond1.gladstonefamily.net | 54.173.35.129 |
| ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)" "pond1.gladstonefamily.net:8080 | 153.254.170.73 |
| LinqiaMetadataDownloaderBot/1.0 (eng@linqia.com)" "blog1.gladstonefamily.net | 34.229.13.202 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "blog.gladstonefamily.net | 216.244.66.242 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "gladstone.name | 216.244.66.241 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "gladstonefamily.net | 216.244.66.242 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "pond.gladstonefamily.net | 216.244.66.242 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "pond1.gladstonefamily.net | 216.244.66.242 |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "gladstonefamily.net | 199.58.86.206 |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "www.gladstonefamily.net | 199.58.86.206 |
| Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)" "gladstonefamily.net | 138.201.30.66 |
| Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)" "pond.gladstonefamily.net | 138.201.59.34 |
| Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)" "pond1.gladstonefamily.net | 138.201.30.66 |
| Python-urllib/2.7" "pond1.gladstonefamily.net | 147.83.200.72 |