S
Sculli
Guest
Well folks, after I got done proof-reading Webster this morning, I figured to have a look at some log files to expose the bandwidth hogs among us (not that it matters at 100GB/month).
Little did I know.
Microsoft is trying to push into the search engine market ahead of their Windows 2006 release so they deploy the spiders, but no, not just one, three of them little buggers crawling QCDN 24/7.
http://search.msn.com/msnbot.htm
Racking up a whopping combined 1.9 GB of traffic withing a 30 day period.
What's even more interesting is that say Google is crawling QCDN as well, but in the log analyzer Google shows up as crawl34.googlebot.com, while the MS bots simply show IP addresses, so that one has no idea what the hell it is, till you actually get to the bottom of it.
To their defense I have to say that the access_log actually does show something like this:
65.54.188.66 - - [30/Aug/2004:00:14:11 -0700] "GET /profile.php?mode=viewprofile&u=348&sid=333d5a9452be27574826b8ed3c25585b HTTP/1.0" 200 25201 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
Be that as it may, while Google and others employ one bot at a time, MS does not, and thus uses up quite a bit of bandwidth.
I have since locked out 65.54.188.XXX, don't care whether QCDN shows up in MSN search or not.
Little did I know.
Microsoft is trying to push into the search engine market ahead of their Windows 2006 release so they deploy the spiders, but no, not just one, three of them little buggers crawling QCDN 24/7.
http://search.msn.com/msnbot.htm
Racking up a whopping combined 1.9 GB of traffic withing a 30 day period.
What's even more interesting is that say Google is crawling QCDN as well, but in the log analyzer Google shows up as crawl34.googlebot.com, while the MS bots simply show IP addresses, so that one has no idea what the hell it is, till you actually get to the bottom of it.
To their defense I have to say that the access_log actually does show something like this:
65.54.188.66 - - [30/Aug/2004:00:14:11 -0700] "GET /profile.php?mode=viewprofile&u=348&sid=333d5a9452be27574826b8ed3c25585b HTTP/1.0" 200 25201 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
Be that as it may, while Google and others employ one bot at a time, MS does not, and thus uses up quite a bit of bandwidth.
I have since locked out 65.54.188.XXX, don't care whether QCDN shows up in MSN search or not.