MSN Crawlers Pawned

After seeing the way MSN crawled my last post, I just realized why Microsoft could never do good in Search Engine Market :)

I wonder why MSN would crawler same page from two different machines. I wonder if a single page can be divided further for crawling. Checkout the screenshots below :)

Why Google Wins

A few minutes later. Three Microsoft machines were crawling the same page ­čśŤ

MSN Crawler Pawned

I have conformed using ARIN that all these IPs belong to Microsoft ­čśÇ

 

Google is Fastest

While testing a piece of code for my squid plugin, I ended up changing permissions of /tmp/ directory to 0755. A few minutes later (after making changes to the code), I tried committing it to fedorapeople.org and got this error

1
2
3
4
5
6
[project@bordeaux youtube_cache]$ git push --all ssh://kulbirsaini@fedorapeople.org/~kulbirsaini/public_git/youtube_cache.git
Enter passphrase for key '/home2/Studies/project/.ssh/id_rsa':
ssh_control_listener bind(): Permission denied
fatal: The remote end hung up unexpectedly
error: failed to push to 'ssh://kulbirsaini@fedorapeople.org/~kulbirsaini/public_git/youtube_cache.git'
[project@bordeaux youtube_cache]$

Well, I couldn’t understand the error and jumped to #fedora-admin. I┬ápasted the error on pastebin. Almost a minute laster, while trying to conquer the error, I Googled the exact error and I was damn surprised to see the paste as first result. What an indexing speed Google has??? See the screen shot for proof :)

Google is Fastest

In #fedora-admin ricky confirmed that nothing was wrong on server side and it should be a local problem with my machine or proxy server at my insti border. A few minutes later, I reliazed the /tmp/ permission thing and changing permissions back to 0777 fixed everything :)