twitter’s robots.txt file look very interesting…http://twitter.com/robots.txt