Invisible side of a Website
By kurtaxl
@kurtaxl (110)
Philippines
September 17, 2009 2:16am CST
Some websites have a file called "robots.txt" in it's root. The file allows or disallows the search engine report the results to you that the website owner don't want to tell you about. But do you know that still you can view these results by typing /robots.txt after the website address; for example, If you want to know what Microsoft hides from you type http://www.microsoft.com/robots.txt and you will find the list of directories that Bill Gates want to hide from you.
1 response
@cmdr001 (371)
• Portugal
17 Sep 09
It's funny actually because I had no idea of that and I didn't quite bothered to search up the information, however, I do have an Apache server up and running and I found weird that something had been looking for a robots.txt file.
Hardly could be a coincidence that someone was seeking such a file on my machine since there are only specific file links to it, but I didn't quite bothered with trying to see what it could be. They got nothing and the access is extremely restricted.
Still, it's a heads up and tells me that some web crawler bots do respect privacy in some way, but I guess most will just ignore that file altogether.
