Back to FAQ Index

Do all robots follow the rules for robots.txt?

Do all robots follow the rules for robots.txt?

The robots.txt file is only a guide and some bad robots ignore it.  Fortunately, there are tools for enforcing compliance with the robots exclusion protocol used in robots.txt, including:

This Q&A was found on: http://nukeseo.com/modules.php?name=FAQ&file=index&myfaq=yes&id_cat=1