Back to FAQ Index

How does robots.txt differ from the ROBOTS META tag?

How does robots.txt differ from the ROBOTS META tag?

robots.txt is used to tell robots which parts of a site should not be visited, whereas the ROBOTS META tag tells a robot how to treat a specific page.

from The Web Robots Pages:
The ROBOTS META tag allows HTML authors to indicate to visiting robots if a document may be indexed, or used to harvest more links. No server administrator action is required.

Note that currently only a few robots implement this.

In this simple example:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
a robot should neither index this document, nor analyse it for links.

Full details on how this tags works is provided:


This Q&A was found on: https://nukeseo.com/modules.php?name=FAQ&file=index&myfaq=yes&id_cat=1