Many people concerned with SEO are familiar with the robots.txt file and the practice of adding robot-based meta tags directly to the HTML page. However, there is another method you can use to deliver the robot exclusion instructions (REP: Robots Exclusion Protocol) to search engine spiders besides these two common methods.

You can pass back in the HTTP Headers to a robot an X-Robots-Tag with the same instructions you'd add into the meta file to instruct the robot how to handle the page.

For example, if you have a PDF file and you want the robot to index the link to it, but not any content in it, you can pass back the following header when you send the contents of the PDF file to the client

X-Robots-Tag: index, noarchive, nosnippet

The following commands appear to be supported at this time

• INDEX|NOINDEX - Tells whether the page may be indexed or not
  
• FOLLOW|NOFOLLOW - Tells whether crawlers may follow links provided on the page or not
  
• ALL|NONE - ALL = INDEX, FOLLOW (default), NONE = NOINDEX, NOFOLLOW
  
• NOODP - tells search engines not to use page titles and descriptions from the ODP on their SERPs.
  
• NOYDIR - tells Yahoo! search not to use page titles and descriptions from the Yahoo! directory on the SERPs.
  
• NOARCHIVE - Google specific, used to prevent archiving (cached page copy)
  
• NOSNIPPET - Prevents Google from displaying text snippets for your page on the SERPs
  
• UNAVAILABLE_AFTER: RFC 850 formatted timestamp - Removes an URL from Google’s search index a day after the given date/time

You can, for example, make an .htaccess file to route all requests for a PDF file through a .php script, which will pass back the header and then server the PDF file to the client. By doing so, in this example, you would tell the search engine spider to index the file/link, but don't archive it, and don't show a short preview in the SERPs.

If you find yourself manually adding the same robots meta tags to every single page, this is a better option. The page content will be that much smaller and you'll save yourself the hassle of mainting the HTML and ensuring uniform tags are used throughout all pages.

Source: http://sebastians-pa...-tags-with-php/