X-Robots-Tag: Control Indexing of Files
The X-Robots-Tag is an HTTP header used to control how search engines index and serve your files. While the traditional Meta Robots tag is strictly limited to HTML documents, the X-Robots-Tag allows you to apply SEO rules to any file type, including PDFs, images, and videos.
1. What is the X-Robots-Tag?
Unlike HTML tags that sit inside the source code of a web page, the X-Robots-Tag is sent as part of the HTTP response header by your web server. When a search engine requests a file, your server replies with headers (like status code and content type) before delivering the actual file. The X-Robots-Tag is included in this invisible conversation.
2. Meta Robots vs. X-Robots-Tag
Why use the X-Robots-Tag if you already have the Meta Robots tag?
- Non-HTML Files: You cannot put a
<meta>tag inside a PDF document, an MP4 video, or a PNG image. If you want to prevent Google from indexing a sensitive PDF, the X-Robots-Tag is your only reliable method. - Global Rules: You can configure your server to apply the X-Robots-Tag to an entire directory or site simultaneously, which is often faster than editing the HTML of thousands of pages individually.
3. Code Examples (Apache & Nginx)
To implement the X-Robots-Tag, you need to modify your server configuration files.
Apache (.htaccess)
To prevent search engines from indexing any PDF files on your site, you would add this to your
.htaccess file:
<FilesMatch "\.(pdf)$">
Header set X-Robots-Tag "noindex, noarchive"
</FilesMatch>Nginx (nginx.conf)
For an Nginx server, you would add the following to your site's configuration block:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, noarchive";
}4. Core Directives
The X-Robots-Tag accepts the exact same directives as the Meta Robots tag:
noindex: Do not index this file in search results.nofollow: Do not follow any links contained within this file (e.g., links inside a PDF).noarchive: Do not show a "Cached" link in the search results.nosnippet: Do not show a text snippet in the search results for this file.
Targeting Specific Bots: You can also target specific crawlers by putting their name before the directive.
Header set X-Robots-Tag "googlebot: noarchive"
Header set X-Robots-Tag "bingbot: noindex"5. How to Check the X-Robots-Tag
Because HTTP headers are invisible on the actual page, they can be tricky to spot. There are two primary ways to check them:
- Browser DevTools: Right-click the page > Inspect > Network Tab. Refresh the page, click on the file you want to inspect, and look under the "Response Headers" section.
- Rank-O-Saur: The easiest method! Rank-O-Saur automatically intercepts HTTP headers. If an X-Robots-Tag is present and blocking indexation, the extension icon will immediately warn you, and the details will be visible in the "Overview" tab.
6. Common Mistakes to Avoid
The robots.txt Trap: Just like the Meta Robots tag, if you block a file in your
robots.txt file, search engines will never see your X-Robots-Tag!
Because they aren't allowed to crawl the URL, they never request the header. If you want to de-index
a file, make sure it is not blocked in robots.txt.
- Conflicting Signals: If you use both a Meta Robots tag (
index) in the HTML and an X-Robots-Tag (noindex) in the header, search engines will generally obey the most restrictive directive (in this case,noindex). - Syntax Errors: A mistyped server configuration file can take down your entire
website with a 500 Internal Server Error. Always back up your
.htaccessornginx.confbefore making changes.
