X-Robots-Tag: Control Indexing of Files
The X-Robots-Tag is an HTTP header used to control how search engines index and serve your files. While the traditional Meta Robots tag is strictly limited to HTML documents, the X-Robots-Tag allows you to apply SEO rules to any file type, including PDFs, images, and videos.
1. What is the X-Robots-Tag?
Unlike HTML tags that sit inside the source code of a web page, the X-Robots-Tag is sent as part of the HTTP response header by your web server. When a search engine requests a file, your server replies with headers (like status code and content type) before delivering the actual file. The X-Robots-Tag is included in this invisible conversation.
2. Meta Robots vs. X-Robots-Tag
Why use the X-Robots-Tag if you already have the Meta Robots tag?
- Non-HTML Files: You cannot put a
<meta>tag inside a PDF document, an MP4 video, or a PNG image. If you want to prevent Google from indexing a sensitive PDF, the X-Robots-Tag is your only reliable method. - Global Rules: You can configure your server to apply the X-Robots-Tag to an entire directory or site simultaneously, which is often faster than editing the HTML of thousands of pages individually.
3. Code Examples (Apache & Nginx)
To implement the X-Robots-Tag, you need to modify your server configuration files.
Apache (.htaccess)
To prevent search engines from indexing any PDF files on your site, you would add this to your
.htaccess file:
<FilesMatch "\.(pdf)$">
Header set X-Robots-Tag "noindex, noarchive"
</FilesMatch>Nginx (nginx.conf)
For an Nginx server, you would add the following to your site's configuration block:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, noarchive";
}4. Core Directives
The X-Robots-Tag accepts the exact same directives as the Meta Robots tag:
noindex: Do not index this file in search results.nofollow: Do not follow any links contained within this file (e.g., links inside a PDF).noarchive: Do not keep a cached copy of the file. Google removed the old “Cached” link from the search results in 2024, so the directive now governs archiving only.nosnippet: Do not show a text snippet in the search results for this file.
Targeting Specific Bots: You can also target specific crawlers by putting their name before the directive.
Header set X-Robots-Tag "googlebot: noarchive"
Header set X-Robots-Tag "bingbot: noindex"5. How to Check the X-Robots-Tag
Because HTTP headers are invisible on the actual page, they can be tricky to spot. There are two primary ways to check them:
- Browser DevTools: Right-click the page > Inspect > Network Tab. Refresh the page, click on the file you want to inspect, and look under the "Response Headers" section.
- Rank-O-Saur: The easiest method! Rank-O-Saur automatically intercepts HTTP headers. If an X-Robots-Tag is present and blocking indexation, the extension icon will immediately warn you, and the details will be visible in the "Overview" tab.
6. Common Mistakes to Avoid
The robots.txt Trap: Just like the Meta Robots tag, if you block a file in your
robots.txt file, search engines will never see your X-Robots-Tag!
Because they aren't allowed to crawl the URL, they never request the header. If you want to de-index
a file, make sure it is not blocked in robots.txt.
- Conflicting Signals: If you use both a Meta Robots tag (
index) in the HTML and an X-Robots-Tag (noindex) in the header, search engines will generally obey the most restrictive directive (in this case,noindex). - Syntax Errors: A mistyped server configuration file can take down your entire
website with a 500 Internal Server Error. Always back up your
.htaccessornginx.confbefore making changes.
7. Quick reference: syntax by server
The directive travels in the HTTP response header, so it works for any file type. The values are the same ones the meta robots tag accepts.
| Context | Example | Applies to |
|---|---|---|
| Apache, .htaccess | Header set X-Robots-Tag "noindex, nofollow" | All matching files, typically via FilesMatch |
| Nginx | add_header X-Robots-Tag "noindex"; | A location block or a specific file type |
| Crawler-specific | X-Robots-Tag: googlebot: noindex | One named crawler only |
| Typical use | PDFs, images, spreadsheets | Files with no head section to carry a meta tag |
8. Frequently asked questions
When should I use the X-Robots-Tag instead of a meta robots tag?
Whenever the resource is not HTML. PDFs, images, spreadsheets and video files have no head section to hold a meta tag, so the HTTP header is the only way to control their indexing. It is also the practical choice for applying a rule to many files at once.
Can I use the X-Robots-Tag and a meta robots tag together?
Yes, but be deliberate. If the two disagree, search engines follow the more restrictive instruction — so a header saying noindex overrides an HTML tag saying index. Conflicting directives across the two layers are a common cause of pages vanishing unexpectedly.
How do I verify that the header is actually being sent?
Request the URL and inspect the response headers, for example with curl -I or the Network tab in your browser's developer tools. Checking the rendered page is not enough, because the directive never appears in the HTML source.
Does the X-Robots-Tag work for any file type?
It works for any resource delivered over HTTP, because the directive travels in the response header rather than the file itself. That makes it applicable to formats that have no way to carry metadata internally.
Read next
