Indexing: From Discovery to Ranking
Indexing is where most SEO projects fail – not ranking. A page can be technically flawless, carry perfect meta tags and still never appear in the search results, because Google deliberately keeps it out of the index. This guide walks through the full chain from discovery to ranking, every status in the page indexing report, and what actually helps in each case.
1. Crawling, indexing and ranking are three different things
These terms get mixed up constantly, even though they describe independent stages. Each stage can fail on its own:
| Stage | What happens | Typical failure |
|---|---|---|
| Discovery | Google learns the URL exists | Orphan page, linked from nowhere |
| Crawling | Googlebot fetches the URL | robots.txt block, server errors, crawl budget |
| Rendering | JavaScript executes, the final DOM is built | Content only appears after user interaction |
| Indexing | Google decides whether the page enters the index | noindex, duplicate, insufficient quality |
| Ranking | The page gets ordered for queries | Missing relevance, strong competition |
The order is mandatory: no discovery, no crawling; no crawling, no indexing; no indexing, no ranking. Before you work on rankings, indexing has to be solid.
Important: being indexed is no guarantee of visibility, and not being indexed is not automatically a bug. On many sites 30 to 60 % of URLs should stay out of the index – filter pages, pagination, thank-you pages. The question is not "why isn't every URL indexed" but "are the right URLs indexed".
2. Discovery: how Google finds URLs at all
Google maintains a list of known URLs that grows from several sources – ordered by practical effectiveness:
- Internal links: the strongest source. A link from your navigation or from a frequently crawled category page usually leads to a fetch within hours.
- External links: equally effective, plus a popularity signal that raises crawl demand.
- XML sitemaps: reliable for discovery, but not a quality signal. A sitemap does not replace internal linking.
- Redirects: a
301makes the target known. - Manual submission: via URL Inspection in Search Console.
- Other traces: mentions in feeds,
hreflangreferences, canonical annotations on other pages, and sometimes even links inside JavaScript files.
An orphan page is the classic discovery failure: it exists, it is reachable, but not a single internal link points to it. Such URLs either never get found or get classified as unimportant.
3. From the queue to the fetch
Known URLs land in a prioritised queue. Priority comes mainly from internal and external linking, past change frequency and the estimated value of the URL. This is where crawl budget bites: on large sites a URL can sit in the queue for weeks with nothing technically wrong.
At fetch time, the status code decides what happens next:
200 OK– content moves on to processing.301/308– Google follows the redirect, indexes the target and consolidates signals there.302/307– Google follows it too, but initially treats it as temporary and keeps crawling the source URL.404/410– the URL drops out of the index after a few fetches.429/5xx– Google retries later. If the state persists for several days, the URL is removed from the index.401/403– no access, no indexing.
Details on every code in the URLs & status codes guide.
4. Rendering: why JavaScript pages are handled differently
After fetching the HTML, Google executes the page in the Web Rendering Service to obtain the final DOM. That is a separate processing step with its own queue. Google has put the median time between crawl and render at a few seconds – the idea that rendered JavaScript sits around for weeks is outdated. On very large sites or under load the step can still be delayed.
Three points remain practically relevant:
- Resources must not be blocked: a
Disallowon/assets/prevents the page from rendering fully. - Content must not require interaction: anything that loads only after a click, scroll or hover does not exist for Google.
- State changes need real URLs: content behind
#fragments is not addressable for indexing.
How to test this is covered in the rendering (SSR vs CSR) guide.
5. Index selection: why Google leaves pages out on purpose
The decisive and least understood step: Google does not index everything it can crawl. Selection follows estimated value. Typical reasons for exclusion:
- Duplicate: a very similar page is already indexed. Google picks one canonical version and folds the rest into it. See duplicate content.
- Too little substance: auto-generated pages, empty categories, tag archives with one post.
- No discernible need: the page does not serve any query better than results already indexed.
- Weak domain-level quality signal: on sites with very many thin URLs, the indexing rate drops for the whole domain.
The most common misreading: "Crawled – currently not indexed" is not a technical bug you can fix with a tag. It is a quality decision. Responding with repeated indexing requests instead of improving the content or the internal linking gets you nowhere.
6. The page indexing report, status by status
The report under Indexing → Pages splits all known URLs into indexed and not indexed with a reason. Here is the full picture with the correct response for each:
| Status | Meaning | Priority |
|---|---|---|
| Server error (5xx) | Server returned an error during the crawl | High – check logs, fix the cause |
| Redirect error | Chain too long, a loop or an empty target | High – reduce to one hop |
| Blocked by robots.txt | Crawling is disallowed | Only if unintended |
| Excluded by "noindex" tag | The directive is working as intended | Only if unintended |
| Soft 404 | Status 200, but the content reads as "not available" | High – return a real 404/410 or add content |
| Not found (404) | The URL no longer exists | Low if intended – otherwise redirect |
| Blocked due to 401/403 | Access denied | High if the page should be public |
| Crawled – currently not indexed | Fetched, but not selected for the index | Medium – quality and internal linking |
| Discovered – currently not indexed | Known, but not yet crawled | Medium – crawl budget and priority |
| Duplicate without user-selected canonical | Google detected a duplicate on its own | Medium – set a canonical or differentiate |
| Duplicate, Google chose a different canonical | Your canonical was overruled | Medium – align your signals |
| Alternate page with proper canonical tag | Everything is correct, signals are consolidated | None |
| Page with redirect | URL redirects, the target gets indexed | None |
| Indexed, though blocked by robots.txt | URL is in the index but its content is unreadable | High – unblock and apply noindex |
7. Fixing "Crawled – currently not indexed"
Google has seen the page and decided against including it. Work through this in order:
- Check whether the URL should be indexed at all. For filter, pagination or thank-you
pages this status is the desired outcome. Then document it deliberately with
noindexor a canonical. - Look for near-duplicates. If three category pages differ only in their title, Google picks one. Merge them or differentiate the content.
- Increase internal linking. URLs known only from the sitemap rarely get indexed. Link them from topically related, frequently crawled pages.
- Reduce click depth. Four clicks from the homepage is a sensible ceiling in practice for pages you want reliably indexed.
- Improve substance. Your own descriptions instead of manufacturer copy, answered questions, original data. See content & readability.
- Reduce the total number of thin URLs. This often works better than any single fix: removing 200,000 thin filter pages from the index improves the indexing rate of the pages that matter.
8. Fixing "Discovered – currently not indexed"
Here Googlebot has never visited the URL. That is a crawl problem, not a quality problem:
- Server performance: the most common cause on large sites. Check the average response time in crawl stats.
- Too many URLs at once: if a deployment creates 500,000 new URLs, Google works through the queue over weeks. Prioritise instead of submitting everything simultaneously.
- Missing internal links: sitemap-only URLs sit at the back of the queue.
- Visibly low value of the pattern: if 90 % of URLs matching a pattern (say
/tag/) were never indexed, Google deprioritises new URLs of the same pattern.
9. Duplicate statuses and canonical conflicts
When Google picks a canonical you did not choose, there is almost always a signal conflict. Google weighs
several signals together and your rel="canonical" is only one of them:
- internal linking (which variant do you link yourself?)
- inclusion in the XML sitemap
- redirects
hreflangannotations- external links to the variants
- HTTPS over HTTP and URL length as tiebreakers
The practical consequence: a rel="canonical" pointing to variant A while all internal links
and the sitemap point to variant B will be ignored. Align every signal before you start tinkering with the
canonical tag itself.
10. Speeding up indexing
There is no switch, but there is a clear order of effectiveness:
- Internal links from strong pages: the most effective and most underrated lever. A link from the homepage or a frequently crawled category page works faster than any submission.
- URL Inspection with "Request indexing": good for individual cases and for diagnosis. The daily allowance per property is small, and repeat requests for the same URL achieve nothing.
- A sitemap with accurate
lastmod: Google uses the value when it is trustworthy. Bulk-updating everything to "today" makes it worthless. - External links and mentions: raise crawl demand durably.
- Indexing API: permitted only for
JobPostingandBroadcastEvent. It is not intended for regular content. - IndexNow: supported by Bing and Yandex, not by Google.
11. Removing content from the index deliberately
| Goal | Correct method | Effect |
|---|---|---|
| Page stays online but should not rank | <meta name="robots" content="noindex"> |
Removed at the next crawl |
| Non-HTML file (PDF, image) | X-Robots-Tag in the HTTP header | Removed at the next crawl |
| Content permanently deleted | 410 Gone (or 404) |
Removed after a few crawls |
| Immediate removal required | Removals tool in Search Console | Temporary, roughly six months |
| Content must not be public at all | Server-side authentication | Permanent and reliable |
The classic mistake: blocking an already indexed URL in robots.txt so that it drops
out of the index. The opposite happens – Google can no longer read the noindex and
keeps the URL in the index without a snippet. Correct order: apply noindex, wait for
removal, then optionally block.
12. Diagnostics: which tools are actually useful
- URL Inspection: the most reliable single-URL answer. Shows the canonical Google chose, the crawl time, the rendered HTML and the indexing status.
- Page indexing report: the only dependable view of the whole set. Crucially: group by pattern instead of working through URL by URL.
- Performance report: a URL with impressions is definitely indexed. That is the fastest positive test.
site:queries: rough only. The result count is an estimate and fluctuates a lot. Usable to check whether a single URL exists, useless as a counting method.- Cached view: no longer available and no longer proof of indexing.
13. Indexing troubleshooting checklist
- URL Inspection: is the URL known? Which canonical did Google pick?
- Check the status code – does the URL return
200? - Test the URL against robots.txt.
- Check meta robots and X-Robots-Tag for
noindex– in the rendered HTML too. - Check the canonical: does it point at itself or elsewhere?
- Count internal links to the URL. Zero links answers most cases.
- Measure click depth from the homepage.
- Is the URL in the sitemap, and is it indexable there?
- Compare rendered HTML with the source – is the main content present?
- Search for near-duplicates within your own domain.
- Group the page indexing statuses by URL pattern.
- Check response time and error rate in crawl stats.
- Only then request indexing – and only once.
Pro tip: Rank-O-Saur surfaces meta robots, X-Robots-Tag, canonical and status code the moment you visit a page. That settles in seconds whether a page can be indexed at all, before you start wondering why it is not.
14. Frequently asked questions
How long does it take Google to index a new page?
On established sites with solid internal linking, usually a few hours to a few days. On new domains or weakly linked URLs it can take weeks. There is no guaranteed timeframe, and repeated indexing requests do not speed it up.
What is the difference between "Crawled" and "Discovered – currently not indexed"?
With "Crawled", Googlebot fetched the page and decided against including it, which is a quality matter. With "Discovered", Google only knows the URL and has not fetched it yet, which is a crawl capacity or priority matter.
Does every URL on my site need to be indexed?
No, and on most sites that is not even desirable. Filter combinations, pagination, internal search results and order confirmations do not belong in the index. What matters is the indexing rate of the pages meant to bring traffic.
Why does Google ignore my canonical tag?
Because the canonical is only one hint among several signals. If internal links, the sitemap, redirects or hreflang annotations point to a different variant, Google decides based on the majority of signals. Align every signal first.
Does the Indexing API help with regular pages?
No. The Google Indexing API is permitted only for JobPosting and BroadcastEvent content. It is not intended for other page types and offers no benefit there. IndexNow, in turn, is used by Bing and Yandex rather than Google.
How do I remove a page from Google's index quickly?
Apply a noindex and then request removal via the Removals tool in Search Console. The tool acts quickly but only temporarily, for about six months. The lasting effect comes from the noindex, a 410 or authentication.
Is a site: query a reliable indexing check?
Only partly. The displayed result count is an estimate and fluctuates noticeably. It works for checking whether a single URL is indexed, but for dependable numbers use the page indexing report in Search Console.
Can a page blocked by robots.txt end up in the index?
Yes. If enough links point to the URL, Google can index it without knowing the content – visible as the status "Indexed, though blocked by robots.txt". To remove it, the URL must be crawlable and serve a noindex.
Read next
