Can This URL Be Indexed?

Why a Page Can Be Crawled But Not Indexed

Crawling and indexing are two separate steps. Crawling is a search engine requesting your URL and reading the response. Indexing is a later, independent decision to store that page and let it compete for search results. A URL can pass every crawl check and still never be indexed.

The mechanical blockers are the easy part. A noindex directive — in the robots meta tag or the X-Robots-Tag header — removes the page from the index outright. A canonical tag pointing at another URL asks search engines to index that other page instead, which is exactly what you want for duplicates and exactly what you do not want on an original page. Redirects hand indexing to the destination URL, and 404, 410 or 5xx responses remove the page from consideration entirely. Robots.txt restrictions are subtler: they block crawling, so the page cannot be read at all, yet the URL may still surface from external links.

Beyond the mechanics sit quality and relevance signals. Google routinely crawls pages it then chooses not to index because they duplicate existing content, add little that is new, or serve no clear search intent. That is why a clean technical report is a necessary condition, not a sufficient one: technical checks alone never guarantee indexing.

To dig further, confirm crawler access with the robots.txt checker, review page-level tags with the SEO meta tag checker, validate discovery with the XML sitemap checker, or run the full crawlability checker.

Frequently Asked Questions

What does indexable mean?

An indexable URL is one that search engines are technically allowed to store and show in search results: it returns a successful response, is not blocked by a noindex directive, and does not point its canonical at another page. Being indexable does not mean a page will be indexed — that remains the search engine's decision.

What is the difference between crawling and indexing?

Crawling is a search engine fetching the URL. Indexing is it deciding to store the page and consider it for search results. A page can be crawled and then not indexed, and a URL blocked from crawling can still be indexed from external links.

How do I check if a page has noindex?

Look for a <meta name="robots" content="noindex"> tag in the page head and for an X-Robots-Tag: noindex HTTP response header. This tool checks both, because either one alone is enough to keep the page out of the index.

Can robots.txt prevent indexing?

No, not reliably. Robots.txt stops crawlers from requesting a URL, but the URL can still be indexed if other pages link to it — with no snippet, because the page was never read. Worse, a blocked page's noindex tag can never be seen. To remove a page, keep it crawlable and use noindex.

What does a canonical tag do?

A canonical tag tells search engines which URL is the preferred version of a set of similar pages. If a page's canonical points elsewhere, that page may be treated as a duplicate and the other URL indexed instead. Canonicals are a strong hint, not a directive.

Why is my page crawled but not indexed?

Usually because the search engine judged the page as thin, near-duplicate, or not worth storing, or because a canonical, redirect, or noindex signal pointed away from it. Technical checks find the mechanical blockers; content quality and relevance explain the rest.

Does a 200 status guarantee indexing?

No. A 200 response only means the page loaded successfully. Search engines still evaluate directives, duplication, and quality before deciding whether to index a URL.

How can I check indexing in Google Search Console?

Open Search Console for your verified property, paste the URL into the URL Inspection bar at the top, and read the coverage report. That is the only way to see whether Google has actually indexed a URL — this free tool checks publicly visible technical signals only.