In Google Search Console, many online shops show a long list under “Crawled - currently not indexed”. The status means that Google fetched the page but did not add it to the index (Search Console Help). Typically, the list contains categories with few products, variants with their own URL and product pages with manufacturer text. This guide shows how to sort the affected URLs by page type, narrow down the cause for each group and decide what to improve, merge or deliberately exclude. If you would rather not do the analysis yourself, it is part of the technical audit in our search engine optimization for shops.
What “Crawled - currently not indexed” means
The page indexing report splits the known URLs of a property into indexed and non-indexed pages and gives a reason for the second group. For “Crawled - currently not indexed”, the description is brief: the page was crawled by Google but not indexed. It may or may not be indexed in the future, and there is no need to resubmit the URL for crawling (Search Console Help). Technically, this is not an error: the server responded, Google received the content and decided against including it.
Google states explicitly in its crawl budget documentation that a fetch does not automatically lead to indexing: after crawling, each page must be evaluated, consolidated and assessed to determine its suitability for the index (Google Crawling Infrastructure). A 200 status code is no free pass either, and there is no promise of inclusion (Google Crawling Infrastructure).
For Google Search, not every page that is crawled will necessarily be indexed.
Google Crawling Infrastructure, Crawl Budget Management
The list is therefore not an error log to be ticked off but feedback on the quality and clarity of the pages. Google itself advises not to assume that every URL on a site is indexed, because some may be duplicates or contain no meaningful information (Search Console Help). What matters is whether important pages are among them: main categories, high-revenue products, brand and guide pages.
Crawled or discovered: two statuses, two different jobs
The same report lists a reason with an almost identical name. With “Discovered - currently not indexed”, Google knows the URL but has not fetched it yet: typically, crawling was expected to overload the site and was rescheduled, and the last crawl date stays empty (Search Console Help). The cause lies in fetching, for example too many URLs, slow responses or parameter spaces from filters. How to control this is described in our article on filter URLs and crawl budget.
Google addresses its crawl budget guidance mainly to sites with more than 1 million pages and weekly changing content, to sites with more than 10,000 pages and daily changes, and to sites with a large portion of their URLs under “Discovered” (Google Crawling Infrastructure). According to Google, these numbers are rough estimates, not exact thresholds (Google Crawling Infrastructure). With “Crawled”, by contrast, the fetch has happened: changes to crawling do not get such a page into the index; what matters is the content and the signals of the page itself.
| Characteristic | Discovered - currently not indexed | Crawled - currently not indexed |
|---|---|---|
| Fetch status | URL known, not fetched yet | page fetched and evaluated |
| Last crawl date in the report | empty | filled in |
| Typical cause in a shop | too many URLs, parameter spaces, server load | thin or duplicate content, unclear canonicalisation |
| First lever | URL inventory and response times | content, canonical, internal linking |
How Google decides between crawling and indexing
After fetching, Google analyses text, key tags such as the title element, alt attributes, images and videos (Google Search Central). In doing so, Google also determines whether a page is a duplicate or canonical. Pages with the same or very similar main content are grouped into a cluster, and the page that is objectively the most complete and useful for search users becomes canonical (Google Search Central). The others count as duplicates and are crawled less often.
According to Google, your own canonical tag is a hint and not a rule. Google evaluates further signals, including HTTP or HTTPS, redirects, presence in the sitemap and rel="canonical" annotations, and may choose differently from the site owner (Google Search Central). If the sitemap lists the variant while the canonical points to the main variant, for example, the outcome becomes hard to predict.
As common indexing problems, Google names low-quality content, robots meta rules and a site design that makes indexing difficult (Google Search Central). A noindex rule appears in the report under its own reason, “URL marked ‘noindex’”. For “Crawled - currently not indexed”, quality and structure of the page usually remain.
For Google Search, a 2xx status code does not mean that indexing worked (Google Crawling Infrastructure). If the content suggests an error or an empty page, Search Console reports a soft 404 (Google Crawling Infrastructure). Monitoring that only checks status codes sees neither.
Which shop pages are typically affected
The example list in the report shows at most 1,000 rows per reason and does not necessarily include all affected URLs (Search Console Help). That is enough for the analysis if you do not read the list URL by URL but sort it by pattern. In shops, this usually yields a few groups, recognizable by path, parameters or page template:
Thin categories
Subcategories with very few items, no text of their own and almost the same list as the parent category.
Variants with their own URL
Color or size variants under their own URL that differ in only one attribute.
Filter and sort URLs
Crawled combinations that show the same product list in a different order.
Paginated series
Page 2 and onwards of a category without its own canonical or reachable only via scripts.
Manufacturer text
Product pages with a description copied word for word from the data sheet that appears at many retailers.
Almost empty pages
Categories without items in stock or product pages with nothing but a “no longer available” note.
The order of work should not follow the size of a group but the value of its pages: a main category that is not indexed weighs more than a long series of sort URLs that do not belong in the index anyway.
Thin categories: when a page carries nothing of its own
In many shops, every level in the category tree automatically gets a URL, even if it holds only a handful of items that appear in the parent category as well. Among the questions Google offers for assessing content quality are whether the content provides substantial value compared to other pages in search results and whether it contains substantial new information rather than just copied or rewritten content (Google Search Central).
A subcategory whose product list can also be seen on the parent page answers both questions with no. An introduction that repeats in every category with a swapped term changes little. How to build category pages from real data instead of templates is described in our article on programmatic SEO for category pages. Essentially, there are three options:
- Merge: the category is folded into its parent, and the old URL redirects there with a 301. The selection remains available as a filter.
- Improve: the category gets content that only it carries: buying guidance, areas of use, a comparison of key attributes. This requires a range that justifies a page of its own.
- Exclude on purpose: if the category is needed for navigation but has no search value of its own, noindex keeps it out of the index. It then does not belong in the sitemap.
When improving categories, product data counts for more than running text: well-maintained attributes can be used for comparisons, filter guidance and buying advice. Where attributes are missing, AI-assisted data enrichment can derive them from existing data and images. The goal is more substance per page, not just more text.
Manufacturer text: when many retailers say the same thing
If many retailers list a product with the same description from the data sheet, Google receives almost the same main content everywhere. In its spam policies, Google names as an example without added value affiliate content whose product descriptions are copied directly from the original merchant without adding original content (Google Search Central). A shop is not an affiliate site, but the pattern is similar. The remedy is information only your own shop has: experience from advice and service, application examples, notes on accessories, your own photos. It pays to maintain this centrally, for example through a connection to a PIM system.
Variants and competing canonicals
If the shop delivers every color or size under its own URL, pages with almost identical main content are created. Google selects one of them as canonical, and the others appear in the report as, for example, “Duplicate without user-selected canonical” or “Duplicate, Google chose different canonical than user” (Search Console Help). Where Google neither classifies a variant as a duplicate nor sees value of its own, experience shows it often remains under “Crawled - currently not indexed”.
For variants on a single page, switched via parameters, Google expects exactly one canonical URL for the product group, usually the base URL without a preselected variant (Google Search Central). For variants on separate pages, by contrast, there is no single canonical URL for the group (Google Search Central). ProductGroup markup also tells Google which products belong together. When a variant deserves its own indexable page is covered in our article on product variants and duplicate content.
<!-- Variant selection on one page: parameter URLs point to the base URL -->
<!-- requested: https://shop.example/winter-jacket-alpine?colour=blue -->
<link rel="canonical" href="https://shop.example/winter-jacket-alpine">
<!-- Paginated category: every page points to itself -->
<!-- requested: https://shop.example/jackets?page=2 -->
<link rel="canonical" href="https://shop.example/jackets?page=2">The annotation is widespread: 67 percent of the mobile pages analyzed by HTTP Archive carry a canonical tag (Web Almanac 2025). The errors lie in the details. On 3.02 percent of the mobile sites analyzed, the canonical only changes during rendering, that is after JavaScript runs (Web Almanac 2025). If the theme switches variants via script, it is therefore worth comparing the delivered and the rendered HTML.
Paginated series, filters and sort orders
For paginated categories, Google recommends giving each page a unique URL, for example with ?page=n, not using the first page as the canonical page of the whole series and giving each page its own canonical URL (Google Search Central). The pages should be linked in sequence with normal HTML links, and Google ignores fragment identifiers after the hash sign (Google Search Central). If page 2 and onwards point to page 1 via canonical, the shop declares them duplicates of the first page, even though they show different products.
Filter and sort URLs are the opposite case. To keep different versions of the same result list out of the index, Google names two options: excluding unwanted URLs with noindex or preventing the crawling of certain URL patterns with robots.txt (Google Search Central). Which option fits which facet is described in detail in the article on controlling filter URLs.
Almost empty pages and sold-out items
A category without items in stock shows “No products found” and still returns status 200. If the content of a page with a 2xx status suggests an error or an empty page, Search Console reports a soft 404 (Google Crawling Infrastructure). Pages just above that threshold, for example with a single item and no further content, are typical candidates for “Crawled - currently not indexed”.
For permanently removed pages, Google recommends the status codes 404 or 410, and a 404 is a strong signal not to crawl the URL again (Google Crawling Infrastructure). If there is a successor, a redirect is better. Items that are temporarily sold out usually stay online, with a clear availability statement. The states in between are covered in our article on unavailable products.
Technical cross-check: does Google see what customers see?
Before any content revision, it is worth checking whether Google receives the content completely. In the live test, the URL Inspection tool shows the tested page including a screenshot and HTTP response headers (Search Console Help). A positive result only means that the URL can be crawled and parsed (Search Console Help). According to Google, a page must, among other things, be standalone or selected as the canonical version, and its quality must be high enough to be worth indexing (Search Console Help). We regularly come across three technical causes:
- Content only after a user action: Google's crawlers do not click buttons and generally do not trigger JavaScript functions that require a user action (Google Search Central). Lists behind “Load more” therefore also need linked pages.
- Rendering queue: pages with status 200 wait to be rendered, sometimes only seconds, sometimes longer (Google Search Central). Important content and the canonical therefore belong in the delivered HTML.
- Very large pages: Googlebot crawls the first 2 MB of a supported file type (Google Search Central), measured uncompressed. Category pages with large embedded data blocks should stay well below that.
Whether Googlebot actually fetches the pages and which status code it receives is shown more reliably by the access log than by any sample. How to analyze it is explained in our article on log file analysis for shops.
Internal linking and sitemap as signals of importance
Google infers the relative importance of a page from linking, among other things from the number of links needed to reach it and the number of links pointing to it (Google Search Central). As a rule of thumb, the more a page is linked within the site, the higher its relative importance (Google Search Central). Googlebot generally does not use search fields when crawling (Google Search Central). Products that can only be found via the internal search are therefore missing as a target. How to plan linking is described in our article on internal linking in the shop.
The sitemap is the second signal. A single sitemap is limited to 50 MB uncompressed or 50,000 URLs; larger inventories have to be split into several sitemaps, which can optionally be submitted together via a sitemap index file (Google Search Central). Because presence in the sitemap is one of the canonicalisation signals, only canonical, indexable URLs belong in it: no variants with a canonical pointing to the main variant, no filter URLs, no pages with noindex.
From report to decision in five steps
The work does not start with individual URLs but with groups, and it does not end with the fix but with the check in the report. The framework for the other audit areas is provided by our guide to the SEO audit for online shops.
- Export and group: export the example URLs for “Crawled - currently not indexed” and sort them by page type: category, variant, filter, paginated page, product, content page.
- Check samples: inspect a few URLs per group in the URL Inspection tool: selected canonical, tested page, last crawl. Live tests are subject to a daily limit per property (Search Console Help).
- Decide per group: improve, merge or exclude on purpose, judged by whether the page answers a search query better than pages already in the index.
- Align signals: after the change, content, canonical, internal links and sitemap must say the same thing.
- Validate and monitor: where the report offers it, start the validation of the fix. It typically takes up to about two weeks, sometimes considerably longer (Search Console Help).
Decision aid by page type
| Page type | Typical finding | Decision |
|---|---|---|
| Category with very few products | hardly any own text, list as in the parent category | merge or improve in a targeted way |
| Variant with its own URL | content almost identical to the main variant | canonical to the base URL or develop into a page of its own |
| Filter or sort URL | same products in a different order | keep out of the index, do not link internally |
| Page 2 and onwards | canonical points to page 1 | own canonical, links in sequence |
| Product with manufacturer text | description identical to other retailers | add own information on use, accessories and service |
| Permanently removed item | status 200 with a notice | 404 or 410, redirect if there is a successor |
For individual pages that have just been revised, “Request indexing” in the URL Inspection tool can be useful. For many pages it is not suitable: there is a limit on the number of indexing requests per day, and a request does not mean the page will appear in the index (Search Console Help). For larger volumes, an up-to-date sitemap with lastmod values is the intended route.
What is realistic: not every URL belongs in the index
According to Google, most sites have at least a few unindexed pages. All important pages should be indexed, while duplicate URLs should not (Search Console Help). The goal is therefore not an empty list but a comprehensible one: every URL on it is excluded on purpose or on the to-do list.
Instead of the total number of non-indexed URLs, keep a list of the pages that should be indexed: main categories, brands, high-revenue products, guides. Their share in the index is the metric. The total may even rise, for example when filter URLs are cleanly excluded.
Time is part of it, too: it can take about a week for Google to crawl and index a new page (Search Console Help). Seasonal pages should therefore go live early, be linked internally and be listed in the sitemap. A fixed URL that is reused every year works better than a promotion page that only appears the day before.
Where the causes sit in the shop system
Many patterns do not arise from individual pages but from settings and templates of the shop system: how SEO URLs for variants are built, whether empty categories are delivered, where the canonical sits in the template and whether a script changes it. If the number of indexing errors rises, a change to the template can be the cause, according to Google (Search Console Help). Conversely, a fix there applies to all affected pages at once. For Shopware, we check the variant and SEO URL configuration, theme templates and custom extensions in the page head, work that is part of everyday life in our Shopware agency.
A good time is an update that is due anyway. Anyone changing the major version because of the end of support for Shopware 6.5 tests templates and extensions anyway and can check the indexing signals in the staging environment at the same time.
Indexing as part of ongoing shop maintenance
The status is rarely settled with a single measure. New categories, range changes and updates create new pages and new candidates for the list. A fixed rhythm makes sense: review the report monthly for new patterns, compare the list of important pages and test changes to the system against the indexing signals before they go live.
Shop check
A free, personal initial assessment that also covers indexing and crawling: request a shop check.
Technical SEO audit
Analysis of the indexing report, grouping by page type, checks of canonicals, sitemap and rendering as part of our search engine optimization.
Implementation in the system
Changes to templates, SEO URLs and extensions as well as better category and product data, with your team or by us.
This article is based on Google's Search Console Help (page indexing report, URL Inspection tool), the Google Crawling Infrastructure documentation (crawl budget management, HTTP status codes), Google Search Central (how Google Search works, URL canonicalisation, sitemaps, Googlebot, pagination, navigation structure for e-commerce sites, structured data for product variants, helpful content, spam policies, JavaScript SEO) and the SEO chapter of the HTTP Archive Web Almanac 2025 (July 2025 crawl). Retrieved on 5 October 2026; Google's documentation is updated continuously.
Not necessarily. Google fetched the page but did not include it, and it may be indexed in the future (Search Console Help). The status becomes a problem when important pages are affected, such as main categories or high-revenue products. For filter URLs, it is usually harmless.
After a genuine revision of individual pages, this can make sense; for unchanged pages, experience shows it achieves little. According to Google, URLs with this status do not need to be resubmitted, the number of requests per day is limited, and a request does not automatically lead to inclusion (Search Console Help).
With “Discovered”, Google has not fetched the URL yet and has rescheduled the crawl, typically because fetching it was expected to overload the site (Search Console Help). There, you start with the number of URLs and server performance. With “Crawled”, the work is about content, canonicalisation and linking.
Usually not. According to Google, important pages should be indexed, duplicate URLs should not (Search Console Help). Variants that differ in only one attribute are usually better represented by the base URL, very small categories often better by the parent category.
According to Google, validating a fix typically takes up to about two weeks, sometimes considerably longer (Search Console Help). Even without a validation, Google updates the numbers as soon as affected pages are crawled again. Plan the analysis over several weeks.
We analyze the report, group the URLs by page type, check samples technically and derive a prioritized list of measures. If you wish, we also take care of the implementation in templates, product data and linking. You can arrange an initial appointment via our contact page.