Faceted navigation is indispensable in e-commerce and at the same time the single largest cause of crawling problems: around 50% of all crawling issues stem from faceted navigation, and together with action parameters they account for roughly 75% (Google Search Central). The reason is combinatorics. A catalogue holding a few thousand products generates millions of theoretically reachable URLs through filters, sorting and pagination. This article provides a decision matrix instead of guesswork: which facets deserve an indexable landing page backed by real search demand, which belong out of the crawl entirely, and why robots.txt disallow and noindex are not interchangeable in search engine optimisation.
How 5,000 products turn into millions of filter URLs
Faceted navigation means users narrow down an assortment across several independent dimensions - brand, colour, size, price range, rating, availability - and additionally control sorting, view and page size. For usability this is a clear win, and without those filters a large catalogue would be practically unusable. For search engines it creates a problem that does not scale linearly but multiplicatively: every combination usually produces its own retrievable URL.
A realistic calculation: a shop carries 5,000 products across 120 categories. It offers filters for 40 brands, 12 colours and 8 sizes, plus 5 sort orders. Those dimensions alone add up to 2.3 million URL variants. Add pagination averaging ten pages per result list and you arrive at roughly 23 million URLs - from a catalogue containing 5,000 products. The ratio of useful to useless URLs tips into the absurd.
This often means a very large number of possible combinations of filters, which translates to a very large number of possible URLs.
Google Search Central, Managing crawling of faceted navigation URLs
The second part of the mechanism is what really matters: a crawler cannot tell whether a new URL is useful. It has to fetch it to find out. Google describes exactly this effect: because faceted URLs appear novel, crawlers typically access a very large number of them before their downstream processes determine that the URLs are in fact useless (Google Search Central). The waste therefore happens before any quality assessment can take effect.
Once it discovers a set of URLs, it cannot make a decision about whether that URL space is good or not unless it crawled a large chunk of that URL space.
Illyes, Google Analyst, Search Off the Record (Google Search Central)
A common misconception says crawl budget only matters for marketplaces with millions of items. But combinatorics works independently of catalogue depth. A shop with 800 products and six filter dimensions comfortably produces six-figure URL volumes. What counts is not the number of products but the number of reachable URL variants - and that depends on the facet logic, not on the assortment. How closely filters and search interlock is covered in our article on search relevance and OpenSearch tuning in the shop.
Why Google names facets as the biggest crawl offender
Google quantified the causes of crawling problems and broke them down in its year-end crawling review for 2025. The result is unambiguous: faceted navigation is by far the largest single item, and together with action parameters it covers roughly three quarters of all reported issues (Google Search Central).
| Cause of crawling issues | Share (Google Search Central) |
|---|---|
| Faceted navigation | 50% |
| Action parameters (cart, sorting, print view) | 25% |
| Irrelevant parameters | 10% |
| Plugins and widgets | 5% |
| Other causes | 2% |
According to Google, two quantities determine how much gets crawled: the crawl capacity limit - the maximum number of simultaneous parallel connections plus the delay between fetches - and crawl demand, which derives from a site's size, update frequency, page quality and relevance (Google Search Central). Both are finite. Whatever flows into filter combinations is missing for new products, updated prices and revised category texts.
Google names concrete thresholds at which active crawl budget management pays off: large sites from around 1 million unique pages with content changing weekly, and medium or larger sites from around 10,000 unique pages with content changing daily (Google Search Central). As a third signal Google points to a high share of URLs carrying the status Discovered - currently not indexed in Search Console (Google Search Central). In practice that status is the most reliable early indicator: it shows Google knows the URLs but no longer spends fetches on them. How to evaluate such signals systematically is described in our SEO audit guide for online shops.
The decision matrix: which facet deserves a landing page
The most common mistake is a blanket decision: either let everything be indexed or block everything. Both waste potential. Some facets cover genuine, standalone search demand and deserve an indexable landing page. The vast majority cover none and belong out of the crawl. That assignment should not follow gut feeling but three testable criteria.
Real search demand
Is the combination actually searched for? Running shoes brand X yes, running shoes black size 44 sorted by price no. Evidence comes from keyword data and from analysing your internal site search.
Sufficient assortment depth
A landing page needs stable results. As a rule of thumb we work with a minimum product count per facet and an assortment that does not drop to zero every week (project experience).
Standalone content
Is there anything to say beyond the filtered list - an introduction, guidance, an FAQ? Without it the page stays a duplicate. The path there is described in programmatic SEO for category pages.
Applied to typical shop facets this yields a workable matrix. It is a starting point, not a dogma: in a fashion shop colour may well carry demand (black evening dress), in a spare parts shop hardly at all.
| Facet | Real search demand | Recommendation |
|---|---|---|
| Category (path URL) | High | Index, link from navigation |
| Brand within category | High | Indexable landing page as path URL |
| Size / fit | Medium | Index selectively, only with proven demand |
| Colour | Sector-dependent | Selective, disallow when in doubt |
| Price range | Low | Disallow |
| Rating / availability | None | Disallow |
| Sale and discount filters | Low | Disallow, run promotions on editorial pages |
| Two or more facets combined | None | Disallow |
| Sorting / view / page size | None | Disallow (action parameters) |
The sale filter deserves an extra remark: discount facets are unattractive from an SEO angle because they return different results every day - and they are legally delicate when price reductions are presented sloppily. Our article on reference prices and discount advertising under German price indication rules covers the requirements. Promotions belong on maintained editorial pages, not in a filter parameter.
When a facet deserves a landing page it should get a descriptive path URL: /running-shoes/brand-x/ instead of /running-shoes/?brand=brand-x. That brings three advantages: the page is unaffected by parameter disallow rules in robots.txt, it can be linked internally in a clean way, and it signals standalone relevance rather than a filter state. The parameter variant stays reserved for everything else.
For filter combinations without results Google explicitly recommends the HTTP status 404 rather than a redirect or a 200 page with a no-results message (Google Search Central). That feels counterintuitive but works: a 404 tells the crawler this branch of the URL space yields nothing. A 200 page without results tells it the opposite - and invites further crawling.
robots.txt or noindex: why this is not an either-or
This is where the costliest misconception about facets sits. robots.txt disallow and noindex are often treated as two routes to the same goal. But they act at different points of the processing chain. Disallow prevents crawling. Noindex prevents indexing - but requires the page to have been fetched first. That is precisely what makes noindex worthless for crawl budget.
Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag.
Google Search Central, Large site owner's guide to managing your crawl budget
The reverse holds as well: a disallow is not a ban on indexing. Google states this unmistakably in the robots.txt specification - the content of a disallowed page cannot be indexed, but the URL itself may still be indexed and shown in search results without a snippet (Google Search Central). Anyone who confuses the two tools produces exactly the outcome they wanted to avoid.
| Measure | Crawling | Indexing | Saves crawl budget |
|---|---|---|---|
| robots.txt disallow | Blocked | URL may appear without snippet | Yes |
| noindex (meta tag) | Allowed and required | Prevented | No |
| rel=canonical | Allowed | Signal, not a directive | Hardly, long term |
| rel=nofollow | Reduces discovery | No indexing signal | Only partly |
| Not linking the facet at all | No discovery | None | Yes |
The combination contradicts itself: if robots.txt forbids crawling, Google does not get to see the noindex in the source - the directive stays ineffective. For filter URLs that are already indexed there is therefore an order of operations: first serve noindex and let the pages be crawled until they drop out of the index, then set the disallow in robots.txt. Activating both at once freezes the status quo.
The blocking itself needs only a compact rule set. In robots.txt Google supports the wildcard * for any sequence of characters and $ for the end of a URL; the file is evaluated up to a size of 500 KiB, content beyond that is ignored, and Google generally caches it for up to 24 hours (Google Search Central). Changes therefore do not take effect immediately - something to plan for during rollouts.
user-agent: *
# Filter facets without their own search demand: do not crawl
disallow: /*?*color=
disallow: /*?*size=
disallow: /*?*price=
disallow: /*?*rating=
# Pure presentation parameters (sorting, view, page size)
disallow: /*?*sort=
disallow: /*?*view=
disallow: /*?*limit=
# Action parameters: not meant for crawlers
disallow: /*?*add-to-cart=
disallow: /*?*wishlist=
disallow: /*?*compare=
disallow: /*?*print=
# Indexable facets use path URLs and stay crawlable:
# /running-shoes/brand-x/ is unaffected by any rule aboveClean parameter conventions instead of organic chaos
If filter URLs are to stay crawlable they have to be machine-readable. Google states a clear convention: the industry standard URL parameter separator is the ampersand. Characters such as comma, semicolon and brackets are hard for crawlers to detect as parameter separators (Google Search Central). Inventing your own filter serialisation makes exactly the analysis harder that you depend on - and it renders robots.txt rules imprecise, because they hook onto parameter names.
# Recommended: standard separator, stable order, one parameter per facet
https://example.com/running-shoes/?brand=brand-x&color=black&size=44
# Problematic: comma, semicolon or brackets as separators
https://example.com/running-shoes/?filter=brand-x,black,44
https://example.com/running-shoes/?filter[brand]=brand-x;color=black
# Problematic: fragment instead of parameter - not crawled
https://example.com/running-shoes/#brand=brand-x&color=black
# Problematic: changing order creates duplicates of the same selection
https://example.com/running-shoes/?color=black&brand=brand-xThe last case is the underrated one: if the same filter selection produces different URLs depending on click order, the URL space multiplies once more - for no benefit whatsoever. One point deserves particular attention: URL fragments behind # are generally not supported by Google Search for crawling and indexing (Google Search Central). They work as a hiding place for filters, but not as carriers of indexable states.
- One parameter per facet, no nested collection parameters
- Fixed, logical parameter order - normalised server-side
- Consistent capitalisation and stable encoding of values
- No session IDs, tracking or random values in crawlable URLs
- Omit default values instead of writing them (
?sort=defaultcreates a duplicate) - Keep parameter names descriptive so
robots.txtrules can target them precisely
In practice server-side normalisation is the most effective single lever: the shop sorts parameters into a fixed order, strips default values and 301-redirects deviating spellings to the canonical form. The URL space shrinks before any crawler ever sees it. In Shopware projects we usually implement this in the routing and listing layer - more on our page about Shopware development.
Canonicals without self-contradiction
The rel=canonical element is popular because it feels like a solution. But it is a hint, not a directive - and it does not solve the crawl problem, at best the duplication problem. Google explicitly rates canonical and rel=nofollow as means of crawl control that are less effective in the long term than blocking crawling (Google Search Central). A canonical only takes effect after the page has been fetched - by which point the budget is already spent.
- Self-referencing on indexable pages: every landing page meant to rank points its canonical at itself - not at the parent category.
- Pagination canonicalises to itself: page 2 of a list points to page 2, not to page 1. A canonical from page 2 to page 1 declares content identical that is not.
- Filter states to the base category: only sensible where content is largely congruent - for example sort variants of the same list.
- Not in combination with disallow: a blocked URL is not fetched, so its canonical is not read at all - the directive goes nowhere.
- No contradiction with internal signals: a page that canonicalises to another one while being prominently linked and listed in the sitemap sends conflicting signals.
That last point is the most frequent finding in our audits: canonicals contradicted by a structure that says the opposite. Search engines weigh the overall picture. When sitemap, internal linking and canonical mark different pages as the right one, the search engine decides for itself - and often differently than intended (project experience).
Internal linking instead of parameter chaos
Crawlers discover URLs through links. That is the cheapest lever of all: what is not linked is typically not found in the first place - so it does not need to be blocked after the fact either. Google points out that rel=nofollow only works as a control mechanism if every single anchor pointing to a URL carries the attribute (Google Search Central). That is exactly where organically grown shops regularly fail: one forgotten link in a footer module or on a sitemap page is enough to reopen the whole branch.
- Link indexable facets as path URLs prominently - from category pages, navigation and editorial content.
- Serve non-indexable filters through controls that do not create crawlable
<a href>links. - Keep filter links out of XML sitemaps - the sitemap is a recommendation list, not a completeness list.
- Give pagination real links, but do not additionally vary page size as a parameter.
- After every template release, check whether new links point into blocked URL spaces.
Internal linking therefore decides twice over: it controls what gets discovered at all, and it distributes link equity to the pages meant to rank. How to build this systematically is described in our article on internal linking in online shops; the overarching thematic structure behind it is covered in the article on content cluster architecture.
Action parameters: cart, sorting, print view
The second largest item in Google's breakdown is action parameters at 25% (Google Search Central). These are URLs that do not represent content but trigger actions or vary presentation: add-to-cart, wishlist, comparison list, print view, sorting, view switching. They share one trait: they are worthless to search engines, and yet they are often embedded as ordinary links in the source.
Cart actions as GET links are especially unpleasant. A crawler follows them and thereby triggers a state change not intended for it - sessions, carts and statistics get distorted, and the shop produces load without any return. Actions that change state belong behind a POST form.
<!-- Problematic: crawlers follow the link and trigger the action -->
<a href="/cart/?add-to-cart=12345">Add to cart</a>
<!-- Better: state-changing action via POST, no crawlable GET link -->
<form method="post" action="/cart/add">
<input type="hidden" name="product" value="12345">
<button type="submit">Add to cart</button>
</form>For presentation parameters such as sorting and view the same logic applies in a milder form: they may exist as links because users need them, but they should be blocked via robots.txt. The effort is small, the effect large - a quarter of all crawling problems hangs on this category (Google Search Central). Cleanly separating state-changing from presentational requests is ultimately an implementation question; we handle it as standard in custom development projects.
AI crawlers make crawl waste economically more expensive
Until a few years ago crawl budget was mainly an SEO topic: wasted fetches cost visibility. By now they cost money on top. The share of automated traffic is high and its composition has shifted. Among verified bots in 2025, around 40% were search engine crawlers and already 20% AI crawlers, with a further 13% SEO bots (Cloudflare Radar). Measured against HTML requests, in early December 2025 around 44% came from non-AI bots while human traffic sat at roughly 47% and AI bots averaged 4.2% across the year; Googlebot alone accounted for 4.5% (Cloudflare Radar).
The decisive point, however, is not volume but the ratio of effort to return. Cloudflare measures this as the crawl-to-refer ratio: how many pages a provider crawls before sending back a single visitor. The spread is dramatic.
| Platform | Crawl-to-refer January 2025 | Crawl-to-refer July 2025 |
|---|---|---|
| Anthropic | 286,930:1 | 38,065:1 |
| OpenAI | 1,217:1 | 1,091:1 |
| Perplexity | 54:1 | 195:1 |
| 3.8:1 | 5.4:1 |
All figures cover the period from January to July 2025 (Cloudflare Radar). The reading is uncomfortable: where classic search crawls in the single digits before a visit comes back, AI crawlers sit three to five orders of magnitude above that. Add momentum: AI crawling triggered by concrete user actions grew more than 15-fold in 2025 (Cloudflare Radar). A bloated filter URL space is therefore grazed several times over - by search engines, by training crawlers and by user-driven AI fetches.
Every unnecessary filter URL is fetched not once but by many bot types - and each fetch creates database load, because filtered lists rarely come from cache. That feeds through to response times, which in turn affect Core Web Vitals and therefore real users. Shrinking the URL space lowers server costs and stabilises performance in one move. We advise on sizing the right infrastructure as part of hosting and maintenance.
The strategic conclusion is not to block AI crawlers across the board - that can cost visibility in generative answers. It is: reduce the crawlable space to what actually has substance, so that every fetch lands on a meaningful page. How to build visibility in AI-driven search surfaces is covered in the article on traffic strategy for Google AI Mode.
Implementation: an order of operations that works
Cleaning up facets is an intervention in indexing and should not be switched on in a single step. This sequence has proven itself in our projects (project experience):
- Inventory: run a full crawl of the shop and record the URL patterns that are actually reachable - not the intended ones, the real ones.
- Log file comparison: analyse which parameter patterns crawlers actually fetch and how much budget leaks there.
- Verify demand: for each facet, prove whether standalone search demand exists - cross-reference keyword data with internal search queries.
- Define the matrix: assign every facet to exactly one class - indexable path URL, blocked parameter or action parameter.
- Build indexable facets: path URLs, standalone content, internal linking, self-referencing canonical.
- Clean up already indexed filter URLs: serve
noindexfirst, wait for deindexing, then set therobots.txtdisallow. - Close the linking: make sure no internal links and no sitemap entries point into blocked spaces any more.
- Measure again: watch crawl stats and the Discovered - currently not indexed status over several weeks.
A realistic horizon is several weeks to months until crawl distribution and index inventory settle - depending on shop size, crawl frequency and starting point. A solid baseline belongs at the beginning. For shops with organically grown filter logic it also pays to look at the wider e-commerce architecture, because facets rarely emerge in isolation; the overarching frame is set by the ranking factors for online shops.
Filter URLs as a growth lever instead of ballast
Faceted navigation is not a problem to be locked away. It is a resource to be sorted. The catalogue already holds the answers to highly specific search queries - they are just buried in a URL space that is 99.9% noise. The task is to extract the signal: turn the few combinations with real demand into standalone landing pages and consistently take the rest out of the crawl.
The economic lever has grown, not shrunk. As long as only Googlebot crawled, a bloated URL space was mainly a missed opportunity. Today, with AI crawlers making up 20% of verified bot traffic (Cloudflare Radar) and crawl-to-refer ratios in the four- to five-digit range (Cloudflare Radar), it is a recurring cost item on top. Both effects point in the same direction - and both can be addressed with the same measure.
Set the decision matrix up cleanly once and you win twice: search engines find what counts faster, and the infrastructure carries less pointless load. We support you with analysis and implementation as part of our SEO services - from taking inventory of the URL space to rolling out the facet architecture. Get in touch if you would like to know how much crawl budget your shop currently loses in filter combinations.
This article is based on data from: Google Search Central - Managing crawling of faceted navigation URLs (combinatorics of filter URLs, overcrawling mechanics, robots.txt as the recommended route, canonical and nofollow as less effective means, separator convention, 404 for empty combinations), Google Search Central - Large site owner's guide to managing your crawl budget (thresholds of 1 million and 10,000 pages, crawl capacity limit and crawl demand, the warning against noindex for budget control), Google Search Central - robots.txt specification (URL indexing despite disallow, wildcards, 500 KiB size limit, 24 hour cache), Google Search Central - Search Off the Record, 2025 year-end crawling report (breakdown of crawling issues: 50% faceted navigation, 25% action parameters, 10% irrelevant parameters, 5% plugins and widgets, 2% other) and Cloudflare Radar (bot and AI crawler traffic, crawl-to-refer ratios January to July 2025, shares of verified bot traffic in 2025). The figures refer to their respective measurement periods and may vary by sector, shop size and point in time.
As a rule yes - but not without exception. The vast majority of filter combinations cover no standalone search demand and typically belong out of the crawl via robots.txt. Facets with proven demand, sufficient assortment depth and standalone content - usually brand within a category, and depending on the sector size or colour - typically benefit from an indexable landing page with a descriptive path URL. The decision should be made per facet on the basis of keyword data, not as a blanket rule.
Because noindex only takes effect after the page has been fetched. Google puts it explicitly: don't use noindex, as Google will still request the page and only drop it once it sees the meta tag (Google Search Central). The fetch - and therefore the cost in crawl capacity and server load - happens regardless. As a rule, only measures that prevent crawling itself save crawl budget: a robots.txt disallow or omitting crawlable links.
As orientation Google names large sites from around 1 million unique pages with weekly changing content, and medium or larger sites from around 10,000 unique pages with daily changing content (Google Search Central). The important nuance: what gets counted is reachable URLs, not products. A shop with a few thousand items can typically exceed that threshold through facet combinations alone. A practical indicator is a high share of URLs with the status Discovered - currently not indexed in Search Console (Google Search Central).
They do not reliably disappear from the index. According to the robots.txt specification, Google cannot index the content of a blocked page, but it may still index the URL itself and show it in search results without a snippet (Google Search Central). For URLs that are already indexed an order of operations is therefore advisable in our experience: first serve noindex and let them be crawled until they drop out of the index, and only then set the disallow. Activating both at once typically prevents exactly the deindexing that was intended.
Two data sources are enough for a first assessment. First, the crawl stats in Search Console: if they show many fetches on parameter URLs while new products are indexed only slowly, that is a clear signal. Second, the server log files: they show which URL patterns bots actually request. As a rule of thumb, based on experience: if a substantial share of bot fetches lands on URLs with filter, sort or action parameters, there is work to do. Our SEO audit guide describes the procedure in detail.
Precisely then - though for a different reason. In July 2025 crawl-to-refer ratios ranged from roughly 5:1 to over 38,000:1 depending on the provider (Cloudflare Radar), while AI crawlers accounted for around 20% of verified bot traffic (Cloudflare Radar). A bloated filter URL space therefore causes load that typically has no return against it. Blocking AI crawlers wholesale is rarely the best answer, because it can cost visibility in generative answers. As a rule it is more effective to reduce the crawlable space to pages with substance.