AI Crawler Control: A Valid TDM Opt-Out in 2026

Almost every SEO discussion currently revolves around the same question: how do I get found by AI systems? This article turns the perspective around without playing it off against visibility. Under Article 4 of the DSM Directive (EU) 2019/790 and Section 44b of the German Copyright Act (UrhG), text and data mining on lawfully accessible works is generally permitted – unless the rightholder has reserved the use, and for content available online that reservation has to be in machine-readable form (EUR-Lex; gesetze-im-internet.de). Anyone who declares nothing effectively permits everything. Anyone who blocks every bot across the board loses answer quality and referral traffic from generative search. The real task is therefore differentiation: separate training crawlers from retrieval crawlers, place the reservation properly in three locations, and enforce it technically where robots.txt reaches its limits. This guide covers the legal framework, the technical implementation and a differentiation strategy that fits your AI visibility and your search engine optimization.

Text and data mining is allowed unless you object

Section 44b UrhG defines text and data mining as the automated analysis of single or multiple digital or digitised works in order to obtain information, in particular about patterns, trends and correlations (gesetze-im-internet.de). Paragraph 2 explicitly permits reproductions of lawfully accessible works for this purpose. Paragraph 3 is the decisive one: such uses are only permitted if the rightholder has not reserved them – and for works available online, a reservation is only effective if it is made in machine-readable form (gesetze-im-internet.de). The German provision implements Article 4 of Directive (EU) 2019/790 of 17 April 2019 (EUR-Lex).

This inverts the familiar copyright logic. Normally: use only with permission. For text and data mining: use is permitted until you object. Recital 18 of the DSM Directive clarifies that for content made publicly available online it is appropriate to reserve rights by machine-readable means – explicitly including metadata and terms and conditions of a website or a service (EUR-Lex). That is exactly where the three-level structure described below comes from. For anyone running an online shop this is not an academic question: product descriptions, guides, reviews and original product photography are usually copyright-relevant investments.

At European level the reservation is now also anchored in regulatory law. Article 53(1)(c) of the AI Act (EU) 2024/1689 obliges providers of general-purpose AI models to put in place a copyright policy and to identify and comply with rights reservations expressed pursuant to Article 4(3) of the DSM Directive (European Commission). The Code of Practice for general-purpose AI models published on 10 July 2025 makes this concrete: signatories commit to using crawlers that read and follow rights reservations expressed via the Robots Exclusion Protocol, and to consider further appropriate machine-readable protocols (European Commission).

Telling training crawlers, retrieval crawlers and agent fetchers apart

The umbrella term AI crawler hides the fact that very different access patterns sit behind it. Google's crawler overview distinguishes three categories: common crawlers used for products such as Google Search, which respect the robots.txt rules, special-case crawlers used by products where there is an agreement between the crawled site and the product, and user-triggered fetchers, that is, fetches initiated by an end user (Google Search Central). Mixing these levels inevitably leads to blunt decisions.

How cleanly this separation can work technically is shown by the Google-Extended token. According to the documentation it is a standalone product token that publishers use to manage whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI. Explicitly: the token does not impact a site's inclusion in Google Search nor is it used as a ranking signal (Google Search Central). You can therefore restrict AI training without touching your organic visibility – a pattern worth transferring to other providers wherever they document separate tokens.

Placing the reservation in three effective locations

Each of the three levels has its own weakness: robots.txt is widely adopted but copyright-agnostic. Terms of use are legally precise but hard to evaluate automatically. TDMRep is machine-readable in the narrow sense but not yet an approved standard. Only together do they form a declaration that crawlers can read and courts can weigh. Implementation belongs in the server configuration and therefore in your hosting.

Level 1: robots.txt with differentiated tokens

robots.txt is the channel that nearly every serious crawler reads. The important point is not to work with User-agent: * and Disallow: /, but to decide per product token. The following pattern leaves classic search untouched, reserves editorial content and original images against AI training, and keeps everything else open:

# 1) Leave classic search untouched
User-agent: Googlebot
Allow: /

# 2) AI training: reserve editorial content and own images
User-agent: Google-Extended
Disallow: /guides/
Disallow: /magazine/
Disallow: /reviews/
Disallow: /media/product-photos/
Allow: /

# 3) Observe unknown collectors first, do not block blindly
User-agent: *
Disallow: /cart/
Disallow: /search/
Disallow: /account/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Level 2: terms of use as the legal anchor

Recital 18 of the DSM Directive explicitly names the terms and conditions of a website as a possible means of reserving rights (EUR-Lex). On their own they are not sufficiently machine-readable, but they carry the substantive precision that robots.txt and TDMRep cannot provide. A robust clause should cover these points:

• An explicit reservation under Section 44b(3) UrhG and Article 4(3) of Directive (EU) 2019/790
• A clear description of the content covered: text, images, video, reviews, structured data
• A delimitation of permitted use: indexing for search services and answering specific queries with source attribution remain allowed
• A pointer to the machine-readable equivalent: robots.txt and /.well-known/tdmrep.json
• A contact route for licensing enquiries naming the responsible function in the company
• A date and version, so the state at the time of a crawl remains verifiable

How heavily this channel is used is shown by a large longitudinal study: in an audit of roughly 14,000 web domains underlying common training corpora, 45 percent of the C4 corpus was restricted through terms of service (Data Provenance Initiative). At the same time the study found substantial inconsistencies between what websites declare in their terms and what their robots.txt actually signals (Data Provenance Initiative). That inconsistency is precisely what you want to avoid.

Level 3: TDMRep as the machine-readable declaration

The TDM Reservation Protocol (TDMRep) was developed as a technical answer to Article 4 of the DSM Directive and has been available as a Final Community Group Report since 10 May 2024; it is not an official W3C standard (w3.org). The model consists of just two properties: tdm-reservation as a boolean (1 = rights reserved, 0 = not reserved) and tdm-policy as a URL where contact details and licensing conditions can be found (w3.org). There are three implementation routes – a file at /.well-known/tdmrep.json, HTTP response headers and HTML metadata (w3.org).

[
  {
    "location": "/",
    "tdm-reservation": 0
  },
  {
    "location": "/guides/",
    "tdm-reservation": 1,
    "tdm-policy": "https://www.example.com/tdm-policy.json"
  },
  {
    "location": "/media/product-photos/",
    "tdm-reservation": 1,
    "tdm-policy": "https://www.example.com/tdm-policy.json"
  }
]

# Option A: HTTP response headers (nginx, guide paths only)
location /guides/ {
    add_header tdm-reservation 1 always;
    add_header tdm-policy "https://www.example.com/tdm-policy.json" always;
}

<!-- Option B: HTML metadata in the head of the page -->
<meta name="tdm-reservation" content="1">
<meta name="tdm-policy" content="https://www.example.com/tdm-policy.json">

The limits of robots.txt: a request, not a lock

Since September 2022 the Robots Exclusion Protocol has been an IETF Proposed Standard as RFC 9309 (rfc-editor.org). The document specifies syntax, error handling and caching, extending the convention Martijn Koster introduced in 1994. It also sets technical limits, such as a parsing limit of at least 500 kibibytes per robots.txt (rfc-editor.org). If you maintain a very long file, put the most important groups near the top.

Crawlers should impose a parsing limit to protect their systems; that limit must be at least 500 kibibytes.

RFC 9309, Robots Exclusion Protocol (IETF)

Something else matters more, though: robots.txt is not access control. It is a statement by the operator about what an automatic client should do – not about what it technically can do. A URL excluded via Disallow remains retrievable over HTTP. Relying on it alone confuses a courtesy rule with a protection mechanism.

• Voluntary compliance: adherence depends on the client's behaviour; there is no technical enforcement
• Identity: the user agent is a freely chosen header and can be spoofed
• Granularity: path rules rarely map exactly onto content categories such as guide or review
• Latency: changes only take effect once the client re-fetches the file – RFC 9309 permits caching (rfc-editor.org)
• Reach: content distributed via feeds, marketplaces or partners escapes your robots.txt entirely

Analyses of the robots.txt landscape show that blocking is applied rather cautiously overall: in an analysis of roughly 140 million websites, block rates even for the best-known SEO crawlers were only 6.3 to 6.5 percent (Ahrefs). News publishers behave differently: at the end of 2023, 48 percent of the most-used news sites across ten countries blocked the crawler of a major AI provider and 24 percent blocked Google's AI token – in Germany the figure for the Google token was the highest at 60 percent (Reuters Institute). A longitudinal study of training corpora found that within a single year more than 5 percent of all tokens in C4 and roughly 28 percent of the most actively maintained sources became fully restricted (Data Provenance Initiative).

The reservation only bites with bot management

The legal reservation and its technical enforcement are two separate jobs. The reservation decides whether a use is permitted. Whether it actually happens is decided by your infrastructure. Four layers work, and they build on one another in the server configuration:

1. Identification: user-agent rules as a first step, but consistently combined with verification via reverse DNS or the provider's published IP ranges – Google documents this verification for its own crawlers explicitly (Google Search Central)
2. Rate limits: capping requests per minute per IP, subnet or autonomous system so that unknown collectors cannot overload the origin
3. Firewall rule set: targeted 403 or 429 responses for verified training crawlers on reserved paths, while product pages stay open
4. Decoupling: an upstream cache layer so crawler load does not hit database and application directly

The fourth point is often underestimated. Crawlers typically generate many requests on detail pages with a low cache hit rate. Our article on the HTTP cache rework in Shopware describes how to absorb that; for asynchronous background processing, see message queue and workers in production. Both reduce the likelihood that heavy bot load turns into a performance problem for real customers.

Server log analysis: who is really crawling, and what does it cost?

Before you write a single rule you need numbers. Analytics tools will not help: crawlers do not execute JavaScript and usually do not appear there. The reliable source is your web server's access logs. Four fields per request matter: timestamp, requested URL, user agent and bytes served, complemented by status code and response time.

1. Retain raw logs for at least 30 days so weekly patterns and campaign periods become visible
2. Group requests by user agent and sort them into four classes: search, retrieval, training, unknown
3. Spot-check claimed identities via reverse DNS and count spoofed entries separately
4. Sum bytes served and response times per class to quantify the real load
5. Compare referral traffic from AI surfaces to build a ratio of crawl effort to return
6. Repeat quarterly and document the changes

In projects we regularly see that a substantial share of bot load falls on paths that produce neither revenue nor visibility – filter combinations, sort parameters, print views (project experience). Those paths can usually be excluded without side effects and relieve the server noticeably before AI-specific rules are even discussed. Ongoing evaluation belongs in the routine operation of your hosting, not in a one-off project.

A differentiation strategy for shops: release what sells

For retailers the decision can be aligned to one simple question: does the content create demand, or is it the product itself? Product data, prices and availability create demand – this is where you want to be found, including by generative systems. Guides, reviews, application knowledge and original imagery, by contrast, are editorial work of your own that produces little return without a back-link.

This differentiation is not a convenience compromise but a consistent application of the law: Section 44b(3) UrhG does not require a blanket reservation, only a declared one (gesetze-im-internet.de). You may draw it as narrowly or as widely as your business model requires – as long as the declaration is machine-readable and free of contradictions. That is exactly what the TDMRep model with its path-based location entries is designed for (w3.org).

Licensing instead of a total block

The reservation is not an end in itself. The TDMRep model deliberately provides a second property, tdm-policy: a URL where contact information and the conditions for obtaining permission to mine the content are available (w3.org). Blocking alone forgoes a revenue channel; blocking while opening a licensing route turns a defensive measure into an offer.

• A lean policy page describing the reserved content classes and a contact route
• A statement of which uses remain permitted without a separate agreement – for example answers with visible source attribution
• A note on whether you can offer structured feeds or interfaces as licensed access
• A named responsibility inside the company so enquiries do not get lost in a shared inbox
• Regular updates, because providers, tokens and protocols keep changing

A reservation without a licensing route is a door without a bell. If you protect content, say in the same breath under which conditions using it would be possible.

XICTRON development team

Technically the effort is manageable: a static JSON file, a header in the server configuration, a paragraph in the terms of use and a policy page. The larger part of the work sits in mapping your shop's content – that is, deciding which paths hold which content class. That mapping combines well with a technical inventory in development and architecture.

Taking back control over AI crawlers

The legal position is clearer than it is often portrayed: without a machine-readable reservation, text and data mining on lawfully accessible works is permitted (gesetze-im-internet.de; EUR-Lex). What is currently unsettled is mainly the exact form a reservation must take – the Higher Regional Court of Hamburg admitted an appeal to the Federal Court of Justice and thereby kept the question open (Legal Tribune Online). Until then the pragmatic answer is to declare the reservation in robots.txt, terms of use and TDMRep at the same time, to enforce it technically only where it matters commercially, and to measure the effect in your server logs. If you would like your crawler traffic analysed, the reservation set properly and bot control configured so your product data stays visible, talk to us – our team supports analysis, implementation and operation.