Between any two orders, a shop serves hundreds of requests that no human triggered. Some of them are wanted: search engines, your own monitoring, preview images for shared links. Others pull the catalogue page by page, put goods into carts they do not pay for, or try out credentials. In 2025, more than 53 percent of all web traffic was automated, up from 51 percent the year before (Thales Bad Bot Report 2026). This article shows how to measure bot traffic in an online shop in the first place, what gives a collector away in the server log, which steps exist between robots.txt and rate limiting, and where the line runs between steering traffic and shutting people out.
How much traffic actually comes from machines
One of the few publicly available series on bot share comes from the analysis of traffic passing through a globally deployed application protection network. For 2025 it reports that bots made up more than 53 percent of all web traffic, up from 51 percent the previous year, while human activity fell to 47 percent (Thales Bad Bot Report 2026). For 2024, the same analysis had already recorded that automated traffic surpassed human activity for the first time in a decade (Imperva Bad Bot Report 2025). None of this says much about a single shop: the series describes a global average across all industries and application types. It works as an order of magnitude for how much noise raw numbers can contain, not as an expected value for your own log.
Within that automation, both reports separate wanted from malicious bots. For 2025 they assign 40 percent of all traffic to the malicious group (Thales Bad Bot Report 2026); the previous year's report put the figure at 37 percent without tying it to a measurement year (Imperva Bad Bot Report 2025). The reference base matters here: in both cases the percentage refers to total measured traffic, not to the bot share. Subtracting one figure from the other to derive the share of benign automation produces a calculation, not one of the two published figures - and it should be labelled as such.
Both analyses are based on the traffic that a globally deployed application protection network sees - not on a sample of German online shops. The 2026 report evaluates the full year 2025 for this purpose (Thales Bad Bot Report 2026). No comparable public series exists for German retail. Anyone who wants to know their own share measures it in their own log; anything else is an assumption borrowed from a foreign population.
What has changed is less the volume than the character. For 2025 the analysis reports that AI-driven bot attacks surged 12.5-fold compared with the previous year, and describes AI agents as a third traffic category alongside wanted and malicious bots (Thales Bad Bot Report 2026). The target is shifting in parallel: 27 percent of bot attacks in 2025 were aimed at interfaces, where the user interface is skipped and the system behind it is addressed directly (Thales Bad Bot Report 2026). For shop operations that typically means the question is no longer whether a request comes from a machine, but what that machine does. Matching against a list of known identifiers falls short of that.
What collectors actually do to a shop
Harvesting prices and the catalogue
The case that draws the least attention is also one of the most common: a program requests category pages, filter combinations and detail pages one after another and writes price, availability and description into a database of its own. The OWASP taxonomy of automated threats lists this case as a separate event under the identifier OAT-011 (OWASP Automated Threats). Nothing of it shows up in the order system. In the server log it does: even intervals between requests, complete coverage of the catalogue instead of a path through it, no images, no stylesheets, no second layer of interaction. Anyone who regularly looks at log file analysis sees such patterns before they show up on the hosting invoice.
Collecting/copying accessible data and/or processed output, for subsequent use such as exploitation of proprietary data, or price-fixing.
OWASP Automated Threats to Web Applications, OAT-011 Scraping
The damage arises along three lines at once. First, the shop spends compute time and bandwidth on requests that generate no revenue - with a catalogue of several tens of thousands of items, a full pass is a noticeable load. Second, your own price structure ends up in someone else's system and reappears there as the basis for automatic undercutting. Third, the extra traffic distorts your own metrics, because it counts towards sessions, page views and load times as long as it is not reported separately.
Blocking carts and grabbing goods
Two patterns mainly affect shops with scarce stock. In the first, a program puts items into a cart and does not pay for them; while the reservation runs, the goods are unavailable to other customers. OWASP lists this as a separate event under OAT-021 and explicitly describes the case where items do not go to checkout yet still contribute to a sold-out state (OWASP Automated Threats). How a shop handles unavailable items therefore also decides how expensive such a blockade becomes - which is the subject of the article on out-of-stock products.
In the second pattern the program genuinely buys, just faster than a human could. OWASP lists the acquisition of goods in a manner that a normal user would be unable to undertake manually under OAT-005 (OWASP Automated Threats). Limited editions, promotional stock and items with hard release times are affected above all. Neither case can be solved through the caller's identifier; both are solved in business logic: how long a reservation holds, how many open carts a session may have, which steps really have to be completed between product page and order confirmation.
Metrics shift unnoticed
The quietest damage hits the numbers that decisions rest on. OWASP lists automated repeated clicking, requesting or submitting of content that affects application-based metrics under OAT-016 (OWASP Automated Threats). In a shop this touches more places than are visible at first glance.
- Conversion rate: bot sessions count in the denominator but do not buy. The rate drops without anything changing in customer behaviour.
- Bounce rate and time on page: a collector requests one page and disappears. Both metrics move towards worse values although the page is unchanged.
- Top products: a complete catalogue pass spreads views evenly and pushes the genuine bestsellers down the ranking.
- Load times from field data: automated requests without images and without scripts look fast. Counting them in means underestimating what real customers experience - why field data beats lab scores applies here in particular.
- On-site search terms: automated queries against the internal search fill the report with terms no customer ever searched for.
Sorting instead of blanket blocking
Anyone who blocks automation across the board reliably blocks something they need as well: the search engine crawler, the preview request when a link is shared, their own availability check, the customer on a corporate network behind a shared address. A workable approach therefore sorts requests along two questions: is the request wanted? And does it identify itself? The four combinations lead to four different responses.
| Case | Typical trait | Response | Tool |
|---|---|---|---|
| Wanted, declared | Fixed identifier, confirmed by reverse DNS, respects the request rate | Let through, steer frequency | robots.txt, crawl control |
| Wanted, undeclared | Many sessions from one address, but buys and fills in forms | Observe instead of block | Allow list, per-session threshold |
| Unwanted, declared | Names itself, ignores exclusions, pulls entire catalogues | State the reservation, then throttle | Opt-out notice, rate limit |
| Unwanted, masked | Rotating addresses, copied identifier, pattern only in behaviour | Measure behaviour, protect logic | Session limits, checkout protection |
The second row is the most expensive one to overlook. Large corporate networks, mobile access and privacy relays bundle many people behind few addresses. Blocking by address typically hits paying customers there, and the outage only surfaces when someone calls. Hence the order: measure first, then sort, then throttle - and block only where the behaviour is unambiguous. Which of these responses a given system can cleanly express depends on how it is built; for Shopware stores that belongs in the ongoing care of the shop.
Detection: what gives a collector away in the log
The basis is the web server's access log, not the analytics tool in the browser. Measuring only in the browser usually misses collectors entirely, because they do not execute JavaScript. The log, by contrast, contains every request - with address, timestamp, path, status code, transferred volume and the caller's identifier. Six signals are typically the most productive.
- Even cadence: human intervals between two pages vary widely. Intervals that barely scatter across hundreds of requests point to a program with a fixed wait time.
- Catalogue completeness: a caller that requests practically every product page exactly once within hours is reading the catalogue rather than browsing it.
- Missing sub-resources: page views without the matching images, fonts and stylesheets usually do not come from a browser with a display.
- Share of 404 and 301: collectors walk over old paths and parameter combinations that no longer appear in the navigation.
- Requests-to-sessions ratio: many requests against very few cookies, or none at all, is a clearer signal than any identifier.
- Address origin: data centre networks are untypical for customers. Rotating access networks with an unchanged behavioural pattern, on the other hand, point to deliberate masking.
The pattern in the sample run is unambiguous: a single address with a multiple of everyone else's requests, almost all of them product pages, and no request for an image or a stylesheet. A browser behaves differently. How to anchor such analyses permanently in operations instead of running them once by hand is part of the hosting question - log retention, rotation, and an analysis job whose output lands somewhere a person reads it.
The caller identifier field is freely chosen. A collector can present itself as a common browser or as a search engine, and both occur. A request can usually only be confirmed through the network: first a reverse lookup of the address, then a forward lookup of the name found, and both results have to match - for Google's search crawler the permitted name spaces are googlebot.com, google.com and googleusercontent.com (Google Search Central). A rule built on the identifier alone typically blocks the honest callers and lets the masked ones through.
Steering in four stages
Stage 1: the notice in robots.txt
robots.txt is the cheapest lever and at the same time the most frequently overestimated one. It is a notice, not a barrier: those who respect it, respect it. The protocol has been standardised as RFC 9309 since 2022 and also settles details that matter day to day - for instance that a cached version should generally not be used for more than 24 hours, and that the parsing limit must be at least 500 kibibytes (RFC 9309). A change therefore does not take effect immediately, and a very large file is not necessarily read in full.
# All callers: exclude search results, cart and parameter floods
User-agent: *
Disallow: /suche
Disallow: /warenkorb
Disallow: /checkout
Disallow: /*?sort=
Disallow: /*?p=
Allow: /
# A single collector with a known identifier: full exclusion
User-agent: <collector-identifier>
Disallow: /
Sitemap: https://www.example.invalid/sitemap.xmlFor content that is read out to train models, a second layer applies. Under German law, text and data mining on lawfully accessible works is permitted in principle, but subject to a declaration by the rights holder - and for works available online that reservation is only effective in machine-readable form (section 44b of the German Copyright Act). How to set such a reservation in practice is covered in detail in the article on the TDM opt-out; anyone who wants to run AI features in house instead will find the framework under AI features without data transfer.
Uses pursuant to subsection (2) sentence 1 are permitted only if the rights holder has not reserved them. A reservation of use for works available online is effective only if it is made in machine-readable form.
Section 44b subsection 3, German Copyright Act
What robots.txt explicitly is not is stated in the standard itself: it is not a substitute for valid content security measures, and listing paths makes those paths publicly discoverable (RFC 9309, section 3). An administration area therefore belongs behind a login, not in an exclusion list. Confusing the two publishes directions to exactly the paths you wanted to hide.
Stage 2: limit the rate instead of blocking
The most effective stage in daily operation is not an exclusion but a ceiling. Setting a rate per address and per path group leaves normal use untouched and slows down precisely what a collector needs: speed. The status code for this has been standardised since 2012 - 429 indicates that too many requests were sent in a given amount of time, and the response may carry a Retry-After header saying how long to wait (RFC 6585).
# Two zones: generous for the catalogue, tight for search and cart
limit_req_zone $binary_remote_addr zone=katalog:10m rate=30r/m;
limit_req_zone $binary_remote_addr zone=suche:10m rate=10r/m;
limit_req_status 429;
server {
location /produkt/ {
limit_req zone=katalog burst=20 nodelay;
add_header Retry-After 60 always;
}
location /suche {
limit_req zone=suche burst=5 nodelay;
add_header Retry-After 120 always;
}
}Three points from the standard that are often missing in practice. First, responses carrying 429 must not be stored by a cache - if an upstream cache serves them anyway, it locks out uninvolved visitors (RFC 6585). Second, the response should explain the condition and include a Retry-After; without it, an honest caller simply retries at once. Third, nobody is obliged to answer with 429 at all: with very many requests from one source every response costs resources itself, which is why the standard explicitly names dropping connections as an appropriate alternative (RFC 6585, section 7.2).
Stage 3: protect the business logic
Rate limits help against volume, not against intent. Against cart blockades and instant buying only the logic itself works: how many open carts a session may have, how long a reservation holds, how many login attempts are allowed per account and per address, whether the stock display names an exact quantity or only a range. The same logic also protects the interfaces through which a headless setup delivers its data - which is the subject of the article on securing the Store API. How the setup behaves under load is settled beforehand and not on campaign day; the procedure is described in the article on load testing and the emergency plan.
- Limit how long a cart reservation holds and release expiring reservations automatically.
- Set a ceiling for open carts and for line items per cart, staggered by product group.
- Count login attempts per account and per source address separately, and cap both counters.
- Switch the stock display to a range for scarce goods instead of publishing the exact quantity.
- Give interfaces their own access key, their own rate limit and their own logging.
- Validate checkout steps on the server instead of relying on the order of steps in the browser.
Stage 4: keep the measurement clean
The last stage costs the least and is forgotten most often: bot traffic belongs in a separate report, not in the bin. Filtering it out loses the early warning signal; counting it in means deciding on shifted numbers. A dedicated segment with its own time series makes sense, so that it stays visible when a new source appears. For visibility in search engines the same data is read from a different angle anyway - what belongs in search engine optimisation is the question of which pages a crawler actually reaches, and how often.
The legal framework in Germany
A systematically harvested catalogue is more than a nuisance. German copyright law grants the database producer the exclusive right to reproduce, distribute and communicate to the public the database as a whole or a part of it that is substantial in nature or scale; repeated and systematic use of insubstantial parts is treated as use of a substantial part where it conflicts with normal exploitation or unreasonably prejudices legitimate interests (section 87b of the German Copyright Act). Whether a specific catalogue meets that threshold is a question of the individual case and belongs in legal review; this article does not replace it. The technical side can be separated from that, see consulting.
Conversely, the question arises whether analysing your own log is permitted when it contains IP addresses. The General Data Protection Regulation states that processing personal data constitutes a legitimate interest to the extent that it is strictly necessary and proportionate for ensuring network and information security, and expressly names defence against deliberate server overload as an example (recital 49 GDPR). In practice that means: record the purpose, limit the retention period, restrict access to the logs, and enter the activity in the record of processing activities.
What this means in operation
Measure
Access logs with sufficient retention, a recurring analysis job, and an output that somebody reads. Without this step every further measure is guesswork.
Sort
Confirmed search engines on an allow list, known collectors in a group of their own, everything else judged by behaviour. The identifier alone decides nothing.
Throttle
Rate limits per path group instead of one global rule, 429 with Retry-After instead of silent connection drops, exceptions for payment and shipping interfaces.
Track
A time series per segment shows when a new source appears and whether a measure worked. Without a before value, the effect stays an assertion.
The most expensive route starts with a block list. It creates work, shuts out customers and misses masked requests, because those simply change identifier. The cheap route starts with a measurement that runs for two weeks and then answers which share of traffic is automated, where it comes from and what it requests. Only then come thresholds - and the first threshold is typically set more generously than feels right, then tightened on the basis of the measurement.
The traffic shares come from the 2026 Bad Bot Report by Thales (analysis of the full year 2025) and from the 2025 Bad Bot Report by Imperva for the year 2024. The classification of attack types follows the OWASP taxonomy Automated Threats to Web Applications (OAT-005, OAT-011, OAT-016, OAT-021). The technical provisions are set out in RFC 9309 (Robots Exclusion Protocol) and RFC 6585 (status code 429). The legal statements rest on sections 87b and 44b of the German Copyright Act and on recital 49 of the General Data Protection Regulation; the procedure for confirming search engine requests is documented by Google Search Central. All figures were checked at source on 5 September 2026.
There is no reliable public figure for that. The widely cited values - 53 percent of all traffic automated, of which 40 percentage points are classified as malicious (Thales Bad Bot Report 2026) - describe a global average across all industries, not German retail. Your own share can typically be determined from the access log within two weeks, and the spread between individual shops is wide in our experience. How the logs are retained for that is part of the hosting question.
As a rule, no. robots.txt is a notice to callers who choose to respect it; the standard itself records that it does not replace valid content security measures and that listed paths become publicly discoverable (RFC 9309). It is still the right first step, because it steers wanted crawlers and forms the basis for a machine-readable reservation of use. Against callers that ignore it, only rate limits and protection in the business logic take effect.
It can, and typically where many people sit behind one address: corporate networks, mobile access, universities, privacy relays. That is why the limit is set per path group rather than globally, started generously and tightened on the basis of the measurement. The right response matters: 429 with a Retry-After header so that an honest caller knows when to try again (RFC 6585). A silent connection drop looks like an outage to the customer.
Not by the identifier - that is freely chosen. What holds up is the double lookup: first a reverse lookup of the accessing address, then a forward lookup of the name found, and both results have to match. For Google the permitted name spaces are googlebot.com, google.com and googleusercontent.com (Google Search Central). For day-to-day operation this check can be built into the analysis instead of being run by hand - a small building block in programming.
The General Data Protection Regulation treats processing personal data as a legitimate interest to the extent that it is strictly necessary and proportionate for network and information security, and expressly names defence against deliberate server overload as an example (recital 49 GDPR). In practice that means: record the purpose, limit the retention period, restrict access, add an entry to the record of processing activities. Assessing the individual case belongs in legal review; this article does not replace it.
That depends on how much of the four stages already exists. Measurement and robots.txt are typically a matter of hours, rate limits a matter of a day, while protecting the business logic depends on how the shop is built - cart, reservation and checkout have to be touched for it. It makes sense to work in that order and measure again after each stage instead of changing everything at once. A short conversation is enough for an assessment of your own case, see contact.