A shop owner can see in Search Console which addresses are indexed. What is missing there: which address a crawler actually requested, at which second, with which status code and how many bytes the server sent back. Those details live in exactly one place - the access log of your own server. It also records what no external report shows: fetches from bots that never identify themselves, requests for addresses that disappeared with the last relaunch, and error responses that only occur under load. How large the non-human share has become is measurable: in early December 2025, 47 percent of HTML requests came from humans and 44 percent from non-AI bots (Cloudflare Radar). Anyone who only estimates the rest is steering their visibility in search on half the data.
What the access log records and what the console leaves out
A line in an access log is a factual record: client address, timestamp, method, requested path, protocol version, status code, response size, referrer and the client identifier. None of it is rounded, none of it is capped at 90 days, none of it is filtered by importance. That is the difference from any report a search provider publishes. The Search Console crawl stats report, for instance, lists only the 20 child domains with the most requests over the last 90 days (Google Search Console Help). A shop with an image host, an editorial system and country subdomains sees an excerpt there, while its own log holds the complete set. Google puts the threshold at which the report becomes worthwhile at 1,000 pages (Google Search Console Help); most catalogues with variants and filters sit well above that.
The second difference is reach. A search provider reports on itself. The log reports on everyone: price comparison services, security scanners, translation services, archives, tools operated by competing merchants, and the share that simply means no good. According to an analysis of network data, automated traffic overtook human traffic in 2024 for the first time in a decade, reaching 51 percent of all web traffic (Imperva); bad bots accounted for 37 percent (Imperva). Those requests create load, consume processing time and distort every report built on page views - and they never appear in the search provider's own numbers.
A standard entry in the combined format holds nine fields. Four of them carry the SEO analysis: the path tells you which address is known at all, the status code tells you what the server answered, the response size exposes silent error pages served with status 200, and the client identifier separates crawlers from visitors. The remaining fields - timestamp, method, protocol version, referrer and client address - answer the questions of rhythm, origin and authenticity.
Who is actually knocking: the distribution of bot traffic
Before looking at a single address, look at the distribution. Across the network of one large infrastructure provider, Googlebot alone generated 4.5 percent of all HTML requests in 2025 - slightly more than all training and answer crawlers combined (Cloudflare Radar). Within verified bot traffic, 40 percent came from search engine crawlers, half as much from AI crawlers and over 13 percent from SEO tools (Cloudflare Radar). A single crawler accounted for more than 28 percent of all verified bot traffic at its most active point (Cloudflare Radar). For a shop this means: the spike that stands out in the log is rarely attributable to the search engine, and controls built for Google do not reach the rest.
Search engine crawlers were responsible for 40% of Verified Bot traffic, with AI crawlers generating half as much (20%). Search engine optimization bots were also quite active, driving over 13% of requests from Verified Bots.
Cloudflare Radar, The 2025 Cloudflare Radar Year in Review
Retail is squarely in view here: in an October 2025 breakdown of AI crawler activity by industry, retail and software together accounted for just over 40 percent of all activity (Cloudflare Radar). That matches the weight of the sector on the web, since 19.2 percent of all mobile pages examined were identified as e-commerce sites (Web Almanac). Anyone running an online retail operation who wants to decide which of these fetches are welcome will find the trade-offs in the article on controlling AI crawlers and the reservation of rights.
| Type of request | Recognisable by | Effect on the budget | Sensible response |
|---|---|---|---|
| Search engine crawlers | identifier plus reverse lookup | consumed predictably | guide paths, remove errors |
| AI and training crawlers | identifier, often unverified | consumes processing time | govern in robots.txt |
| SEO and analysis tools | fixed identifiers, fixed networks | no benefit for search | limit the rate |
| Disguised requests | identifier does not match the address | pure load | reject in front of the application |
The limits at which a fetch stops
A crawler does not download files of arbitrary size. Googlebot fetches no more than the first two megabytes of any single address; anything beyond that is neither rendered nor indexed (Google Search Central). For other Google crawlers without their own specification the ceiling is 15 megabytes per address, regardless of file type (Google Search Central). Both sound generous, and for the document itself they are: the HTML of a typical page weighs 35 kilobytes at the median (Web Almanac). It only becomes critical when a shop writes product data, image lists or entire state objects inline into the page. Then the part carrying the structured markup slides past the limit - and in the log the fetch still looks like a clean 200 response.
log_format crawl '$remote_addr $time_iso8601 "$request" '
'$status $body_bytes_sent $request_time '
'"$http_referer" "$http_user_agent"';
access_log /var/log/nginx/shop-crawl.log crawl;The response size in the log is therefore not a side note but a measurement. A category page that normally serves 60 kilobytes and suddenly answers with 4 kilobytes has delivered either an error or an empty result list - both with status code 200 and therefore invisible to any check that looks only at the code. Together with the response time this produces a series you can follow over weeks. How the same figure plays out in loading behaviour is covered in the article on Core Web Vitals and loading time; loading time optimisation ultimately also governs the rate at which crawling happens.
If the site gets slower, if the server answers with 5xx or signals throttling with 429, the crawl rate drops immediately (Google Search Central). That is the lever which works without anyone touching it: a server that answers briskly receives more fetches - and a server under constant load receives fewer, without any message appearing anywhere.
robots.txt: what the standard says and what the web contains
robots.txt is the first file a crawler requests, and in the log it is the most reliable marker for the start of a run. The standard requires a parsing limit of at least 500 kibibytes (IETF), and Google enforces exactly that ceiling (Google Search Central). According to the standard a crawler should not cache the file for longer than 24 hours (IETF); Google likewise generally caches it for up to 24 hours (Google Search Central). A redirect chain should be followed for at least five consecutive hops (IETF) - sending robots.txt across several host changes risks it being treated as absent at the end.
- A server error on robots.txt pauses crawling of the site for twelve hours; during that window only the file itself is requested (Google Search Console Help).
- If the error persists, Google works with the last successfully fetched copy until the 30th day after the original request (Google Search Console Help).
- File size is rarely the problem: only 0.1 percent of sites exceed the 500-kilobyte ceiling (Web Almanac).
- The rhythm of ad crawlers is not coincidence: registered addresses are fetched roughly every two weeks (Google Search Console Help).
Looking at the whole population puts things in perspective. Only 84.9 percent of all robots.txt requests receive a 200 response at all (Web Almanac), on 13 percent of sites the request runs into nothing (Web Almanac), and 1.8 percent serve a completely empty file for which the standard offers no interpretation (Web Almanac). Almost everything is governed through the catch-all: 77.04 percent of files name the wildcard as the addressee (Web Almanac), while Googlebot is named explicitly in only 6.66 percent (Web Almanac). Rules for the training crawler GPTBot have become roughly half again as common within a year and now appear on 4.5 percent of sites (Web Almanac); llms.txt, intended as a signpost, is found on 2.13 percent (Web Almanac). These figures come from a corpus of 16,213,084 websites (Web Almanac) - they describe not the individual case but the normal state a shop competes against.
A path blocked in robots.txt is not fetched - and so no crawler reads the noindex that sits on the page. In the log you therefore see nothing at all for excluded addresses, even though they may be in the index. If you want an address out of the index, allow the fetch and place the instruction in the head or as an HTTP header. The wider picture fits: only 2.4 percent of mobile pages set a noindex at all (Web Almanac), and 10.3 percent carry invalid elements in the head (Web Almanac) - where canonical, noindex and hreflang lose their effect.
Crawl budget: at which size it starts to matter
Crawl budget is a term used more often than it is needed. Google considers deliberate management necessary only for large sites from around one million unique pages with weekly changing content (Google Search Central), and for very rapidly changing content already from 10,000 unique pages (Google Search Central). The second case affects more shops than it first appears: variants, filter combinations, sort orders and page numbers create addresses in the five-digit range without anyone having created them. Which of them may exist at all is a decision of information architecture - the article on faceted navigation and its filter addresses sets out the trade-offs in detail.
Fetches per directory
How does crawling spread across categories, products, filters and search results? A catalogue where half the fetches land on filter addresses does not have a budget problem, it has a structure problem.
Status codes per area
An area with a striking number of 404 or 301 responses reveals stale links. A genuine 404 removes the address from the queue, while an error page served as a success keeps being fetched and consumes budget (Google Search Central).
Response time per template
Product pages, categories and the internal search answer at different speeds. The slowest template determines how much gets fetched overall - a case for caching and server operations.
First fetch of new addresses
The gap between publication and first fetch is the most honest indicator of how reachable a structure is. It exists only in the log.
One detail that regularly causes bafflement is the traffic spike with no visible trigger. Often there is no event behind it, only a rhythm: an ad crawler, a feed reconciliation, a weekly security scan. Once those patterns have been named, nobody keeps hunting for a cause that does not exist - and the spikes that genuinely are new become visible instead.
From raw log to decision
The volume is off-putting at first. A typical page generates 77 requests per view at the median (Web Almanac) - document, stylesheets, scripts, fonts and images combined. In a shop with five-digit visitor numbers that produces millions of lines per day. The first step is therefore not analysis but scoping: take resources out of view, reduce to document fetches, separate by identifier. What remains is a volume that standard command-line tools can handle - with no additional software and without data leaving the house.
Those two commands alone already lead to a decision. If a notable share of fetches lands on the internal search, processing time is being spent on addresses that have no business in the index. If the share of 301 responses stays in double-digit percentages, internal links still point at old paths - a task for internal linking, not for the redirect table. And a block of 500 responses clustered into a few minutes belongs in uptime and performance monitoring rather than in the SEO analysis.
- Fetches per status code and week. Breaks in the trend expose relaunch leftovers and server problems before they show up in visibility.
- Ratio of documents to resources. If a crawler mostly pulls images and scripts, the priority of your links is off.
- First fetch of new products. The gap between creation and first fetch answers whether the structure holds.
- Addresses without a single fetch. Comparing the sitemap against the log names exactly those pages no crawler has seen yet.
Verify instead of believe
The client identifier is a free text field anyone can set. A notable share of requests claiming to be a search engine originates from networks that belong to no search engine - the same analysis that attributes 37 percent of traffic to bad bots (Imperva) also explains why an identifier on its own proves nothing. The dependable route is a reverse lookup of the client address followed by a forward check: the name returned must belong to the expected domain, and that name must resolve back to the same address. Only then may a line be counted as a search engine fetch.
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1For ongoing analysis it is enough to store the result per network range rather than per line - otherwise the check takes longer than the analysis itself. Skipping the step means measuring a crawl increase that never happened and deriving measures from it that change nothing. The same reasoning applies to the defensive layer as to custom-built applications: the filter belongs in front of the application, not inside it.
What the error side of the log reveals
Google treats an error category as a problem when more than 5 percent of all requests fail there on a given day, for example during DNS resolution (Google Search Console Help). That threshold can be reproduced in your own log, and more finely: per template, per directory, per hour of the day. Particularly rewarding is the hunt for responses that are formally correct and substantively wrong. An error page served as a success keeps being fetched and consumes budget (Google Search Central); in the log it shows up only through the response size.
| Observation in the log | Likely cause | Check | Consequence of doing nothing |
|---|---|---|---|
| 200 with a strikingly small response | error page or empty result list | compare response size per template | address stays in the queue |
| Many 301s in one directory | stale internal links | check link targets in the source | budget flows into redirects |
| 429 or 5xx in waves | overload or throttling | evaluate response time per hour | the crawl rate drops |
| 404 with a referrer from the shop | broken internal links | group the referrer field | visitors and crawlers run into nothing |
For completeness the opposite direction belongs here too: addresses that exist but never appear in the log at all. They arise when a page is neither in the sitemap nor linked internally. A single sitemap holds at most 50,000 addresses or 50 megabytes (Google Search Central) - large catalogues therefore need several files and an index above them. Without it, part of the catalogue stays undiscovered, and in the log that is precisely the gap nobody searches for. How closely this connects to duplicate content is shown in the article on product variants and duplicate content.
Retention, data protection and the right scope
An access log contains client addresses and therefore personal data. For SEO analysis the full address is needed at exactly one point: when verifying crawlers. After that the result of the check is enough. A clean scope therefore separates two streams - a short-lived raw log for operations and security, and a shortened copy stripped of addresses for analysis over months. Which retention periods come together here and how to implement them without losing data is covered in the article on the deletion concept and retention periods.
The raw log stays short and serves operations. Alongside it a condensed copy is built without the client address: date, hour, path pattern, status code, response size, response time and the verified crawler class. That copy is small enough to keep for years, answers every SEO question and makes the analysis independent of the retention period of the raw data. Building it takes the same order of work that deliverability of shop emails demands: record every source first, then build the chain.
Where the log is blind
Your own log shows only what reaches your own server. If a delivery network sits in front of it, that network answers part of the requests itself and the origin log is missing that part. How often this happens can be quantified: only 35 percent of HTML responses are served through a delivery network, the rest comes straight from the origin server (Web Almanac). For roughly two thirds of sites the crawler traffic is therefore fully contained in the own log; for the remaining third the analysis has to merge both sources. Anyone already operating that intermediate layer will find the trade-offs in the article on cache architecture in the shop.
The second blind spot is rendering. Resources such as stylesheets and scripts are held by the rendering service for up to 30 days, regardless of HTTP caching instructions (Google Search Central). A script replaced today can therefore produce no fetch in the log for weeks and still be used during rendering - in an outdated version. Conversely, the share of fetches that could be answered from a cache is vanishingly small: ten years ago it was 0.026 percent, today it is 0.017 percent of all fetches (Google Search Central). In practice that means almost every crawler fetch really does reach the server, and almost every one leaves a line behind.
An approach for the first four weeks
Log analysis rarely fails on technology and often on sequence. Starting with the crawl budget question means measuring noise. Starting with scope means having a time series after four weeks that holds - and a baseline against which every later change can be measured.
- Week 1, collection. Set up a dedicated log format with response time and response size, define rotation and retention, separate the streams for operations and analysis.
- Week 2, verification. Collect crawler identifiers, confirm them through reverse and forward lookups, store the result per network range.
- Week 3, structure. Evaluate fetches per directory, status code and template, count filter and search addresses separately, compare the sitemap against the log.
- Week 4, decision. Adjust rules, clean up redirects, prioritise slow templates. A structured SEO audit for the shop places the findings in the wider context.
What reading the log changes day to day
The economic frame is not small: German online retail most recently turned over 92.3 billion euros net, an increase of 3.9 percent (HDE), and the online share of total retail stood at 13.5 percent (HDE). In a market of that size, every category a crawler fails to reach is a position missing in competition. The log answers no question about rankings or revenue - it answers the question before that: was the page fetched at all, in what state, how often and with what result? Only afterwards is a discussion about content worth having.
For day-to-day operations a slim report is then enough: fetches per crawler class, status distribution, response time per template, first fetch of new addresses. Four figures, monthly, measured against your own baseline. If one of them breaks out, the cause is usually found quickly - provided the log was there beforehand. If you would rather not build that yourself, it makes sense to look at technical search engine optimisation and operating the shop platform together; a review conversation typically clarifies quickly which of the four figures is needed first.
Google Search Central: Inside Googlebot: demystifying crawling, fetching, and the bytes we process (2026), Crawling December: HTTP caching and The how and why of Googlebot crawling (2024), How Google interprets the robots.txt specification, Large site owner's guide to managing your crawl budget and Build and submit a sitemap (2026). Google Search Console Help: Crawl stats report (2026). IETF: RFC 9309 - Robots Exclusion Protocol (2022), sections 2.3.1.2, 2.4 and 2.5. Web Almanac 2025, chapters SEO, Page Weight, CDN and Ecommerce, corpus of 16,213,084 websites. Cloudflare Radar: The 2025 Cloudflare Radar Year in Review (December 2025). Imperva: 2025 Bad Bot Report using network data from 2024. HDE: Online-Monitor 2026. Every figure is documented with a deep link and a verbatim quotation in the source register of this article.
It is a good entry point, but an excerpt. The report lists only the 20 child domains with the most requests over the last 90 days (Google Search Console Help), and Google itself considers it largely dispensable for sites with fewer than 1,000 pages (Google Search Console Help). Above all it reports on a single provider. Everything else that knocks - price comparison services, scanners, training crawlers - exists only in your own log.
Google considers deliberate budget management necessary only from around one million unique pages with weekly changing content, and from 10,000 unique pages where content changes very rapidly (Google Search Central). The analysis itself typically pays off much earlier: error patterns, dead links and slow templates appear regardless of catalogue size. A structured shop audit provides a frame for it.
Not from the identifier, which anyone can set. What holds up is a reverse lookup of the client address followed by a forward check: the name returned must belong to the expected domain and resolve back to the same address. That the effort pays off is shown by the share of bad bots at 37 percent of all traffic (Imperva).
Consider them separately: the raw log with client addresses stays as short as operations permit, because it contains personal data. The condensed copy without addresses may be kept long and carries the time series. Which retention periods come together in a shop is set out in the article on the deletion concept and retention periods.
Only those that honour the file - it is a request, not a barrier. In 77.04 percent of files on the web only the catch-all is present anyway (Web Almanac), and Googlebot is named explicitly in just 6.66 percent (Web Almanac). Anyone who really wants to stop access works in front of the application, for example through network ranges and rate limiting. For training crawlers the legal side comes on top, which concerns the reservation of rights.
First check whether they are in a sitemap and reachable internally. A single sitemap holds at most 50,000 addresses or 50 megabytes (Google Search Central), so larger catalogues need several files with an index above them. After that it is usually a question of internal linking: what no page links to is generally not fetched either.