End-to-End Test Automation for Online Shops

Deployments run without a maintenance window, a rollback is rehearsed, monitoring reports outages within seconds. One question stays open in many e-commerce projects all the same: how does anyone find out before go-live that the checkout no longer holds? That is exactly the gap automated end-to-end tests close. They walk the order path the way a customer walks it: find a product, add it to the cart, enter an address, pick a payment method, place the order. This article shows which paths belong in the automated suite, how dependable test data comes about, why flaky tests are more dangerous than missing ones, and where in the release chain the checks should run.

Why the order path is the most expensive blind spot

Online retail is everyday life in Germany: 83 % of people aged 16 to 74 have bought or ordered something online at least once, which corresponds to around 52 million people (Federal Statistical Office). Within the three months before the survey, 67 % of the same age group shopped online (Federal Statistical Office). Anyone running a shop is therefore serving a mainstream channel, not a niche one. If that channel fails, revenue fails with it, immediately and without warning.

At the same time the checkout is the most sensitive stretch in the whole system. The average cart abandonment rate is 70.22 % (Baymard Institute); that figure is an average across 50 separate studies on shopping cart abandonment (Baymard Institute). An average checkout flow is 5.1 steps long and contains 11.3 form fields, and 17 % of users have abandoned an order because of checkout complexity (Baymard Institute). On a path that already produces this much friction, every technical defect costs twice. Our article on checkout optimization covers how to reduce the friction itself; this one is about noticing a defect before it goes live.

Which paths belong in automation

Full coverage is neither achievable nor sensible. End-to-end tests are slow, expensive to maintain and sensitive to interface changes. The benefit arises where an outage costs money or legal certainty. For a shop, experience points to five stretches, and the order in which they are built follows the damage an outage would cause.

Priority follows revenue impact, not complexity. A broken faceted search is annoying; a checkout that throws a server error on submit costs a day of revenue. That is why the guest order comes first: it is the shortest route to the money and still touches price calculation, tax logic, shipping rules, payment integration and order confirmation. If you may automate only a single test, automate this one.

1. Walk through a guest order - pick an item on the product page, change the quantity, check the cart, enter an address, choose shipping and payment, place the order, then compare order number and total on the confirmation page against the expected value.
2. Walk through a signed-in order - sign in with an account, take over the stored delivery address, set a different billing address, place the order, then find that order again in the account overview.
3. Payment against the provider sandbox - redirect, successful payment, cancelled payment and failed payment as three separate cases. The cancellation is the most important one, because it has to preserve the cart.
4. Registration and password change - create an account with a fresh address, follow the confirmation link from the test environment mailbox, sign in with the new password.
5. Purchase rejection when stock runs out - put an item with a stock level of 1 into two carts in parallel and submit both orders. The second attempt has to decline cleanly instead of overselling.

One path deserves particular attention because it carries legal weight: the button that triggers the order. Under section 312j (3) of the German Civil Code, the trader's obligation is met only if the button is labelled legibly with nothing other than the words "order with obligation to pay" or a correspondingly unambiguous formulation (German Civil Code). That label often does not survive theme adjustments, translation runs and page builder blocks. An automated test that compares the button label word for word takes ten lines and prevents a defect that touches the formation of the contract itself.

Test data: where most suites fall over

The most common cause of unusable end-to-end tests is not the tests but their data. A test that adds a particular item to the cart depends on that item's stock, price, tax rate, visibility, category assignment and sales channel. If the item is maintained in the production system and the test data set is copied from it, every editorial change alters the test result. The suite then reports a failure although the code is unchanged - and trust in its reports drops with each such case.

• Test data belongs in version control. Storing the cases as a file next to the code makes it possible to change and roll them back together with the code.
• Every run starts from a defined state. Set up the database, load the data set, rebuild the index. A run that builds on the result of the previous run is not repeatable.
• Every test creates what it needs. A test that requires an account creates one itself with a unique address. Shared accounts make two parallel runs interfere with each other.
• No real personal data. Test addresses, test cards and mailboxes of the test environment. Production data in a test environment is a data protection problem, not a testing advantage.
• The data set models the edges, not the average. An item with stock 1, an item with tiered pricing, an item with a deviating tax rate, a cart above the free shipping threshold.

#!/usr/bin/env bash
set -euo pipefail

# Every run starts from the same state: schema, data set, index.
bin/console database:migrate --all
bin/console fixtures:load --set=e2e-checkout
bin/console dal:refresh:index

# The data set holds exactly the cases the path needs:
# single-stock-item stock 1, checks the rejection on oversell
# tiered-price-item tiered price from 10 units
# reduced-tax-item deviating tax rate
# free-shipping-cart cart above the free shipping threshold

bin/console cache:clear --env=test
echo "Test data set ready: $(date --iso-8601=seconds)"

The second stumbling block is payments. A real payment must not be triggered in an automated run, while a skipped payment leaves the decisive part of the path unchecked. The answer lies in the payment providers' sandboxes: they supply test card numbers for success, decline and timeout, and they mirror the callbacks to the shop the way production does. That makes it possible to check the complete return including the order status change without money moving.

Flakiness: why green runs can deceive

A flaky test is one that sometimes passes and sometimes fails with unchanged code. It is worse than a missing test, because it undermines trust in the entire suite. An evaluation of the full test corpus at Google from 2016 puts the share at around 1.5 % of all test runs reporting a flaky result (Google Testing Blog). Related to the tests themselves, this affects almost 16 % of the corpus, which carries some level of flakiness (Google Testing Blog).

The share of failure reports is even more telling: about 84 % of the observed transitions from pass to fail come from a flaky test (Google Testing Blog). Put differently, out of six red runs only one points to a genuine defect on average. The arithmetic behind that is sobering - in an average project with around 1,000 individual tests (Google Testing Blog) and a flakiness rate of 1.5 %, roughly 15 tests fail per run and tie up expensive investigation time (Google Testing Blog).

• Fixed waiting times. A two-second pause is too long on the development machine and too short on a loaded build agent. Wait for conditions, not for the clock.
• Animations and lazily loaded areas. A button still fading in accepts the click and loses it. Wait for the settled state, not for visibility.
• Shared data between parallel runs. Two runs ordering the same item meet at the stock level.
• Time and time zone. A test that calculates differently at midnight is a test with built-in randomness. Pin the clock inside the run.
• Third-party systems without a sandbox. When a service does not answer dependably, a controlled stand-in belongs in its place - plus a separate, infrequent run against the real system.

// Wait for states, not for seconds: the most common cause of flaky runs
// is a fixed waiting time.
test('guest checkout completes and shows the order number', async ({ page }) => {
  await page.goto('/p/single-stock-item');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await expect(page.getByTestId('cart-count')).toHaveText('1');

  await page.goto('/checkout/confirm');
  await fillAddress(page, guestAddress());
  await page.getByLabel('Invoice').check();

  // Legally mandated label of the order button
  const submit = page.getByTestId('confirm-submit');
  await expect(submit).toHaveText('order with obligation to pay');

  await submit.click();
  await expect(page.getByTestId('order-number')).toHaveText(/^SW\d{5,}$/);
  await expect(page.getByTestId('order-total')).toHaveText('49.90 EUR');
});

Execution in the release chain

Tests nobody waits for have no effect. The recommendation from the DORA research is clear: keep the test suite fast, developers should be able to get feedback from automated tests in less than ten minutes both on local workstations and from the continuous integration system (DORA). Ten minutes is unrealistic for a complete end-to-end suite - which is why it is staged. Fast checks run on every change, slow ones before release, very slow ones nightly.

stages: [quick, e2e, smoke]

unit:
  stage: quick
  script:
    - composer install --no-interaction
    - vendor/bin/phpunit --testsuite unit
  timeout: 5 minutes

e2e-core:
  stage: e2e
  script:
    - ./ci/seed-testdata.sh
    - npx playwright test --project=chromium --grep @core
  artifacts:
    when: always
    paths: [playwright-report/, test-results/]
    expire_in: 14 days
  timeout: 20 minutes

smoke-preview:
  stage: smoke
  script:
    - npx playwright test --project=chromium --grep @smoke
  environment: preview
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

The smoke stage after rollout is where end-to-end tests and operations mesh. It does not check business logic but the question of whether the rolled-out state can be operated at all - and it is the trigger for falling back to the previous state. In an environment with blue-green rollout this test decides whether traffic is switched to the new side. If it fails, the old state stays active and nobody has to intervene by hand at night. The linked article describes the technical side of switching in detail.

A performance budget belongs in the same stage. The target value for loading experience is a Largest Contentful Paint of 2.5 seconds or less (web.dev), measured at the 75th percentile of page loads, split between mobile and desktop devices (web.dev). A smoke test that collects the value for home page, category page and product page and raises a warning when it is exceeded catches performance regressions before they show up in monitoring as a slow trend. Background on the metrics themselves is in our article on Core Web Vitals.

One detail decides the value of the whole chain: the run has to leave evidence behind. A screenshot on failure, a recording of the session, console output and the response times of the requests involved. Without that archive every investigation starts with an attempt to reproduce the failure by hand - and with a flaky test that rarely works. When the cause sits in data access, a look at the slow query log covering the same period takes you further.

What stays manual

Automation does not replace testing by hand, it shifts it. The DORA recommendation names the manual parts explicitly: perform manual test activities such as exploratory testing, usability testing and acceptance testing throughout the delivery process (DORA). The reason is simple: an automated test checks what somebody described as correct beforehand. It finds deviations from the expectation but no wrong expectations. What is missing from the expectation is typically missing from the requirements specification as well - and only surfaces when a human walks the path without a script.

The limit becomes especially clear with accessibility. The annual report on the most visited home pages shows an average of 56.1 detected errors per page (WebAIM Million 2026), an increase of 10.1 % over the previous year's analysis, which found 51 errors per page (WebAIM Million 2026). At the same time the report records that all automated tools have limitations and not every conformance failure can be detected automatically (WebAIM Million 2026). An automated check in the suite catches the machine-detectable violations; operability with keyboard and screen reader stays a task for people. Our accessibility audit article describes the approach.

• A first order by a real person on a real device after every larger rebuild
• Operating the checkout with the keyboard alone, once per release cycle
• Checking order and shipping mails in several mail clients
• Visual inspection of price display, unit price and shipping cost notice on narrow screens
• Exploratory testing of new features without a test plan, with the explicit brief to break them
• Acceptance by the business department based on real business cases

Introduction in four weeks

Setting this up rarely fails on the technology and often on the sequence. Starting with broad coverage leaves you with 200 flaky tests after three weeks and nobody to maintain them. The sustainable route starts with a single path that runs dependably and grows from there. This split has proven itself in development projects:

1. Week 1 - foundation. Set up the test environment, create the test data set as a file, write the seeding script. Outcome: one command produces a reproducible state.
2. Week 2 - first path. The guest order as the only test, run twenty times in a row. Only when twenty out of twenty runs pass does the second test join.
3. Week 3 - core paths and chain. Signed-in order, payment with its three outcomes, registration. Integration into the release chain with archiving of screenshots and recordings.
4. Week 4 - smoke and fallback. Smoke test against preview and live state, wiring to the automatic fallback, assignment of responsibility for red runs.

From week five onwards, maintenance decides the value of the suite. Three metrics are enough to see whether it is healthy. They can be derived from the release chain reports and belong in the same archive as the operational figures from the peak season load test.

A coverage figure in percent, by contrast, is a poor steering value for end-to-end tests. It rewards many small tests and punishes the few long paths that carry the actual benefit. More useful is a list of the business cases an outage would make impossible, next to a note on which of them is checked automatically. Management understands that list too - and it works as the basis for a shop check in which the current state is assessed once from the outside.