Competitor price monitoring in the UK and EU: what is actually legal

The question that comes before any other

Before anyone asks what a price crawler can do, they ask whether they are allowed to run one. It is the right first question, and most answers online are wrong in one of two directions.

Half say scraping is illegal and you will be sued, which is not true. The other half say public data is public and you can do what you like, which is also not true and the more dangerous of the two, because that belief is what gets people into real trouble.

The honest answer is that watching a competitor's public prices is lawful in the UK and the EU in almost every case that matters to a small or medium sized shop. The exceptions are specific, knowable and mostly about how you collect and use the data rather than whether you collect it. This article draws the map for the UK first, then Germany, Austria and the EU rules underneath them, because many UK shops sell into Europe and their competitors sit there too.

One paragraph of housekeeping

I am not a lawyer and this is not legal advice. I build these systems and I read the statutes and judgments that shape them, and I can show you where the lines are and which side of them I keep my work on. If your situation is unusual (I say below what unusual looks like), book an hour with a solicitor who does IT and competition law. It is the best value hour you will buy this year.

Prices are facts, and facts belong to nobody

Everything starts here, and it is the point most people skip.

Copyright protects works. A product description somebody wrote can be a work. A product photo is a work. A shop's category copy and its blog can be protected too, and I come back below to what that means for a crawler.

A price is not a work. Neither is a stock level, a delivery time, a product name or a barcode number. They are facts, and copyright in the UK and across Europe protects the expression of an idea, never the underlying fact. You cannot own the fact that a competitor sells a particular coffee grinder for £249 any more than you can own the fact that it rained in Leeds on Tuesday.

So when a competitor says "you cannot use our prices", the first answer is that the price itself is not theirs in the way they think. What they might have is one of four other things: a right in the collection of prices, a contract with you about how you use their website, a claim about the way you collected the data, or a competition law problem with what you do next. Those are the real arguments, and I take them one at a time.

The database right, and why a shop's own prices usually fall outside it

Europe has a peculiar right that exists almost nowhere else. The Database Directive of 1996 created the sui generis database right, which protects a collection of data because somebody made a substantial investment in obtaining, verifying or presenting its contents.¹ Extract or reuse all or a substantial part of the contents without permission and you infringe, even though every single item is a plain fact. The directive also catches repeated and systematic extraction of insubstantial parts when it conflicts with the normal exploitation of the database.

The UK still has this right in the Copyright and Rights in Databases Regulations 1997, regulation 16 for infringement.² Since Brexit it only protects databases made by UK nationals, residents or companies (regulation 18), so a German competitor's database made after 2020 is not protected in the UK by this right, and a UK shop's new database is no longer protected in the EU by it.³ Germany has the same right in sections 87a to 87e of its copyright act, Austria in sections 76c to 76e of its own.⁴

It is the argument opponents of scraping reach for first, and it is weaker than it looks, for three reasons.

Investment in creating data does not count. In British Horseracing Board v William Hill (2004), the Court of Justice of the European Union held that the protected investment is the one made in obtaining existing data, not in creating it.⁵ The racing authority decided which horses ran in which race, and that creation of data did not earn the right. A shop is in the same position: it did not go out and collect its prices from the world, it set them. On the reasoning of that case, the effort of deciding prices is not the investment the right rewards. Lawyers call this the spin off argument. National courts do not all apply it with equal enthusiasm, so treat it as a strong starting point, not a guarantee.

Forty products are not a substantial part. Watching the forty products you actually compete on, out of a competitor's catalogue of six thousand, is not a substantial part by quantity. Mirroring the whole catalogue every hour is a different conversation.

The harm has to reach the investment. In the Latvian job portal case (2021), the Court held that extraction only infringes if it risks the maker's ability to recoup its investment, typically by offering a substitute for the database.⁶ Earlier, in Innoweb (2013), it found that a dedicated search engine which queried a car sales site in real time and showed its results to the public did infringe, because it offered the whole database through a different front door.⁷ That pair is the boundary: monitoring for your own pricing decisions sits on one side, republishing a competitor's catalogue as your own service sits on the other. One caveat for UK readers: the 2021 judgment came after Brexit, so it binds courts in Germany and Austria but only persuades a UK court.

Terms of use: what the Ryanair cases decided

If the database right does not protect the prices, the next thing a competitor reaches for is the website terms. Somewhere in a footer link is a paragraph saying automated access is prohibited. Does that bind you?

It might, and most of what we know comes from one airline. Ryanair has spent years suing price comparison sites and online travel agents that scrape its fares. Two decisions matter.

In Ryanair v PR Aviation (2015), the Court of Justice held that where a database is protected by neither copyright nor the database right, the directive does not stop its owner from restricting use by contract, subject to national law.⁸ Put simply, if the data falls outside the database right, the terms can still cover it. Scraping opponents quote this one, and it is real.

A year earlier, Germany's Federal Court of Justice decided a Ryanair case against an online flight broker. It held that screen scraping in breach of the airline's terms was not, on its own, unfair obstruction of a competitor, and that ticking a box that accepted the terms was not a technical protection measure.⁹ The court said the picture could change where a scraper overcomes technical measures that are there to stop it.

Put together, this is the position most practitioners work from. Terms can matter, especially where you actively accepted them by creating an account and ticking a box. A footer link you never saw is much weaker, and whether a visit to a public page creates a contract at all is doubtful in England, Germany and Austria alike. Even where terms bind, the remedy is a contract claim, not a finding that you competed unfairly. My practical rule follows from that: I do not create accounts on competitor sites to crawl them, and I do not crawl anything that requires accepting terms to see.

The line that matters most: logins and technical barriers

Almost every scraping dispute that ended badly for the scraper has one of two features. The data sat behind a login, or the scraper defeated something put there to stop it.

Logins are the clear case. If you have to authenticate to see the data, it is not public, you have almost certainly accepted terms, and you are inside someone else's system on their conditions. The UK's Computer Misuse Act 1990 makes it an offence to cause a computer to secure access you know is unauthorised.¹⁰ Germany's criminal code punishes gaining access to data specially secured against unauthorised access by overcoming that security (section 202a).¹¹ Austria's equivalent (section 118a of its criminal code) requires overcoming a specific security measure plus an intent such as causing a disadvantage.¹² None of these laws was written with price monitoring in mind, which is exactly why you do not want to find out how a court would apply them to yours.

Technical barriers are subtler. If a site rate limits you and your crawler slows down, that is good behaviour. If the site blocks you and you route around the block, or it shows a CAPTCHA and you pay a service to solve it, you have done the thing the German court flagged. The site has said, in the only language a server speaks, that it does not want this, and you have overruled it.

My rule is simple. Public pages, no login, and if the site pushes back the crawler stops rather than pushes harder. That costs some data. It is still the right trade.

Unfair competition: thin in the UK, sharper in Germany and Austria

The UK has no general law of unfair competition. Passing off protects goodwill against misrepresentation, which a price crawler does not touch. So in England the realistic claims against a polite crawler are the database right, the contract and, for abusive load, the Computer Misuse Act.

Germany and Austria are different. Both have an unfair competition act (UWG), and it is the law competitors actually use against each other. The German act names targeted obstruction of a competitor as unfair (section 4 number 4, which was number 10 when the Ryanair case was decided).¹³ Austria covers the same ground through its general clause in section 1.¹⁴ The 2014 Ryanair decision is the reassuring anchor: collecting public prices to inform your own decisions is competing on merit. It is, in fact, what competition is.

The UWG bites when the crawling itself does damage, or when the data is used to do damage. The German court said the scraping in its case must not measurably slow the airline's site. Crawling so hard that a small shop slows down for real customers, or has to buy more servers, is obstruction. Copying their descriptions and photos and republishing them is a separate wrong. Every one of those is within your control.

Text and data mining: the exception and the opt out

A crawler never fetches a price in isolation. It fetches a page, and the page contains descriptions, images and category copy that can be works. Copying that page, even briefly, is a reproduction.

In the EU the Digital Single Market Directive of 2019 created two exceptions for text and data mining. Article 3 is for research organisations. Article 4 lets anyone with lawful access reproduce works, and extract from databases, for text and data mining, unless the rights holder has expressly reserved that use, and for content made publicly available online the reservation has to be machine readable.¹⁵ Germany implemented Article 4 in section 44b of its copyright act and extended it to databases in section 87c.¹⁶ Austria put it in section 42h paragraph 6 and applied it to databases through section 76d.¹⁷ Copies may be kept only as long as the mining needs them.

The UK is the odd one out. Its text and data mining exception (section 29A) covers only research for a non commercial purpose.¹⁸ A UK business relies instead on the narrow temporary copies exception in section 28A, which covers transient copies that are an integral part of a technical process, and which by its own words does not apply to databases.¹⁹ The practical consequence is the same everywhere: extract the facts, throw the page copy away at once, and never store or reuse the descriptions and images.

The opt out is where most of the change has come in the last two years, driven by AI training rather than price monitoring. Sites now publish reservations in robots.txt, in meta tags and through the TDM Reservation Protocol, a W3C community report that defines a tdm-reservation header and a well known file.²⁰ My crawlers read robots.txt under the robots exclusion standard²¹ and the TDM signals, and where a site has reserved its content they take nothing from it.

Load: what polite looks like in numbers

"Low enough that the target would not notice" sounds vague, so here is the arithmetic. The daily request load a crawler puts on one site is

$R = n \times f$

where $R$ is requests per day, $n$ is the number of product pages you watch on that site and $f$ is how many times a day you fetch each one. It is a rate, so the ratios and rates section of Foundations applies.

Take an invented Bristol shop, harboursidegrinders.example, that watches 40 competing grinders on each of three competitor sites, four times a day, so $R = 40 \times 4 = 160$. That is 160 requests a day per competitor, about one every nine minutes. One shopper comparing grinders for half an hour loads more pages than that. If the shop spaces requests at least 30 seconds apart and never runs two at once, there is no plausible argument that it burdened anyone's server. The same shop mirroring 6,000 products every hour would make 144,000 requests a day per site, and that is where obstruction, the database right and every other problem in this article start.

GDPR: where product data stops and personal data starts

A price is not personal data. Neither is a product, a category, a shipping rate or a stock level. For most of what a price monitor collects, data protection law is not engaged.

A careless crawler picks up personal data anyway, in three places.

Reviews. Product pages carry reviews with names, towns and sometimes photos. Store "Anna from Brighton finds this grinder loud" and you are processing personal data about someone who has never heard of you. Article 14 of the GDPR, which the UK kept as the UK GDPR, then requires you to tell her within a reasonable period and at the latest within one month, unless that would involve disproportionate effort, an exemption you would have to justify and document.²² In the UK the regulator you would justify it to is the Information Commission (formerly the ICO), which replaced the Information Commissioner under the Data (Use and Access) Act 2025.²³ I do not collect reviews. If a client wants sentiment, aggregate star ratings are enough.

Sole traders. Many small shops, in the UK and especially in Germany and Austria, are run by individuals under their own name. The trading name, the address and the legal notice page are personal data. My crawlers do not touch legal notice or contact pages and store no merchant identity beyond the domain.

Marketplace sellers. On Amazon or eBay the seller name on a listing is often a person. Watching a listing's price is fine. Building a profile of who sells what needs a lawful basis you can write down.

The pattern repeats: the data you want is unproblematic, and the trouble comes from collecting more than you want because it sat on the same page.

Marketplaces are a different country

Amazon, eBay, Zalando and the other marketplaces deserve their own section, because the contract is different. As a seller you have signed their seller agreement and accepted their conditions of use, so the weak footer link argument does not help you there, and a suspended seller account costs far more than any pricing data is worth.

So for marketplaces I use what the platform offers. Amazon's Selling Partner API includes a Product Pricing API that returns pricing and offer information, including competitive pricing data, for products in the Amazon catalogue, and Amazon describes it as the basis for repricers.²⁴ Where no API covers the need, I use a licensed data provider who has its own arrangement with the platform and carries that risk. I do not point a crawler at a marketplace on a client's behalf and hope. That is less a legal judgement than a commercial one: the marketplace holds all the power.

The competition law trap nobody warns you about

Now the risk that matters most and that almost no scraping guide mentions, because it has nothing to do with scraping.

In the UK, Chapter I of the Competition Act 1998 prohibits agreements and concerted practices between businesses that restrict competition in the UK.²⁵ In the EU, Article 101 of the Treaty on the Functioning of the European Union does the same across borders, and Germany (section 1 of the GWB) and Austria (section 1 of the Cartel Act 2005) mirror it at home.²⁶ Price coordination is the most serious category, and fines can reach 10% of worldwide turnover.

Watching your competitors' public prices and reacting to them on your own is not an agreement. It is what independent competition looks like. The European Commission's 2023 guidelines on horizontal cooperation, in their chapter on information exchange, treat genuinely public information as lower risk, while warning that exchanges through a third party, including a shared algorithm or pricing tool fed with competitors' sensitive data, can be a concerted practice.²⁷

Here is how monitoring goes wrong.

Sharing the data. Two competitors who compare notes about what their monitoring shows have moved from monitoring to exchange.

Coordinating through the repricer. The UK has the textbook case. Two online poster sellers on Amazon agreed not to undercut each other and used automated repricing software, configured by each of them, to carry the agreement out. The Competition and Markets Authority fined one of them £163,371 in 2016; the other escaped a fine through leniency.²⁸ The software was legal. Using it to enforce an agreement was not.

Resale price maintenance. This one is for brands. In July 2018 the European Commission fined Asus, Denon & Marantz, Philips and Pioneer more than €111 million in total for restricting online retailers' prices, and it noted that monitoring tools let the manufacturers track retail prices and intervene quickly when retailers cut them.²⁹ Monitoring your retailers is lawful. Using it to pressure them back up to a minimum price is not. If you are a brand watching your own stockists, read this paragraph twice.

Hub and spoke. In the Eturas case (2016), a Lithuanian online travel booking system sent its travel agencies a message capping discounts. The Court of Justice held that agencies aware of the message could be presumed to have joined the practice unless they publicly distanced themselves from it or reported it.³⁰ A monitoring provider or marketplace that passes pricing intentions between competitors can create the same problem.

The rule I give clients: monitoring data goes into your own pricing decisions and nowhere else. Not to a trade association, not to a group chat of friendly competitors, not to a manufacturer who asks nicely. And the repricing rules should let a person explain every price in terms of your own costs and strategy, not in terms of what a competitor will do next.

Good conduct as evidence

Everything above comes down to conduct, and conduct is easier to defend when it is written down. These cost nothing and change the conversation if a letter ever arrives.

It identifies itself. The crawler sends a user agent that names the operator and links to a page explaining what it does and how to ask it to stop. A request to stop is honoured.

It reads robots.txt and means it. Disallowed paths are not fetched, crawl delays are honoured, TDM reservations are honoured.

It is slow. A few fetches a day per product, one request at a time, with gaps between them.

It backs off. A 429, a 503 or a block makes it wait and retry later. If it keeps happening, it stops and tells me. It does not rotate proxies and does not solve CAPTCHAs.

It has a written purpose. One page in plain language: what is collected, from whom, why, how long it is kept and who sees it. This is also most of a record of processing if one is ever needed, and it is the document you hand a solicitor.

The specification I build to

Public pages only, no accounts, no logins, no paywalls. An honest user agent with a contact page. Robots.txt, crawl delays and TDM reservations respected. A request rate well below what one human shopper generates. Product facts only: name, identifiers, price, availability, delivery cost and variant attributes. No reviews, no reviewer names, no legal notice data, no seller profiles. Page copies discarded once the facts are extracted, descriptions and images never stored or reused. Extracted facts kept for a documented rolling window. Marketplaces through their APIs or a licensed provider. Output that stays inside the client's own pricing process and is never shared with competitors or used to enforce resale prices. A one page purpose document, reviewed whenever the scope changes.

Your raw data stays yours, and the terms for code and models are in the analytics and data science terms.

What I will and will not crawl

I will monitor competitors' public product pages and category listings for price, availability, delivery and promotions. I will monitor a manufacturer's direct shop if you are a retailer, and retailers' public prices if you are a brand, with the competition law warning above written into the brief. I will monitor price comparison sites. I will track your own marketplace listings through the marketplace's API.

I will not create accounts or crawl behind a login. I will not defeat blocks, rotate proxies or solve CAPTCHAs. I will not crawl marketplaces directly. I will not collect reviews, reviewer identities, legal notice pages or seller profiles. I will not store or republish descriptions or images. I will not build anything whose output is meant to reach a competitor or to police a retailer's prices. And I will not crawl a site that has asked me to stop.

The checklist

If you are judging a monitoring setup, yours or a vendor's, ask these ten questions. A good answer to all of them puts you in the space where the whole retail sector operates. A bad answer to any one shows you where your risk is.

1. Is everything collected from public pages without a login?
2. Does the crawler identify itself and give a contact?
3. Does it respect robots.txt, crawl delays and TDM reservations?
4. Is the request rate low enough that the target would not notice?
5. Does it stop when blocked rather than route around the block?
6. Is the data limited to product and price facts, with no reviews, names or contact details?
7. Are page copies discarded and descriptions and images left alone?
8. Are marketplaces handled through APIs or licensed data, not direct crawling?
9. Is there a written purpose document, and does it match what the system does?
10. Does the output stay inside your own pricing decisions, never reaching a competitor or policing resale prices?

Ten yeses and the question you started with has a boring answer: yes, you are allowed. The interesting question is what to do with the data, which I cover in beating price wars in homogeneous product markets and game theory in growth strategy.

Footnotes

1. Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, Article 7 (sui generis right), https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:31996L0009 ↩

2. The Copyright and Rights in Databases Regulations 1997 (SI 1997/3032), regulation 16 (acts infringing database right), https://www.legislation.gov.uk/uksi/1997/3032/regulation/16 ↩

3. The Copyright and Rights in Databases Regulations 1997, regulation 18 (qualification for database right), as amended for the end of the transition period, https://www.legislation.gov.uk/uksi/1997/3032/regulation/18 ↩

4. Germany, Urheberrechtsgesetz, §§ 87a and 87b, https://www.gesetze-im-internet.de/urhg/__87b.html; Austria, Urheberrechtsgesetz, § 76d, https://www.ris.bka.gv.at/Dokumente/Bundesnormen/NOR40241419/NOR40241419.html ↩

5. Court of Justice of the European Union, Case C-203/02, The British Horseracing Board Ltd and Others v William Hill Organization Ltd, judgment of 9 November 2004, ECLI:EU:C:2004:695, https://curia.europa.eu/juris/liste.jsf?num=C-203/02&language=en ↩

6. Court of Justice of the European Union, Case C-762/19, CV-Online Latvia SIA v Melons SIA, judgment of 3 June 2021, ECLI:EU:C:2021:434, https://curia.europa.eu/juris/liste.jsf?num=C-762/19&language=en ↩

7. Court of Justice of the European Union, Case C-202/12, Innoweb BV v Wegener ICT Media BV and Wegener Mediaventions BV, judgment of 19 December 2013, ECLI:EU:C:2013:850, https://curia.europa.eu/juris/liste.jsf?num=C-202/12&language=en ↩

8. Court of Justice of the European Union, Case C-30/14, Ryanair Ltd v PR Aviation BV, judgment of 15 January 2015, ECLI:EU:C:2015:10, https://curia.europa.eu/juris/liste.jsf?num=C-30/14&language=en ↩

9. Bundesgerichtshof, judgment of 30 April 2014, I ZR 224/12, Flugvermittlung im Internet, report by Legal Tribune Online, https://www.lto.de/recht/hintergruende/h/bgh-urteil-izr22412-screen-scraping-flugdaten-automatisiert-auslesen-ryanair-reiseportal ↩

10. Computer Misuse Act 1990, section 1 (unauthorised access to computer material), https://www.legislation.gov.uk/ukpga/1990/18/section/1 ↩

11. Germany, Strafgesetzbuch, § 202a (Ausspähen von Daten), https://www.gesetze-im-internet.de/stgb/__202a.html ↩

12. Austria, Strafgesetzbuch, § 118a (Widerrechtlicher Zugriff auf ein Computersystem), https://www.ris.bka.gv.at/Dokumente/Bundesnormen/NOR40254197/NOR40254197.html ↩

13. Germany, Gesetz gegen den unlauteren Wettbewerb, § 4 (Mitbewerberschutz), https://www.gesetze-im-internet.de/uwg_2004/__4.html ↩

14. Austria, Bundesgesetz gegen den unlauteren Wettbewerb, § 1, https://www.ris.bka.gv.at/Dokumente/Bundesnormen/NOR40279746/NOR40279746.html ↩

15. Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market, Articles 3 and 4, https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32019L0790 ↩

16. Germany, Urheberrechtsgesetz, § 44b (Text und Data Mining), https://www.gesetze-im-internet.de/urhg/__44b.html, and § 87c (1) no. 4, https://www.gesetze-im-internet.de/urhg/__87c.html ↩

17. Austria, Urheberrechtsgesetz, § 42h (Text und Data Mining), https://www.ris.bka.gv.at/Dokumente/Bundesnormen/NOR40241407/NOR40241407.html ↩

18. Copyright, Designs and Patents Act 1988, section 29A (copies for text and data analysis for non commercial research), https://www.legislation.gov.uk/ukpga/1988/48/section/29A ↩

19. Copyright, Designs and Patents Act 1988, section 28A (making of temporary copies), https://www.legislation.gov.uk/ukpga/1988/48/section/28A ↩

20. W3C Text and Data Mining Reservation Protocol Community Group, TDM Reservation Protocol (TDMRep), final community group report, 10 May 2024, https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/ ↩

21. Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman, RFC 9309, Robots Exclusion Protocol, IETF, September 2022, https://www.rfc-editor.org/rfc/rfc9309 ↩

22. UK GDPR, Article 14 (information where personal data have not been obtained from the data subject), paragraphs 3 and 5, https://www.legislation.gov.uk/eur/2016/679/article/14 ↩

23. Data (Use and Access) Act 2025, section 118 (abolition of the Information Commissioner), https://www.legislation.gov.uk/ukpga/2025/18/section/118 ↩

24. Amazon, Selling Partner API, Product Pricing API documentation, https://developer-docs.amazon.com/sp-api/docs/product-pricing-api ↩

25. Competition Act 1998, section 2 (the Chapter I prohibition), https://www.legislation.gov.uk/ukpga/1998/41/section/2 ↩

26. Germany, Gesetz gegen Wettbewerbsbeschränkungen, § 1, https://www.gesetze-im-internet.de/gwb/__1.html; Austria, Kartellgesetz 2005, § 1, https://www.ris.bka.gv.at/Dokumente/Bundesnormen/NOR40065786/NOR40065786.html ↩

27. European Commission, Guidelines on the applicability of Article 101 of the Treaty on the Functioning of the European Union to horizontal co-operation agreements, OJ C 259, 21 July 2023, chapter 6 (information exchange), https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52023XC0721%2801%29 ↩

28. Competition and Markets Authority, Online seller admits breaking competition law, press release, 21 July 2016, https://www.gov.uk/government/news/online-seller-admits-breaking-competition-law ↩

29. European Commission, Antitrust: Commission fines four consumer electronics manufacturers for fixing online resale prices, press release IP/18/4601, 24 July 2018, https://ec.europa.eu/commission/presscorner/detail/en/IP_18_4601 ↩

30. Court of Justice of the European Union, Case C-74/14, Eturas UAB and Others v Lietuvos Respublikos konkurencijos taryba, judgment of 21 January 2016, ECLI:EU:C:2016:42, https://curia.europa.eu/juris/liste.jsf?num=C-74/14&language=en ↩