Cloudflare Introduced the New Robotcop Feature to Enforce.txt Policies
1m, 6s
2
14:54, 11.12.2024
Overview of the New Robotcop Feature
Cloudflare has announced a new feature in its audit panel called Robotcop. This tool helps monitor how various AI services comply with the policies specified in robots.txt files and allows users to automatically enforce these policies at the network level. With the rise of generative AI, which actively scans the internet to collect data for training models, many website owners have started updating their robots.txt policies to restrict access not only for web crawlers but also for AI bots.
Thanks to the new feature from Cloudflare, website owners can get a report on the number of requests from various bots, including those from AI services such as ChatGPT, Google Gemini, and others. The audit panel allows users to filter requests by category and receive information on policy violations.
How Robotcop Helps with Traffic Management and Policy Violations
The Robotcop feature provides reports on policy violations for each bot. Users can see which traffic violates the established rules and receive popup hints with violation details. The “Most Popular Paths” section highlights traffic that violates the site’s policies. Users can also apply robots.txt rules through an advanced firewall rule, automatically blocking violators.
This feature is already available to all Cloudflare users, providing additional tools for effectively managing site access and protecting against unwanted bots.
Was this article helpful to you?
Yes
No
VPS popular offers
-10%
Linux
€ 49.99 /mo
Billed annually
-7.1%
Windows
€ 21 /mo
Billed annually
-15.6%
Linux
€ 38 /mo
Billed annually
-20.5%
Linux
€ 57 /mo
Billed annually
-10%
Linux
€ 7.7 /mo
Billed annually
-20.5%
Linux
€ 95 /mo
Billed annually
-18.4%
Windows
€ 24 /mo
Billed annually
-13.1%
Linux
€ 7 /mo
Billed annually
-10%
Windows
€ 28.99 /mo
Billed annually
-22.2%
Linux
€ 33 /mo
Billed annually
