Cloudflare Unveils Ethical AI Scandal: Perplexity’s Web Crawler Bypasses Restrictions

Login to Continue Learning

AI search startup Perplexity has found itself in the middle of a controversy, facing allegations that it bypassed restrictions designed to prevent its web crawlers from accessing certain websites. According to a report by Cloudflare, the company has been able to sidestep these protections by concealing its true identity and sneaking past website restrictions meant to protect content.

**Perplexity Accused of Bypassing Website Restrictions**

Cloudflare conducted an in-depth investigation after setting up a hidden webpage with crawler restrictions that was not linked or indexed publicly. Despite these protections, Cloudflare claims that Perplexity’s systems were still able to access the page and include its content in search results, confirming the unauthorized ways the company tends to collect data.

**Response from Both Sides**

In a unique start to his ruling against President Donald Trump’s administration on September 30, U.S. District Court Judge William Young included a scanned handwritten note sent to his office. The note read: “Trump has pardons and tanks – what do you have?” At the top of Young’s opinion in AAUP v. Rubio, which ruled that Trump’s effort to deport foreign-born student protesters was unconstitutional.

October 8, 2025

Perplexity has denied the accusations against it, stating that the investigation did not consider openness and transparency. The company believes that Cloudflare has either exaggerated findings or misread the situation. In response, Cloudflare asserts that Perplexity’s actions are a breach of its terms of service and unethical. As a result, Perplexity has been removed from the list of verified bots, and Cloudflare plans to tighten restrictions overall.

**The Broader Debate**

This incident highlights a larger issue: the war for accessing content and its monetization. It also raises questions about how AI models source their data and the gray practices that arise as these systems become more powerful and commercialized. Cloudflare CEO Matthew Prince has been vocal about the risks these models pose to content creators and publishers, leading the company to introduce a charge for accessing content and automatically blocking AI crawlers.

The situation underscores the need for transparency and accountability in the AI ecosystem, particularly concerning data sourcing practices.

📚 Reading Comprehension Quiz

10 Points

According to the report, what action did Perplexity take that led to controversy?

A. It openly shared all its data sources. B. It bypassed website restrictions designed to protect content. C. It tightened its own restrictions on web crawlers. D. It removed itself from the list of verified bots.

Please login or register to take the quiz and earn points!