Web owners gain efficiency as OpenAI and Cloudflare pilot helps AI filter noise to index unseen pages

Cloudflare and OpenAI have shared early results from a research pilot that helps ChatGPT discover hundreds of millions of new web pages. According to Cloudflare, the project aims to make AI discovery more efficient for users and website owners alike. The initiative focuses on helping AI systems find relevant, timely information while reducing wasted requests. This collaboration addresses the challenge of optimising sites for AI when discovery mechanisms are unclear.

The pilot, which began in July, used data from approximately one per cent of Cloudflare’s total network. OpenAI used these signals to identify and index pages that were previously unknown to its search index. This allows ChatGPT to draw on more sources when answering user questions. The process creates a feedback loop that improves both content discovery and the quality of information presented to users.

What we know

Cloudflare’s network provides specific signals that help OpenAI’s search crawler find useful pages across the web. OpenAI then independently evaluates and ranks these pages based on relevance and helpfulness. The pilot revealed that automated traffic can make nonexistent pages appear interesting, such as bots repeatedly requesting missing files. Recognising these patterns allows the system to cut wasted work before fetching the page.

The research also tested signals that identify and flag potentially harmful pages. A legitimate site might have a single compromised page, such as a fake login form planted by an attacker. By linking back to relevant sources, ChatGPT Search gives people a way to explore original content. This work complements OpenAI’s advances in GPT-6 and Intelligent UI, which aim to provide more visual and interactive experiences.

Defender's checklist

• Monitor server logs for unusual patterns of automated traffic requesting non-existent files.
• Verify that redirects are specific and do not send all addresses to the homepage.
• Check for duplicate content that might confuse crawlers or dilute page value.
• Ensure login forms and sensitive pages are secure against compromise or fake overlays.

The background in plain language

Web crawlers are automated programs that browse the internet to index content for search engines or AI systems. Discoverability refers to how easily these systems can find and understand your website’s content. In the past, this meant ranking high in search results. Now, it involves helping AI identify specific, useful information. Teams usually need to ensure their sites send clear signals about what is valuable and accessible, rather than relying on links alone.

What to do and how to stay safe: Cloudflare

• Review your server logs to identify and block malicious bots that waste resources.
• Ensure your website’s structure helps crawlers find specific, useful content easily.
• Monitor for signs of compromised pages, such as unexpected redirect behaviour.
• Once the vendor provides an update, review any new tools for managing AI discovery.

Step-by-step guide: AI Governance for Small Teams: Practical Controls and Cost

General security guidance from the TechWire Beat newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

How many new pages did the pilot help discover?

The pilot enabled OpenAI to discover and index hundreds of millions of new web pages.

What percentage of Cloudflare's network was used?

The joint pilot used a participating cohort of approximately one per cent of Cloudflare’s total network.

Does this mean ChatGPT is now indexing all websites?

No, the pilot focused on a specific cohort and helped discover previously unknown pages within that scope.