ÜBER DIESE EPISODE
AI companies increasingly crawl and use content from the open web, while creators and publishers often receive neither meaningful traffic nor compensation in return.
In this episode, Kim Isenberg speaks with Stephanie Cohen, Chief Strategy Officer at Cloudflare, about the company’s plan to reshape that relationship. They discuss why Cloudflare is giving website owners greater control over AI bots, the shift from Pay Per Crawl to Pay Per Use, and whether major AI companies are genuinely prepared to pay for the content their systems use.
The conversation also explores Cloudflare’s new rules for distinguishing between search, training, and AI-agent traffic, the difficult relationship between Google Search and AI answers, and whether independent creators and small publishers can realistically benefit from this emerging market.
Finally, Stephanie addresses the bigger questions: Should Cloudflare have the power to determine which bots can access the web? What happens if AI answers continue replacing clicks? And can the open web survive without a new economic model for original content?
Topics include:
- How AI crawlers use online content
- Pay Per Crawl versus Pay Per Use
- Getting creators and publishers compensated
- Cloudflare’s new AI-bot controls
- Search, training, and agent traffic
- Google’s role in the changing web economy
- The future of independent publishing
- Whether the open web can survive the AI transition
TRANSKRIPT 🔗
speaker-1: What do you say to critics who argue that Cloudflare is moving from internet infrastructure into private rulemaking for
speaker-0: We as a company have a mission which is to help build a better internet and we take this very seriously.
speaker-1: If a bot cannot clearly say I'm here for search or I'm here for training or AI answers, Cloudflare may block it more strictly. Does that make life harder for smaller AI starters?
speaker-0: Yeah, we have lots of incentive and…