Publishers Unblock Openai’s Crawler As Tiktok’s Parent Bytedance Boasts A 25x Faster Web Scraper

By Author: jamescolin
Total Articles: 215
Comment this article

OpenAI has recently secured a massive $6.6 billion in funding, bringing its valuation to $157 billion, a testament to the company's market dominance in generative artificial intelligence (GenAI). Backed by notable investors like Microsoft, NVIDIA, and Thrive Capital, OpenAI has positioned itself as a leader in the AI space. However, even with this influx of capital, OpenAI predicts losses exceeding $5 billion this year. Despite this, the company remains confident, forecasting over $11 billion in revenue for the upcoming year, largely driven by its AI products like ChatGPT.

A crucial component of OpenAI’s growth strategy is its ability to gather data from across the web using its web crawler, GPTBot. This crawler scrapes content to help train its AI models, which require vast datasets to produce high-quality results. Initially, many websites blocked GPTBot from accessing their content, with over 33% of websites implementing restrictions. Yet, recent reports indicate a shift, with the number of sites blocking OpenAI’s crawler dropping to 25%. Additionally, major news publishers have softened their stance, reducing their ...
... block rate from 90% to 50%.

This change in attitude among publishers can partly be attributed to new partnerships OpenAI has formed with key content providers, including TIME, NewsCorp, Reddit, and Condé Nast. These collaborations may offer mutual benefits, allowing publishers to play a role in shaping the future of AI while OpenAI gains access to valuable content. Still, not all publishers who have unblocked GPTBot are doing so willingly. The Onion, for example, unintentionally unblocked the crawler during a recent migration to a new hosting service. The outlet’s CEO, Ben Collins, was quick to dismiss the possibility of partnering with OpenAI, referring to it as a “Plagiarism Machine.”

The broader debate surrounding the use of AI web crawlers is complex. AI models require massive amounts of data for training, but this comes with legal and ethical questions. The use of web scrapers like GPTBot and ByteDance’s Bytespider, which was recently revealed to be even faster than GPTBot, has raised concerns about data privacy and copyright infringement. Moreover, ByteDance’s Bytespider reportedly ignores robots.txt files, which are used by websites to block unauthorized access, causing further alarm.

The issue becomes even more complicated in light of geopolitical tensions between the U.S. and China. ByteDance, the parent company of TikTok, is already under scrutiny for its handling of U.S. data, and the aggressive use of its web crawler has sparked additional worries. As U.S. companies like OpenAI and Chinese firms like ByteDance continue to expand their AI capabilities, questions about who controls access to online data and how that data is used are becoming increasingly urgent.

Publishers now face a critical decision: Should they embrace AI companies and unblock their web crawlers, potentially benefiting from new opportunities? Or should they hold out for better legal protections and assurances before giving AI firms access to their content? The answer to this dilemma may shape the future of both the publishing industry and AI development.

Read More: https://www.techdogs.com/tech-news/td-newsdesk/publishers-unblock-openais-crawler-as-tiktoks-parent-bytedance-boasts-a-25x-faster-web-scraper