Chinese internet search giant Baidu appears to have started blocking the online search engines of Alphabet’s Google and Microsoft’s Bing from scraping content derived out of the mainland firm’s Wikipedia-style service, a Post survey found.
A recent update of Baidu Baike’s robots.txt – a file that tells search engine crawlers which uniform resource locators, commonly known as web addresses, can be accessed from a site – has outright blocked the ability of the Googlebot and Bingbot crawlers to index content from the Chinese platform.
That update appears to have been made some time on August 8, according to records on internet archive service the Wayback Machine. It also showed that earlier on the same day Baidu Baike still allowed Google and Bing to browse and index its online repository of nearly 30 million entries, with only part of its website designated as off limits.
Do you have questions about the biggest topics and trends from around the world? Get the answers with SCMP Knowledge, our new platform of curated content with explainers, FAQs, analyses and infographics brought to you by our award-winning team.
This initiative shows Beijing-based Baidu’s increased effort to safeguard its online assets, as demand for vast troves of data have increased for training and building artificial intelligence (AI) models and applications.
That followed US social news aggregation platform and forum Reddit’s move in July, when it blocked various search engines, except Google, from indexing its online posts and discussions. Google has a multimillion dollar deal with Reddit that gives it the right to scrape the social media platform for data to train its AI services.
Since OpenAI released ChatGPT on November 30, 2022, major search platforms Google and Microsoft have sought to obtain more data for use in their own generative artificial intelligence systems. Photo: Shutterstock alt=Since OpenAI released ChatGPT on November 30, 2022, major search platforms Google and Microsoft have sought to obtain more data for use in their own generative artificial intelligence systems. Photo: Shutterstock>
Even Microsoft last year threatened to cut off access to its internet-search data, which it licenses to rival search engine operators, if they did not stop using it as the basis for their chatbots and other generative AI (GenAI) services, according to a Bloomberg report.
By comparison, the Chinese version of online encyclopaedia Wikipedia has 1.43 million entries to date, which are made accessible to search engine crawlers.
Following Baidu Baike’s robots.txt update, the Post’s survey of Google and Bing on Friday found many entries – probably from older cached content – from the Wikipedia-style service still come up in the US search platforms’ results.
Representatives from Baidu, Google and Microsoft did not immediately reply to requests for comment on Friday.
More than two years after the groundbreaking launch of OpenAI‘s ChatGPT, many large AI developers around the world are striking deals with content publishers for access to quality content to for their GenAI projects.
GenAI refers to the algorithms and services, such as ChatGPT, that are used to create new content, including audio, code, images, text, simulations and videos.
OpenAI, for example, in June forged a deal with American news magazine Time that gives it access to all the archived content from more than 100 years of the publication’s history.
This article originally appeared in the South China Morning Post (SCMP), the most authoritative voice reporting on China and Asia for more than a century. For more SCMP stories, please explore the SCMP app or visit the SCMP’s Facebook and Twitter pages. Copyright © 2024 South China Morning Post Publishers Ltd. All rights reserved.
Copyright (c) 2024. South China Morning Post Publishers Ltd. All rights reserved.
Source Agencies
