Developer docs
API playgroundTry for free, no card

Search company profiles

Common Crawl

Full company profile

uuid0007wrm

Namestring
Common Crawl
Legal namestring
Common Crawl Foundation
Websiteurl
commoncrawl.org
Company typeenum
Private
Founded yearint
2007
Descriptiontext

Common Crawl Foundation is a California 501(c)(3) nonprofit founded in 2007 by Gil Elbaz, operating from Beverly Hills, CA, with a lean 1-10 person engineering team. The organization maintains a free, open repository of web crawl data, currently containing over 300 billion pages spanning 15 years of internet history and adding 3-5 billion new pages each month through its proprietary ccBot crawler. Core products include the Web Crawl Data Repository (distributed in WARC, WAT, and WET formats), Web Graphs at the host and domain level (latest release covering April-June 2026 with 247.3M host nodes / 6.3B edges and 121.1M domain nodes / 3.9B edges), a CDXJ/URL Index with API access, and a recently launched AI Agent for natural-language search. The dataset is cited in over 10,000 research papers and is used by AI companies including OpenAI for training large language models, alongside academic and web-science research.

The technical stack centers on a distributed crawling infrastructure (ccBot) with data processing pipelines built in Python, Spark/PySpark, and the WebGraph framework, producing outputs in standardized web-archive formats (WARC, WAT, WET) plus a ZipNum-based CDX index. Distribution is global and entirely free, delivered via AWS S3 (under Amazon's Open Data Sponsorship Program, which eliminates the petabyte-scale hosting cost burden) and HTTPS endpoints, with an additional distribution surface on Hugging Face for ML practitioners.

The business model is nonprofit donation- and sponsorship-funded rather than transactional: there is no pricing, no paid tier, and no direct revenue from data licensing. AWS provides infrastructure at no cost through its Open Data Sponsorship Program. Operating geographies are global with no restrictions, and customer segments split between academic researchers (primary persona), AI/ML companies (primary business unit, including OpenAI and various AI labs), and web accessibility researchers. The organization faces ongoing regulatory and reputational pressure from major news publishers (CNN, NBC, USA Today, News/Media Alliance) regarding the inclusion of paywalled content in the archive, which it has begun addressing through an Opt-Out Registry.

Short descriptiontext

Common Crawl Foundation is a 501(c)(3) nonprofit that maintains a free, open repository of 300+ billion web pages spanning 15 years, used by AI labs including OpenAI, academic researchers, and web scientists for training language models and conducting research.

Operating statusenum
Operating
Ownership categoryenum
Headcount rangeband
11–50
akta.pro rankint
HeadquartersBeverly Hills, United States
HQ citystring
Beverly Hills
HQ countrystring
United States
HQ regionstring
North America
Markets served

Serves global market

Offices1 record

Each record includes

City, Country, Type, Description, Source

Keyword5 values
open web data, web crawl archive, web graph datasets, training data corpus, open data repository
Industry3 codes
PRIMARYBPAMAAAL
Web Archiving, Cache & Snapshot Services
SECONDARYHDAEAGAH
Enterprise Search, Indexing & Content Discovery
Hardware, Technology & Telecom/Enterprise Platforms & Databases/Enterprise Content Management (ECM) & Document Management
SECONDARYBPAMAAAI
News & Content Aggregation Portals
NAICS code2 codes
  • Web Search Portals and All Other Information Services51929
  • Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services5182
SIC code1 code
  • Services-Computer Programming, Data Processing, Etc.7370
Product category
Open Web Data Infrastructure
GTM motion1 record

Each record includes

Type, Description, Source

Revenue model2 records
1Donations and Sponsorships
TypeData Monetisation
Description

As a 501(c)(3) nonprofit, Common Crawl relies on donations from individuals and corporate sponsors to fund operations. The organization actively seeks corporate sponsors to partner with for their non-profit open data mission.

commoncrawl.org
2AWS Open Data Sponsorship
TypeData Monetisation
Description

Cloud hosting infrastructure is provided free of charge through Amazon Web Services' Open Data Sponsorship Program, significantly reducing operational costs

commoncrawl.org
Marketing channels5 records

Each record includes

Title, Type, Stage, Description, Source

Distribution channels3 records

Each record includes

Title, Type, Scope, Target buyer, Description, Source

Cost components4 values
Infrastructure, Technology or R&D, Personnel, Operations
Pricing details1 tier
1Free open access to all web crawl data
ModelFreemiumBilling cadencePay-as-you-go
Notes

All web crawl data is provided free of charge. No pricing tiers or paid subscriptions exist - all data can be freely downloaded from AWS S3 or HTTPS endpoints.

commoncrawl.org
GTM typeB2B
B2B
Offering typeSoftware
Software
Core offering1 text field

Common Crawl operates a large-scale web crawler (ccBot) that systematically archives the open web and publishes the resulting data as a free, open repository. Its core offerings include bulk web crawl archives in WARC/WAT/WET formats, host-level and domain-level web graphs, CDXJ and URL indexes, and an Index API, all hosted on AWS S3 under the AWS Open Data Sponsorship Program and accessible to anyone without charge.

Differentiator
Functional benefit
Problem solved
Quantifiable outcome1 of 2 values shown
  • Over 10,000 research papers have cited Common Crawl data
+1 more record
Product overview1 text field

Common Crawl is a 501(c)(3) nonprofit organization that maintains a free, open repository of web crawl data. The core offering consists of the Web Crawl Data Repository containing over 300 billion web pages, supplemented by Web Graphs (host-level and domain-level hyperlink graphs), CDXJ Index (capture index format), URL Index (for discovering URLs across crawls), and WARC/WAT/WET file formats for raw and processed crawl data. Additional offerings include CCBot (the crawler), AI Agent (natural language search), and integration with Hugging Face for dataset access. The organization publishes 3-5 billion new pages monthly and data is hosted on AWS S3 under their Open Data Sponsorship Program.

Product and service5 records
1Web Crawl Data Repository
CategoryOpen Web Data Repository
Description

Free, open repository of web crawl data containing over 300 billion pages spanning 15 years, with 3-5 billion new pages added each month. Data is distributed in WARC, WAT, and WET formats on AWS S3 for researchers, academics, and AI companies.

2Web Graphs
CategoryWeb Graph Datasets
Description

Host-level and domain-level web graphs derived from crawl data using the WebGraph framework, used for ranking analysis, link spam detection, and large-scale graph research.

3CDXJ Index
CategorySearch Index
Description

Capture Index JSON (CDXJ) line-format index enabling efficient lookup of archived web content by URL and timestamp, using the ZipNum CDX format for billions of pages.

4URL Index
CategorySearch Index
Description

Index service for searching and discovering URLs across all crawl archives, including a columnar URL index for enhanced querying.

5Index API (CDX Server API)
CategoryDeveloper API
Description

Public API at index.commoncrawl.org for querying crawl data, supporting wildcard URL queries, pagination, and JSON output for efficient programmatic lookup of archived URLs.

Scale indicator6 records

Each record includes

Type, Value, Description, Source

Partnership2 partners
Strategic tierCoreTypeOthers
Description

AWS provides hosting for Common Crawl's data through their Open Data Sponsorship Program, allowing the nonprofit to store and distribute petabytes of web crawl data at no cost. This is critical infrastructure for the organization's mission.

Strategic tierCoreTypeTechnology or Integration
Description

Common Crawl uses the WebGraph framework (from University of Milano) for computing web graph properties and rankings. This open-source software enables large-scale graph analysis of the crawled data.

Recent move6 records

Each record includes

Date, Type, Title, Description, Source

Expansion highlight5 records

Each record includes

Type, Description

Peers10 records
TypeBroad incumbent
Description

Diffbot is a commercial web-scale knowledge extraction company that builds structured knowledge graphs from crawled web data. It addresses the same underlying market of structured web knowledge used for AI and enterprise search, but via a paid product rather than free open data.

TypeDirect peer
Description

The Internet Archive's Wayback Machine is the most direct peer — a nonprofit operating large-scale web archiving infrastructure (WARC files, historical snapshots) freely accessible to the public. Both organizations preserve historical web content at scale using similar file formats and serve overlapping research and AI-training use cases.

TypeDirect peer
Description

Hugging Face hosts Common Crawl's datasets and serves as a primary distribution channel for ML practitioners consuming them. Both operate as open-data infrastructure for the AI community and share overlapping users (AI researchers and model builders).

TypeDirect peer
Description

Wikimedia Foundation operates a similar nonprofit, open-data model for Wikipedia and structured knowledge, frequently used as training data alongside Common Crawl. Both serve as foundational open-data infrastructure for AI and academic research with comparable governance and funding structures.

TypeOthers
Description

Common Sense Media is a nonprofit producing open datasets for AI safety, child development, and media evaluation. It shares the nonprofit open-data model for AI training and research, although the focus areas differ.

6Archive.today
TypeEmerging player
Description

Archive.today is a smaller web archiving service capturing on-demand snapshots of web pages. While it operates at far smaller scale than Common Crawl, it addresses the same archival use case and competes for the same publisher opt-out and copyright narrative around AI training data.

TypeEmerging player
Description

Curlie and similar community-curated web directories historically served as open indexes of web content for research and discovery. They share Common Crawl's nonprofit, community-driven ethos but operate at much smaller scale and with manual curation.

TypeOthers
Description

AWS's Open Data Sponsorship Program hosts Common Crawl and a broader catalog of petabyte-scale open datasets (e.g., NASA, NIH genomic data). It is an ecosystem enabler rather than a competitor, but its terms directly shape Common Crawl's cost structure and distribution reach.

9Open Web Search (open-source web search initiative)
TypeEmerging player
Description

The Open Web Search EU initiative is a European open-web-indexing project aiming to provide an open alternative to commercial search indexes. It is comparable to Common Crawl in mission (open web data) but operates as a research consortium rather than a continuously archived dataset.

10Pushshift (now defunct, archived datasets)
TypeEmerging player
Description

Pushshift historically provided free open datasets of social media data for research, similar in mission to Common Crawl but focused on Reddit and social platforms. Although largely defunct, its archived datasets remain a comparable open-data precedent in AI research.

Market position
Strengths4 records

Each record includes

Headline, Details, Source

Weaknesses1 record

Each record includes

Headline, Details, Source

Competitive moat4 records

Each record includes

Type, Details

Key risks5 records

Each record includes

Headline, Details, Source

Key highlights6 records

Each record includes

Headline, Details, Source

Customer concentration

Classification, Details

Named customers3 records

Each record includes

Name, Industry, Type, Use case, Source, UUID

Segment3 records

Each record includes

Title, Type, Primary, Description, Pain point addressed, Use case, Source

Ideal customer profile3 records

Each record includes

Profile, Firmographic size, Sales motion, Sales cycle length, Buying structure, Purchase trigger, Buyer persona, Geography, Industry vertical, Primary use case, Description, Pain points, Evidence proof points, Target buyer

Technology focused
Yes
API detail
Has APIbool
Yes

Docs URL, Description

AI capability2 records

Each record includes

Type, Description, Source

AI maturity
App detail

Has app

Feature4 records

Each record includes

Title, Differentiator, Description, Source

Core technology
Revenue estimate
Valuation estimate
Number of profiles
Profiles5 records

Each record includes

Name, Designation, Designation category, Overview, Profile commentary, Source

No data
No data
Funding overview

Funding stage, Last funding date, Total funding USD

Funding rounds

Each record includes

Round, Amount USD, Date, Pre money valuation, Total investors, Investors, News

Investors

Each record includes

Name, Type, Date of entry, Rounds participated, Website

Funding detail is available on the Subscription and Enterprise plan.Contact sales →

M&A

Each record includes

Name, Acquisition type, Announced date, Completed date, Status, Website, News

Investment

Each record includes

Name, Round, Announced date, Lead investor, Website, News

M&A and investment is available on the Subscription and Enterprise plan.Contact sales →

Common Crawl

Open Web Data Infrastructurecommoncrawl.org

Common Crawl Foundation is a 501(c)(3) nonprofit that maintains a free, open repository of 300+ billion web pages spanning 15 years, used by AI labs including OpenAI, academic researchers, and web scientists for training language models and conducting research.

What Common Crawl does

Common Crawl Foundation is a California 501(c)(3) nonprofit founded in 2007 by Gil Elbaz, operating from Beverly Hills, CA, with a lean 1-10 person engineering team. The organization maintains a free, open repository of web crawl data, currently containing over 300 billion pages spanning 15 years of internet history and adding 3-5 billion new pages each month through its proprietary ccBot crawler. Core products include the Web Crawl Data Repository (distributed in WARC, WAT, and WET formats), Web Graphs at the host and domain level (latest release covering April-June 2026 with 247.3M host nodes / 6.3B edges and 121.1M domain nodes / 3.9B edges), a CDXJ/URL Index with API access, and a recently launched AI Agent for natural-language search. The dataset is cited in over 10,000 research papers and is used by AI companies including OpenAI for training large language models, alongside academic and web-science research.

The technical stack centers on a distributed crawling infrastructure (ccBot) with data processing pipelines built in Python, Spark/PySpark, and the WebGraph framework, producing outputs in standardized web-archive formats (WARC, WAT, WET) plus a ZipNum-based CDX index. Distribution is global and entirely free, delivered via AWS S3 (under Amazon's Open Data Sponsorship Program, which eliminates the petabyte-scale hosting cost burden) and HTTPS endpoints, with an additional distribution surface on Hugging Face for ML practitioners.

The business model is nonprofit donation- and sponsorship-funded rather than transactional: there is no pricing, no paid tier, and no direct revenue from data licensing. AWS provides infrastructure at no cost through its Open Data Sponsorship Program. Operating geographies are global with no restrictions, and customer segments split between academic researchers (primary persona), AI/ML companies (primary business unit, including OpenAI and various AI labs), and web accessibility researchers. The organization faces ongoing regulatory and reputational pressure from major news publishers (CNN, NBC, USA Today, News/Media Alliance) regarding the inclusion of paywalled content in the archive, which it has begun addressing through an Opt-Out Registry.

Common Crawl firmographics

Firmographics
Name
Common Crawl
Legal name
Common Crawl Foundation
Website
https://commoncrawl.org
Company type
Private
Founded year
2007
Operating status
Operating
Headcount range
11–50 employees
Short description
Common Crawl Foundation is a 501(c)(3) nonprofit that maintains a free, open repository of 300+ billion web pages spanning 15 years, used by AI labs including OpenAI, academic researchers, and web scientists for training language models and conducting research.
Ownership category
akta.pro rank

Common Crawl industry classification

Industry
Product category
Open Web Data Infrastructure
NAICS
Web Search Portals and All Other Information Services (51929), Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services (5182)
SIC
Services-Computer Programming, Data Processing, Etc. (7370)
akta.pro industry
PRIMARYBPAMAAAL
Web Archiving, Cache & Snapshot Services
SECONDARYHDAEAGAH
Enterprise Search, Indexing & Content Discovery
Hardware, Technology & Telecom/Enterprise Platforms & Databases/Enterprise Content Management (ECM) & Document Management
SECONDARYBPAMAAAI
News & Content Aggregation Portals

Keywords

  • Open web data
  • Web crawl archive
  • Web graph datasets
  • Training data corpus
  • Open data repository

Where Common Crawl is headquartered

Location

Headquarters

HQ city
Beverly Hills
HQ country
United States
HQ region
North America

Offices1 record

Markets served

Common Crawl business model

Business model
GTM type
B2B
Offering type
Software
Cost components
Infrastructure, Technology or R&D, Personnel, Operations

Revenue model

  1. Donations and Sponsorships: As a 501(c)(3) nonprofit, Common Crawl relies on donations from individuals and corporate sponsors to fund operations. The organization actively seeks corporate sponsors to partner with for their non-profit open data mission.
  2. AWS Open Data Sponsorship: Cloud hosting infrastructure is provided free of charge through Amazon Web Services' Open Data Sponsorship Program, significantly reducing operational costs

Pricing tiers

ModelBillingPrice
FreemiumPay-as-you-goFree open access to all web crawl data

Go-to-market motion1 record

Distribution channels3 records

Marketing channels5 records

Common Crawl product offering

Product offering

Core offering

Common Crawl operates a large-scale web crawler (ccBot) that systematically archives the open web and publishes the resulting data as a free, open repository. Its core offerings include bulk web crawl archives in WARC/WAT/WET formats, host-level and domain-level web graphs, CDXJ and URL indexes, and an Index API, all hosted on AWS S3 under the AWS Open Data Sponsorship Program and accessible to anyone without charge.

Product overview

Common Crawl is a 501(c)(3) nonprofit organization that maintains a free, open repository of web crawl data. The core offering consists of the Web Crawl Data Repository containing over 300 billion web pages, supplemented by Web Graphs (host-level and domain-level hyperlink graphs), CDXJ Index (capture index format), URL Index (for discovering URLs across crawls), and WARC/WAT/WET file formats for raw and processed crawl data. Additional offerings include CCBot (the crawler), AI Agent (natural language search), and integration with Hugging Face for dataset access. The organization publishes 3-5 billion new pages monthly and data is hosted on AWS S3 under their Open Data Sponsorship Program.

Differentiator

Problem solved

Functional benefit

Products and services

  • Web Crawl Data Repository Free, open repository of web crawl data containing over 300 billion pages spanning 15 years, with 3-5 billion new pages added each month. Data is distributed in WARC, WAT, and WET formats on AWS S3 for researchers, academics, and AI companies.
  • Web Graphs Host-level and domain-level web graphs derived from crawl data using the WebGraph framework, used for ranking analysis, link spam detection, and large-scale graph research.
  • CDXJ Index Capture Index JSON (CDXJ) line-format index enabling efficient lookup of archived web content by URL and timestamp, using the ZipNum CDX format for billions of pages.
  • URL Index Index service for searching and discovering URLs across all crawl archives, including a columnar URL index for enhanced querying.
  • Index API (CDX Server API) Public API at index.commoncrawl.org for querying crawl data, supporting wildcard URL queries, pagination, and JSON output for efficient programmatic lookup of archived URLs.

Quantifiable outcome

  • Over 10,000 research papers have cited Common Crawl data
  • +1 more outcomes

Companies that use Common Crawl

Customer profile

Named customers3 records

Segments3 records

Ideal customer profiles3 records

Common Crawl technology and API

Technology

Technology focussed Yes

API detail

Has API
Yes
API docs
API detail

Core technology

AI maturity

App detail

AI capability2 records

Feature4 records

Common Crawl partnerships and signals

Strategic signal

Partnerships

Two partnerships are on record, tiered core.

  • Amazon Web Services (AWS)coreOthersAWS provides hosting for Common Crawl's data through their Open Data Sponsorship Program, allowing the nonprofit to store and distribute petabytes of web crawl data at no cost. This is critical infrastructure for the organization's mission.
  • WebGraph FrameworkcoreTechnology or IntegrationCommon Crawl uses the WebGraph framework (from University of Milano) for computing web graph properties and rankings. This open-source software enables large-scale graph analysis of the crawled data.

Scale indicators6 records

Recent moves6 records

Expansion highlights5 records

Common Crawl competitors and assessment

Company assessment

Broad incumbents

  • Diffbot: Diffbot is a commercial web-scale knowledge extraction company that builds structured knowledge graphs from crawled web data. It addresses the same underlying market of structured web knowledge used for AI and enterprise search, but via a paid product rather than free open data.

Direct peers

  • Internet Archive: The Internet Archive's Wayback Machine is the most direct peer — a nonprofit operating large-scale web archiving infrastructure (WARC files, historical snapshots) freely accessible to the public. Both organizations preserve historical web content at scale using similar file formats and serve overlapping research and AI-training use cases.
  • Hugging Face: Hugging Face hosts Common Crawl's datasets and serves as a primary distribution channel for ML practitioners consuming them. Both operate as open-data infrastructure for the AI community and share overlapping users (AI researchers and model builders).
  • Wikimedia Foundation: Wikimedia Foundation operates a similar nonprofit, open-data model for Wikipedia and structured knowledge, frequently used as training data alongside Common Crawl. Both serve as foundational open-data infrastructure for AI and academic research with comparable governance and funding structures.

Others

  • Common Sense Media (Common Sense): Common Sense Media is a nonprofit producing open datasets for AI safety, child development, and media evaluation. It shares the nonprofit open-data model for AI training and research, although the focus areas differ.
  • AWS Open Data Sponsorship Program: AWS's Open Data Sponsorship Program hosts Common Crawl and a broader catalog of petabyte-scale open datasets (e.g., NASA, NIH genomic data). It is an ecosystem enabler rather than a competitor, but its terms directly shape Common Crawl's cost structure and distribution reach.

Emerging players

  • Archive.today: Archive.today is a smaller web archiving service capturing on-demand snapshots of web pages. While it operates at far smaller scale than Common Crawl, it addresses the same archival use case and competes for the same publisher opt-out and copyright narrative around AI training data.
  • Curlie / Open Directory Project successors: Curlie and similar community-curated web directories historically served as open indexes of web content for research and discovery. They share Common Crawl's nonprofit, community-driven ethos but operate at much smaller scale and with manual curation.
  • Open Web Search (open-source web search initiative): The Open Web Search EU initiative is a European open-web-indexing project aiming to provide an open alternative to commercial search indexes. It is comparable to Common Crawl in mission (open web data) but operates as a research consortium rather than a continuously archived dataset.
  • Pushshift (now defunct, archived datasets): Pushshift historically provided free open datasets of social media data for research, similar in mission to Common Crawl but focused on Reddit and social platforms. Although largely defunct, its archived datasets remain a comparable open-data precedent in AI research.

Market position

Strengths4 records

Weaknesses1 record

Competitive moat4 records

Key risks5 records

Key highlights6 records

Customer concentration

Common Crawl social profiles

Digital presence

Common Crawl financial estimates

Financial estimate

Revenue estimate

Valuation estimate

Common Crawl leadership team

Management profile

Number of profiles

Profiles5 records

Common Crawl funding detail

Funding detail

Funding overview

Funding rounds

Investors

Funding detail is available on the Subscription and Enterprise plan.Contact sales →

Common Crawl M&A and investment

M&A and investment

M&A

Investments

M&A and investment is available on the Subscription and Enterprise plan.Contact sales →

Frequently asked questions about Common Crawl

What does Common Crawl do?

Common Crawl operates a large-scale web crawler (ccBot) that systematically archives the open web and publishes the resulting data as a free, open repository. Its core offerings include bulk web crawl archives in WARC/WAT/WET formats, host-level and domain-level web graphs, CDXJ and URL indexes, and an Index API, all hosted on AWS S3 under the AWS Open Data Sponsorship Program and accessible to anyone without charge.

Is Common Crawl a public or private company?

Common Crawl is a private company. It is classified as nonprofit foundation owned and is currently operating.

When was Common Crawl founded?

Common Crawl was founded in 2007. It employs 11 to 50 people.

Where is Common Crawl based?

Common Crawl is headquartered in Beverly Hills, United States, in the North America region.

How does Common Crawl make money?

Two revenue lines are on record. Donations and Sponsorships are the primary driver. The others are AWS Open Data Sponsorship.

Who are Common Crawl's main competitors?

Diffbot is listed as a broad incumbent. Direct peers are Internet Archive, Hugging Face and Wikimedia Foundation. Others are Common Sense Media (Common Sense) and AWS Open Data Sponsorship Program. Emerging players are Archive.today, Curlie / Open Directory Project successors, Open Web Search (open-source web search initiative) and Pushshift (now defunct, archived datasets).

Does Common Crawl have an API?

Yes. Common Crawl provides public API access via the Index API at index.commoncrawl.org for querying crawl data. The CDX Server API allows querying URLs from specific crawls, with support for wildcard queries, pagination, and JSON output. They also offer an open data repository accessible via AWS S3 and HTTPS endpoints. Developer documentation is at index.commoncrawl.org.

What industry is Common Crawl in?

Common Crawl's product category is Open Web Data Infrastructure. Its primary akta.pro industry code is BPAMAAAL, Web Archiving, Cache & Snapshot Services, with a secondary code of HDAEAGAH, Enterprise Search, Indexing & Content Discovery. Its NAICS code is 51929 and its SIC code is 7370.

Unlock the full company data

50 free credits on sign-up, no credit card required.

Contact sales
Live signals
Search Engine JournalSome Sites Use llms.txt Like robots.txt, Common Crawl FindsCommon Crawl analyzed over 500,000 llms.txt files, finding 68% originated from plugins or templates and 22% had no links. The analysis also noted 6% included AI usage guidelines and 32 sites appeared to deny CCBot, though none blocked it outright in robots.txt.HackerNoonHow to Find Vulnerable WordPress Sites Using Common CrawlA developer demonstrates using Amazon Athena to query Common Crawl's S3 archive for WordPress sites, filtering by URL patterns and fetch status. The method costs $0.40 for 1 TB of scanned data and yields a list of vulnerable sites. The author then uses JavaScript to exploit a plugin vulnerability and promotes his BlockBots.org project.Android HeadlinesOne in Three New Web Pages Show Signs of AI Writing, Pew Study FindsA Pew Research study analyzing over half a million web pages found that more than 35% of those published after ChatGPT's launch in November 2022 showed signs of AI authorship. The research, which utilized the Open Pangram tool on data from the Common Crawl archive, revealed significant disparities in AI usage across domain types, with .com sites showing substantially higher rates than .org or .edu/.gov domains.Pew Research CenterMethodologyThe article outlines the methodology used to analyze the prevalence of AI-authored content on the internet by sampling 490,000 pages from Common Crawl data between January 2021 and July 2026. Researchers utilized Pangram's editlens_Llama-3.2-3B model to detect AI authorship, validating results against their commercial Pangram 3.3 model which showed a 96% agreement rate. The study found that 35% of pages with post-ChatGPT publication dates in the July 2026 crawl exhibited signs of AI authorship.DigidayEuropean publishers are getting hit harder by AI bot scraping, report findsA report by TollBit indicates that European publishers are experiencing significantly higher rates of AI bot scraping and lower referral traffic compared to their North American counterparts, with robots.txt instructions being ignored nearly three times more often. Analysts attribute this disparity to the growing demand for non-English content in Large Language Models, as evidenced by increased data collection from European domains by providers like Common Crawl. However, conflicting data from Cloudflare and DataDome suggests that while Europe has a higher share of bot traffic relative to total visits, absolute bot volumes remain higher in North America.Search Engine JournalCommon Crawl Published A Manual For Being Visible To AI, I Automated ItCommon Crawl, a nonprofit, maintains an extensive archive used for training AI models, with recent efforts to automate checking if sites are included or blocked from this dataset. The article discusses the role of the CCBot crawler, blocking practices by website owners, and a new automated tool for site owners to audit their visibility in Common Crawl. It emphasizes the importance of crawl priority and technical configurations for site inclusion in AI training data.Idrc-CrdiInclusion by design: making African languages visible to AIThe Data Science for Social Impact lab at the University of Pretoria launched the Africa Common Crawl (AfriCC) initiative to make African languages visible to AI systems, addressing how web crawlers and language identification tools systematically bypass African-language content due to their English and Global North-focused design. The initiative mobilised researchers and volunteers across 19 African countries who submitted approximately 700 seed URLs in 34 languages, resulting in a collection of about 29 million web pages and contributing over 343,000 African-language pages to the January 2026 Common Crawl dataset — an 18% increase from the previous monthly update. The project demonstrates that much African-language content already exists online but remains invisible to AI training pipelines due to how data collection begins and how automated tools classify languages, and aims to influence the pre-training stage of AI development rather than relying solely on post-hoc fine-tuning.Search Engine JournalUS Publishers Demand Common Crawl Stop Scraping Their ContentDigital Content Next sent a cease-and-desist letter to Common Crawl, demanding it stop scraping and remove copyrighted publisher content from its datasets. The letter argues that copyright law is not an opt-out regime, and Common Crawl has not commented on the notice.Search Engine LandPublishers push Common Crawl to stop collecting content for AI trainingDigital Content Next sent Common Crawl a cease-and-desist letter demanding it stop scraping and distributing protected publisher content, including paywalled and subscriber-only articles. DCN alleges Common Crawl failed to honor opt-out requests, while Common Crawl denies misleading publishers. The dispute could shape how much publisher content AI search engines can use without permission.BloombergNews Organizations Push Back Against Web Archive Used For AIMajor news organizations including CNN, NBC, and USA Today are part of a group of 20 publishers that have opted out of having their content saved in Common Crawl's online repository, a web archive used by artificial intelligence companies for training chatbots. The News/Media Alliance, representing newspapers and magazines, sent a letter to Common Crawl urging the nonprofit to honor publishers' requests to remove content from dozens of websites and prohibit unauthorized use of their work for AI purposes. The action reflects growing tensions between traditional media companies and AI firms over the use of copyrighted content for training language models.