Common Crawl Web Corpus

Publicly accessible historical web crawl archives containing WARC and related data, with periodic releases that support web-scale text analysis.

This source is cataloged by RIDLY for discovery. Data access is provided by the original publisher under its applicable terms. RIDLY can assist with sourcing inquiries or evaluating licensing options.

Data coverage

Researchers use the corpus to study online content and build machine-learning pipelines. Copyright, privacy and acceptable-use considerations still apply to downstream use.

Research and commercial use

Review the source documentation before using data for analytics, product development, model training or commercial redistribution. Availability, permitted use and any required approvals depend on the provider's terms.

Need a tailored source?

Tell us the data fields, geography, population, time period and intended use. We can review the requirements and discuss potential suppliers or a managed sourcing process.

Request sourcing assistance