🍷 FineWeb

15 trillion tokens of the finest data the 🌐 web has to offer

What is it?

The 🍷 FineWeb dataset consists of more than 15T tokens of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 datatrove library, our large scale data processing library.

🍷 FineWeb was originally meant to be a fully open replication of 🦅 RefinedWeb, with a release of the full dataset under the ODC-By 1.0 license. However, by carefully adding additional filtering steps, we managed to push the performance of 🍷 FineWeb well above that of the original 🦅 RefinedWeb, and models trained on our dataset also outperform models trained on other commonly used high quality web datasets (like C4, Dolma-v1.6, The Pile, SlimPajama, RedPajam2) on our aggregate group of benchmark tasks.

That said, we think there is still room for additional filtering and improvement and intend to continue exploring how to improve the dataset quality in coming versions of 🍷 FineWeb.

What is being released?

Along with the dataset, which includes all CommonCrawl dumps since 2013, we also share all the code needed to fully reproduce our processing setup using the 🏭 datatrove library here. To enable full replication of our results, we have also published the small ablation models we have trained using nanotron to validate the dataset and compare it with other reference datasets. You will find them here, with checkpoints every 1000 steps. We have also published our evaluation results here. Our evaluation setup is available here.

You will find details on the different processing decisions we took and some interesting explorations of deduplication methods and differences between CommonCrawl dumps in our technical report to be published in the coming days.

How to download and use 🍷 FineWeb

You can load the full dataset or a specific crawl/dump (see table below). Dumps have the format CC-MAIN-(year)-(week number).

Using 🏭 datatrove

from datatrove.pipeline.readers import ParquetReader

# limit determines how many documents will be streamed (remove for all)
# to fetch a specific dump: hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10
data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb/data", limit=1000) 
for document in data_reader():
    # do something with document
    print(document)

###############################    
# OR for a processing pipeline:
###############################

from datatrove.executor import LocalPipelineExecutor
from datatrove.pipeline.readers import ParquetReader
from datatrove.pipeline.filters import LambdaFilter
from datatrove.pipeline.writers import JsonlWriter

pipeline_exec = LocalPipelineExecutor(
    pipeline=[
        ParquetReader("hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10", limit=1000),
        LambdaFilter(lambda doc: "hugging" in doc.text),
        JsonlWriter("some-output-path")
    ],
    tasks=10
)
pipeline_exec.run()

Using huggingface_hub

from huggingface_hub import snapshot_download
folder = snapshot_download(
                "HuggingFaceFW/fineweb", 
                repo_type="dataset",
                local_dir="./fineweb/",
                allow_patterns="data/CC-MAIN-2023-50/*")

For faster downloads, make sure to install pip install huggingface_hub[hf_transfer] and set the environment variable HF_HUB_ENABLE_HF_TRANSFER=1.

Using datasets

from datasets import load_dataset
fw = load_dataset("HuggingFaceFW/fineweb", name="CC-MAIN-2024-10", split="train", streaming=True)

Breakdown by dump/crawl

Dataset performance evaluation and ablations

We conducted our dataset performance ablations and evaluations by training a series of 1.8B parameters models on 27 billion tokens. To compare 🍷 FineWeb with other datasets, we also trained one of these 1.8B models per target dataset, on 350 billion tokens sampled from it (or the entire dataset when its size was < 350 billion tokens).

Hyper-parameters for ablation models

The detailed configurations for training the 1.8B parameters ablation model can be found here (link will be added soon).

Ablation evaluation benchmarks

To conduct the ablations for each of our dataset filtering choices, we selected a set of benchmarks which we identified as “high-signal” benchmarks. These benchmarks were selected according to the following criteria:

• small variance between runs trained on different samplings of the same dataset
• performance increasing monotically during training (or close)
• separation between runs on datasets of known quality (C4, The Pile, RedPajama) higher than the variance between runs with various modeling/data seeds

We used the following list of benchmark for our ablation runs:

• commonsense_qa (acc_norm)
• hellaswag (acc/acc_norm)
• openbookqa (acc/acc_norm)
• piqa (acc/acc_norm)
• siqa (acc/acc_norm)
• winogrande (acc/acc_norm)
• sciq (acc/acc_norm)
• arc (acc/acc_norm)
• mmlu (acc/acc_norm)

To compare runs we consider an aggregate score, the average of the scores for these tasks.

The prompts for all these benchmarks are formatted in order to compute and compare the log-likelihood of the full answers for each multiple choice question. All the implementation details for the benchmarks are available in lighteval here.

Comparison with other datasets

We compared 🍷 FineWeb with the following datasets:

• RefinedWeb
• C4
• Dolma v1.6 (the CommonCrawl part)
• The Pile
• SlimPajama
• RedPajama2 (deduplicated)

You will find these models on this collection. We have uploaded checkpoints at every 1000 training steps. You will also find our full evaluation results here.

Note: The plot is smoothed by averaging 5k steps in a rolling window.

Dataset card for 🍷 FineWeb

Dataset Summary

This dataset was created by processing 95 CommonCrawl dumps comprising web data crawled from the summer of 2013 to March of 2024. 🍷 FineWeb includes a variety of domains and topics in English and is primarily intended to be used as a research artifact on public data in the context of pretraining dataset for large language models. The CommonCrawl data was carefully processed, filtered and deduplicated with the 🏭 datatrove library, resulting in the largest publicly available clean LLM pretraining dataset, counting around 15 trillion tokens (gpt2 tokenizer).

Dataset Structure

Data Instances

The following is an example sample from the dataset. It is part of the CC-MAIN-2021-43 and was crawled on 2021-10-15T21:20:12Z.

{
   "text": "This is basically a peanut flavoured cream thickened with egg yolks and then set into a ramekin on top of some jam. Tony, one of the Wedgwood chefs, suggested sprinkling on some toasted crushed peanuts at the end to create extra crunch, which I thought was a great idea. The result is excellent.",
   "id": "<urn:uuid:e5a3e79a-13d4-4147-a26e-167536fcac5d>",
   "dump": "CC-MAIN-2021-43",
   "url": "<http://allrecipes.co.uk/recipe/24758/peanut-butter-and-jam-creme-brulee.aspx?o_is=SimilarRecipes&o_ln=SimRecipes_Photo_7>",
   "date": "2021-10-15T21:20:12Z",
   "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-43/segments/1634323583083.92/warc/CC-MAIN-20211015192439-20211015222439-00600.warc.gz",
   "language": "en",
   "language_score": 0.948729,
   "token_count": 69
}

Data Fields

• text (string): the main text content
• id (string): original unique identifier for this sample from CommonCrawl
• dump (string): the CommonCrawl dump this sample was a part of
• url (string): url to the original page where text was present
• date (string): crawl date (from CommonCrawl)
• file_path (string): s3 path for the individual CommonCrawl warc file containing this sample
• language (string): en for all the samples in this dataset
• language_score (float): language prediction score (0.01.0) as reported by the fastText language classifier
• token_count (int): number of tokens when applying the gpt2 tokenizer to this sample

Data Splits

The default subset includes the entire dataset. If you would like to only use the data from a particular CommonCrawl dump, you can use the dump name as a subset. You will find the full list of available dumps on the table above. From experiments we have run, not all dumps give the same performance. For relatively small trainings (<400 billion tokens) we recommend using the recent CC-MAIN-2023-50 and CC-MAIN-2024-10. We will share more details about these experiments on our future technical report.

Dataset Creation

Curation Rationale

While multiple open-weights models have regularly been released in recent months, these releases often do not include the model's training data. With 🍷 FineWeb we aim to provide the open source community with a very large clean pretraining dataset that can be used to push the envelope on truly open source models (open source models where data is also released).

Source Data

The source data consists of webpages crawled by the CommonCrawl foundation over the 2013-2024 time period.

We then extracted the main page text from the html of each webpage, carefully filtered each sample and deduplicated each individual CommonCrawl dump/crawl.

While we originally intended to deduplicate the dataset as a whole, our ablations showed that training on a sampling of individually deduplicated dumps/crawls outperformed training on a sampling of all the dumps/crawls deduplicated together. We will discuss this further in our technical report.

Data processing steps

We used the 🏭 datatrove library to process the data. You can find a working script that launches the entire processing pipeline here.

The data processing pipeline consists of:

1. Url Filtering, removing documents originating from Malicious and NSFW websites, using both block-list as well as subwords detection
2. Trafilatura text extraction on the raw HTML from CommonCrawl’s warc files
3. FastText LanguageFilter, removing any document with en language score lower than 0.65
4. Quality filtering
   1. Gopher Repetition / Quality
   2. C4 Quality filters except terminal_punct rule
   3. FineWeb custom filters, consisting of heuristics for removing list-like documents, documents with repeated lines and documents with likely wrong line formatting.
5. MinHash deduplication with each crawl deduplicated individually (5-grams, 14x8 hash functions)
6. PII Formatting to anonymize email and public IP addresses

Annotations

We augment the original samples with the language, language_score and token_count annotations. The language related annotations are automatically generated by our language filter. token_count is generated by applying the gpt2 tokenizer to the text column.

Personal and Sensitive Information

We anonymize email addresses and public IP addresses.

For emails, we apply a regex pattern and replace any occurrence of an email address with either email@example.com or firstname.lastname@example.org. For IP addresses, we also employ a regex pattern and then further filter to only anonymize IP addresses allocated for public networks. Matched IP addresses are then replaced with one of the following randomly generated IP addresses, which at the time of dataset creation were not responding to ping requests: 22.214.171.124, 126.96.36.199, 188.8.131.52, 184.108.40.206, 220.127.116.11, and 18.104.22.168. We decided against applying regex patterns for phone numbers due to the high false positive rate.

Despite our efforts, given that 🍷 FineWeb is sourced from the internet at large, it is very likely that some personable identifiable information (PII) will be present. If you find your own PII in 🍷 FineWeb and would like it removed, please fill out our PII removal form.

Considerations for Using the Data

Social Impact of Dataset

With the release of this dataset we aim to make model training more accessible to the machine learning community at large.

While multiple open-weights models with strong performance have been publicly released in the past, more often than not these releases are not accompanied by the corresponding training dataset. This is unfortunate as the dataset specificities and characteristics have been demonstrated to have a very large impact and role in the performances of the models. As the creation of a high quality training dataset is a fundamental requirement to training an LLM capable of excelling at downstream tasks, with 🍷 FineWeb we (a) not only make the dataset creation process more transparent, by sharing our entire processing setup including the codebase used, we also (b) help alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.

Discussion of Biases

Efforts were made to minimize the amount of NSFW and toxic content present in the dataset by employing filtering on the URL level. However, there are still a significant number of documents present in the final dataset that could be considered toxic or contain harmful content. As 🍷 FineWeb was sourced from the web as a whole, any harmful biases typically present in it may be reproduced on our dataset.

We deliberately avoided using machine learning filtering methods that define text quality based on the similarity to a “gold” source such as wikipedia or toxicity classifiers as these methods have been known to disproportionately remove content in specific dialects and overclassify as toxic text related to specific social identities, respectively.

Other Known Limitations

As a consequence of some of the filtering steps applied, it is likely that code content is not prevalent in our dataset. If you are training a model that should also perform code tasks, we recommend you use 🍷 FineWeb with a code dataset, such as The Stack v2. You should also probably consider complementing 🍷 FineWeb with specialized curated sources (such as Wikipedia, for example) as they will likely have better formatting than the wikipedia content included in 🍷 FineWeb (we did not tailor the processing to individual websites).

Additional Information

Licensing Information

The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.

Future work

We plan to not only continue but also expand our efforts to create open-source high quality training datasets and to improve 🍷 FineWeb itself in future iterations.

Citation Information

@software{penedo2024fineweb,
  author = {Penedo, Guilherme and Kydlíček, Hynek and von Werra, Leandro and Wolf, Thomas},
  title = {FineWeb},
  month = April,
  year = 2024,
  doi = { 10.57967/hf/2092 },
  url = {https://huggingface.co/datasets/HuggingFaceFW/fineweb}
}