esCorpius-m: A massive multilingual crawling corpus with a focus on Spanish Gutiérrez Fandiño, Asier Pérez Fernández, David Armengol-Estapé, Jordi Griol Barres, David Kharitonova, Ksenia Callejas Carrión, Zoraida Corpus Dataset Massive In recent years, transformer-based models have played a significant role in advancing lan- guage modeling for natural language processing. However, they require substantial amounts of data and there is a shortage of high-quality non-English corpora. Some recent initiatives have introduced multilingual datasets obtained through web crawling. However, there are notable limitations in the results for some languages, including Spanish. These datasets are either smaller compared to other languages or suffer from lower quality due to insufficient cleaning and deduplication. In this paper, we present ESCORPIUS-M, a multilingual corpus extracted from around 1 petabyte of Common Crawl data. It is the most extensive corpus for some languages with such a level of high-quality content extraction, cleanliness, and deduplication. Our data curation process involves an efficient cleaning pipeline and various deduplication methods that maintain the integrity of document and paragraph boundaries. We also ensure compliance with EU regulations by retaining both the source web page URL and the WARC shared origin URL 2024-02-07T10:58:59Z 2024-02-07T10:58:59Z 2023 info:eu-repo/semantics/article https://hdl.handle.net/10481/88558 10.3390/app132212155 eng MCIN/AEI/10.13039/501100011033 NextGenerationEU/PRTR http://creativecommons.org/licenses/by-nc/4.0/ info:eu-repo/semantics/openAccess Atribución-NoComercial 4.0 Internacional MDPI