Version 5.1 builds upon the same raw corpus as version 5. Thanks to improvements in filtering procedure, the official subset extracted as version 5.1 is now higher in quantity for almost all language pairs (but ga, de, sl and et). Quality measured extrinsically through MT for several language pairs shows also improvement in quality.
Corpora sizes and download links are available from ParaCrawl's website (https://paracrawl.eu/v5-1).
This is the official release to be used in WMT20. Stay tuned for more news and follow us on twitter @ParaCrawl.
The latest release of the ParaCrawl OpenSource Pipeline (Bitextor) is available on Github.
The corpus and software are released as part of the ParaCrawl project co-financed by the European Union through the Connecting Europe Facility (CEF). This release used an existing toolchain that will be refined throughout the project and expanded to cover all official EU languages (23 languages parallel with English).
The corpora are released under the Creative Commons CC0 license ("no rights reserved"). (https://creativecommons.org/share-your-work/public-domain/cc0/)