ParaCrawl Release v1

A newer version is available

See Latest Releases

Language

Sentences

Source Words

Russian

12,061,155

157,061,045

1,078,819,759

Bonus Release (Low resource languages) - Last Updates on Sep 2021

Khmer

65,113

1,511,950

21,560,446

21,565,078

65,113

1,511,950

Burmese

31,374

661,577

40,590,354

40,595,755

31,374

661,577

Nepali

92,084

2,941,031

36,454,553

36,466,101

92,084

2,941,031

Pashto

26,321

692,651

2,587,950

2,593,163

26,321

692,651

Singhalese

217,407

5,791,982

38,720,907

38,724,422

217,407

5,791,982

Somali

14,879

506,201

28,387,922

28,396,227

14,879

506,201

Swahili

132,517

3,696,543

84,605,506

132,517

3,696,543

Tagalog

248,684

6,327,801

108,260,601

248,684

6,327,801

Bonus Release - Last Updates on Sep 2021

Polish-Czech

New

24,001,403

288,826,678

24,001,403

288,826,678

6,055,618,075

28,559,061,699

Ukrainian

New

13,354,365

505,831,880

13,354,365

505,831,880

235,700,383

5,832,658,894

Chinese

New

14,170,585

217,604,664

14,170,585

217,604,664

1,207,487,761

8,953,713,029

Russian

5,377,911

101,312,142

5,377,911

101,312,142

491,941,804

492,260,972

English-Korean

4,002,441

61,963,744

4,002,441

61,963,744

Dutch-French

2,687,331

60,504,313

2,687,331

60,504,313

38,164,560

770,141,393

Polish-German

916,522

18,883,576

916,522

18,883,576

11,060,105

202,765,359

Language

Crawled Websites

Download

Details

In the proceedings of WMT 2019 release 3 of the corpus is used. For WMT 2018, the FILTERED v1.0 of the released corpus was used.

Two formats of BiCleaner files are provided. You can download either the TMX format or plaint TXT format. To effectively transform a TMX to a tab-separated text file Download TMXT tool.

Bulgarian

4,762

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	7.4GB	248,555,951	1,564,051,100
BiCleaner v5.0	692MB	2,586,277	55,725,444
RAW v4.0	10.2GB	288,395,110	1,552,588,179
BiCleaner v4.0	284MB	1,039,885	21,109,546
RAW v3.0	10.2GB	288,395,110	1,552,588,179
BiCleaner v3.0	371MB	1,704,762	28,243,306
Zipporah v3.0	217MB	821,464	17,578,839

Croatian

8,889

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	8.33GB	273,330,006	1,738,164,401
BiCleaner v5.0	477MB	1,861,590	43,464,197
RAW v4.0	12.4GB	411,950,164	1,996,212,922
BiCleaner v4.0	246MB	1,002,053	19,904,218
RAW v3.0	12.4GB	411,950,164	1,996,212,922
BiCleaner v3.0	265MB	1,568,947	23,531,438
Zipporah v3.0	260MB	1,004,131	18,004,931
BiCleaner v2.0	345MB	1,455,841	21,387,649
Zipporah v2.0	295MB	933,190	16,655,718

Czech

14,335

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	20GB	665,535,115	4,025,512,842
BiCleaner v5.0	1.2GB	5,280,149	117,385,158
RAW v4.0	33.9GB	1,189,317,247	5,621,562,488
BiCleaner v4.0	697MB	2,981,949	48,918,151
RAW v3.0	33.9GB	1,189,317,247	5,621,562,488
BiCleaner v3.0	913MB	5,862,521	75,316,848
Zipporah v3.0	4.1GB	17,058,282	139,211,417
BiCleaner v2.0	1.2GB	5,488,589	69,182,264
Zipporah v2.0	4.5GB	15,846,424	123,222,290
RAW v1.0	21.5GB	818,784,053	-
Filtered v1.0	285MB	10,020,250	78,743,955
BiCleaner v1.2	237MB	2,367,609	38,913,821
Zipporah v1.2	529MB	9,982,508	78,943,174

Danish

19,776

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	17.8GB	447,743,455	3,347,135,236
BiCleaner v5.0	1066MB	4,606,183	106,565,546
RAW v4.0	24.8GB	586,535,848	3,484,768,564
BiCleaner v4.0	553MB	2,414,895	48,240,290
RAW v3.0	24.8GB	586,535,848	3,484,768,564
BiCleaner v3.0	737MB	4,891,462	67,200,201
Zipporah v3.0	565MB	1,194,589	22,476,008

Dutch

17,887

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	32GB	1,101,087,006	6,792,400,704
BiCleaner v5.0	2.8GB	10,596,717	233,087,345
RAW v4.0	47.8GB	1,760,140,259	8,239,317,278
BiCleaner v4.0	1.7GB	5,659,268	108,197,376
RAW v3.0	47.8GB	1,760,140,259	8,239,317,278
BiCleaner v3.0	2.2GB	10,408,489	143,294,712
Zipporah v3.0	1.1GB	3,291,804	56,744,571
BiCleaner v2.0	2.7GB	9,342,505	127,895,866
Zipporah v2.0	1.2GB	2,922,152	50,402,591
RAW v1.0	37.5GB	1,506,033,538	-
Filtered v1.0	168MB	2,560,472	45,149,412
BiCleaner v1.2	581MB	6,185,906	100,284,153
Zipporah v1.2	276MB	2,556,523	45,169,303

Estonian

9,522

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	4.5GB	168,091,382	915,074,587
BiCleaner v5.0	338MB	1,387,869	30,858,140
RAW v4.0	8.4GB	342,677,535	1,522,504,098
BiCleaner v4.0	202MB	853,422	16,537,397
RAW v3.0	8.4GB	342,677,535	1,522,504,098
BiCleaner v3.0	198MB	1,064,078	17,725,513
Zipporah v3.0	214MB	1,163,994	17,105,752
BiCleaner v2.0	245MB	960,276	15,633,491
Zipporah v2.0	271MB	1,122,289	12,820,311
RAW v1.0	4.4GB	191,183,197	-
Filtered v1.0	74.7MB	1,298,103	13,134,231

Finnish

11,028

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	13.5GB	460,181,215	2,731,068,033
BiCleaner v5.0	704MB	3,097,223	66,385,933
RAW v4.0	20.6GB	736,050,617	3,494,554,815
BiCleaner v4.0	469MB	2,156,069	41,564,859
RAW v3.0	20.6GB	736,050,617	3,494,554,815
BiCleaner v3.0	693MB	3,944,929	54,984,783
Zipporah v3.0	432MB	966,145	14,175,421
BiCleaner v2.0	985MB	3,632,447	49,751,376
Zipporah v2.0	459MB	831,170	12,692,508
RAW v1.0	12.7GB	504,805,915	-
Filtered v1.0	34.5MB	544,335	8,420,501
BiCleaner v1.2	177MB	1,982,774	29,979,317
Zipporah v1.2	59.8MB	621,728	9,481,646

French

48,498

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	128GB	4,273,819,421	24,983,683,983
BiCleaner v5.0	13.9GB	51,316,168	1,178,317,233
RAW v4.0	183GB	6,429,921,903	28,529,875,306
BiCleaner v4.0	9.9GB	31,374,161	664,924,148
RAW v3.0	183GB	6,429,921,903	28,529,875,306
BiCleaner v3.0	9.3GB	40,417,114	640,273,938
Zipporah v3.0	15GB	39,615,885	791,250,385
BiCleaner v2.0	11.4GB	37,823,646	600,029,874
Zipporah v2.0	16.5GB	37,743,429	754,045,036
RAW v1.0	111GB	4,235,725,445	-
Filtered v1.0	2.1GB	27,622,881	546,401,428
BiCleaner v1.2	2.6GB	25,380,067	428,397,408
Zipporah v1.2	3.9GB	33,108,141	648,244,663

German

67,977

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	142.8GB	5,038,103,659	27,994,213,177
BiCleaner v5.0	11.07GB	36,936,714	929,818,868
RAW v4.0	211GB	7,387,809,953	32,358,035,774
BiCleaner v4.0	5.4GB	16,264,450	307,786,150
RAW v3.0	211GB	7,387,809,953	32,358,035,774
BiCleaner v3.0	8.5GB	31,358,551	502,903,379
Zipporah v3.0	28.7GB	61,349,218	809,954,481
BiCleaner v2.0	9.8GB	27,702,949	456,442,715
Zipporah v2.0	26.3GB	55,849,341	740,849,699
RAW v1.0	121GB	4,591,582,415	-
Filtered v1.0	1.8GB	36,351,593	476,398,001
BiCleaner v1.2	1.8GB	17,378,982	302,274,816
Zipporah v1.2	3.3GB	40,546,537	522,204,110

Greek

11,343

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	16.2GB	640,502,801	3,768,712,672
BiCleaner v5.0	1122MB	3,830,643	88669279
RAW v4.0	24.7GB	740,094,469	3,340,324,438
BiCleaner v4.0	572MB	1,985,233	38,322,532
RAW v3.0	24.7GB	740,094,469	3,340,324,438
BiCleaner v3.0	878MB	4,533,175	57,752,932
Zipporah v3.0	454MB	992,220	18,958,591

Hungarian

9,522

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	14.47GB	461,181,772	3,208,285,083
BiCleaner v5.0	1069MB	4,187,051	104,292,635
RAW v4.0	16.5GB	622,224,794	2,590,060,050
BiCleaner v4.0	482MB	1,901,342	30,835,267
RAW v3.0	16.5GB	622,224,794	2,590,060,050
BiCleaner v3.0	456MB	3,160,496	32,151,740
Zipporah v3.0	360MB	1,023,875	17,235,595
BiCleaner v2.0	669MB	308,2486	31,764,228
Zipporah v2.0	338MB	902,412	15,054,278

Irish

1,283

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	4.66GB	64,628,733	667,211,260
BiCleaner v5.0	190MB	782,769	21,909,039
RAW v4.0	6.8GB	156,189,807	1,194,451,883
BiCleaner v4.0	75.9MB	357,399	8,241,515
RAW v3.0	6.8GB	156,189,807	1,028,019,178
BiCleaner v3.0	117MB	607,734	15,473,067
Zipporah v3.0	154MB	744,375	14,525,892
BiCleaner v2.0	150MB	573,451	14,813,115
Zipporah v2.0	191MB	732,783	14,269,940

Italian

31,518

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	66GB	2,251,771,798	13,150,606,108
BiCleaner v5.0	6.02GB	22,100,078	533,512,632
RAW v4.0	91.3GB	3,333,886,336	14,519,224,940
BiCleaner v4.0	3.6GB	12,162,239	260,361,435
RAW v3.0	91.3GB	3,333,886,336	14,519,224,940
BiCleaner v3.0	4.3GB	14,439,190	308,244,744
Zipporah v3.0	6.9GB	1,368,691	269,587,549
BiCleaner v2.0	5.0GB	17,224,855	264,324,830
Zipporah v2.0	6.7GB	12,252,492	231,025,420
RAW v1.0	45.1GB	1,727,688,019	-
Filtered v1.0	593MB	8,318,493	155,973,063
BiCleaner v1.2	963MB	11,790,134	147,402,459
Zipporah v1.2	1.2GB	12,065,631	212,026,083

Latvian

3,557

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	5GB	176,113,669	1,069,218,155
BiCleaner v5.0	286MB	1,019,003	23,656,140
RAW v4.0	7.7GB	262,685,954	1,371,257,575
BiCleaner v4.0	158MB	553,060	10,996,032
RAW v3.0	7.7GB	262,685,954	1,371,257,575
BiCleaner v3.0	218MB	1,009,860	15,058,052
Zipporah v3.0	125MB	434,479	7,742,539
BiCleaner v2.0	270MB	1,133,362	16,744,306
Zipporah v2.0	137MB	386,447	6,066,997
RAW v1.0	4.9GB	173,585,643	-
Filtered v1.0	16MB	242,227	4,250,040
BiCleaner v1.2	43.3MB	406,742	6,995,228
Zipporah v1.2	26.9MB	241,546	4,247,908

Lithuanian

4,678

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	4.92GB	198,101,611	963,384,230
BiCleaner v5.0	375MB	1,270,933	27,214,054
RAW v4.0	7.8GB	294,568,032	1,198,118,449
BiCleaner v4.0	231MB	844,643	15,087,805
RAW v3.0	7.8GB	294,568,032	1,198,118,449
BiCleaner v3.0	273MB	1,368,691	19,471,370
Zipporah v3.0	128MB	432,724	6,727,629
BiCleaner v2.0	330MB	1,133,362	16,744,306
Zipporah v2.0	144MB	386,447	6,066,997

Maltese

672

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	173MB	3,693,930	38,492,028
BiCleaner v5.0	38MB	177,244	4,252,814
RAW v4.0	723MB	17,602,902	164,119,571
BiCleaner v4.0	39.9MB	195,510	4,100,912
RAW v3.0	723MB	17,602,902	164,119,571
BiCleaner v3.0	38.1MB	227,499	4,429,648
Zipporah v3.0	33.9MB	154,038	2,143,321
BiCleaner v2.0	41.6MB	198,537	3,884,509
Zipporah v2.0	38.6MB	137,318	1,919,196

Polish

13,357

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	22.6GB	723,052,912	4,123,972,411
BiCleaner v5.0	1.6GB	6,382,371	145,802,939
RAW v4.0	33.8GB	1,259,312,618	5,555,536,170
BiCleaner v4.0	970MB	3,503,276	65,618,419
RAW v3.0	33.8GB	1,259,312,618	5,555,536,170
BiCleaner v3.0	1.3GB	6,806,796	94,612,131
Zipporah v3.0	709MB	1,748,865	29,205,973
BiCleaner v2.0	1.6GB	5,787,436	81,662,507
Zipporah v2.0	692MB	1,488,056	26,329,826
RAW v1.0	24.4GB	984,884,968	-
Filtered v1.0	85.7MB	1,275,162	22,092,316
BiCleaner v1.2	330MB	3,270,262	55,467,253
Zipporah v1.2	143MB	1,269,300	22,079,132

Portuguese

18,887

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	34.8GB	1,068,161,866	6,537,298,891
BiCleaner v5.0	3.3GB	13,860,663	299,634,135
RAW v4.0	50.6GB	1,763,439,122	8,465,738,356
BiCleaner v4.0	2GB	8,141,940	156,125,200
RAW v3.0	50.6GB	1,763,439,122	8,465,738,356
BiCleaner v3.0	2GB	11,698,633	171,495,357
Zipporah v3.0	1.1GB	3,834,613	79,794,493
BiCleaner v2.0	2.3GB	9,740,600	148,240,776
Zipporah v2.0	1.1GB	3,454,349	72,592,387
RAW v1.0	34.1GB	1,357,911,799	-
Filtered v1.0	222MB	2,809,381	57,392,721
BiCleaner v1.2	611MB	6,436,491	93,021,518
Zipporah v1.2	366MB	3,056,920	60,180,429

Romanian

9,335

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	15.2GB	510,209,923	3,034,045,929
BiCleaner v5.0	728MB	2,870,687	62,189,306
RAW v4.0	22.7GB	793,759,210	4,059,255,214
BiCleaner v4.0	498MB	1,952,043	39,882,223
RAW v3.0	22.7GB	793,759,210	4,059,255,214
BiCleaner v3.0	592MB	3,284,810	49,494,227
Zipporah v3.0	621MB	2,766,706	38,673,891
BiCleaner v2.0	713MB	2,684,189	39,958,916
Zipporah v2.0	607MB	2,537,851	34,596,458
RAW v1.0	17.2GB	635,709,587	-
Filtered v1.0	105MB	2,459,752	32,800,110
BiCleaner v1.2	159MB	1,592,692	27,531,812
Zipporah v1.2	151MB	2,459,408	32,806,629

Slovak

7,980

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	6.05GB	269,067,288	1,416,750,646
BiCleaner v5.0	568MB	2,365,339	45,636,383
RAW v4.0	8.9GB	334,903,774	1,418,785,612
BiCleaner v4.0	359MB	1,591,831	26,711,854
RAW v3.0	8.9GB	334,903,774	1,418,785,612
BiCleaner v3.0	540MB	2,759,451	35,247,648
Zipporah v3.0	163MB	599,671	11,134,061

Slovenian

5,016

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	4.07GB	175,682,959	1,003,867,134
BiCleaner v5.0	406MB	1,406,645	31,855,427
RAW v4.0	6GB	208,466,320	967,461,921
BiCleaner v4.0	197MB	660,161	14,489,659
RAW v3.0	6GB	208,466,320	967,461,921
BiCleaner v3.0	309MB	1,386,819	19,915,661
Zipporah v3.0	178MB	478,444	9,060,846

Spanish

36,211

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	80.4GB	2,674,900,280	16,598,620,402
BiCleaner v5.0	9.6GB	38,971,348	897,891,704
RAW v4.0	111GB	3,959,845,706	18,128,847,778
BiCleaner v4.0	6.1GB	21,987,267	476,409,854
RAW v3.0	111GB	3,959,845,706	18,128,847,778
BiCleaner v3.0	6.4GB	30,535,457	491,951,545
Zipporah v3.0	8.6GB	24,634,419	505,890,391
BiCleaner v2.0	7.2GB	25,473,946	412,852,386
Zipporah v2.0	8.7GB	21,286,014	437,009,844
RAW v1.0	60.9GB	2,368,243,619	-
Filtered v1.0	1.3GB	16,001,341	325,745,201
BiCleaner v1.2	1.8GB	17,511,545	303,161,256
Zipporah v1.2	2.2GB	18,197,039	366,172,313

Swedish

13,616

RAW v5.0
BICLEANER v5.0 TMX BICLEANER v5.0 TXT

	File Size	Sentence Pairs	English Words
RAW v5.0	16.54GB	620,338,561	3,496,650,816
BiCleaner v5.0	1542MB	6,079,175	138,264,978
RAW v4.0	22.5GB	739,146,200	3,217,514,612
BiCleaner v4.0	857MB	3,476,729	70,088,534
RAW v3.0	22.5GB	739,146,200	3,217,514,612
BiCleaner v3.0	967MB	4,960,282	79,278,861
Zipporah v3.0	675MB	1,913,487	34,574,753

Bonus Release

Dutch-French

7,700

RAW
BICLEANER TMX BICLEANER TXT

	File Size	Sentence Pairs	Dutch Words	French Words
RAW	1.8GB	38,164,560	770,141,393	817,973,481
BiCleaner	752MB	2,687,331	60,504,313	64,650,034

Polish-German

5,549

RAW
BICLEANER TMX BICLEANER TXT

	File Size	Sentence Pairs	Polish Words	German Words
RAW	479MB	11,060,105	202,765,359	198,442,547
BiCleaner	216MB	916,522	18,883,576	20,271,637

Extra Languages in release v1.0

Russian

14,035

RAW v1.0 FILTERED v1.0

	File Size	Sentence Pairs	English Words
RAW v1.0	38GB	1,078,819,759	-
Filtered v1.0	637MB	12,061,155	157,061,045

Releases 4 and earlier included unaligned sentences in the raw file with one side empty. Release 5 removes these sentences from the raw file, explaining why the raw sizes dropped.
FILTERED v1.0 of the corpus is very rough and it is significantly refined in new releases.

Bitextor

ParaCrawl Open Source pipeline

Bitextor is a tool for automatically harvesting bitexts from multilingual websites.

View release on Github

Quickstart

Docker

If you want to easily install Bitextor, just use the Docker version:

docker pull bitextor/bitextor docker run --name bitextor bitextor/bitextor

Docker Image

Corset

Filtering

Searching

Perform searches on ParaCrawl corpora or get filtered subsets from it using Corset.

View release on Github

KEOPS

Quality Evaluation

KEOPS provides a complete tool for manual evaluation of parallel sentences and other linguistic tasks.

View release on Github

Bicleaner

Classifier

Bicleaner (bicleaner-classify) is a tool in Python that aims at detecting noisy sentence pairs in a parallel corpus. It indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0). Sentence pairs considered very noisy are scored with 0.

View release on Github

Third Party Releases

JParaCrawl

JParaCrawl is the largest publicly available English-Japanese parallel corpus created by NTT Communication Science Laboratories. It was created by largely crawling the web and automatically aligning parallel sentences.

Read More

Citing ParaCrawl

Research

If you want to cite ParaCrawl, please refer to: ParaCrawl: Web-Scale Acquisition of Parallel Corpora

Broader/Continued Web-Scale Provision of Parallel Corpora for European Languages

VersionRelease Date

Checkout the bonus releases

A Japanese version is availablethrough JParaCrawl (3rd Party Releases)

Version
Release Date

A Japanese version is available
through JParaCrawl (3rd Party Releases)