A newer version is available
See Latest Releases
Language
 
Sentences
Source Words
Russian
12,061,155
157,061,045
1,078,819,759
Bonus Release (Low resource languages) - Last Updates on Oct. 2024
English-Azerbaijani
3,158,025
47,117,416
3,158,025
47,117,416
3,158,025
47,117,416
336,067,622
5,513,087,127
English-Tajik
343,401
5,513,041
343,401
5,513,041
343,401
5,513,041
11,394,854
244,332,910
English-Armenian
1,988,287
35,997,571
1,988,287
35,997,571
1,988,287
35,997,571
31,671,924
233,771,297
English-Khmer v1
65,113
1,511,950
21,560,446
21,565,078
65,113
1,511,950
English-Burmese v1
31,374
661,577
40,590,354
40,595,755
31,374
661,577
English-Nepali v1
92,084
2,941,031
36,454,553
36,466,101
92,084
2,941,031
English-Pashto
26,321
692,651
2,587,950
2,593,163
26,321
692,651
English-Singhalese
217,407
5,791,982
38,720,907
38,724,422
217,407
5,791,982
English-Somali
14,879
506,201
28,387,922
28,396,227
14,879
506,201
English-Swahili
132,517
3,696,543
84,605,506
84,605,506
132,517
3,696,543
English-Tagalog
248,684
6,327,801
108,260,601
108,260,601
248,684
6,327,801
Bonus Release - Last Updates on October 2024
English-Hindi  
New
4,712,564
74,000,000
4,712,564
74,000,000
4,712,564
74,000,000
English-Indonesian  
New
7,133,323
109,000,000
7,133,323
109,000,000
7,133,323
109,000,000
English-Khmer v2  
New
1,501,304
23,000,000
1,501,304
23,000,000
1,501,304
23,000,000
English-Korean v2  
New
7,709,312
114,000,000
7,709,312
114,000,000
7,709,312
114,000,000
English-Lao  
New
1,994,053
27,000,000
1,994,053
27,000,000
1,994,053
27,000,000
English-Burmese v2  
New
1,666,530
28,000,000
1,666,530
28,000,000
1,666,530
28,000,000
English-Nepali v2  
New
2,243,954
32,000,000
2,243,954
32,000,000
2,243,954
32,000,000
English-Thai  
New
2,175,890
22,000,000
2,175,890
22,000,000
2,175,890
22,000,000
English-Vietnamese  
New
6,291,407
93,000,000
6,291,407
93,000,000
6,291,407
93,000,000
Polish-Czech
24,001,403
288,826,678
24,001,403
288,826,678
6,055,618,075
28,559,061,699
English-Ukrainian
13,354,365
505,831,880
13,354,365
505,831,880
235,700,383
5,832,658,894
English-Chinese
14,170,585
217,604,664
14,170,585
217,604,664
1,207,487,761
8,953,713,029
English-Russian
5,377,911
101,312,142
5,377,911
101,312,142
491,941,804
492,260,972
English-Korean
4,002,441
61,963,744
4,002,441
61,963,744
Dutch-French
2,687,331
60,504,313
2,687,331
60,504,313
38,164,560
770,141,393
Polish-German
916,522
18,883,576
916,522
18,883,576
11,060,105
202,765,359
Language Crawled Websites Download Details
In the proceedings of WMT 2019 release 3 of the corpus is used. For WMT 2018, the FILTERED v1.0 of the released corpus was used.
Two formats of BiCleaner files are provided. You can download either the TMX format or plaint TXT format. To effectively transform a TMX to a tab-separated text file Download TMXT tool.
Bulgarian 4,762
  File Size Sentence Pairs English Words
RAW v5.0 7.4GB 248,555,951 1,564,051,100
BiCleaner v5.0 692MB 2,586,277 55,725,444
RAW v3.0 10.2GB 288,395,110 1,552,588,179
BiCleaner v3.0 371MB 1,704,762 28,243,306
Zipporah v3.0 217MB 821,464 17,578,839
Croatian 8,889
  File Size Sentence Pairs English Words
RAW v5.0 8.33GB 273,330,006 1,738,164,401
BiCleaner v5.0 477MB 1,861,590 43,464,197
RAW v3.0 12.4GB 411,950,164 1,996,212,922
BiCleaner v3.0 265MB 1,568,947 23,531,438
Zipporah v3.0 260MB 1,004,131 18,004,931
BiCleaner v2.0 345MB 1,455,841 21,387,649
Zipporah v2.0 295MB 933,190 16,655,718
Czech 14,335
  File Size Sentence Pairs English Words
RAW v5.0 20GB 665,535,115 4,025,512,842
BiCleaner v5.0 1.2GB 5,280,149 117,385,158
RAW v3.0 33.9GB 1,189,317,247 5,621,562,488
BiCleaner v3.0 913MB 5,862,521 75,316,848
Zipporah v3.0 4.1GB 17,058,282 139,211,417
BiCleaner v2.0 1.2GB 5,488,589 69,182,264
Zipporah v2.0 4.5GB 15,846,424 123,222,290
RAW v1.0 21.5GB 818,784,053 -
Filtered v1.0 285MB 10,020,250 78,743,955
BiCleaner v1.2 237MB 2,367,609 38,913,821
Zipporah v1.2 529MB 9,982,508 78,943,174
Danish 19,776
  File Size Sentence Pairs English Words
RAW v5.0 17.8GB 447,743,455 3,347,135,236
BiCleaner v5.0 1066MB 4,606,183 106,565,546
RAW v3.0 24.8GB 586,535,848 3,484,768,564
BiCleaner v3.0 737MB 4,891,462 67,200,201
Zipporah v3.0 565MB 1,194,589 22,476,008
Dutch 17,887
  File Size Sentence Pairs English Words
RAW v5.0 32GB 1,101,087,006 6,792,400,704
BiCleaner v5.0 2.8GB 10,596,717 233,087,345
RAW v3.0 47.8GB 1,760,140,259 8,239,317,278
BiCleaner v3.0 2.2GB 10,408,489 143,294,712
Zipporah v3.0 1.1GB 3,291,804 56,744,571
BiCleaner v2.0 2.7GB 9,342,505 127,895,866
Zipporah v2.0 1.2GB 2,922,152 50,402,591
RAW v1.0 37.5GB 1,506,033,538 -
Filtered v1.0 168MB 2,560,472 45,149,412
BiCleaner v1.2 581MB 6,185,906 100,284,153
Zipporah v1.2 276MB 2,556,523 45,169,303
Estonian 9,522
  File Size Sentence Pairs English Words
RAW v5.0 4.5GB 168,091,382 915,074,587
BiCleaner v5.0 338MB 1,387,869 30,858,140
RAW v3.0 8.4GB 342,677,535 1,522,504,098
BiCleaner v3.0 198MB 1,064,078 17,725,513
Zipporah v3.0 214MB 1,163,994 17,105,752
BiCleaner v2.0 245MB 960,276 15,633,491
Zipporah v2.0 271MB 1,122,289 12,820,311
RAW v1.0 4.4GB 191,183,197 -
Filtered v1.0 74.7MB 1,298,103 13,134,231
Finnish 11,028
  File Size Sentence Pairs English Words
RAW v5.0 13.5GB 460,181,215 2,731,068,033
BiCleaner v5.0 704MB 3,097,223 66,385,933
RAW v3.0 20.6GB 736,050,617 3,494,554,815
BiCleaner v3.0 693MB 3,944,929 54,984,783
Zipporah v3.0 432MB 966,145 14,175,421
BiCleaner v2.0 985MB 3,632,447 49,751,376
Zipporah v2.0 459MB 831,170 12,692,508
RAW v1.0 12.7GB 504,805,915 -
Filtered v1.0 34.5MB 544,335 8,420,501
BiCleaner v1.2 177MB 1,982,774 29,979,317
Zipporah v1.2 59.8MB 621,728 9,481,646
French 48,498
  File Size Sentence Pairs English Words
RAW v5.0 128GB 4,273,819,421 24,983,683,983
BiCleaner v5.0 13.9GB 51,316,168 1,178,317,233
RAW v3.0 183GB 6,429,921,903 28,529,875,306
BiCleaner v3.0 9.3GB 40,417,114 640,273,938
Zipporah v3.0 15GB 39,615,885 791,250,385
BiCleaner v2.0 11.4GB 37,823,646 600,029,874
Zipporah v2.0 16.5GB 37,743,429 754,045,036
RAW v1.0 111GB 4,235,725,445 -
Filtered v1.0 2.1GB 27,622,881 546,401,428
BiCleaner v1.2 2.6GB 25,380,067 428,397,408
Zipporah v1.2 3.9GB 33,108,141 648,244,663
German 67,977
  File Size Sentence Pairs English Words
RAW v5.0 142.8GB 5,038,103,659 27,994,213,177
BiCleaner v5.0 11.07GB 36,936,714 929,818,868
RAW v3.0 211GB 7,387,809,953 32,358,035,774
BiCleaner v3.0 8.5GB 31,358,551 502,903,379
Zipporah v3.0 28.7GB 61,349,218 809,954,481
BiCleaner v2.0 9.8GB 27,702,949 456,442,715
Zipporah v2.0 26.3GB 55,849,341 740,849,699
RAW v1.0 121GB 4,591,582,415 -
Filtered v1.0 1.8GB 36,351,593 476,398,001
BiCleaner v1.2 1.8GB 17,378,982 302,274,816
Zipporah v1.2 3.3GB 40,546,537 522,204,110
Greek 11,343
  File Size Sentence Pairs English Words
RAW v5.0 16.2GB 640,502,801 3,768,712,672
BiCleaner v5.0 1122MB 3,830,643 88669279
RAW v3.0 24.7GB 740,094,469 3,340,324,438
BiCleaner v3.0 878MB 4,533,175 57,752,932
Zipporah v3.0 454MB 992,220 18,958,591
Hungarian 9,522
  File Size Sentence Pairs English Words
RAW v5.0 14.47GB 461,181,772 3,208,285,083
BiCleaner v5.0 1069MB 4,187,051 104,292,635
RAW v3.0 16.5GB 622,224,794 2,590,060,050
BiCleaner v3.0 456MB 3,160,496 32,151,740
Zipporah v3.0 360MB 1,023,875 17,235,595
BiCleaner v2.0 669MB 308,2486 31,764,228
Zipporah v2.0 338MB 902,412 15,054,278
Irish 1,283
  File Size Sentence Pairs English Words
RAW v5.0 4.66GB 64,628,733 667,211,260
BiCleaner v5.0 190MB 782,769 21,909,039
RAW v3.0 6.8GB 156,189,807 1,028,019,178
BiCleaner v3.0 117MB 607,734 15,473,067
Zipporah v3.0 154MB 744,375 14,525,892
BiCleaner v2.0 150MB 573,451 14,813,115
Zipporah v2.0 191MB 732,783 14,269,940
Italian 31,518
  File Size Sentence Pairs English Words
RAW v5.0 66GB 2,251,771,798 13,150,606,108
BiCleaner v5.0 6.02GB 22,100,078 533,512,632
RAW v3.0 91.3GB 3,333,886,336 14,519,224,940
BiCleaner v3.0 4.3GB 14,439,190 308,244,744
Zipporah v3.0 6.9GB 1,368,691 269,587,549
BiCleaner v2.0 5.0GB 17,224,855 264,324,830
Zipporah v2.0 6.7GB 12,252,492 231,025,420
RAW v1.0 45.1GB 1,727,688,019 -
Filtered v1.0 593MB 8,318,493 155,973,063
BiCleaner v1.2 963MB 11,790,134 147,402,459
Zipporah v1.2 1.2GB 12,065,631 212,026,083
Latvian 3,557
  File Size Sentence Pairs English Words
RAW v5.0 5GB 176,113,669 1,069,218,155
BiCleaner v5.0 286MB 1,019,003 23,656,140
RAW v3.0 7.7GB 262,685,954 1,371,257,575
BiCleaner v3.0 218MB 1,009,860 15,058,052
Zipporah v3.0 125MB 434,479 7,742,539
BiCleaner v2.0 270MB 1,133,362 16,744,306
Zipporah v2.0 137MB 386,447 6,066,997
RAW v1.0 4.9GB 173,585,643 -
Filtered v1.0 16MB 242,227 4,250,040
BiCleaner v1.2 43.3MB 406,742 6,995,228
Zipporah v1.2 26.9MB 241,546 4,247,908
Lithuanian 4,678
  File Size Sentence Pairs English Words
RAW v5.0 4.92GB 198,101,611 963,384,230
BiCleaner v5.0 375MB 1,270,933 27,214,054
RAW v3.0 7.8GB 294,568,032 1,198,118,449
BiCleaner v3.0 273MB 1,368,691 19,471,370
Zipporah v3.0 128MB 432,724 6,727,629
BiCleaner v2.0 330MB 1,133,362 16,744,306
Zipporah v2.0 144MB 386,447 6,066,997
Maltese 672
  File Size Sentence Pairs English Words
RAW v5.0 173MB 3,693,930 38,492,028
BiCleaner v5.0 38MB 177,244 4,252,814
RAW v3.0 723MB 17,602,902 164,119,571
BiCleaner v3.0 38.1MB 227,499 4,429,648
Zipporah v3.0 33.9MB 154,038 2,143,321
BiCleaner v2.0 41.6MB 198,537 3,884,509
Zipporah v2.0 38.6MB 137,318 1,919,196
Polish 13,357
  File Size Sentence Pairs English Words
RAW v5.0 22.6GB 723,052,912 4,123,972,411
BiCleaner v5.0 1.6GB 6,382,371 145,802,939
RAW v3.0 33.8GB 1,259,312,618 5,555,536,170
BiCleaner v3.0 1.3GB 6,806,796 94,612,131
Zipporah v3.0 709MB 1,748,865 29,205,973
BiCleaner v2.0 1.6GB 5,787,436 81,662,507
Zipporah v2.0 692MB 1,488,056 26,329,826
RAW v1.0 24.4GB 984,884,968 -
Filtered v1.0 85.7MB 1,275,162 22,092,316
BiCleaner v1.2 330MB 3,270,262 55,467,253
Zipporah v1.2 143MB 1,269,300 22,079,132
Portuguese 18,887
  File Size Sentence Pairs English Words
RAW v5.0 34.8GB 1,068,161,866 6,537,298,891
BiCleaner v5.0 3.3GB 13,860,663 299,634,135
RAW v3.0 50.6GB 1,763,439,122 8,465,738,356
BiCleaner v3.0 2GB 11,698,633 171,495,357
Zipporah v3.0 1.1GB 3,834,613 79,794,493
BiCleaner v2.0 2.3GB 9,740,600 148,240,776
Zipporah v2.0 1.1GB 3,454,349 72,592,387
RAW v1.0 34.1GB 1,357,911,799 -
Filtered v1.0 222MB 2,809,381 57,392,721
BiCleaner v1.2 611MB 6,436,491 93,021,518
Zipporah v1.2 366MB 3,056,920 60,180,429
Romanian 9,335
  File Size Sentence Pairs English Words
RAW v5.0 15.2GB 510,209,923 3,034,045,929
BiCleaner v5.0 728MB 2,870,687 62,189,306
RAW v3.0 22.7GB 793,759,210 4,059,255,214
BiCleaner v3.0 592MB 3,284,810 49,494,227
Zipporah v3.0 621MB 2,766,706 38,673,891
BiCleaner v2.0 713MB 2,684,189 39,958,916
Zipporah v2.0 607MB 2,537,851 34,596,458
RAW v1.0 17.2GB 635,709,587 -
Filtered v1.0 105MB 2,459,752 32,800,110
BiCleaner v1.2 159MB 1,592,692 27,531,812
Zipporah v1.2 151MB 2,459,408 32,806,629
Slovak 7,980
  File Size Sentence Pairs English Words
RAW v5.0 6.05GB 269,067,288 1,416,750,646
BiCleaner v5.0 568MB 2,365,339 45,636,383
RAW v3.0 8.9GB 334,903,774 1,418,785,612
BiCleaner v3.0 540MB 2,759,451 35,247,648
Zipporah v3.0 163MB 599,671 11,134,061
Slovenian 5,016
  File Size Sentence Pairs English Words
RAW v5.0 4.07GB 175,682,959 1,003,867,134
BiCleaner v5.0 406MB 1,406,645 31,855,427
RAW v3.0 6GB 208,466,320 967,461,921
BiCleaner v3.0 309MB 1,386,819 19,915,661
Zipporah v3.0 178MB 478,444 9,060,846
Spanish 36,211
  File Size Sentence Pairs English Words
RAW v5.0 80.4GB 2,674,900,280 16,598,620,402
BiCleaner v5.0 9.6GB 38,971,348 897,891,704
RAW v3.0 111GB 3,959,845,706 18,128,847,778
BiCleaner v3.0 6.4GB 30,535,457 491,951,545
Zipporah v3.0 8.6GB 24,634,419 505,890,391
BiCleaner v2.0 7.2GB 25,473,946 412,852,386
Zipporah v2.0 8.7GB 21,286,014 437,009,844
RAW v1.0 60.9GB 2,368,243,619 -
Filtered v1.0 1.3GB 16,001,341 325,745,201
BiCleaner v1.2 1.8GB 17,511,545 303,161,256
Zipporah v1.2 2.2GB 18,197,039 366,172,313
Swedish 13,616
  File Size Sentence Pairs English Words
RAW v5.0 16.54GB 620,338,561 3,496,650,816
BiCleaner v5.0 1542MB 6,079,175 138,264,978
RAW v3.0 22.5GB 739,146,200 3,217,514,612
BiCleaner v3.0 967MB 4,960,282 79,278,861
Zipporah v3.0 675MB 1,913,487 34,574,753
Bonus Release
Dutch-French 7,700
  File Size Sentence Pairs Dutch Words French Words
RAW 1.8GB 38,164,560 770,141,393 817,973,481
BiCleaner 752MB 2,687,331 60,504,313 64,650,034
Polish-German 5,549
  File Size Sentence Pairs Polish Words German Words
RAW 479MB 11,060,105 202,765,359 198,442,547
BiCleaner 216MB 916,522 18,883,576 20,271,637
Extra Languages in release v1.0
Russian 14,035 RAW v1.0 FILTERED v1.0
  File Size Sentence Pairs English Words
RAW v1.0 38GB 1,078,819,759 -
Filtered v1.0 637MB 12,061,155 157,061,045
  • Releases 4 and earlier included unaligned sentences in the raw file with one side empty. Release 5 removes these sentences from the raw file, explaining why the raw sizes dropped.
  • FILTERED v1.0 of the corpus is very rough and it is significantly refined in new releases.