Skip to content

Creating super-parallel corpora of more than 1500+ unique languages for NLP research

License

Notifications You must be signed in to change notification settings

ehsanasgari/1000Langs

Repository files navigation

1000Langs is a super-parallal corpora crawler for multilingual NLP and Computational Linguistics

Developer and maintainer: Ehsaneddin Asgari (asgari [at] cis [dot] lmu [dot] de)
Please feel free to report any technical issue by sending an email or reporting an issue here.
PI: Prof. Hinrich Schuetze


Installation and running

The 1000Langs is written in Python3. For the detailed installation and running guideline see the installation guideline .


List of covered languages

The 1000Langs super-parallel crawler currently supports crawling of 1500+ unique languages, that are crawled through multiple sources of bible corpora. Here you can find the list of languages covered by 1000Langs crawler to date (Dec 2018). These language are subject to change based on changes in the crawling sources.

Language ISO 639-3 Code Language Name Max number of verses available in a translation
aah Abu' Arapesh 7904
aai Arifama-Miniafia 7959
aak Ankave 7959
aau Abau 7959
aaz Amarasi 6592
abi Abidji 7925
abt Ambulas 7959
abx Inabaknon 7942
aby Aneme Wake 7959
aca Achagua 4418
acc NaN 7930
acd Gikyode 7911
ace Achinese 29713
acf Saint Lucian Creole French 7958
ach Acoli 7957
acn Achang 7949
acr Achi 7959
acu Achuar-Shiwiar 7934
ade Adele 7956
adh Adhola 7959
adi Adi 5053
adj Adioukrou 7955
adl Galo 5057
adz Adzera 1533
aeb Tunisian Arabic 5059
aer Eastern Arrernte 5059
aeu Akeu 7958
aey Amele 9492
afr Afrikaans 5059
agd Agarabi 7959
agg Angor 7959
agm Angaataha 7959
agn Agutaynen 7955
agr Aguaruna 7946
agt Central Cagayan Agta 7958
agu Aguacateco 7935
agw Kahua 5057
agx Aghul 1151
ahk Akha 8433
aia Arosi 7946
aii Assyrian Neo-Aramaic 7520
aim Aimol 5059
ain Ainu (Japan) 5052
aji Ajië 5059
ajz Amri Karbi 5059
aka Akan 7959
akb Batak Angkola 7958
ake Akawaio 7958
akh Angal Heneng 7959
akp Siwu 7959
ald Alladian 5058
alj Alangan 7923
alp Alune 7912
alq Algonquin 5058
als Tosk Albanian 7957
alt Southern Altai 7950
alz Alur 7957
ame Yanesha' 7911
amf Hamer-Banna 7954
amh Amharic 7957
amk Ambai 7954
amm Ama (Papua New Guinea) 7959
amn Amanab 7959
amp Alamblak 7959
amr Amarakaeri 7959
amu Guerrero Amuzgo 7947
anm Anal 5059
ann Obolo 7956
anv Denya 7944
any Anyin 6130
aoj Mufian 7959
aom Ömie 7959
aon Bumbita Arapesh 7959
aoz Uab Meto 7953
apb Sa'a 5059
ape Bukiyip 7959
apn Apinayé 7957
apr Arop-Lokep 7959
apt Apatani 5056
apu Apurinã 5057
apw Western Apache 5057
apy Apalaí 7958
apz Safeyoka 7959
arb Standard Arabic 31103
arl Arabela 7959
arn Mapudungun 7959
arp Arapaho 1151
ary Moroccan Arabic 5058
arz Egyptian Arabic 31102
asg Cishingini 7955
asm Assamese 5059
aso Dano 7959
ata Pele-Ata 9540
atb Zaiwa 7959
atd Ata Manobo 7958
atg Ivbie North-Okpela-Arhe 7958
atq Aralle-Tabulahan 7942
att Pamplona Atta 5059
auc Waorani 7958
aui Anuki 4169
auy Awiyaana 7959
ava Avaric 7952
avn Avatime 7957
avt Au 7959
avu Avokaya 7593
awa Awadhi 7910
awb Awa (Papua New Guinea) 7959
awx Awara 1666
ayp North Mesopotamian Arabic 7958
ayr Central Aymara 30700
azb South Azerbaijani 31103
azg San Pedro Amuzgos Amuzgo 7957
azz Highland Puebla Nahuatl 7947
bak Bashkir 7940
ban Balinese 7944
bao Waimaha 8010
bav Vengo 7957
bba Baatonum 30920
bbb Barai 13142
bbc Batak Toba 31166
bbj Ghomálá' 5058
bbk Babanki 5059
bbo Northern Bobo Madaré 7956
bbr Girawa 7959
bcc Southern Balochi 1885
bch Bariai 10705
bci Baoulé 5058
bcl Central Bikol 7957
bco Kaluli 1651
bcw Bana 7898
bdd Bunama 7959
bdh Baka (South Sudan) 6592
bdq Bahnar 5059
bea Beaver 678
bef Benabena 7959
bel Belarusian 5057
bem Bemba (Zambia) 5057
ben Bengali 31099
beq Beembe 5057
bex Jur Modo 7957
bfd Bafut 7956
bfo Malba Birifor 7941
bgr Bawm Chin 7867
bgs Tagabawa 7958
bgt Bughotu 7861
bgz Banggai 7945
bhg Binandere 2836
bhl Bimin 9820
bhw Biak 7956
bhz Bada (Indonesia) 7915
bib Bissa 7947
big Biangai 7959
bim Bimoba 28202
bis Bislama 5055
biu Biete 5059
biv Southern Birifor 7942
bjp Fanamaket 2018
bjr Binumarien 10705
bjv Bedjond 7957
bkd Binukid 7959
bkl Berik 7938
bkq Bakairí 7958
bku Buhid 7930
bkv Bekwarra 7959
bla Siksika 1071
blt Tai Dam 7943
blw Balangao 7279
blz Balantak 7958
bmh Kein 7959
bmk Ghayavi 678
bmq Bomu 7957
bmr Muinane 7950
bmu Somba-Siawari 9625
bmv Bum 7959
bnj Eastern Tawbuid 7944
bno Bantoanon 5058
bnp Bola 14602
boa Bora 7651
bod Tibetan 23145
boj Anjam 7958
bom Berom 5058
bon Bine 7959
bov Tuwuli 7957
box Buamu 7944
bpr Koronadal Blaan 7928
bps Sarangani Blaan 7927
bqc Boko (Benin) 31073
bqj Bandial 5057
bqp Busa 7958
bre Breton 5057
bru Eastern Bru 7921
bsc Bassari 5057
bsn Barasana-Eduria 7548
bsp Baga Sitemu 4541
bss Akoose 7942
btd Batak Dairi 7933
bth Biatah Bidayuh 5057
bts Batak Simalungun 31172
btt Bete-Bendi 7895
btx Batak Karo 7954
bud Ntcham 7957
buk Bugawac 7959
bul Bulgarian 31101
bum Bulu (Cameroon) 5060
bus Bokobaru 19425
bvc Baelelea 7951
bvd Baeggu 5058
bvr Burarra 5059
bvz Bauzi 7800
bwd Bwaidoka 1533
bwq Southern Bobo Madaré 7946
bwu Buli (Ghana) 7836
bxh Buhutu 4281
bxr Russia Buriat 7827
byr Baruya 9492
byx Qaqet 8044
bzd Bribri 7959
bzh Mapos Buang 7959
bzi Bisu 5059
bzj Belize Kriol English 7953
caa Chortí 7956
cab Garifuna 7947
cac Chuj 7918
caf Southern Carrier 7959
cag Nivaclé 5059
cak Kaqchikel 7959
cao Chácobo 7959
cap Chipaya 7958
caq Car Nicobarese 5057
car Galibi Carib 7929
cat Catalan 5058
cav Cavineña 7957
cax Chiquitano 7956
cbc Carapana 7815
cbi Chachi 5059
cbk Chavacano 7959
cbr Cashibo-Cacataibo 7959
cbs Cashinahua 7959
cbt Chayahuita 5059
cbu Candoshi-Shapra 7959
cbv Cacua 7940
cco Comaltepec Chinantec 7895
ccp Chakma 7958
cdf Chiru 7957
ceb Cebuano 31102
ceg Chamacoco 7958
cek Eastern Khumi Chin 7896
ces Czech 31172
cfm Falam Chin 30534
cgc Kagayanen 7948
chd Highland Oaxaca Chontal 7959
che Chechen 5049
chf Tabasco Chontal 7958
chj Ojitlán Chinantec 5058
chk Chuukese 31078
cho Choctaw 1127
chq Quiotepec Chinantec 5058
chr Cherokee 7957
chv Chuvash 7950
chz Ozumacín Chinantec 7958
cjo Ashéninka Pajonal 7931
cjp Cabécar 7952
cjv Chuave 7959
ckb Central Kurdish 7952
cko Anufo 7913
cle Lealao Chinantec 7959
clu Caluyanun 5058
cly Eastern Highland Chatino 7958
cme Cerma 7958
cmn Mandarin Chinese 31039
cmo Central Mnong 7937
cmr Mro-Khimi Chin 7955
cnh Hakha Chin 31087
cni Asháninka 7775
cnk Khumi Chin 7986
cnl Lalana Chinantec 7949
cnt Tepetotutla Chinantec 7948
coe Koreguaje 7737
cof Colorado 7878
cok Santa Teresa Cora 7959
con Cofán 7898
cop Coptic 6400
cor Cornish 5058
cot Caquinte 7919
cou Wamey 7936
cpa Palantla Chinantec 7958
cpb Ucayali-Yurúa Ashéninka 7957
cpc Ajyíninka Apurucayali 7959
cpu Pichis Ashéninka 7959
cpy South Ucayali Ashéninka 6128
crh Crimean Tatar 31163
crj Southern East Cree 5058
crk Plains Cree 5056
crl Northern East Cree 5058
crm Moose Cree 5057
crn El Nayar Cora 7955
crs Seselwa Creole French 5058
crt Iyojwa'ja Chorote 5053
crx Carrier 5059
csk Jola-Kasa 7954
cso Sochiapam Chinantec 7957
csy Siyin Chin 5055
ct1 NaN 30987
cta Tataltepec Chatino 5058
ctd Tedim Chin 7957
ctp Western Highland Chatino 7957
ctu Chol 28203
cub Cubeo 7953
cuc Usila Chinantec 7956
cui Cuiba 7936
cuk San Blas Kuna 7958
cul Culina 7952
cut Teutila Cuicatec 7958
cux Tepeuxila Cuicatec 5058
cwe Kwere 7956
cwt Kuwaataay 7912
cya Nopala Chatino 7958
cym Welsh 7520
czt Zotung Chin 7957
daa Dangaléat 7955
dad Marik 7942
daf NaN 5058
dah Gwahatike 7959
dan Danish 31169
dbq Daba 7831
ddn Dendi (Benin) 7960
ded Dedua 10420
des Desano 7744
deu German 31102
dga Southern Dagaare 7957
dgc Casiguran Dumagat Agta 7953
dgi Northern Dagara 7952
dgr Dogrib 7957
dgz Daga 7959
dhg Dhangu-Djangu 678
dhm Zemba 5058
did Didinga 7388
dig Digo 7954
dik Southwestern Dinka 7953
dis Dimasa 5058
div Dhivehi 2810
djk Eastern Maroon Creole 7882
djr Djambarrpuyngu 5058
dks Southeastern Dinka 5058
dln Darlong 31102
dnj Dan 5058
dnw Western Dani 7903
dob Dobu 7959
dop Lukpa 7950
dos Dogosé 7932
dow Doyayo 5058
dru Rukai 5051
dsh Daasanach 7958
dtb Labuk-Kinabatangan Kadazan 7904
dtp Kadazan Dusun 30471
dts Toro So Dogon 7857
dud Hun-Saare 5055
due Umiray Dumaget Agta 7957
dug Duruma 7938
duo Dupaninan Agta 7939
dur Dii 5058
dwr Dawro 7959
dww Dawawa 7959
dyi Djimini Senoufo 7957
dyo Jola-Fonyi 7959
dyu Dyula 31170
dzo Dzongkha 7902
ebk Eastern Bontok 7959
eka Ekajuk 7958
ell Modern Greek (1453-) 31104
emi Mussau-Emira 8092
emp Northern Emberá 7752
enb Markweeta 7906
eng English 31104
enq Enga 7959
enx Enxet 23148
epo Esperanto 5057
eri Ogea 7958
ese Ese Ejja 7959
esi North Alaskan Inupiatun 5058
esk Northwest Alaska Inupiatun 7959
ess Central Siberian Yupik 7861
est Estonian 5028
esu Central Yupik 5059
etr Edolo 3216
etu Ejagham 5058
eus Basque 7948
eve Even 1151
ewe Ewe 5057
ewo Ewondo 5056
eza Ezaa 5059
faa Fasu 7959
fad Wagi 1151
fai Faiwol 7959
fal South Fali 7957
fao Faroese 5054
far Fataleka 7951
fas Persian 31103
ffm Maasina Fulfulde 7958
fij Fijian 7957
fin Finnish 31134
fmu Far Western Muria 1148
fon Fon 5058
for Fore 7959
fra French 31170
frd Fordata 7866
fub Adamawa Fulfulde 5057
fue Borgu Fulfulde 7958
fuf Pular 7957
fuh Western Niger Fulfulde 7957
fuq Central-Eastern Niger Fulfulde 7958
fuv Nigerian Fulfulde 7959
gag Gagauz 7954
gah Alekano 7959
gai Borei 5059
gam Kandawo 7959
gaw Nobonob 7959
gbi Galela 7948
gbr Gbagyi 5058
gde Gude 7940
gdg Ga'dang 5058
gdn Umanakaina 7959
gdr Wipi 7959
geb Kire 7958
gfk Patpatar 7959
ghe Southern Ghale 5059
ghs Guhu-Samane 7959
gid Gidar 5058
gil Gilbertese 5059
giz South Giziga 5058
gjn Gonja 5059
gkp Guinea Kpelle 5059
gla Scottish Gaelic 5058
gle Irish 6941
glk Gilaki 1250
glv Manx 5057
gmv Gamo 7958
gna Kaansa 7835
gnb Gangte 5059
gnd Zulgo-Gemzek 7887
gng Ngangam 7957
gnn Gumatj 5059
gnw Western Bolivian Guaraní 7959
gof Gofa 7958
gog Gogo 7957
gom Goan Konkani 337
gor Gorontalo 7954
gqr Gor 7957
grc Ancient Greek (to 1453) 7939
grt Garo 7813
gso Southwest Gbaya 7958
gub Guajajára 30187
guc Wayuu 7621
gud Yocoboué Dida 5057
gug Paraguayan Guaraní 5058
guh Guahibo 7884
gui Eastern Bolivian Guaraní 7914
guj Gujarati 30928
guk Gumuz 7958
gul Sea Island Creole English 7959
gum Guambiano 7957
gun Mbyá Guaraní 7957
guo Guayabero 7800
guq Aché 7957
gur Farefare 7958
guu Yanomamö 7958
gux Gourmanchéma 7950
gvc Guanano 7765
gvf Golin 7959
gvl Gulay 7958
gvn Kuku-Yalanji 5055
gwi Gwichʼin 7940
gwr Gwere 7951
gya Northwest Gbaya 5058
gym Ngäbere 7804
gyr Guarayu 7936
hae Eastern Oromo 7958
hag Hanga 7694
hak Hakka Chinese 7908
hat Haitian 7949
hau Hausa 5058
hch Huichol 7958
heb Hebrew 31152
heg Helong 7959
heh Hehe 7958
hif Fiji Hindi 7863
hig Kamwe 7911
hil Hiligaynon 7958
hin Hindi 7957
hix Hixkaryána 7959
hla Halia 7959
hmo Hiri Motu 31104
hmr Hmar 4951
hne Chhattisgarhi 7959
hnj Hmong Njua 5058
hnn Hanunoo 7946
hns Caribbean Hindustani 7832
hop Hopi 5059
hot Hote 7958
hra Hrangkhol 5059
hrv Croatian 5057
hto Minica Huitoto 7956
hub Huambisa 7744
hui Huli 35420
hun Hungarian 31170
hus Huastec 7958
huu Murui Huitoto 7931
huv San Mateo Del Mar Huave 7958
hvn Sabu 7946
hwc Hawai'i Creole English 5059
hye Armenian 5053
ian Iatmul 7959
iba Iban 7940
ibo Igbo 5058
icr Islander Creole English 7956
ifa Amganad Ifugao 31090
ifb Batad Ifugao 31097
ife Ifè 7953
ifk Tuwali Ifugao 7759
ifu Mayoyao Ifugao 7941
ify Keley-I Kallahan 30632
ign Ignaciano 7957
ike Eastern Canadian Inuktitut 5058
ikk Ika 31104
ikt Inuinnaqtun 1886
ikw Ikwere 7959
ilb Ila 5059
ilo Iloko 7957
imo Imbongu 7959
inb Inga 7940
ind Indonesian 31172
ino Inoke-Yate 7959
iou Tuma-Irumu 13151
ipi Ipili 7965
iqw Ikwo 5059
iri Rigwe 7959
irk Iraqw 7957
iry Iraya 7912
isd Isnag 5059
isl Icelandic 7951
ita Italian 31102
itl Itelmen 1151
itv Itawit 7949
ium Iu Mien 5059
ivb Ibatan 5059
ivv Ivatan 5058
iws Sepik Iwam 7958
ixi NaN 7958
ixl Ixil 7957
izr Izere 7987
izz Izii 31095
jaa Jamamadí 5058
jac Popti' 7956
jae Yabem 18719
jam Jamaican Creole English 7966
jav Javanese 7959
jbu Jukun Takum 7942
jic Tol 7922
jiv Shuar 7947
jmc Machame 7958
jpn Japanese 31104
jra Jarai 5059
jun Juang 7956
jvn Caribbean Javanese 7931
kaa Kara-Kalpak 7941
kac Kachin 30944
kan Kannada 7958
kao Xaasongaxango 7946
kaq Capanahua 7959
kaz Kazakh 31062
kbc Kadiwéu 7959
kbh Camsá 7900
kbm Iwal 7957
kbo Keliko 5059
kbp Kabiyè 7955
kbq Kamano 10540
kbr Kafa 7683
kby Manga Kanuri 879
kcg Tyap 7952
kck Kalanga 5788
kdc Kutu 7959
kde Makonde 5059
kdh Tem 1071
kdi Kumam 5058
kdj Karamojong 7958
kdl Tsikimba 7959
kdp Kaningdon-Nindem 7957
kek Kekchí 7958
ken Kenyang 7948
keo Kakwa 7961
ker Kera 7956
kew West Kewa 9448
kez Kukele 7936
kgf Kube 10422
kgk Kaiwá 7959
kgp Kaingang 7959
kgr Abun 7947
kha Khasi 5059
khk Halh Mongolian 5059
khm Khmer 7959
khq Koyra Chiini Songhay 7957
khs Kasua 5059
khy Kele (Democratic Republic of Congo) 5045
khz Keapara 7959
kia Kim 7955
kij Kilivila 7918
kik Kikuyu 7945
kin Kinyarwanda 5058
kix Khiamniungan Naga 5057
kjb Q'anjob'al 7910
kje Kisar 7843
kjh Khakas 7954
kjs East Kewa 7958
kkc Odoodee 5341
kki Kagulu 7958
kkj Kako 7946
kle Kulung (Nepal) 7959
kln Kalenjin 5059
klt Nukna 2921
klv Maskelynes 7958
kmd Majukayang Kalinga 7886
kmg Kâte 6171
kmh Kalam 7959
kmk Limos Kalinga 7959
kmm Kom (India) 5057
kmo Kwoma 7958
kmr Northern Kurdish 7941
kms Kamasau 7959
kmu Kanite 7959
kmy Koma 7948
kne Kankanaey 7947
knf Mankanya 7957
knj Western Kanjobal 7959
knk Kuranko 7900
kno Kono (Sierra Leone) 7877
knv Tabo 7958
kog Cogui 7958
kor Korean 31101
kpf Komba 7959
kpg Kapingamarangi 30555
kpj Karajá 7959
kpq Korupun-Sela 7958
kpr Korafe-Yegha 7959
kpv Komi-Zyrian 7932
kpw Kobon 7959
kpx Mountain Koiali 7958
kpz Kupsabiny 7955
kqa Mum 1071
kqc Doromu-Koki 8034
kqe Kalagan 7959
kqf Kakabai 4165
kql Kyenele 678
kqo Eastern Krahn 5058
kqp Kimré 7956
kqs Northern Kissi 5058
kqw Kandas 5996
kqy Koorete 7955
krc Karachay-Balkar 7954
kri Krio 7926
krj Kinaray-A 7957
krl Karelian 7954
kru Kurukh 7957
krx Karon 1151
ksb Shambala 5057
ksc Southern Kalinga 5059
ksd Kuanua 31103
ksf Bafia 5057
ksj Uare 4235
ksp Kaba 4854
ksr Borong 9624
kss Southern Kisi 5059
ktb Kambaata 7950
ktj Plapo Krumen 7935
ktm Kurti 2472
kto Kuot 7958
ktu Kituba (Democratic Republic of Congo) 5057
kua Kuanyama 5059
kub Kutep 7928
kud 'Auhelawa 7958
kue Kuman (Papua New Guinea) 7959
kuj Kuria 5059
kum Kumyk 5058
kup Kunimaipa 7958
kus Kusaal 7954
kvg Kuni-Boazi 8075
kvj Psikye 5057
kvn Border Kuna 7958
kwd Kwaio 7959
kwf Kwara'ae 7877
kwi Awa-Cuaiquer 7917
kwj Kwanga 7958
kxc Konso 7957
kxm Northern Khmer 7938
kxw Konai 3057
kyc Kyaka 17192
kyf Kouya 7937
kyg Keyagana 9716
kyq Kenga 7958
kyu Western Kayah 7959
kyz Kayabí 7575
kze Kosena 7958
kzf Da'a Kaili 7870
kzj NaN 7955
lac Lacandon 7955
lai Lambya 5053
laj Lango (Uganda) 7956
lam Lamba 7939
lao Lao 7948
lap Laka (Chad) 5058
lat Latin 35608
lav Latvian 5056
law Lauje 7904
lbb Label 678
lbj Ladakhi 5059
lbk Central Bontok 6273
lcm Tungag 9625
lcp Western Lawa 5059
lee Lyélé 7915
lef Lelemi 7957
leg NaN 5058
leh Lenje 5058
lem Nomaande 7956
leu Kara (Papua New Guinea) 7957
lew Ledo Kaili 7953
lex Luang 7886
lgg Lugbara 7957
lgl Wala 7878
lgm Lega-Mwenga 5057
lhi Lahu Shi 5053
lhu Lahu 7935
lia West-Central Limba 7923
lid Nyindrou 9623
lif Limbu 7959
lin Lingala 7518
lip Sekpele 7955
lis Lisu 7950
lit Lithuanian 5055
ljp Lampung Api 7904
llg Lole 7958
lln Lele (Chad) 7958
lmk Lamkang 7959
lmp Limbum 5059
lnd Lundayeh 7956
lob Lobi 7960
loe Saluan 7946
log Logo 5055
lok Loko 7956
lol Mongo 5056
lom Loma (Liberia) 5058
loq Lobala 5058
loz Lozi 5058
lsi Lashi 5037
lsm Saamia 7957
ltz Luxembourgish 7954
luc Aringa 7955
lug Ganda 5058
luo Luo (Kenya and Tanzania) 5057
lus Lushai 7956
lwo Luwo 7818
lww Lewo 7835
lzh Literary Chinese 7102
maa San Jerónimo Tecóatl Mazatec 7959
mad Madurese 29777
maf Mafa 5055
mag Magahi 7899
mah Marshallese 31074
mai Maithili 7958
maj Jalapa De Díaz Mazatec 7959
mak Makasar 7886
mal Malayalam 7955
mam Mam 30992
maq Chiquihuitlán Mazatec 7958
mar Marathi 7956
mas Masai 5058
mau Huautla Mazatec 7958
mav Sateré-Mawé 7959
maw Mampruli 7902
maz Central Mazahua 7850
mbb Western Bukidnon Manobo 30206
mbc Macushi 7959
mbd Dibabawon Manobo 5059
mbh Mangseng 7957
mbi Ilianen Manobo 5059
mbj Nadëb 7959
mbl Maxakalí 7959
mbs Sarangani Manobo 7959
mbt Matigsalug Manobo 7926
mca Maca 7959
mcb Machiguenga 7959
mcd Sharanahua 9315
mcf Matsés 7959
mck Mbunda 5059
mcn Masana 5057
mco Coatlán Mixe 7950
mcp Makaa 7959
mcq Ese 7957
mcu Cameroon Mambila 7937
mda Mada (Nigeria) 7936
mdy Male (Ethiopia) 30744
med Melpa 7959
mee Mengen 7957
mej Meyah 7939
mek Mekeo 7959
men Mende (Sierra Leone) 7929
meq Merey 7949
met Mato 7918
meu Motu 31099
mf1 NaN 7939
mfe Morisyen 7957
mfg Mogofin 1885
mfh Matal 7952
mfi Wandala 7945
mfk North Mofu 7956
mfq Moba 7942
mfy Mayo 7958
mfz Mabaan 7884
mgc Morokodo 2098
mgh Makhuwa-Meetto 7959
mgo Meta' 7959
mgr Mambwe-Lungu 5059
mhi Ma'di 7955
mhl Mauwake 7959
mhr Eastern Mari 7946
mhx Maru 7920
mhy Ma'anyan 7906
mib Atatláhuca Mixtec 7958
mic Mi'kmaq 7958
mie Ocotepec Mixtec 7956
mif Mofu-Gudur 7907
mig San Miguel El Grande Mixtec 7957
mih Chayuco Mixtec 7958
mil Peñoles Mixtec 7955
mim Alacatlatzala Mixtec 7952
min Minangkabau 7945
mio Pinotepa Nacional Mixtec 7954
mip Apasco-Apoala Mixtec 7958
miq Mískito 7956
mir Isthmus Mixe 7512
mit Southern Puebla Mixtec 7953
miy Ayutla Mixtec 7958
miz Coatzospan Mixtec 7956
mjc San Juan Colorado Mixtec 7958
mjw Karbi 5058
mkd Macedonian 5057
mkl Mokole 7904
mkn Kupang Malay 7959
mks Silacayoapan Mixtec 7958
mlg Malagasy 7958
mlh Mape 7959
mlp Bargam 7957
mmn Mamanwa 5059
mmo Mangga Buang 7958
mmx Madak 10420
mna Mbula 13795
mnb Muna 7957
mnf Mundani 7958
mnk Mandinka 7958
moa Mwan 7956
moc Mocoví 5058
mog Mongondow 7904
moh Mohawk 879
mop Mopán Maya 7904
mor Moro 7957
mos Mossi 7958
mox Molima 7958
mpg Marba 7958
mph Maung 676
mpm Yosondúa Mixtec 7951
mpp Migabac 4025
mps Dadibi 30314
mpt Mian 7959
mpx Misima-Panaeati 12161
mqb Mbuko 7954
mqj Mamasa 7958
mqy Manggarai 5059
mrg Mising 5058
mri Maori 5057
mrw Maranao 7940
msa Malay (macrolanguage) 5057
msb Masbatenyo 7959
msc Sankaran Maninka 2941
mse Musey 5058
msk Mansaka 7957
msm Agusan Manobo 7936
msy Aruamu 7958
mta Cotabato Manobo 7904
mtg Una 5059
mti Maiwa (Papua New Guinea) 7957
mtj Moskona 7953
mto Totontepec Mixe 7958
mtp Wichí Lhamtés Nocten 7958
mua Mundang 5057
mug Musgu 5057
muh Mündü 7482
mup Malvi 7959
mur Murle 7828
mux Bo-Ung 7959
muy Muyang 7958
mva Manam 7959
mvc NaN 7882
mvn Minaveha 7958
mvp Duri 7950
mwc Are 4415
mwl Mirandese 879
mwm Sar 5058
mwp Kala Lagaw Ya 1506
mwq Mün Chin 7930
mwv Mentawai 7958
mww Hmong Daw 7950
mxb Tezoatlán Mixtec 7958
mxp Tlahuitoltepec Mixe 7958
mxq Juquila Mixe 7957
mxt Jamiltepec Mixtec 7918
mxv Metlatónoc Mixtec 7958
mya Burmese 30944
myb Mbay 7961
myk Mamara Senoufo 7958
myu Mundurukú 7959
myv Erzya 7955
myw Muyuw 7959
myx Masaaba 7958
myy Macuna 7944
mza Santa María Zacatepec Mixtec 7958
mzh Wichí Lhamtés Güisnay 5051
mzk Nigeria Mambila 7952
mzl Mazatlán Mixe 5059
mzm Mumuye 7947
mzw Deg 7909
mzz Maiadomu 678
nab Southern Nambikuára 7959
naf Nabak 7958
nak Nakanai 7958
naq Khoekhoe 5058
nas Naasioi 8096
nav Navajo 5058
naw Nawuri 7959
nbc Chang Naga 5057
nbe Konyak Naga 5059
nbl South Ndebele 5059
nbu Rongmei Naga 5056
nca Iyo 7959
nch Central Huasteca Nahuatl 7952
ncj Northern Puebla Nahuatl 7957
ncl Michoacán Nahuatl 7958
ncq Northern Katang 5059
nct Chothe Naga 5058
ncu Chumburung 7686
ndc Ndau 5058
nde North Ndebele 5057
ndi Samba Leko 5058
ndj Ndamba 7959
ndo Ndonga 5058
ndy Lutos 7958
ndz Ndogo 7938
neb Toura (Côte d'Ivoire) 7881
nep Nepali (macrolanguage) 5059
nfa Dhao 7958
nfr Nafaanra 7930
ngc Ngombe (Democratic Republic of Congo) 5051
ngp Ngulu 7959
ngu Guerrero Nahuatl 7933
nhe Eastern Huasteca Nahuatl 7958
nhg Tetelcingo Nahuatl 6950
nhi Zacatlán-Ahuacatlán-Tepetzintla Nahuatl 7959
nho Takuu 7958
nhr Naro 7959
nhu Noone 7950
nhw Western Huasteca Nahuatl 7952
nhx Isthmus-Mecayapan Nahuatl 7958
nhy Northern Oaxaca Nahuatl 7952
nia Nias 31164
nif Nek 3779
nii Nii 7957
nij Ngaju 7945
nim Nilamba 7982
nin Ninzo 7959
niy Ngiti 5058
njb Nocte Naga 5051
njm Angami Naga 5058
njn Liangmai Naga 5057
njo Ao Naga 5059
njz Nyishi 5059
nkf Inpui Naga 7957
nki Thangal Naga 5059
nko Nkonya 7958
nlc Nalca 7956
nld Dutch 31103
nlg Gela 5059
nma Maram Naga 5057
nmf Tangkhul Naga (India) 5059
nmh Monsang Naga 5058
nmo Moyon Naga 5057
nmw Nimoa 1226
nmz Nawdm 7951
nnb Nande 7958
nng Maring Naga 5059
nnh Ngiemboon 5058
nnl Northern Rengma Naga 5057
nno Norwegian Nynorsk 5057
nnp Wancho Naga 5057
nnq Ngindo 7959
nnw Southern Nuni 7954
noa Woun Meu 7959
nob Norwegian Bokmål 5057
nod Northern Thai 7938
nog Nogai 7936
nop Numanggang 11952
nor Norwegian 31169
not Nomatsiguenga 7959
nou Ewage-Notu 7957
nph Phom Naga 5058
npi Nepali (individual language) 7958
npl Southeastern Puebla Nahuatl 7952
npo Pochuri Naga 5058
npy Napu 7797
nre Southern Rengma Naga 5059
nrf Jèrriais 1071
nri Chokri Naga 5056
nsa Sangtam Naga 5058
nse Nsenga 5059
nsm Sumi Naga 5058
nsn Nehan 7958
nso Pedi 5057
nss Nali 2211
nst Tase Naga 5057
nsu Sierra Negra Nahuatl 7951
ntp Northern Tepehuan 7763
ntr Delo 7931
ntu Natügu 5057
nuj Nyole 7959
nuy Nunggubuyu 5058
nuz Tlamacazapa Nahuatl 1883
nvm Namiae 7959
nwb Nyabwa 7920
nwi Southwest Tanna 7949
nwx Middle Newar 8435
nxd Ngando (Democratic Republic of Congo) 5057
nya Nyanja 5059
nyf Giryama 7956
nyn Nyankole 7956
nyo Nyoro 7958
nyy Nyakyusa-Ngonde 7957
nza Tigon Mbembe 5059
nzi Nzima 7953
nzm Zeme Naga 5058
obo Obo Manobo 7958
ojb Northwestern Ojibwa 7949
ojs Severn Ojibwa 7949
oku Oku 7959
okv Orokaiva 7958
old Mochi 7958
omb East Ambae 5058
omw South Tairora 7959
ong Olo 7959
ons Ono 5058
ood Tohono O'odham 7568
opm Oksapmin 7959
ori Oriya (macrolanguage) 7944
ory Odia 5059
oss Ossetian 7955
otd Ot Danum 7905
ote Mezquital Otomi 7957
otm Eastern Highland Otomi 7958
otn Tenango Otomi 7958
otq Querétaro Otomi 7958
ots Estado de México Otomi 7958
oym Wayampi 5058
ozm Koonzime 7912
pab Parecís 7955
pad Paumarí 7959
pag Pangasinan 7942
pah Tenharim 7881
pam Pampanga 7958
pan Panjabi 7940
pao Northern Paiute 7040
pap Papiamento 5059
pau Palauan 7939
pbb Páez 7923
pbc Patamona 7942
pbi Parkwa 7952
pck Paite Chin 31098
pcm Nigerian Pidgin 7959
pdc Pennsylvania German 5057
pdt Plautdietsch 5057
pes Iranian Persian 5058
pez Eastern Penan 7954
pfe Peere 7584
pib Yine 7958
pio Piapoco 7958
pir Piratapuyo 7753
pis Pijin 7920
pjt Pitjantjatjara 7507
pkb Pokomo 7899
plg Pilagá 5058
pls San Marcos Tlacoyalco Popoloca 7958
plt Plateau Malagasy 5058
plu Palikúr 7956
plw Brooke's Point Palawano 7958
pma Paama 7959
pmf Pamona 7812
pmx Poumei Naga 5058
pne Western Penan 5057
pny Pinyin 7959
poe San Juan Atzingo Popoloca 8436
poh Poqomchi' 7958
poi Highland Popoluca 7958
pol Polish 7953
pon Pohnpeian 7960
por Portuguese 31167
pos Sayula Popoluca 5058
pot Potawatomi 2078
poy Pogolo 5059
ppk Uma 7941
ppo Folopa 7958
pps San Luís Temalacayuca Popoloca 5059
prf Paranan 7958
pri Paicî 7957
prk Parauk 31103
prq Ashéninka Perené 5057
prs Dari 5057
pse Central Malay 7958
pss Kaulong 7957
ptp Patep 7958
ptu Bambam 7958
pui Puinave 7954
pwg Gapapaiwa 7958
pww Pwo Northern Karen 7936
pxm Quetzaltepec Mixe 31008
qub Huallaga Huánuco Quechua 7958
quc K'iche' 7958
quf Lambayeque Quechua 7950
qug Chimborazo Highland Quichua 5058
quh South Bolivian Quechua 7959
qul North Bolivian Quechua 7944
qup Southern Pastaza Quechua 7728
quw Tena Lowland Quichua 7958
quy Ayacucho Quechua 7937
quz Cusco Quechua 31072
qva Ambo-Pasco Quechua 5059
qvc Cajamarca Quechua 7810
qve Eastern Apurímac Quechua 7959
qvh Huamalíes-Dos de Mayo Huánuco Quechua 7455
qvi Imbabura Highland Quichua 51
qvm Margos-Yarowilca-Lauricocha Quechua 7455
qvn North Junín Quechua 7949
qvo Napo Lowland Quechua 7875
qvs San Martín Quechua 7883
qvw Huaylla Wanca Quechua 7644
qvz Northern Pastaza Quichua 7813
qwh Huaylas Ancash Quechua 7885
qxh Panao Huánuco Quechua 7945
qxl Salasaca Highland Quichua 5057
qxn Northern Conchucos Ancash Quechua 7947
qxo Southern Conchucos Ancash Quechua 7531
qxr Cañar Highland Quichua 112
rad Rade 5055
rai Ramoaaina 7958
rap Rapanui 7906
rar Rarotongan 5058
rav Sampang 7952
raw Rawang 7957
rej Rejang 7942
rel Rendille 7942
rgu Ringgou 7079
ria Riang (India) 5058
rim Nyaturu 7957
rjs Rajbanshi 7955
rkb Rikbaktsa 7959
rmc Carpathian Romani 7958
rme Angloromani 97
rml Baltic Romani 879
rmn Balkan Romani 1152
rmo Sinte Romani 7952
rmq Caló 1151
rmy Vlax Romani 7967
rnl Ranglong 7957
ron Romanian 9141
roo Rotokas 7958
rop Kriol 5059
row Dela-Oenale 7958
rro Waima 7957
rub Gungu 741
ruf Luguru 7959
rug Roviana 7947
run Rundi 7957
rup Macedo-Romanian 1151
rus Russian 31102
rwo Rawa 7957
sab Buglere 7542
sag Sango 7958
sah Yakut 5055
saj Sahu 7854
san Sanskrit 5059
sas Sasak 7909
sat Santali 5059
sba Ngambay 7957
sbd Southern Samo 7920
sbe Saliba 4388
sbl Botolan Sambal 7957
sck Sadri 7959
sco Scots 1151
sda Toraja-Sa'dan 7907
sdm Semandang 7751
seh Sena 5058
ses Koyraboro Senni Songhai 7957
sey Secoya 7952
sgb Mag-antsi Ayta 7941
sgw Sebat Bet Gurage 7955
sgz Sursurunga 7958
shn Shan 7955
shp Shipibo-Conibo 7959
shu Chadian Arabic 5059
sig Paasaal 7954
sil Tumulung Sisaala 7912
sim Mende (Papua New Guinea) 7958
sin Sinhala 5058
sja Epena 7930
sld Sissala 7928
slk Slovak 5057
sll Salt-Yui 7953
slv Slovenian 7585
smk Bolinao 7958
sml Central Sama 7946
smo Samoan 7941
smt Simte 7957
sna Shona 5059
snc Sinaugoro 7959
snd Sindhi 5058
snf Noon 5056
snn Siona 7956
snp Siane 7958
snw Selee 7959
sny Saniyo-Hiyewe 7958
som Somali 31108
sop Songe 5057
soq Kanasi 10858
sot Southern Sotho 5062
soy Miyobe 7957
spa Spanish 35337
spl Selepet 7958
spm Akukem 1685
spp Supyire Senoufo 7905
sps Saposa 8244
spy Sabaot 7945
sqi Albanian 31102
sri Siriano 7957
srm Saramaccan 7774
srn Sranan Tongo 7941
srp Serbian 5058
srq Sirionó 5059
ssd Siroi 11149
ssg Seimat 7959
ssw Swati 5058
ssx Samberigi 9490
stn Owa 7954
stp Southeastern Tepehuan 7924
sua Sulka 7957
suc Western Subanon 5059
sue Suena 7958
suk Sukuma 7953
sun Sundanese 7957
sur Mwaghavul 7959
sus Susu 7957
suz Sunwar 31104
swe Swedish 31170
swh Swahili (individual language) 5059
swp Suau 11728
sxb Suba 7959
sxn Sangir 7903
syb Central Subanen 5059
syl Sylheti 5059
tab Tabassaran 7879
tac Lowland Tarahumara 7895
taj Eastern Tamang 7922
tam Tamil 7943
tap Taabwa 7958
taq Tamasheq 7956
tat Tatar 7952
tav Tatuyo 7909
taw Tai 7959
tbc Takia 7952
tbg North Tairora 19853
tbk Calamian Tagbanwa 5058
tbl Tboli 5059
tbo Tawala 7952
tbw Tagbanwa 5058
tby Tabaru 7896
tbz Ditammari 31008
tca Ticuna 7742
tcc Datooga 7957
tcs Torres Strait Creole 5059
tcz Thado Chin 5059
tdt Tetun Dili 7079
ted Tepo Krumen 7935
tee Huehuetla Tepehua 7949
tel Telugu 30742
tem Timne 7957
teo Teso 7958
ter Tereno 7958
tet Tetum 5059
tew Tewa (USA) 3169
tfr Teribe 7637
tgk Tajik 7954
tgl Tagalog 31102
tgo Sudest 8089
tgp Tangoa 7957
tha Thai 7957
thk Tharaka 7932
thl Dangaura Tharu 7959
tif Tifal 7959
tih Timugon Murut 34555
tik Tikar 7957
tim Timbe 7959
tir Tigrinya 7949
tiy Tiruray 7957
tke Takwane 5058
tku Upper Necaxa Totonac 7958
tlb Tobelo 7946
tlf Telefol 7958
tlj Talinga-Bwisi 7959
tmc Tumak 7959
tmd Haruai 5059
tna Tacana 7959
tnc Tanimuca-Retuarã 3237
tnk Kwamera 7958
tnn North Tanna 7941
tnp Whitesands 5056
tnr Ménik 7920
tob Toba 5058
toc Coyutla Totonac 7921
tod Toma 11207
toi Tonga (Zambia) 5057
toj Tojolabal 7958
ton Tonga (Tonga Islands) 31102
too Xicotepec De Juárez Totonac 7958
top Papantla Totonac 7958
tos Highland Totonac 7911
tpa Taupota 678
tpi Tok Pisin 35589
tpm Tampulma 30835
tpp Pisaflores Tepehua 7958
tpt Tlachichilco Tepehua 7957
tpz Tinputz 7959
tqo Toaripi 7959
trc Copala Triqui 5058
tro Tarao Naga 5059
trp Kok Borok 5059
trs Chicahuaxtla Triqui 7957
tsg Tausug 5059
tsn Tswana 9303
tso Tsonga 5058
tsw Tsishingini 7957
tsz Purepecha 5056
ttc Tektiteko 7959
tte Bwanabwana 7958
ttq Tawallammat Tamajaq 5057
ttr Tera 31083
tuc Mutu 12880
tue Tuyuca 7956
tuf Central Tunebo 7894
tui Tupuri 5058
tuk Turkmen 30867
tul Tula 5059
tum Tumbuka 5059
tuo Tucano 7939
tur Turkish 30984
tvk Southeast Ambrym 7959
twb Western Tawbuid 7958
twi Twi 7519
twu Termanu 7946
txq Tii 7958
txu Kayapó 7675
tyv Tuvinian 5056
tzh Tzeltal 5058
tzj Tz'utujil 7959
tzo Tzotzil 7958
tzt NaN 7906
ubr Ubir 9489
ubu Umbu-Ungu 7959
udm Udmurt 7955
udu Uduk 7958
uig Uighur 5059
ukr Ukrainian 31103
upv Uripiv-Wala-Rano-Atchin 7954
ura Urarina 7752
urb Urubú-Kaapor 7869
urd Urdu 7900
uri Urim 2511
urk Urak Lawoi' 7922
urt Urat 7962
urw Sop 1199
ury Orya 7763
usa Usarufa 8232
usp Uspanteco 7958
uvh Uri 5059
uvl Lote 7959
uzn Northern Uzbek 5055
vag Vagla 7958
vap Vaiphei 5058
var Huarijio 5058
ven Venda 5055
vid Vidunda 7959
vie Vietnamese 23136
viv Iduna 7959
vmy Ayautla Mazatec 7959
vol Volapük 103
vun Vunjo 7957
vut Vute 7957
waj Waffa 7959
wal Wolaytta 22939
wap Wapishana 8067
war Waray (Philippines) 7957
wat Kaninuwa 2665
way Wayana 7957
wba Warao 7958
wbm Wa 5058
wbp Warlpiri 5059
wed Wedau 4192
wer Weri 7957
whk Wahau Kenyah 5057
wib Southern Toussian 4159
wim Wik-Mungkan 5056
wiu Wiru 7957
wlx Wali (Ghana) 31102
wmt Walmajarri 3384
wmw Mwani 7944
wnc Wantoat 10418
wnu Usan 7958
wob Wè Northern 7928
wos Hanga Hundi 8006
wrk Garrwa 3527
wrs Waris 9624
wsg Adilabad Gondi 5059
wsk Waskia 13609
wuv Wuvulu-Aua 7941
wwa Waama 7955
xal Kalmyk 5051
xav Xavánte 7428
xbi Kombio 1384
xed Hdi 7957
xho Xhosa 5058
xla Kamula 10172
xmm Manado Malay 7955
xnn Northern Kankanay 5059
xog Soga 5058
xon Konkomba 7958
xpe Liberia Kpelle 5059
xrb Eastern Karaboro 7925
xsb Sambal 7958
xsi Sio 10420
xsm Kasem 7957
xsr Sherpa 7957
xsu Sanumá 7799
xtd Diuxi-Tilantongo Mixtec 7953
xtm Magdalena Peñasco Mixtec 7958
xtn Northern Tlaxiaco Mixtec 7958
xuo Kuo 7953
yaa Yaminahua 7918
yad Yagua 7870
yal Yalunka 7955
yam Yamba 7929
yan Mayangna 5053
yao Yao 5057
yap Yapese 5057
yaq Yaqui 7958
yas Nugunu (Cameroon) 7723
yat Yambeta 7935
yaz Lokaa 7958
ybb Yemba 7949
yby Yaweyuha 7957
ycn Yucuna 7948
yim Yimchungru Naga 5059
yka Yakan 7958
yle Yele 7957
yli Angguruk Yali 7911
yml Iamalele 7958
yom Yombe 7517
yon Yongkom 7958
yor Yoruba 5058
yrb Yareba 7875
yre Yaouré 7958
yss Yessan-Mayo 7958
yua Yucateco 5059
yue Yue Chinese 31097
yuj Karkar-Yuri 7959
yut Yopno 10419
yuw Yau (Morobe Province) 7958
yva Yawa 7950
zaa Sierra de Juárez Zapotec 7951
zab Western Tlacolula Valley Zapotec 7953
zac Ocotlán Zapotec 7958
zad Cajonos Zapotec 7959
zae Yareni Zapotec 7956
zai Isthmus Zapotec 7931
zam Miahuatlán Zapotec 7958
zao Ozolotepec Zapotec 7944
zar Rincón Zapotec 7929
zas Santo Domingo Albarradas Zapotec 7948
zat Tabaa Zapotec 7958
zav Yatzachi Zapotec 7958
zaw Mitla Zapotec 7956
zca Coatecas Altas Zapotec 7958
zho Chinese 5058
zia Zia 7959
ziw Zigula 7959
zlm Malay (individual language) 23145
zoc Copainalá Zoque 5058
zom Zou 5059
zos Francisco León Zoque 7957
zpc Choapan Zapotec 7953
zpi Santa María Quiegolani Zapotec 7953
zpl Lachixío Zapotec 7958
zpm Mixtepec Zapotec 7720
zpo Amatlán Zapotec 7937
zpq Zoogocho Zapotec 7953
zpt San Vicente Coatlán Zapotec 7958
zpu Yalálag Zapotec 7958
zpv Chichicapan Zapotec 7958
zpz Texmelucan Zapotec 7950
zsm Standard Malay 7943
zsr Southern Rincon Zapotec 7958
ztq Quioquitani-Quierí Zapotec 7922
zty Yatee Zapotec 7922
zul Zulu 5059
zyb Yongbei Zhuang 5057
zyp Zyphe Chin 7954

Comparison of 1000Langs and the massive parallel corpora

Here we compare language coverage of massively parallel corpus and 1000Langs:

download

In 1199 out iso codes of 1526 total, 1000Langs crawled larger max verses for the iso code! In 912 out iso codes of 1239 total intersection, 1000Langs crawled larger max verses for the iso code!

For a detailed comparison please see 'reports>comparison.xlsx'.

About

Creating super-parallel corpora of more than 1500+ unique languages for NLP research

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages