first release

asmelashteka · Dec 5, 2021 · 3794561 · 3794561
commit 3794561
Show file tree

Hide file tree

Showing 10 changed files with 11,825 additions and 0 deletions.
diff --git a/HornMT.json b/HornMT.json
diff --git a/HornMT.xlsx b/HornMT.xlsx
diff --git a/README.md b/README.md
@@ -0,0 +1,80 @@
+# HornMT 
+
+The `HornMT` repository contains data and the associated metadata for the project [Machine Translation Benchmark Dataset for Languages in the Horn of Africa](https://lesan.ai/benchmark). The goal is to create a multi-way parallel corpus that will serve as a benchmark to accelerate progress in machine translation research and production systems for languages in the Horn of Africa.
+
+
+## Supported Languages
+
+ Language      | ISO 639-3 code
+ ------------- | -------------
+Afar	| aaf
+Amharic	| amh
+English	| eng
+Oromo	| dorm
+Somali	| Sam
+Tigrinya	| tir
+
+
+## Content
+
+`data/` contains one text file per language and each file contains news snippets in the same order for each language.
+
+```
+data
+├── aar.txt
+├── amh.txt
+├── eng.txt
+├── orm.txt
+├── som.txt
+└── tir.txt
+```
+
+`metadata.tsv` contains tab separated data describing each news snippet. The metadata contains the following fields.
+
+- **Scope** - describes whether the news is global or local. It takes two values: Global news and Local news.
+- **Category** - News category covering the following 12 topics
+  - Art and Culture
+  - Business and Economy
+  - Conflicts and Attacks
+  - Disaster and Accidents
+  - Entertainment
+  - Environment
+  - Health
+  - International Relations
+  - Law and Crime
+  - Politics
+  - Science and Technology
+  - Sport
+- **Source** - List of one or more URLs from which the news content is extracted or based on. 
+- **Domain** - TLD corresponding to the URL(s) in Source.
+- **Date** - The publication date of the source article. The format is yyyy-mm-dd.
+
+## Other formats
+
+All the data and associated metadata together in one file is also available in other file formats.
+
+`HornMT.xlsx` - data and associated metadata in xlsx format.
+
+`HornMT.json` - data and associated metadata in json format.
+
+Below is an example row.
+
+```json
+{
+   "data":{
+      "eng":"The World Meteorological Organisation reports that the ozone layer is damaged to its worst extent ever in the Arctic.",
+      "aaf":"Baad Metrolojih Eglali Areketekeh Addal Ozonih qelu faxe waktik lafetle calat biyakisem xayose.",
+      "amh":"የአለማ የአየር ንብረት ድርጅት በአርክቲክ አካባቢ ያለው የኦዞን ምንጣፍ ከፍተኛ ጉዳት እንደደረሰበት አስታወቀ፡፡",
+      "orm":"Dhaabbanni Meetiroolojii Addunyaa baqqaanni oozonii Arkiitik keessatti gara sadarkaa isa hamaa haga ammaatti akka miidhame gabaase.",
+      "som":"Ururka Saadaasha Hawada Adduunka ayaa ku warramaya in lakabka ozoneka ee Ka koreeya dhulka baraflayda uu waxyeelladii abid ugu darnaa soo gaadhay.",
+      "tir":"ውድብ ሜትሮሎጂ ዓለም ኣብ ኣርክቲክ ዝርከብ ናሕሲ ኦዞን ኣዝዩ ብዝኸፍአ ደረጃ ከምዝተጎድአ ሓቢሩ፡፡"
+   },
+   "metadata":{
+      "scope":"Global",
+      "category":"Science and Technology",
+      "source":"https://www.independent.co.uk/environment/climate-change/ozone-layer-damaged-by-unusually-harsh-winter-2263653.html",
+      "domain":"www.independent.co.uk",
+      "date":"2011-04-05"
+   }
+}
+```
diff --git a/data/aar.txt b/data/aar.txt
diff --git a/data/amh.txt b/data/amh.txt
diff --git a/data/eng.txt b/data/eng.txt
diff --git a/data/orm.txt b/data/orm.txt
diff --git a/data/som.txt b/data/som.txt
diff --git a/data/tir.txt b/data/tir.txt
diff --git a/metadata.tsv b/metadata.tsv