This project puts a key emphasis on the efficient management of the dataset while addressing potential challenges related to memory and time complexity.The focus lies in generating language-specific files, denoted as 'en-xx.xlsx,' for diverse languages such as English (en), Swahili (sw), and German (de). Additionally, the project entails the creation of distinct JSONL files for English (en), Swahili (sw), and German (de), comprising test, train, and dev data sets. Moreover, the endeavor aims to produce a consolidated JSON file that incorporates translations from English to all other languages, encompassing both 'id' and 'utt' data for the training sets.
To set up the Python3 development environment and install the necessary dependencies, follow these steps:
- Clone this repository to your local machine:
https://github.com/lawdneo/CG_cat.git
- Create a virtual environment (recommended):
python3 -m venv venv
Activate the virtual environment:
source venv/bin/activate
- Install the required dependencies:
pip install -r requirements.txt
- Data Import:
Having already obtained the MASSIVE Dataset mentioned in the Data File, Ensure that this dataset is accessible in your project directory. Save it under the
datafolder in the project directory.
Using a merge algorithm to form the requests locales instead of a recursive algorithm. The merge algorithm was done using pandas.