feat: atlas pull and push scripts | FC-55

[Amr-Nash]

Description

This pull request provides the CI scripts and developer tooling for the Atlas Translations Management Design
proposal related to OEP-58.

Testing instructions

Testing the pull_translations:

make ATLAS_OPTIONS='--repository=Zeit-Labs/openedx-translations --revision=fc_55_sample' pull_translations

When the above command is executed:

1. The None-English translations will be pulled from openedx-translations repo into I18N/ directory.
2. Then each language file will be split and distributed to the modules one by one
3. In the end, the pulled translations will be removed and only the split translations can be found in the modules' directories

If the script is working as intended:

1. The above command should not produce errors when executed.
2. The pulled translations should be correctly distributed among the modules in the expected paths.
3. The I18N directory should not exist after the script has been executed.
4. The app should have all its translations in place and can now be deployed.

Testing the extract_translations:

make extract_translations

When the above command is executed:

1. The English translations from all en.lproj/Localizable.strings files in all modules will be extracted
2. The keys of each translation entry will be updated to contain the name of the module followed by a dot e.g. "Module_name.OLD_KEY_of_the_entry".
3. Then all of the updated entries will be put in one file: I18N/en.lproj/Localizable.strings

If this script is working as intended:

1. The above command should not produce errors when executed.
2. Every entry from the English translation of each module should exist in I18N/en.lproj/Localizable.strings with the 3) corresponding comment, if present, placed before it.
3. The keys in I18N should contain the name of the module from which the entry originated, as specified.
4. The values and comments should be identical to the original ones.

Status

We'll merge to this pull request into this branch multiple pull requests. We'll keep this open until we're ready for review.

This pull request is useful to watch progress.

Q and A

1. When are source strings extracted? what do the files look like?

• An automated action will be developed to extract the English sources if they are changed in the IOS repo, then the action will push the sources to the translations repo.
• The sources will be in one file: I18N/en.lproj/Localizable.strings and it will look like this:

"Module_one.FIRST.ENTRY.KEY.IN.MODULE.ONE" = "English Translation for the first entry in module one.";
"Module_one.SECOND.ENTRY.KEY.IN.MODULE.ONE" = "English Translation for the second entry in module one.";
.
.
"Module_two.FIRST.ENTRY.KEY.IN.MODULE.TWO" = "Translation for the first entry in module two";
.
.

2. Does anything happen to those files before they are added to openedx-translations

The only extracted file is the English sources from the modules in the IOS repo as it is the only translation that lives in that repo.
TBC

3. When translation files are pulled via atlas what do the files look like?

When the translation files are pulled via atlas the files' structure is going to look like the structure below:

I18N/
----ar.lproj/
--------Localizable.strings
----uk.lproj/
--------Localizable.strings
----es.lproj/
--------Localizable.strings

As an example the es.lproj/Localiztion.strings would look like:
I18N/es.lproj/Localizable.strings

"Module_one.FIRST.ENTRY.KEY.IN.MODULE.ONE" = "Traducción al inglés de la primera entrada del módulo uno.";
"Module_one.SECOND.ENTRY.KEY.IN.MODULE.ONE" = "Traducción al inglés de la segunda entrada del módulo uno.";
.
.
"Module_two.FIRST.ENTRY.KEY.IN.MODULE.TWO" = "Traducción de la primera entrada del módulo dos.";
.
.

4) What is done to those files after atlas pull to make them work in the app build process?

After atlas pull is completed, all language translation files are retrieved, similar to the process described in the previous question. Subsequently, a Python script is executed to process each file. This script iterates through each translation file, splits it, removes the module name from the keys, and organizes each entry into its corresponding module.

For instance, the I18N/es.lproj/Localizable.strings file is split into two separate files:

1. Module_one/Module_one/es.lproj/Localizable.strings
2. Module_two/Module_two/es.lproj/Localizable.strings

Inside Module_one/Module_one/es.lproj/Localizable.strings, the content would resemble:

"FIRST.ENTRY.KEY.IN.MODULE.ONE" = "English translation of the first entry in module one.";
"SECOND.ENTRY.KEY.IN.MODULE.ONE" = "English translation of the second entry in module one.";
.
.
.

Similarly, inside Module_two/Module_two/es.lproj/Localizable.strings, it would appear as:

"FIRST.ENTRY.KEY.IN.MODULE.TWO" = "English translation of the first entry in module two.";
.
.
.

Once the Python script completes its task and the files are split, the make script proceeds to remove the entire I18N directory and its contents.

5) Why is this needed?

This new process is being introduced to have the best combination of Developer Experience and Translator Experience.

The best experience for Translators requires combining source strings into as few transifex resources as possible. The best experience for Engineers requires splitting translation source files to fit within the modular architecture.

6) What should live in openedx translations?

Translations that were done by the open-edx translators are going to be stored in openedx translations.

7) What should live in atlas?

Atlas is the tool that we are using to pull the translations from openedx translations; therefor, no translations are stored in atlas.

8) What should live in the openedx-app-ios repo?

In the IOS app repo only the English sources are going to be stored.

TODO

• Fix the python3-localizable pip package for python3 compatibility
• Install openedx-atlas in the virtual env
• Refactor the code into a single Python script to improve code reuse
• Add CI scripts for make extract_translations
• Test with the iOS team (Volo and others)
• Review by Brian Smith

Related issues

• Closes [iOS] Internationalization Infrastructure #106

[openedx-webhooks]

Thanks for the pull request, @Amr-Nash! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[OmarIthawi]

Note: CLA has been submitted but needs a day or two to reflect on GitHub.

[OmarIthawi]

Thanks @Amr-Nash, this pull request needs the following:

• squash the commits
• use conventional commits
• rename make combine_translations to make extract_translations

[OmarIthawi]

Thanks @Amr-Nash. I've reviewed the README and suggested many edits. Please adopt and revise and let me know when it's ready.

You'll be getting another review regarding the Python code from either me or @iamjazzar today.

[Amr-Nash]

@OmarIthawi Thank you for your feedback. It is much appreciated and I am waiting for the script review.

[iamjazzar]

I've done an initial review, but I haven't tested the functionality yet. The code appears to be clean, but there are numerous nested for loops and some ambiguous names.

[OmarIthawi]

@iamjazzar

but there are numerous nested for loops

Any suggestion how to improve that?

[OmarIthawi]

Added some notes.

[OmarIthawi]

Posting yesterday's review:

[Amr-Nash]

Hello @brian-smith-tcril ,
Oman and Jazzar have reviewed the PR multiple times by now. Could you please give it a look?

[brian-smith-tcril]

Overall this is looking great! The code seems reasonable, so assuming it has been tested that part has a 👍 from me!

I left a couple suggestions in the README that I hope will make the instructions a little more clear.

Please let me know if you have any questions!

[OmarIthawi]

Thanks @brian-smith-tcril! Yes, the manual testing part is what we focused on.

@Amr-Nash please address Brian notes and let's squash the code once more.

[Amr-Nash]

Thanks @brian-smith-tcril for your feedback,
@OmarIthawi I squashed and pushed again. I think it is ready now

[OmarIthawi]

Thanks @Amr-Nash! I've tested it and it worked locally.

We need an iOS engineer to pull and compile, but overall I think it's ready for merge.

[OmarIthawi]

@volodymyr-chekyrta would you mind chekcing this pull request? We'd love a test compile from one of your team so we can continue with the other GitHub Actions pipeline steps.

[OmarIthawi]

Please add the language renaming as well.

[RawanMatar89]

after testing the instructions and build the app using Ukraine language on simulator (iPhone 15 pro max) I got

SignIn View	Registration view

[volodymyr-chekyrta]

Tested ✅
Works fine for me 👍

[volodymyr-chekyrta]

How about removing this section from the README?
It seems that the PR description is sufficient for testing

[OmarIthawi]

@Amr-Nash please replace the underscore for iOS.

[Amr-Nash]

@OmarIthawi Done, please check.
If everything is ok I'll squash and push again.

[openedx-webhooks]

@Amr-Nash 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.