Exclude .zip files from repository & CI

[miloshavlicek]

I would propose to set up a CI pipe to build artifacts like .zip/.json files are (and exclude them from this repo), because .zip files are treated by GIT as a blob and .json files are only artifacts.

Why?
At this time it is very slow to clone the repo because it contains many .zip files. Each time you change the .zip file, the repo size will grow by the size of the .zip file.

It would be also easier for a maintainer, because he would no longer need to run grunt commands manually and commit the changes back to the repo.

CI pipe can build it and upload it to API automatically.

[aristath]

This is actually something we've been investigating and we're hoping to do.

There are just a few technical difficulties we're trying to solve before we can do this.
The zip files need to be in the repo 'cause that's how our API implementation works, it pulls the zip files directly from the repo, and the API is the actual JSON files.
The plan is to do the following:

• Separate the language files (.po) to a branch which will be then set as the primary branch of the repo. This branch will start from a clean slate to avoid all the git-clone delays. An alternative proposal is to have them in a separate repo (let's call it l10n-api).
• When a commit is pushed on the .po files, run a CI task.
• The CI task should detect which languages were modified in which project. These are used as params in the grunt task to avoid updating the API for languages that didn't actually get an update.
• The CI task should run something like
  grunt --locale=cs_CZ project=Avada
grunt --locale=cs_CZ project=fusion-builder
  for each language and project that has updated translations.
• The CI task should then git-commit & push the changes to the dedicated API branch or repo (still under discussion)

Building the CI script is probably going to be the most time-consuming part here 'cause there's many edge-cases and things to consider, but we're hoping to do it soon because you're right, the current structure is sub-optimal. 👍

That being said, the current implementation may not be perfect but it works and we're therefore prioritizing dev tasks on the theme & builder. This will be done, but I don't have an ETA.

[miloshavlicek]

@aristath So, the whole API "logic" is the repository code and the client's update checks are done via JSON checksum, am I right?

From my point of view, it is not best practice to process artifacts this way (store them in a GIT repo). What do you think about the (npm package) scenario below?

• Commit .po files only (with .zip and .json files removed)
• Pull request merge would trigger Github Workflows build Action
  • Download current npm package content
  • Check which files have changed, copy them to original content, get its language codes and folder names and run grunt --locale=[language code] project=[project] for each modified file
  • Pack it as npm package and publish it as a new version (API will be available on unpkg.com/:package)

You can then set up your current API as a proxy to unpkg.com/:package

If you want I can try to implement it.

[aristath]

@aristath So, the whole API "logic" is the repository code and the client's update checks are done via JSON checksum, am I right?

The "API" is basically just this: https://raw.githubusercontent.com/Theme-Fusion/Localization-l10n/master/api-Avada.json
This was created using the official w.org API response as a template, and there are separate files for each project(Avada, Fusion-Builder etc).

Each language in that JSON has as updated field that stores the datetime the .zip file was last compiled, a version field which is retrieved from the actual .po file and then the package field which contains the URL reference to that compiled file.
These get generated from the createJSON grunt task, so as long as the logic there remains the same for updating the time, version & URLs we should be OK. The actual URLs can be anything and it's relatively easy to change them from there, as long as there's a reference to the raw zip file 😉

To be honest I haven't worked that much with npm packages or unpkg so I'm not sure how involved implementing the CI for them would be. 🤔

Changing the client API in Avada, Fusion-Builder etc will be relatively easy, all we'll need to do it change the remote URLs they check and use the new ones instead of the ones from this repo.