Move website from gh-pages branch to master branch under website/ folder

[naskio]

What type of PR is this?

Enhancement of website build and deployment process.

What this PR does / why we need it:

Current solution:

Armada's website is a static site built with Jekyll and hosted on GitHub Pages. Currently:

• It's source code resides in the gh-pages branch.
• Documentation is written in Markdown format and stored in the docs/ folder within the master branch.
• The workflow Deploy GH Pages, located at .github/workflows/pages.yml within the master branch, is responsible
  for copying the documentation files from the master branch to the gh-pages branch.
• We use the Deploy from a branch option as source for building and deploying the website to GitHub Pages.
• The build and deployment process is triggered by pushing changes to the gh-pages branch, either through a pull
  request or through the Deploy GH Pages workflow.
• The website is accessible at https://armadaproject.io.

Issues with the current solution:

• Having a separate branch gh-pages for the website source code increases complexity and maintenance overhead.
• Keeping documentation in sync between the master and gh-pages branches requires additional effort and can lead to
  inconsistencies.
• A few weeks ago, protection rules were added to the gh-pages branch to prevent direct pushes. As a result,
  the Deploy GH Pages workflow can no longer push changes to the gh-pages branch. The only way to update
  the website now is through a pull request.

Proposed Solution:

To address the issues mentioned above, we propose the following setup:

• Relocate Source Code: Move the website source code from the gh-pages branch to the master branch under
  a website/ folder.
• Centralize Documentation: Keep all documentation in Markdown format within the docs/ folder in the master
  branch, using it as the source for the website content.
• Automate with GitHub Actions: Use GitHub Actions to build and deploy the website directly from the master
  branch.

This setup will simplify the build and deployment process, making it easier to maintain and update the website. It will
also ensure that the documentation is always in sync with the website content and is accessible from the IDE, GitHub,
and the website.

Note

More efforts have been made to ensure that the website works correctly on forks to allow previewing changes. my PR is available for preview at https://naskio.github.io/armada/.

Which issue(s) this PR fixes:

Fixes #3715

Special notes for your reviewer:

Once this PR is merged, we would also need to:

• Set "Source" to "GitHub Actions" in "Settings > Pages > Build and deployment > Source".
• Set "Custom domain" to armadaproject.io in "Settings > Pages > Build and deployment > Custom domain".
• Get rid of the gh-pages branch (Optional).

There's also few more minor issues that need to be addressed:

• The file docs/developer/etc-localdev.md doesn't exist, but it's referenced in docs/developer/README.md.
• The file docs/quickstart/img/lookout.png doesn't exist, but it's referenced in docs/demo.md.
• At the website/index.html we refer to /user#queue, /user#job and /user#job-set but these sections don't
  exist.

[Sharpz7]

Hey @naskio, thanks for working on this.

I would be hesitant to move to this solution because of the requirement that the website would have to be built locally every time you want to make a change to a README and have it reflected on the website (Unless I am mistake, this is the reason for the way it was originally)

We would either need to enforce CI to check that the website is built to latest (a process we already do for the auto-generated https://github.com/armadaproject/armada/blob/master/docs/python_armada_client.md in the python client), or just let the website get stale without pushes.

If more work can be done, maybe a Github Action that builds the website and opens a PR automatically? Is that possible?

I apologize if I am missing something - I am definitely in support of removing the current system if we can address this problem / agree it is okay.

[naskio]

Hi @Sharpz7,

The website will be built by the workflow .github/workflows/deploy-website.yml.
To update the documentation, you need to:

1. Make changes to files in the docs/ folder
2. Once your changes are merged into the main branch, the workflow will automatically:
   • Build a new version of the website
   • Publish it to GitHub Pages

You don't really need to build the website locally unless you want to develop something related to the website.

This approach is preferable to using a gh-pages branch as it eliminates potential duplication issues and keeps everything in the main branch.

If we need to generate some files before building the website, we can add it to the workflow .github/workflows/deploy-website.yml as well.

[Sharpz7]

Hey @naskio

Most of this change is made up of website-based files (HTML Files, CSS Files etc). They can be removed from this change if it is being built in CI?

I.e. I don't see how "Relocate Source Code: to /website" and "The website will be built by the workflow" are both needed at the same time? If we are building it in CI, we don't need a copy in git.

That is more what I was thinking about when I wrote my original comment.

[naskio]

@Sharpz7
Actually, the website has Jekyll source files (Markdown, HTML templates, CSS, JS) which are different from the output (static files).
The CI will take the Jekyll website source (Markdown content, HTML templates, CSS and JS sources) and generate the static website (rendered HTML, CSS and JS) and publish them to GitHub Pages.

[Sharpz7]

@naskio ah okay - it doesn't look like files we should need in the main repo (a lot of them do not look like source, they look like things that have been downloaded from other places), but I trust your judgement, since you have been leading these efforts.

Thank you, and thanks for answering my questions :))

I think you will need someone else with maintainer privileges to get this merged fully.