[Azure Update Center] Serve directory listing instead of redirecting to mirrors (ending in HTTP/404)

[dduportal]

ref. #2649 (comment)

This issue tracks the work related to moving back workloads from mirrors to front Apache to allow directory listing.

As described in the linked comment, we want to have ALL files available in Apache (e.g. azure.updates.jenkins.io), and we'll only redirect requests to mirrors(.updates.jenkins.io) on heavily downloaded files.

[dduportal]

Update:

• Initial PR on the update center2 generation: chore(wrappers - publish.sh) only send JSON files to mirrors update-center2#816
  • This PR was deployed successfully
  • The directory listing are back:
    • https://azure.updates.jenkins.io/current/
  • While directory with an index.html are served by Apache like it was before:
    • https://azure.updates.jenkins.io
    • https://azure.updates.jenkins.io/current/latest
  • And the usual JSON files are served by the mirrors:
    • https://azure.updates.jenkins.io/update-center.json -> https://mirrors.updates.jenkins.io/update-center.json -> mirrors
    • https://azure.updates.jenkins.io/dynamic-2.476/update-center.json -> https://mirrors.updates.jenkins.io/dynamic-2.476/update-center.json
  • Jenkins Controllers are still working well with the current azure.updates.jenkins.io
  • 🔴 The "JSON wrapped in HTML" files are not redirected to mirrors (example: https://azure.updates.jenkins.io/current/update-center.json.html)
  • 🔴 The tools /updates path is not providing a directory listing: https://azure.updates.jenkins.io/updates/
  • 🟡 We don't have a favicon: https://azure.updates.jenkins.io/favicon.ico (HTTP/404)

Next steps:

• We have to cover the whole /updates path along with the crawler to:
  • Provide all files to Apache (it's only published to mirrors right now) to have a listing directoris at https://azure.updates.jenkins.io/
• We need to fix the .json.html wrapped files to be served by mirrors
• Favicon ? ;)

[dduportal]

Update:

• https://azure.updates.jenkins.io/updates/ is no serving a directory listing, but its content is served by mirrors
  • Fixed by chore(publish) add the 2 Apache destinations for publication to allow directory listing crawler#151
• https://azure.updates.jenkins.io/current/update-center.json.html is now served by the mirrors
  • Fixed by chore(wrappers/publish) new Azure UC: send all .json.html requests to mirrors update-center2#817