Move list of extensions to external file

[heitortsergent]

The list of extensions available on the Explore extensions page used to be a JSON file, and we migrated it to be included as part of the versioned files with the Grafana docs move.

Those extensions aren't necessarily versioned the same way that k6 is, so we should move them to a separate place that can represent a single source of truth and then update the documentation to use that file to list them in all versions of the k6 docs.

[szkiba]

A PoC for the external registry:
https://github.com/grafana/k6-extension-registry

[heitortsergent]

@szkiba thanks for working on this! From what I understand, we could use this file as the source for the documentation: https://grafana.github.io/k6-extension-registry/extensions.json?

@jdbaldry would it be possible for us to read that file and then feed it into a Markdown file? 🤓

[szkiba]

From what I understand, we could use this file as the source for the documentation: https://grafana.github.io/k6-extension-registry/extensions.json?

Exactly.
https://grafana.github.io/k6-extension-registry/registry.json can also be used. extensions.json fits the previous structure a little better (it doesn't have the entry called "k6", there is no "repo" object, it changes less frequently, etc.)

In the meantime, things have changed, I will describe it in a new comment.

I think the extension documentation should be moved outside of the k6 releases. The generation can initially be done with a simple schedule (1-2 times per day). Later, we will be able to solve the change detection and trigger the generation only when a change has occurred.

[jdbaldry]

Nice, yeah we can do this!

There's a couple of ways of the top of my head that you may want to consider:

1. Sync the data to the website repository and use Hugo templates. This gives us the most flexibility in HTML structure. We would generate the page directly from the data using the Hugo templating language. The downside is that changes in the data wouldn't be immediately available in grafana/docs-base image. I could extend the make docs procedure to support data mounting though and then as long as you have both repositories checked out, you're good to develop in tandem.
2. Generate a Markdown page source file from the JSON. This introduces a generation step. It is common in other repositories but can be a hurdle for contributors.

In either case, we probably need to write down what's happening somewhere. I'm happy to do that as part of any PR if you tell me where you'd like that documentation to live.

[szkiba]

Hi @heitortsergent , @jdbaldry ,

The next milestone of automated registry management has been completed.

The grafana/k6-extension-registry repository contains the extension registry. The source of the registry contains the most important properties, these are supplemented by automation using the GitHub API (e.g. number of stars, versions, etc.).

In the event of a change, other repositories are notified with a repository_dispatch event. Good examples of this are the grafana/k6-extension-catalog and grafana/k6-extension-list repositories. k6-extension-catalog automatically generates JSON catalog files and publishes them using GitHub Pages. k6-extension-list is an example, it automatically generates JSON extension list and markdown extension list in README.md file. It pushes the change to a branch and opens a pull request.

I think k6-docs should work similarly, i.e. a repository_dispatch event would inform about extension registry changes. The workflow downloads the registry from https://grafana.github.io/k6-extension-registry/registry.json and generates what it needs from it, and then publishes the generated extensions documentation.

Will this work for you?

[jdbaldry]

Nice!

Yeah I think repository dispatch with that generates changes from the registry JSON seems great to me.

[szkiba]

Sounds good.

I added the grafana/k6-docs repository to the list of repositories to be notified.
In the event of an extension registry change, a repository_dispatch event will be received, the event_type will be extension-registry-changed. There are no other parameters, the latest version of the extension registry can be downloaded from https://grafana.github.io/k6-extension-registry/registry.json

Let me know if I can help you with anything else regarding the registry.

[jdbaldry]

Thank you for your work!

I'll arrange with @heitortsergent when he's back and get something set up to generate the documentation from that registry :)

[szkiba]

Hey @jdbaldry , I made a solution proposal to update the extension list in the documentation automatically:

#1731

If you have some time, I would appreciate your feedback.