This proposal is to add a Kafka administration UI to the Strimzi project.
Kafka out of the box provides a number of useful command tools to perform common administrative tasks such as creating topics, viewing consumer group status, and consuming record content. While these are helpful, they are not always the simplest or most convenient way of monitoring and performing administrative tasks on a Kafka cluster. In these cases a dedicated administration UI, focused on common administration tasks, would not only enable a user to complete their task more quickly and simply than previously possible, but also enable a more positive user experience of Kafka and its capabilities.
This proposal will describe a Kafka administration UI capability. It will describe example capabilities which could be offered, cover at a high level how the UI code itself could be implemented (with more low level detail to be provided as required), how this could be delivered, as well as possible future work and how this could integrate with Strimzi today.
At a high level, I would propose a UI implemented as follows:
- It would be a TypeScript based UI (using Babel (MIT license) to provide latest ECMAScript capabilities in a cross browser compatible manner), using the React framework (MIT license)
- That the last 2 major versions of the following browsers are supported (via Babel transpiling/polyfilling):
- Google Chrome
- Microsoft Edge
- Mozilla Firefox
- Apple Safari
- This UI is hosted and provided via an Express server (MIT license)
- This UI is built in a modular/metadata driven manner - allowing for easy extensibility, modification, and dynamic behaviour at runtime
- Uses a GraphQL API (MIT license)
- Should be fully translatable, and accessible (using a superset of US Section 508 and WCAG 2.1 rulesets).
I would suggest that the UI implementation to follow the Model View Controller (MVC) pattern, with clear separation between business logic, and the view logic that renders the state of the UI. This would enable/allow the view logic to be modified/swapped in/out as required, but keep the same business logic. I would suggest using the Carbon design system as the view layer for the contributed UI, given it is open source, is supported, and I can provide both the design and implementation using it. As an example, the following is a mock up of how a Topics page could look using the Carbon design system (Apache 2 license), which would allow a user to view, create, edit and delete topics in a given cluster:
By maintaining and enforcing a thin view layer, this should not stop other view layers being designed, implemented and contributed in other frameworks, such as PatternFly or Material for example.
I would also suggest the use of TypeScript for this UI. TypeScript (vs regular JavaScript) is better fit for larger UI code bases, by enforcing typing and structure definition. The main benefit of this is the early detection of potential bugs in code early at 'compile' rather than runtime - e.g a function being called with an unexpected type. In addition, given this typing and structure, IDEs can then use this to enable easier navigation of the codebase (eg jump to implementation, type-ahead support). Secondly, it introduces a number of more typical programming language constructs, such as generics and interfaces, which will make the code base more approachable to those who have not worked with a UI previously. The main benefit to the Strimzi-ui however will be the ability to use (and enforce) shared data structures across view layer implementations, enabling the easy addition and contribution of other view layer implementations.
I would also suggest using Express as a server for this UI. This is due to its small footprint, modularity and available 3rd party modules, such as helmet for http security, Passport.js for authentication and authorization support or the graphql and express-graphql modules for GraphQL server support.
This UI would also be provided with a full set of supporting elements - such as end to end tests, automation (examples of which are in this section) to manage common tasks, and documentation around implementation approach, UI best practice and so on.
The below shows how these pieces could integrate.
Where:
- 'strimzi-api' is a backend server for Kafka data/requests
- 'Supporting elements' could be other Kafka related deployments such as Mirror Maker, Kafka Connect and Cruise control. These could also be non Kafka related deployments (not deployed as a part of Strimzi), such as LDAP servers, metric stores etc.
In addition to the implementation detailed above, I would propose that any UI contributed to Strimzi be managed and developed as follows:
- Be developed in a dedicated repository inside the Strimzi Github organisation
- Both the UI server and client are contained in this repo - worked as sub modules in a monorepo
- That design/discussion/implementation and defect issues are kept solely in this repository
- All development is done in a behavioural manner - focusing on the end user, and the task they are trying to achieve (see topic list for an example)
- That the repository operate in a cloud like CI/CD manner; lots of small/little and often deliveries of new function, appropriately sized and gated, allowing for the UI to be shipped at any time
- That automation, such as GitHub actions for example, is used as much as possible to support this CI/CD model - eg, automated dependency updates, automated testing run on PR, issue triage etc
A UI is more than just what a user sees. It also needs to be supported by backend data, services, and resources. To this end, and in bearing in mind the cloud development model mentioned above, all UI capability (both front and backend) should be provided in a phased manner. I would suggest initially that a topic listing page is the first deliverable, but before that can happen, prerequisites need to be discussed, agreed and implemented.
In order to provide any UI, the whole stack which will support it will need to be discussed and considered. This will include (but not be limited to):
- Setting up the required UI build/packaging for deployment to Strimzi
- Integrating the UI with Strimzi itself
- A Kafka backend server for Kafka data/requests the UI server can call, covered by proposal
- UI server configurability/capability - including aspects such as;
- Transport security
- User session, authentication and authorization capabilities)
- Cluster metadata - listener addresses, number of brokers, etc
- Maintenance/tracing support
- UI client side capabilities
- State design
- Configuration/capability discovery from the server
- Routing and navigation logic
- Maintenance/tracing capabilities
I am more than happy to elaborate and collaborate on any of these points either in the proposal, or via a number of follow on issues, worked as a part of the UI's development.
Given the prerequisites are satisfied, the first user task I would suggest is implemented is a listing of all the topics (along with partition and replica information per topic) in a given cluster. This will exercise all the prerequisites, as well as offer a user of Strimzi a new way to interact with their deployment. Using a gherkin behavioural syntax to describe the capability, it would:
Scenario: Seeing all available topics:
Given I am a user of the UI
When I navigate to the topic listing
Then see all topics in my Kafka cluster
Scenario: Viewing a specific topic (when it exists in the cluster):
Given I am a user of the UI
When I navigate to the topic listing
And I filter for topic 'SampleTopic'
Then I see topic 'SampleTopic' in the topic list
And Topic 'SampleTopic' has '3' partitions and '2' replicas
Scenario: Viewing a specific topic (when it does not exist in the cluster):
Given I am a user of the UI
When I navigate to the topic listing
And I filter for topic 'SampleTopic'
Then I am told topic 'SampleTopic' does not exist in the cluster
Additional Kafka capabilities which could be added later include (but would not be limited to):
- Create, update and delete topics on the topics list page (re-contextualised as a Topics page - as per mock up)
- View Consumer group status for a given topic, or the whole cluster
- View the Brokers in a given cluster, their configuration, and where appropriate, allow modification of broker configuration
- Provide details of Kafka listener/bootstrap addresses, along with sample configuration, to allow streamline client connectivity
If desired, additional Strimzi provided capabilities, such as Kafka Users, could be leveraged/added/managed through this UI also (for example, allowing access to the UI and its capabilities, general CRUD updates to them etc).
Finally, one capability which may be of interest (which may have bearing on how this is surfaced in Strimzi itself) is having a single UI, which can manage multiple Kafka clusters deployed via the Strimzi operator.
In all of these cases, capabilities can be added in a prioritised order, and should be added in a phased manner themselves (for example, add a view of the brokers and their configuration, then the ability to modify select configuration).
This UI could be deployed as a part of Strimzi as follows:
Where:
- The UI is deployed standalone, akin to how the Kafka bridge, Kafka Connect are currently for example (so a new CRD would be required, with configuration needed to reference which cluster(s) to interact with).
- The UI's spec would contain 0 to N Kafka clusters the UI will operate against, which will:
- Contain 1 or more 'backend' entries. These represent backend sources of data which can be surfaced in the UI.
- One of these entries will be the
strimzi-api(ie each Kafka kind will contain its own api server).
- One of these entries will be the
- Each of these entries will define items such as the address to use to connect to it, any auth credentials required, Transport security required, etc.
- Contain 1 or more 'backend' entries. These represent backend sources of data which can be surfaced in the UI.
- The UI's spec will contain general configuration for the UI - such as certificates to expose to clients etc.
This could look as follows in a CR configured by a user (note that common fields such as image, or readiness and liveness probes have been omitted here for conciseness, but would be present in a full CR).:
...
spec:
clientCert: <mounted TLS cert> # (1)
clusters:
- name: 'dev-cluster' # (2)
uiConfig: <mounted config map> # (3)
backend:
- name: 'strimzi-api' # (4)
type: [admin/...] # specific type to identify admin server? # (5)
address: 'https://route-to-strimzi-api-for-dev-cluster:port' # (6)
version: 1 # (7)
tls: # (8)
cert: <mounted tls certificate>
version: TLS_1.3
authentication: # (9)
host: 'https://route-to-provider:port'
type: <provider type> # (9.1)
registrationPath: '/oidc' # (9.2)
tokenPath: '/token' # (9.3)
basicAuthPath: '/user' # (9.4)
...
Where:
- Optional, certificate used between client and UI server. If omitted, UI server will serve via http rather than https.
- Required, string - the name of this cluster. Should be the same value as
metadata.namein the Kafka CR. - Optional, a config map containing JSON. If provided, values in this config map will override default configuration values.
- Required, string - a unique identifier of this 'backend'.
- Required, string - the type of this 'backend' - so we can have subtypes for admin etc.
- Required, string - endpoint address for this backend.
- Required, integer - the version of the API this UI will use.
- Optional, object. Contains tls configuration (certificate to use, protocol versions etc. If omitted, traffic between these two endpoints will be in the clear.
- Optional, object. Contains authentication configuration for the UI to allow a user to login and view that backend
- Required, string - the type of authentication that this 'backend' supports (bearer token, basic auth)
- Optional, string - depending on authentication type, UI may need to register OIDC callback urls and generate a client id/secret
- Optional, string - path to use for token exchange (bearer)
- Optional, string - depending on authentication type (basic), UI may need to render own login screen and then POST to validate user/password
I am suggesting this approach for the following reasons:
- It follows the model of similar supporting capabilities currently available in Strimzi
- It allows for multiple Kafka clusters to be managed via a single UI (a potential future work item)
This approach does have a few assumptions. These being:
- A
strimzi-apibeing deployed as a part of the Kafka cluster (ie one per namespace/cluster). This deployment is then referenced in the UI's CR (as above). - A handshake/metadata exchange will be required for the UI to discover and integrate with each backend. I expect this exchange to be the retrieval of a GraphQL schema, so they can be combined/unified by the UI server, as well as any other metadata appropriate to that backend (eg Kafka version of the cluster a
strimzi-apiis configured to use). - The backends registered with the UI will need to be version aware and backwards compatible - ie version 2 of an api also needs to support the version 1 api.
- Both the UI, and any backend requiring authentication and authorization (eg
strimzi-api) will need to align on how authentication and authorization will work. My suggestion would be to have a separation between resources (eg topics) and actions upon them, and the backing implementation which represent/persist them. As shown above, this could be provided via KafkaUsers, or other mechanisms, such as OAuth.
I would also suggest a name from the CRD of kafka-web-ui.
The above is a suggestion of how the UI could be deployed as a part of a Strimzi deployment. Given the scope/range of options available, I would suggest the exact implementation details are decided upon in a follow on issue, worked as a part of the UI's development/delivery to Strimzi.
This section describes how a UI deployed as a part of Strimzi could be secured. The following is based on the work done as a part of the security proposal, and how the UI would/could utilise the available security mechanisms.. The exact implementation, as well as any other required discussion around security I would suggest is had as a part of the UI's development/delivery to Strimzi.
Provide a mechanism for UI users to log into Strimzi and make authorized requests to Kafka.
UX would be impacted if a user had to provide credentials for every request made by the UI, so instead a user should log in once, and credentials are then automatically appended to requests. This session should have a maximum lifetime, at which point a user will need to re-authenticate. In addition, sessions should expire due to inactivity. A user should also be able to log out of a UI to allow for user switching.
This session must be shared between HTTP and WebSocket traffic – as the UI will be executing graphql queries, mutations and subscriptions using the Apollo graphql stack.
- Admin – Graphql server, supporting HTTP and websocket connections admin server proposal
- Express – UI server, handling sessions for the client
- Client – Browser running the strimzi UI, executing graphql queries/subscriptions
- Kafka – brokers (connected to via admin clients)
- Auth – external auth provider, oauth for Kafka (and possible oauth dance for UI)
The UI is being hosted by an Express server. Express has session middleware - https://www.npmjs.com/package/express-session - that can create and persist server-side sessions, using a cookie as a key to hydrate a session into the incoming express request. Proposed session store is a redis container as it's a lightweight key/value (and the session will just be storing a token value).
Authentication in node/Express can be achieved through by http://www.passportjs.org/ (MIT license) which provides a large collection of “strategies” for authenticating a user. For purposes of the proposal, the below diagrams show an oidc flows for a UI oauth dance, although other authentication mechanisms could be used.
Note – the log in flow is assuming an oauth dance to retrieve a user token. It could however be any type of token, with an accompanying passport strategy to inspect and validate it. In addition, calls to the IDP will only be required when a token needs to be either retrieved for the first time, or refreshed if it or the session has expired. This will be determined by looking at the token previously retrieved (if any) kept in the redis session store. Finally, depending on the passport strategy and kafka auth mechanism, the UI may need to render its own basic login page.
I would expect that the main development effort for a UI will be in a new repository in the Strimzi Github organisation. Given the discussion in strimzi/strimzi-kafka-operator#2540 and #9 , the source backend Kafka data would be via the Strimzi-admin codebase. At first this would provide support for the various Kafka backend calls needed to surface any required information for the UI, but could be extended to provide other sources of data for use in the UI also. In addition, depending on how and where the UI is provided in Strimzi, changes may be needed in the strimzi-kafka-operator to either define a UI CRD, or deploy a UI deployment if configured to do so via the operator. I would very much welcome discussion around this, and what would make the most sense for the Strimzi project.
- A Vert.X based UI Server: The server hosting the static files for a UI will also need to support the UI in number of ways. It may need to do session management, enforce security, process responses and perform other general logic. In my experience, Express can be easily augmented to have these capabilities (and more) in a highly configurable manner, while maintaining a small footprint (vs say a whole JVM) and performance. Generally speaking as well, it is part of the defacto stack for React UIs, alongside things like Webpack for build.
The proposal covers at a high level how and what a UI for Strimzi could provide. For more lower level detail, please see the Strimzi-ui repository, which is being populated with further, lower level details of how the UI would work.