Multi-Cluster Pod Deployment with Fabric8 Operator - Best Practices and API Support?

[aswinayyolath]

I'm currently working on a POC involving an Operator using Java and the fabric8 K8s client with an interesting use case. The Operator runs in one primary K8s cluster and applies a CR in that cluster. However, the Operator's role goes further - it should create and manage pods across multiple Kubernetes clusters based on the applied CR.

• Does fabric8 provide APIs that directly support such multi-cluster deployments?
• What are the best practices for implementing this type of cross-cluster resource management using Fabric8?

Potential Approach

One approach I am considering involves defining the CRD such that the CRs include details of the target clusters, such as their URLs and credentials. The Operator would use this information to interact with the Kubernetes API server of the target clusters to create resources.

spec:
  ...
  target:
    clusterUrl: <K8S Cluster URL>
    secret: <SecretName>

The secret field would reference a K8s Secret that contains the kubeconfig or authentication details required to access the target cluster. The Operator could then use these details to establish a KubernetesClient instance dynamically and create the necessary resources.

So these are my questions

1. Is there a more efficient way or best practice for achieving this kind of cross-cluster deployment with fabric8?
2. Does fabric8 already offer APIs or built-in features that support multi-cluster interactions, or is managing multiple KubernetesClient instances manually the way to go?
3. Are there existing examples or patterns that could help in securely handling kubeconfigs and credentials for connecting to multiple clusters?

I would appreciate any insights, examples, or advice on implementing this use case effectively and securely.

[manusa]

Does fabric8 already offer APIs or built-in features that support multi-cluster interactions

No, each Kubernetes Client instance has a Config instance that is applicable for a single cluster.

This means that if you want to target multiple clusters, you'll need a kuberntesClient instance for each cluster.

So potentially you'd have a KubernetesClient instance as your controller and then, you'd either create an ephemeral KubernetesClient instance for each external cluster operation or maintain a pool of KubernetesClients for each of the clusters you're managing.

[aswinayyolath]

Thank you for the detailed explanation! 😊 To ensure I've understood this correctly:

Single KubernetesClient per Cluster

• Each KubernetesClient instance must have its own Config, and this is only applicable for a single cluster.

Multiple Client Strategies

• For ephemeral operations, I would create a KubernetesClient instance on-demand for each external cluster operation, using the appropriate kubeconfig.

Alternatively, I could maintain a pool of KubernetesClient instances, one for each cluster, if I need frequent operations across multiple clusters.

Questions for Further Clarification

• Are there any best practices or potential pitfalls I should consider when managing and maintaining a pool of KubernetesClient instances?
• How should I handle authentication securely when creating these clients, especially when dealing with kubeconfigs for multiple clusters? Would using Kubernetes Secrets be the recommended approach?
• Are there specific libraries or utilities in Fabric8 that could help manage these multiple KubernetesClient instances more efficiently?

Thanks again for your guidance! Any further insights or best practices would be greatly appreciated.