Proposal: new node kind K8s kind

[GrigoriyMikhalkin]

I'm not sure if it really should be a part of containerlab, but i have an example case when it would be useful.

In our lab setup mini-lab we run network switches which need to be connected to service running in K8s cluster created by kind. For that we need to run kind and nodes in a same network.

Right now, for such case, you would have to pre-create a new network, before running kind and containerlab, and then clean it afterwards. Introducing new kind node kind would simplify such setup for users. Since network management would be done by containerlab.

Of course, one potential drawback: containerlab would have to use kind as it's dependency if we would want to implement this.

[hellt]

what if we drive this not through API integration with kind, but by shelling out the calls to kind CLI and, if needed, to create a kind config file?

[GrigoriyMikhalkin]

So, create kind config and call kind CLI? That could work. But then docs have to be very explicit that this kind node is dependent on kind CLI. Not sure how much userfriendly is it)

[hellt]

I am not against experimenting with adding kind as a dep either. If you want to explore that path we can definitely check it out first

[GrigoriyMikhalkin]

Great, i will hack on this next weekend then!

[networkop]

I think if you just want your kind nodes to be on the same network as clab nodes, you could use KIND_EXPERIMENTAL_DOCKER_NETWORK when creating a kind cluster and point it at the network created by the containerlab?

But this would simply mean that kind and network nodes are connected to the same bridge -- not the exact k8s cluster replica.

The "right" way would be to create a network fabric and then attach kind nodes to it, changing NodeIPs to be the fabric-facing IPs (as opposed to eth0 IPs). I think this would be a lot harder to achieve with kind

[GrigoriyMikhalkin]

I think if you just want your kind nodes to be on the same network as clab nodes, you could use KIND_EXPERIMENTAL_DOCKER_NETWORK when creating a kind cluster and point it at the network created by the containerlab?

Thanks, yep, this solution would work for my case)

The "right" way would be to create a network fabric and then attach kind nodes to it, changing NodeIPs to be the fabric-facing IPs (as opposed to eth0 IPs). I think this would be a lot harder to achieve with kind

I was thinking that kind nodes could be used basically as linux nodes in containerlab. So you would be able to establish node-to-node connections in a similar way. Do you mean, that this part would be hard to achieve?

Also, regarding node IPs. Just checked, internally nodes assigned to the same IP as container. But probably i misunderstood you.

[networkop]

what I meant with nodeIPs is the following. Let's suppose you have a kind cluster connected to the same bridge as your containerized network fabric. All inter-node comms will happen over the eth0 interface, which for the network fabric is an out of band interface. This could very well be your design.
But in my case, I'm thinking of testing various integration scenarios with the likes of calico or cilium, where I'd need to peer with next-hop TOR switches over an in-band interface. This would mean I need a new eth1 interface connected to the in-band port of my TOR, this interface would need a unique IP and I'd need to change the nodeIP to be this eth1's IP. So this change from eth0's IP to eth1 is quite difficult as kind hard-codes a lot of domain names to eth0 (which is the main reason they've moved to non-default docker bridge -- to use docker built-in DNS).

[hellt]

@networkop
what if you do the following

1. deploy the underlay with containerlab where mgmt network is the default clab network 172.20.20.0/24
2. at the same time create a bridge called fabric
3. nodes of the clab topo file can connect their network interfaces (eth1, etc.) to that bridge using the bridge kind and the appropriate endpoints config.
4. kind will use fabric bridge as a KIND_EXPERIMENTAL_DOCKER_NETWORK

[networkop]

@hellt it'll work, it'll get you half way but it's a bit hacky. One thing I'd like to do is test BGP using IPv6 LLA neighbour discovery and this would only work on p2p links.
I think it may be possible to use kind nodes (without kind itself) and pass the kubeadm commands as exec. this way you can just connect them using standard containerlab yaml file, no need for special bridges. This will need an extra feature in containerlab to assign static IPs on veth interfaces.

[hellt]

The work is being done in #1110

[steiler]

If you are fine with deploying the kind cluster(s) via an external workflow, you can since today (only main branch, release as of today) use the "ext-container" node kind.
Via this kind pre-existing, non containerlab-managed containers can be wired. For more details take a look at the doc at https://github.com/srl-labs/containerlab/blob/main/docs/manual/kinds/ext-container.md