Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for --internal network? #23

Open
dkoshkin opened this issue Mar 8, 2023 · 4 comments · May be fixed by #25
Open

Support for --internal network? #23

dkoshkin opened this issue Mar 8, 2023 · 4 comments · May be fixed by #25

Comments

@dkoshkin
Copy link

dkoshkin commented Mar 8, 2023

Just found this project and super excited, works great for the default or other non --internal docker networks.

$ docker run --rm --name nginx -d nginx
$ docker inspect nginx --format '{{.NetworkSettings.IPAddress}}'
172.17.0.2
$ curl -m 1 -I 172.17.0.2
HTTP/1.1 200 OK
Server: nginx/1.23.3
Date: Wed, 08 Mar 2023 03:44:47 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 13 Dec 2022 15:53:53 GMT
Connection: keep-alive
ETag: "6398a011-267"
Accept-Ranges: bytes

But it doesn't work if I use an --internal network

$ docker network create --internal internal
$ docker run --rm --name nginx-internal -d nginx --network internal
# note that NetworkSettings.IPAddress is empty
$ docker inspect nginx-internal --format '{{.NetworkSettings.Networks.internal.IPAddress}}'
$ curl -m 1 -I 172.28.0.2
curl: (28) Connection timed out after 1002 milliseconds

Was wondering if its even possible for this to work with an internal network?

@gregnr
Copy link
Member

gregnr commented Mar 8, 2023

Hey @dkoshkin, thanks for reaching out. Without --internal, Docker creates a bridge network between the container and the host (Linux VM) to give it external access. Since adding --internal removes this bridge, you no longer get access to the Wireguard tunnel that lives on the Linux VM. (not entirely true - see below)

Curious what your use case is using --internal with this tool?

@jimmidyson
Copy link

We use it to simulate air-gapped (specifically no egress) networks in docker.

@gregnr
Copy link
Member

gregnr commented Apr 14, 2023

Got it. Yeah this is a bit of an interesting case, as typically air-gap would exclude all networks, including your host (but of course, that makes dev/debugging hard).

The solution here would be to somehow create a connection only between the --internal container and the macOS host, without passing through the Linux host's network namespace. I'll have to think more about this one.

@gregnr
Copy link
Member

gregnr commented Apr 22, 2023

Okay I did a deep dive on this today. This is what I discovered (for those curious):

How internal works

  • I was partially wrong about the bridge network above. For internal networks, a bridge network is created. This makes sense of course - how else would multiple containers talk to each other within the internal network?
  • The difference between an internal bridge and a regular bridge actually just comes down to a few iptables rules:
    • If internal, 2 additional iptables rules are added to the Linux host:
      • iptables -I DOCKER-ISOLATION-STAGE-1 ! -d 172.20.0.0/16 -i br-f653b152ce48 -j DROP
      • iptables -I DOCKER-ISOLATION-STAGE-1 ! -s 172.20.0.0/16 -o br-f653b152ce48 -j DROP
      • Uses br-f653b152ce48 as the example internal bridge interface and 172.20.0.0/16 as the example subnet
    • Basically Docker is saying, "If any packet from the internal subnet is going anywhere other than the same network, drop it"
    • There are a few additional NAT rules for a normal bridge and missing for an internal bridge that will masquerade when going to an external network - not relevant here

Solution

We need to respect the above isolation rules for internal networks while still giving access to the Wireguard VPN. I think the solution is to add 2 iptables rules to the Linux host:

  • iptables -I DOCKER-USER -o chip0 -j ACCEPT
  • iptables -I DOCKER-USER -i chip0 -j ACCEPT

Basically anytime a packet is headed to or from the chip0 (Wireguard) interface, accept it.
Fyi, DOCKER-USER is a custom iptables chain created by Docker intended to be augmented by end users. See:
https://docs.docker.com/network/iptables/

I've tested the above and it works. I think it is safe to add these rules as defaults (rather than some sort of opt-in), because this is more-or-less the default behaviour if you were to run Docker on a vanilla Linux host.

@gregnr gregnr linked a pull request Apr 22, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants