diff --git a/README.md b/README.md index cc802a6..bed83d3 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ If you've had problems with ingress-nginx, cert-manager, LetsEncrypt ACME HTTP01 ## One-line install ```shell -kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.1.2/deploy.yml +kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.2.0/deploy.yml ``` If you're using [ingress-nginx](https://kubernetes.github.io/ingress-nginx/) and [cert-manager](https://github.com/jetstack/cert-manager), it will work out of the box. See detailed installation and testing instructions below. @@ -72,7 +72,7 @@ The `dig` should show the external load balancer IP address. The first `curl` sh ### Step 1: Install hairpin-proxy in your Kubernetes cluster ```shell -kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.1.2/deploy.yml +kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.2.0/deploy.yml ``` If you're using `ingress-nginx`, this will work as-is. @@ -119,3 +119,15 @@ This time, the first `dig` should show an internal service IP address (generally NOTE: CoreDNS is a cache, so even if you see the `rewrite` rules in Step 2, it will take another minute or two before the queries resolve correctly. Be patient. You may wish to `watch -n 1 dig subdomain.example.com` to see when this changeover happens. At this point, cert-manager's self-check will pass, and you'll get valid LetsEncrypt certificates within a few minutes. + +### Step 4: (Optional) Install hairpin-proxy-etchosts-controller DaemonSet + +Note that the CoreDNS rewrites above only cover access within containers, while the iptables rewrite applies to the Node itself. This mismatch causes a problem if your node itself needs to access something behind your ingress. An example is if you're hosting your own container registry with [trow](https://github.com/ContainerSolutions/trow) and it's behind the ingress. If you follow only steps 1-3 above, you'll experience image pull failures because the Docker daemon (running on the Node directly, not in a container) can't access your registry. + +To resolve this, we need to rewrite the DNS on the Node itself. The Node does not use CoreDNS, so we can instead rewrite `/etc/hosts` to point to the IP address of the `hairpin-proxy-haproxy` service. This runs as a DaemonSet, so that it can modify each Node's copy of `/etc/hosts`. + +To install this DaemonSet: + +```shell +kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.2.0/deploy-etchosts-daemonset.yml +``` diff --git a/deploy-etchosts-daemonset.yml b/deploy-etchosts-daemonset.yml new file mode 100644 index 0000000..1506a94 --- /dev/null +++ b/deploy-etchosts-daemonset.yml @@ -0,0 +1,36 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + app: hairpin-proxy-etchosts-controller + name: hairpin-proxy-etchosts-controller + namespace: hairpin-proxy +spec: + selector: + matchLabels: + app: hairpin-proxy-etchosts-controller + template: + metadata: + labels: + app: hairpin-proxy-etchosts-controller + spec: + serviceAccountName: hairpin-proxy-controller-sa + containers: + - image: compumike/hairpin-proxy-controller:0.2.0 + name: main + command: ["/app/src/main.rb", "--etc-hosts", "/app/etchosts"] + volumeMounts: + - name: etchosts + mountPath: /app/etchosts + resources: + requests: + memory: "50Mi" + cpu: "10m" + limits: + memory: "100Mi" + cpu: "50m" + volumes: + - name: etchosts + hostPath: + path: /etc/hosts + type: File diff --git a/deploy.yml b/deploy.yml index d1da576..0135d6e 100644 --- a/deploy.yml +++ b/deploy.yml @@ -23,7 +23,7 @@ spec: app: hairpin-proxy-haproxy spec: containers: - - image: compumike/hairpin-proxy-haproxy:0.1.2 + - image: compumike/hairpin-proxy-haproxy:0.2.0 name: main resources: requests: @@ -151,7 +151,7 @@ spec: runAsUser: 405 runAsGroup: 65533 containers: - - image: compumike/hairpin-proxy-controller:0.1.2 + - image: compumike/hairpin-proxy-controller:0.2.0 name: main resources: requests: diff --git a/hairpin-proxy-controller/src/main.rb b/hairpin-proxy-controller/src/main.rb index aa8b6fd..8848e12 100755 --- a/hairpin-proxy-controller/src/main.rb +++ b/hairpin-proxy-controller/src/main.rb @@ -3,6 +3,8 @@ require "k8s-ruby" require "logger" +require "optparse" +require "socket" class HairpinProxyController COMMENT_LINE_SUFFIX = "# Added by hairpin-proxy" @@ -32,7 +34,9 @@ def fetch_ingress_hosts end }.flatten all_tls_blocks = all_ingresses.map { |r| r.spec.tls }.flatten.compact - all_tls_blocks.map(&:hosts).flatten.compact.sort.uniq + hosts = all_tls_blocks.map(&:hosts).flatten.compact + hosts.filter! { |host| /\A[A-Za-z0-9.\-_]+\z/.match?(host) } + hosts.sort.uniq end def coredns_corefile_with_rewrite_rules(original_corefile, hosts) @@ -68,10 +72,69 @@ def check_and_rewrite_coredns end end + def dns_rewrite_destination_ip_address + Addrinfo.ip(DNS_REWRITE_DESTINATION).ip_address + end + + def etchosts_with_rewrite_rules(original_etchosts, hosts) + # Returns a String represeting the original /etc/hosts file, modified to include a rule for + # mapping *hosts to dns_rewrite_destination_ip_address. This handles kubelet and the node's Docker engine, + # which does not go through CoreDNS. + # This is an idempotent transformation because our rewrites are labeled with COMMENT_LINE_SUFFIX. + + # Extract base configuration, without our hairpin-proxy rewrites + our_lines, original_lines = original_etchosts.strip.split("\n").partition { |line| line.strip.end_with?(COMMENT_LINE_SUFFIX) } + + ip = dns_rewrite_destination_ip_address + hostlist = hosts.join(" ") + new_rewrite_line = "#{ip}\t#{hostlist} #{COMMENT_LINE_SUFFIX}" + + if our_lines == [new_rewrite_line] + # Return early so that we're indifferent to the ordering of /etc/hosts lines. + return original_etchosts + end + + (original_lines + [new_rewrite_line]).join("\n") + "\n" + end + + def check_and_rewrite_etchosts(etchosts_path) + @log.info("Polling all Ingress resources and etchosts file at #{etchosts_path}...") + hosts = fetch_ingress_hosts + + old_etchostsfile = File.read(etchosts_path) + new_etchostsfile = etchosts_with_rewrite_rules(old_etchostsfile, hosts) + + if old_etchostsfile.strip != new_etchostsfile.strip + @log.info("/etc/hosts has changed! New contents:\n#{new_etchostsfile}\nWriting to #{etchosts_path}...") + File.write(etchosts_path, new_etchostsfile) + end + end + def main_loop + etchosts_path = nil + + OptionParser.new { |opts| + opts.on("--etc-hosts ETCHOSTSPATH", "Path to writable /etc/hosts file") do |h| + etchosts_path = h + raise "File #{etchosts_path} doesn't exist!" unless File.exist?(etchosts_path) + raise "File #{etchosts_path} isn't writable!" unless File.writable?(etchosts_path) + end + }.parse! + + if etchosts_path && etchosts_path != "" + @log.info("Starting in /etc/hosts mutation mode on #{etchosts_path}. (Intended to be run as a DaemonSet: one instance per Node.)") + else + etchosts_path = nil + @log.info("Starting in CoreDNS mode. (Indended to be run as a Deployment: one instance per cluster.)") + end + @log.info("Starting main_loop with #{POLL_INTERVAL}s polling interval.") loop do - check_and_rewrite_coredns + if etchosts_path.nil? + check_and_rewrite_coredns + else + check_and_rewrite_etchosts(etchosts_path) + end sleep(POLL_INTERVAL) end