Allow Consul DNS to recurse on haproxy nodes

[ewdurbin]

PR #486 introduced resolution of hosts for haproxy backends via consul, but did not give it any way to resolve external DNS for the console/pythonanywhere or jobs.pyfound.org/applytojob.com backends.

This adds a recursive DNS setup on the haproxy nodes so that they can successfully and fully resolve external services.

[ewdurbin]

Before:

vagrant@loadbalancer:/etc/consul.d$ dig _console._tcp.service.consul SRV @127.0.0.1 -p 8600

; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> _console._tcp.service.consul SRV @127.0.0.1 -p 8600
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35362
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_console._tcp.service.consul.	IN	SRV

;; ANSWER SECTION:
_console._tcp.service.consul. 0	IN	SRV	1 1 443 www.pythonanywhere.com.

;; Query time: 2 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1) (UDP)
;; WHEN: Thu Aug 29 16:15:55 UTC 2024
;; MSG SIZE  rcvd: 99

After:

vagrant@loadbalancer:/etc/consul.d$ dig _console._tcp.service.consul SRV @127.0.0.1 -p 8600

; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> _console._tcp.service.consul SRV @127.0.0.1 -p 8600
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16638
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_console._tcp.service.consul.	IN	SRV

;; ANSWER SECTION:
_console._tcp.service.consul. 0	IN	SRV	1 1 443 www.pythonanywhere.com.

;; ADDITIONAL SECTION:
www.pythonanywhere.com.	0	IN	A	35.173.69.207

;; Query time: 8 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1) (UDP)
;; WHEN: Thu Aug 29 16:15:37 UTC 2024
;; MSG SIZE  rcvd: 115

[github-actions]

Don't forget to pull the latest changes on salt.nyc1.psf.io!