Kkavish/compatibility test

[kkavish]

This PR builds and runs Consul, Nomad and other dependencies using a docker image and then runs all the tests using the image to test whether a particular version of consul template is compatible with a particular version of consul CE/ENT.

Go through the Readme.md file for more details.

Read about the test results here - https://hashicorp.atlassian.net/browse/NET-8062

[david-yu]

Approving README

[david-yu]

@lkysow or @tgross Could you take a look to see if this looks good?

[tgross]

@kkavish there's a lot about this PR that's not well-motivated by the README or even the PR description. We're running Docker-in-Docker here? Why the rsyslogd? Why are we reading from cgroups? Shouldn't these tests run in CI and not exist in a Makefile?

[tgross]

This script doesn't appear to exist

[kkavish]

@kkavish there's a lot about this PR that's not well-motivated by the README or even the PR description. We're running Docker-in-Docker here? Why the rsyslogd? Why are we reading from cgroups? Shouldn't these tests run in CI and not exist in a Makefile?

@tgross

1. Docker-in-Docker so that we can build multiple images with different versions of CT and Consul together and then run the tests without changing the actual binaries on the system manually. The current Unit Tests require Vault, Consul, Nomad binaries to be present in the PATH so that it can run them and then run the unit tests. With Docker-in-Docker running compatibility tests will be cumbersome.
2. Apparently Nomad requires rsyslogd to run, when I tried running Nomad inside docker without rsyslog it crashed.
3. For cgroups - Cgroup v2 errors on nomad 1.7.2 nomad#19587, Docker needs access to all cgroups (I guess) to run in privileged mode. Only then can we run Nomad in privileged mode. More on this:

• Running nested inside docker starts failing with Ubuntu Jammy: writing file /sys/fs/cgroup/buildah-<random>/cgroup.procs: Operation not supported containers/podman#14884
• podman-in-docker: building image fails on cgroupv2-enabled systems containers/podman#12559 (comment)

With respect to CI, could you please let me know what is on your mind to integrate this better with Consul Template.

Thanks for taking time out to review this.

[tgross]

`. Docker-in-Docker so that we can build multiple images with different versions of CT and Consul together and then run the tests without changing the actual binaries on the system manually. The current Unit Tests require Vault, Consul, Nomad binaries to be present in the PATH so that it can run them and then run the unit tests. With Docker-in-Docker running compatibility tests will be cumbersome.

Why do we care about "the system" here... tests like should be running in CI, where the whole environment gets thrown away when we're done. Right?

2. Apparently Nomad requires rsyslogd to run, when I tried running Nomad inside docker without rsyslog it crashed.

It definitely does not require rsyslogd, but I can't tell you why you're running into trouble because you're not even running Nomad in this test rig anymore and never had any scripts or configuration in this PR to do so.

3. For cgroups - Cgroup v2 errors on nomad 1.7.2 nomad#19587, Docker needs access to all cgroups (I guess) to run in privileged mode. Only then can we run Nomad in privileged mode. More on this:

It's possible I've missed it, but as far as I can tell none of the tests actually run Nomad workloads, so it's not like we need to run a real Nomad client that can run Docker here, right?

With respect to CI, could you please let me know what is on your mind to integrate this better with Consul Template.

Well it's your PR... what exactly did you have in mind for this test if it's not going to be running in CI?

[kkavish]

`. Docker-in-Docker so that we can build multiple images with different versions of CT and Consul together and then run the tests without changing the actual binaries on the system manually. The current Unit Tests require Vault, Consul, Nomad binaries to be present in the PATH so that it can run them and then run the unit tests. With Docker-in-Docker running compatibility tests will be cumbersome.

Why do we care about "the system" here... tests like should be running in CI, where the whole environment gets thrown away when we're done. Right?

2. Apparently Nomad requires rsyslogd to run, when I tried running Nomad inside docker without rsyslog it crashed.

It definitely does not require rsyslogd, but I can't tell you why you're running into trouble because you're not even running Nomad in this test rig anymore and never had any scripts or configuration in this PR to do so.

Sorry, I re-tried and it was docker which did not start without rsyslog.

3. For cgroups - Cgroup v2 errors on nomad 1.7.2 nomad#19587, Docker needs access to all cgroups (I guess) to run in privileged mode. Only then can we run Nomad in privileged mode. More on this:

It's possible I've missed it, but as far as I can tell none of the tests actually run Nomad workloads, so it's not like we need to run a real Nomad client that can run Docker here, right?

It definitely does not run a Nomad workload, but it does query Nomad variables, services, etc in the tests. So, I think a running Nomad client is required.

With respect to CI, could you please let me know what is on your mind to integrate this better with Consul Template.

Well it's your PR... what exactly did you have in mind for this test if it's not going to be running in CI?

The problem with running this in CI is how do you input what versions of Consul and CT to tests without actually changing the versions manually. Also, this was not meant to run in CI rather manually by configuring any version of Consul or Vault or Nomad against any version of CT. And hence the question.

[tgross]

It definitely does not run a Nomad workload, but it does query Nomad variables, services, etc in the tests. So, I think a running Nomad client is required.

That would require a Nomad server and agent, not a client. But again, it doesn't matter at all if you're not actually running Nomad here.

The problem with running this in CI is how do you input what versions of Consul and CT to tests without actually changing the versions manually.

Typically that's done with either a matrix test (via GHA's features) or something like how we do our Consul compatibility tests on Nomad where we let the test itself construct the matrix: https://github.com/hashicorp/nomad/blob/main/e2e/consulcompat/shared_download_test.go

Also, this was not meant to run in CI rather manually by configuring any version of Consul or Vault or Nomad against any version of CT.

Why wouldn't you run it in CI so that we know that new PRs don't break backwards compatibility? No one is ever going to run these tests again by hand if that's the only way they can be run.

[kkavish]

@tgross thanks, sure let me have a look at Consul-Nomad compat tests and update this PR accordingly.

However, I guess the compat matrix created through this approach might still hold and @david-yu can accept the matrix. Your thoughts on this please.

[david-yu]

@kkavish The matrix looks good to me. Thank you for doing this, this was asked by a customer.

[kkavish]

@kkavish The matrix looks good to me. Thank you for doing this, this was asked by a customer.

thanks @david-yu , I'll work on making this part of the CI as well as suggested by @tgross