goroutine leak in template runner

[tgross]

A Nomad user reported an issue where template runner View.poll goroutines were being leaked when using templates with many dependencies.

An investigation found that when the runner's Run method is in-flight and the runner concurrently receives Stop, it's possible for the watcher to stop dependencies via stop and then immediately recreate them. Because the Stop method has returned, Nomad's template hook will assume that all goroutines have been stopped, and this causes the View.poll goroutines to be leaked. The race condition for this case is very narrow, but is most likely to occur when the user has both a very large number of dependencies, short secret leases, and the template Wait configuration is nil.

I've got a PR fixing this in hashicorp/consul-template#1898. Once that's been merged and released, we'll need to bump the dependency in Nomad. Looking to ship this fix in the next patch version of Nomad, as well as backports.

[tgross]

hashicorp/consul-template#1898 has been merged and we've sent a build to the folks who reported it so they can confirm it entirely fixes the issue. Once that's been done we'll get that dependency bump shipped here.