[NET-1484] Support for namespaces, partitions in consul endpoints

[roncodingenthusiast]

TODO

• KV list endpoint
• KV keys endpoint
• KV get endpoint
• Catalog node endpoint
• Catalog nodes endpoint
• catalog service endpoint
• catalog services endpoint
• Docs

How to test

Note: ensure you have consul running

previous behavior is preserved

• insert into kv in default namespace and partition

consul kv put "hashi/test" "default"

• create a file in.tpl insert the following to retrieve that key

{{ key "hashi/test" }}

• ensure it can still read the key

consul-template -template "in.tpl:out.txt" -once

• verify: your out.tpl file should have:

default

specify namespace/partition

• create a new consul namespace/partition (sorry these names aren't too creative :D)

consul partition create -name "partition"
consul namespace create -name "partition-ns" -partition "partition"

• store kv in the new partition/namespace created above

consul kv put -partition "partition" -namespace "partition-ns" "hashi/test" "part
ition"

• update your in.tpl to include namespace and partition

{{ key "hashi/test?partition=partition&ns=partition-ns" }}

• verify: your out.tpl file should have:

partition

[komapa]

I understand this is "alpha" but this does not work at all with range services and different namespaces. You get the same services as the first range services call because (I am guessing), you did not update the cache key to consider any of the new QUERY params (namespace, partition, peer)

Also, this functionality is pretty useless unless there is also range namespaces implemented. Right now, we need to do something like:

{{- range $k, $n := (plugin "consul" "namespace" "list" "--format" "json" | parseJSON) -}}

[jkirschner-hashicorp]

Hi @komapa,

It sounds like you have a specific scenario in mind that you need to work, but unfortunately doesn't work with the current implementation. To help get folks aligned on this, can you share that scenario and sample Consul Template syntax you'd hope would work for that scenario?

[komapa]

We just want to get all services for all namespaces that we care about:

{{- $namespaces := sprig_list "default" -}}
{{- range $k, $n := (plugin "consul" "namespace" "list" "--format" "json" | parseJSON) -}}
    {{- /* TODO: Figure out how we can check for $n.Meta.external-source eq kubernetes, instead */ -}}
    {{ if eq $n.Description "Auto-generated by consul-k8s" }}
        {{- $namespaces = sprig_append $namespaces $n.Name -}}
    {{- end -}}
{{- end -}}

{{- range $k, $namespace := $namespaces }}
{{- range services (printf "?ns=%s&partition=default" $namespace) }}
{{ $namespace }}: {{ .Name }}
{{- end }}
{{- end }}

[jkirschner-hashicorp]

And based on this comment from before:

Also, this functionality is pretty useless unless there is also range namespaces implemented.

I assume you actually want something like this?

{{- range namespaces "?partition=default"}
{{- range services (printf "?ns=%s&partition=default" .NamespaceName) }}
{{ $namespace }}: {{ .Name }}
{{- end }}
{{- end }}

In other words, is the below an accurate summary?

• UX Challenge: The top loop in your example is just to construct the list of namespaces to iterate over, which is unnecessary if range namespaces is implemented. This workaround is ugly, but functional.
• Bug: However, there's a bug that no workaround can resolve: range services is always returning results from the first namespace in the array (index 0 in range namespaces loop).

[komapa]

And based on this comment from before:

Also, this functionality is pretty useless unless there is also range namespaces implemented.

I assume you actually want something like this?

{{- range namespaces "?partition=default"}
{{- range services (printf "?ns=%s&partition=default" .NamespaceName) }}
{{ $namespace }}: {{ .Name }}
{{- end }}
{{- end }}

In other words, is the below an accurate summary?

• UX Challenge: The top loop in your example is just to construct the list of namespaces to iterate over, which is unnecessary if range namespaces is implemented. This workaround is ugly, but functional.
• Bug: However, there's a bug that no workaround can resolve: range services is always returning results from the first namespace in the array (index 0 in range namespaces loop).

Yes, this is perfect summary. Thank you!

[jkirschner-hashicorp]

Since both Consul and Vault have namespaces, it might make sense to have it be range consulNamespaces or something like that with variable name .ConsulNamespace to make it unambiguous (in case iterating over a Vault namespace tree is ever added, or to just make it clear to the reader that we're talking about a Consul namespace not a Vault one).

[jkirschner-hashicorp]

@komapa : The Bug described above should be fixed by this PR (to be included in the next Consul Template release): #1874.

The UX Challenge will need to be addressed separately in the future.