Move masters to StatefulSet

[jimmyjones2]

I think there is an issue with the master configuration that would cause issues if all master nodes were lost simultaneously:

• The data/client nodes would require restarting as Zen doesn't appear to re-query the elasticsearch-discovery DNS record after initial startup, so they wouldn't be able to find the newly provisioned master pods Master nodes with unpredictable names pires/kubernetes-elasticsearch-cluster#197
• Masters store state to disk, if this is lost, it can cause serious data loss across the cluster. Currently this is written directly into the container filesystem, which is also bad practice (PVs should be used).

Both of these issues would be resolved by moving to a StatefulSet for the masters, with a small PV associated to each and publishNotReadyAddresses: true

If there is agreement I'm happy to implement.