ES_PLUGINS_INSTALL breaks boot sequence

[rewt]

After several different stages of debugging why my fresh instances are failing to boot, I've tracked it down to the ES_PLUGINS_INSTALL option. Without ES_PLUGINS_INSTALL, pods boot fine wihtout issue. Once ES_PLUGINS_INSTALL added, pods fail. I'm using the stateful master yaml files.

It seems any Pires image from 6.3.0 to latest shares same behaviour.

Steps to re-create

1. Download fresh es-master-stateful.yaml
2. Update image version to 6.4.0 (6.3.0 throws license error)
3. Update storage settings to match platform
4. Add ES_PLUGINS_INSTALL

ex.

- name: "ES_PLUGINS_INSTALL"
  value: "mapper-murmur3" 

Pod will load plugin, but then die.

Is there issue with stateful storage path? Unfortunately, I'm not able to find any logs.

[rewt]

Same issue on stateless configuration as well. Very curious - this was just working fine recently. What happened?

[rewt]

Pod log:

-> Downloading mapper-murmur3 from elastic
[=================================================] 100%
-> Installed mapper-murmur3
[2018-10-10T19:04:23,328][INFO ][o.e.n.Node ] [es-master-stateless-68bc4b596-6h5j7] initializing ...
[2018-10-10T19:04:24,207][INFO ][o.e.e.NodeEnvironment ] [es-master-stateless-68bc4b596-6h5j7] using [1] data paths, mounts [[/data (/dev/sda1)]], net usable_space [17.4gb], net total_space [29gb], types [ext4]
[2018-10-10T19:04:24,207][INFO ][o.e.e.NodeEnvironment ] [es-master-stateless-68bc4b596-6h5j7] heap size [247.5mb], compressed ordinary object pointers [true]
[2018-10-10T19:04:24,209][INFO ][o.e.n.Node ] [es-master-stateless-68bc4b596-6h5j7] node name [es-master-stateless-68bc4b596-6h5j7], node ID [CADIFrrJRRSSk5mD8sMFOA]
[2018-10-10T19:04:24,209][INFO ][o.e.n.Node ] [es-master-stateless-68bc4b596-6h5j7] version[6.3.1], pid[1], build[default/tar/eb782d0/2018-06-29T21:59:26.107521Z], OS[Linux/4.15.0-1023-azure/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b11]
[2018-10-10T19:04:24,209][INFO ][o.e.n.Node ] [es-master-stateless-68bc4b596-6h5j7] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms256m, -Xmx256m, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-10-10T19:04:28,216][WARN ][o.e.d.c.s.Settings ] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [aggs-matrix-stats]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [analysis-common]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [ingest-common]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [lang-expression]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [lang-mustache]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [lang-painless]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [mapper-extras]
[2018-10-10T19:04:31,802][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [parent-join]
[2018-10-10T19:04:31,802][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [percolator]
[2018-10-10T19:04:31,802][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [rank-eval]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [reindex]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [repository-url]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [transport-netty4]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [tribe]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-core]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-deprecation]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-graph]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-logstash]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-monitoring]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-rollup]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-security]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-sql]
[2018-10-10T19:04:31,804][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-upgrade]
[2018-10-10T19:04:31,804][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-watcher]
[2018-10-10T19:04:31,805][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded plugin [mapper-murmur3]
[2018-10-10T19:04:31,806][INFO ][o.e.x.s.a.s.FileRolesStore] [es-master-stateless-68bc4b596-6h5j7] parsed [0] roles from file [/elasticsearch/config/roles.yml]