es-data nodes exceeding Xmx memory

[fortuneFelix]

Hello,
We are running version 5.6.0 in our K8 1.10.3 cluster with docker version 1.13.1.
The es-data nodes, constantly exceed the configured the Xmx value.
Settings:
ES_JAVA_OPTS=-Xms2048m -Xmx2048m
Kubernetes memory limit: 4096M

Full JVM arguments:
-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms2048m, -Xmx2048m, -Des.path.home=/elasticsearch

The es-data nodes grow beyond 4GB and then are getting killed.

The container we run is:
quay.io/pires/docker-elasticsearch-kubernetes:5.6.0

Any idea why this is happening? Any advice on how to improve this so es-data nodes stay within their limits?

Thanks a ton for any advice/help!

• Felix

[bw2]

I've run a stable cluster with -Xms3900m -Xmx3900m or more.
I think memory usage will depend on what you're doing in terms of data size, indexing, and queries.
(https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html)
Why do you expect 2048m to be sufficient?

[msmaverick2018]

Are you running into issue where JVM does not run under cgroup limitations set by the container and looks at host memory for limitations. Check the link below:
https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed/

-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap
-XshowSettings:vm -version