OpenSearch JVM crash

[dgilling]

Please provide a brief summary of the bug

Seeing a crash on OpenSearch 2.6.0. 

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (virtualSpaceNode.cpp:190), pid=1, tid=270
#  fatal error: Failed to uncommit metaspace.
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.6+10 (17.0.6+10) (build 17.0.6+10)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (17.0.6+10, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xed5fe2]  metaspace::VirtualSpaceNode::uncommit_range(MetaWordImpl**, unsigned long)+0x1c2
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /usr/share/opensearch/core.1)
#
# JFR recording file will be written. Location: /usr/share/opensearch/hs_err_pid1.jfr
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

--------------- S U M M A R Y ------------

Host: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz, 20 cores, 157G, Amazon Linux release 2 (Karoo)
Time: Wed Mar 1 03:58:30 2023 UTC elapsed time: 2262.216714 seconds (0d 0h 37m 42s)

--------------- T H R E A D ---------------

Current thread (0x00007fa135249b50): VMThread "VM Thread" [stack: 0x00007f90798a4000,0x00007f90799a4000] [id=270]

Stack: [0x00007f90798a4000,0x00007f90799a4000], sp=0x00007f90799a22e0, free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xed5fe2] metaspace::VirtualSpaceNode::uncommit_range(MetaWordImpl**, unsigned long)+0x1c2
V [libjvm.so+0xb8ee88] metaspace::Metachunk::uncommit_locked()+0x38
V [libjvm.so+0x53f414] metaspace::ChunkManager::return_chunk(metaspace::Metachunk*)+0x134
V [libjvm.so+0xb927ce] metaspace::MetaspaceArena::~MetaspaceArena()+0xbe
V [libjvm.so+0x5942b6] ClassLoaderMetaspace::~ClassLoaderMetaspace()+0x36
V [libjvm.so+0x58a39e] ClassLoaderData::~ClassLoaderData()+0x12e
V [libjvm.so+0x59025f] ClassLoaderDataGraph::purge(bool)+0x3f
V [libjvm.so+0x6efcd0] G1CollectedHeap::prepare_heap_for_mutators()+0x30
V [libjvm.so+0x70c6bc] G1FullCollector::complete_collection()+0x5c
V [libjvm.so+0x6e6404] G1CollectedHeap::do_full_collection(bool, bool, bool) [clone .part.0]+0x164
V [libjvm.so+0x76a4bb] VM_G1CollectFull::doit()+0x3b
V [libjvm.so+0xee364a] VM_Operation::evaluate()+0xea
V [libjvm.so+0xee4ff3] VMThread::evaluate_operation(VM_Operation*)+0x103
V [libjvm.so+0xee5544] VMThread::inner_execute(VM_Operation*)+0x1d4
V [libjvm.so+0xee582f] VMThread::run()+0xbf
V [libjvm.so+0xe62020] Thread::call_run()+0xc0
V [libjvm.so+0xc187e1] thread_native_entry(Thread*)+0xe1

Register to memory mapping:

RAX=0x00007fa13c4e8000 points into unknown readable memory: 0x0000000000000058 | 58 00 00 00 00 00 00 00
RBX=0x00007f8fff880000 is pointing into metadata
RCX=0x00007fa13b5b9e4b: <offset 0x0000000001018e4b> in /usr/share/opensearch/jdk/lib/server/libjvm.so at 0x00007fa13a5a1000
RDX=0x00000000000000be is an unknown value
RSP=0x00007f90799a22e0 points into unknown readable memory: 0x0000000000010000 | 00 00 01 00 00 00 00 00
RBP=0x00007f90799a2320 points into unknown readable memory: 0x00007f90799a2340 | 40 23 9a 79 90 7f 00 00
"logs/hs_err_pid1.log" 265286L, 46323744B

Please provide steps to reproduce where possible

Run OpenSearch 2.6.0 Docker image with OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (build 17.0.6+10, mixed mode, sharing)

Expected Results

Not crash

Actual Results

JVM crashed referencing invalid memory

What Java Version are you using?

OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (build 17.0.6+10, mixed mode, sharing)

What is your operating system and platform?

Ubuntu

How did you install Java?

Bundled with opensearchproject/opensearch:2.6.0

Did it work before?

No response

Did you test with the latest update version?

TBD

Did you test with other Java versions?

TBD

Relevant log output

No response

[karianna]

@dgilling Is that a fully patched Ubuntu and the latest version of OpenSearch? Is this occurring on one host or all hosts?

[stream-coder]

@dgilling Is this problem solved?

[dgilling]

Yes, this can be closed. Was unrelated.

[wind57]

we are having the same crash report and this was the only place on the web where we found mentioning it :|