Reaper Error: Error getting pending compactions attribute from JMX

[asgeek]

What happened?

Reaper is continuously showing this error, but it doesn’t cause the pod to restart. Both the Cassandra Cluster and Reaper are functioning as expected.

Did you expect to see something different?

Get rid of the exception from the Reaper logs.

How to reproduce it (as minimally and precisely as possible):

Environment

• K8ssandra Operator version:

  v1.18.0

  * Kubernetes version information: `v1.29.5+k3s1` * Kubernetes cluster kind:```

K3s

* Manifests:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
name: my-cluster-2
namespace: k8ssandra-operator
spec:
cassandra:
serverVersion: "4.0.0"
softPodAntiAffinity: true
storageConfig:
cassandraDataVolumeClaimSpec:
storageClassName: ceph-rbd
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
config:
jvmOptions:
heapSize: 512M
cassandraYaml:
read_request_timeout_in_ms: 10000
range_request_timeout_in_ms: 15000
write_request_timeout_in_ms: 10000
networking:
hostNetwork: false
resources:
requests:
cpu: "2"
memory: 8Gi
limits:
cpu: "2"
memory: 32Gi
datacenters:
- metadata:
name: e1
k8sContext: e1
size: 3
- metadata:
name: e2
k8sContext: e2
size: 3
reaper:
containerImage:
tag: "3.6.1"
# tried to fix the error disabling telemetry, but it didn't help
telemetry:
mcac:
enabled: false
prometheus:
enabled: false
vector:
enabled: false
stargate:
containerImage:
registry: docker-proxy.domain.com
tag: v1.0.77
heapSize: 384Mi
resources:
requests:
cpu: "1"
memory: 1Gi
limits:
cpu: "2"
memory: 4Gi

* Reaper Logs:

ERROR [2024-09-18 07:46:35,566] [ForkJoinPool-1-worker-41] i.c.m.ICassandraManagementProxy - Error getting pending compactions attribute from JMX
javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:type=Compaction,name=PendingTasks
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at jdk.remoteref/jdk.jmx.remote.internal.rmi.PRef.invoke(Unknown Source)
at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source)
at java.management.rmi/javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:904)
at io.cassandrareaper.management.jmx.JmxCassandraManagementProxy.getPendingCompactions(JmxCassandraManagementProxy.java:426)
at io.cassandrareaper.management.CompactionProxy.getPendingCompactions(CompactionProxy.java:97)
at io.cassandrareaper.management.ClusterFacade.listCompactionStatsDirect(ClusterFacade.java:575)
at io.cassandrareaper.service.MetricsService.grabAndStoreCompactionStats(MetricsService.java:124)
at io.cassandrareaper.service.Heart.updateMetricsForNode(Heart.java:275)
at io.cassandrareaper.service.Heart.lambda$updateMetricsForClusters$6(Heart.java:222)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

**Anything else we need to know?**:

* Cassandra Cluster is healthy
* Reaper UI is accessible
* Cluster information is visible in the Reaper UI



┆Issue is synchronized with this [Jira Story](https://datastax.jira.com/browse/K8OP-257) by [Unito](https://www.unito.io)
┆Issue Number: K8OP-257