Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Alerts not working with custom detection rule #1227

Open
duzvik opened this issue Aug 8, 2024 · 3 comments
Open

[BUG] Alerts not working with custom detection rule #1227

duzvik opened this issue Aug 8, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@duzvik
Copy link

duzvik commented Aug 8, 2024 •
edited
Loading

What is the bug?
Alerts not working with custom detection rule.

How can one reproduce the bug?
A new detection rule(yaml) adde with API call /_plugins/_security_analytics/rules?category=windows
Enable this rule in "Active rules" section
Alerting doesn't work.
If I disable custom rule - everything works fine.

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

  • Docker deployment
  • opensearch 2.11.1
  • opensearch security analytics plugin 2.11.1.0

Do you have any additional context?
yaml file:

title: Suspicious DLL Loaded via CertOC.EXE
id: 84232095-ecca-4015-b0d7-7726507ee793
related:
    - id: 242301bc-f92f-4476-8718-78004a6efd9f
      type: similar
status: test
description: Detects when a user installs certificates by using CertOC.exe to load the target DLL file.
references:
    - https://twitter.com/sblmsrsn/status/1445758411803480072?s=20
    - https://github.com/elastic/protections-artifacts/commit/746086721fd385d9f5c6647cada1788db4aea95f#diff-fe98e74189873d6df72a15df2eaa0315c59ba9cdaca93ecd68afc4ea09194ef2
    - https://lolbas-project.github.io/lolbas/Binaries/Certoc/
author: Nasreddine Bencherchali (Nextron Systems)
date: 2023/02/15
modified: 2024/03/05
tags:
    - attack.defense_evasion
    - attack.t1218
logsource:
    category: process_creation
    product: windows
detection:
    selection_img:
        - Image|endswith: '\certoc.exe'
        - OriginalFileName: 'CertOC.exe'
    selection_cli:
        CommandLine|contains|windash: ' -loaddll '
    selection_paths:
        CommandLine|contains:
            - '\appdata\local\temp\'
            - '\desktop\'
            - '\downloads\'
            - '\users\public\'
            - 'c:\windows\tasks\'
            - 'c:\windows\temp\'
    condition: all of selection_*
falsepositives:
    - Unknown
level: high

API request to POST /_plugins/_security_analytics/rules/_search?pre_packaged=false
shows that generated query looks like:

"queries": [
            {
              "value": """(action: "processcreate") AND (((process.command_line: _ws_*\-loaddll_ws_*) OR (process.command_line: _ws_*\/loaddll_ws_*)) AND ((process.command_line: *\\appdata\\local\\temp\\*) OR (process.command_line: *\\desktop\\*) OR (process.command_line: *\\downloads\\*) OR (process.command_line: *\\users\\public\\*) OR (process.command_line: *c\:\\windows\\tasks\\*) OR (process.command_line: *c\:\\windows\\temp\\*)) AND ((winlog.event_data.Image: *\\certoc.exe) OR (OriginalFileName: "CertOC.exe")))"""
            }
          ]

seems _ws_ is used for space.

error log:

[2024-08-08T20:21:25,750][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [os03] Detected cluster change event for destination migration
[2024-08-08T20:21:25,906][INFO ][o.o.p.PluginsService     ] [os03] PluginService:onIndexModule index:[.opensearch-sap-windows-alerts/rRdWamgnSNyZD6QLiHdbCg]
[2024-08-08T20:21:26,003][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [os03] Detected cluster change event for destination migration
[2024-08-08T20:21:26,009][INFO ][o.o.a.a.AlertIndices     ] [os03] Index mapping of .opensearch-sap-windows-alerts-history-2024.08.08-1 is updated
[2024-08-08T20:21:26,013][INFO ][o.o.a.a.AlertIndices     ] [os03] Index mapping of .opensearch-sap-windows-findings-2024.08.08-1 is updated
[2024-08-08T20:21:26,594][INFO ][o.o.a.a.AlertIndices     ] [os03] Index mapping of .opensearch-sap-windows-alerts is updated
[2024-08-08T20:21:26,599][INFO ][o.o.a.a.AlertIndices     ] [os03] Index mapping of .opensearch-sap-windows-alerts-history-2024.08.08-1 is updated
[2024-08-08T20:21:26,606][INFO ][o.o.a.a.AlertIndices     ] [os03] Index mapping of .opensearch-sap-windows-alerts is updated
[2024-08-08T20:21:26,606][WARN ][o.o.t.OutboundHandler    ] [os03] send message failed [channel: Netty4TcpChannel{localAddress=/172.20.0.4:42574, remoteAddress=172.20.0.2/172.20.0.2:9300}]
java.lang.IllegalArgumentException: can not write type [class org.opensearch.commons.alerting.model.Alert$State]
	at org.opensearch.core.common.io.stream.StreamOutput.getWriter(StreamOutput.java:821) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeGenericValue(StreamOutput.java:838) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.BaseTermQueryBuilder.doWriteTo(BaseTermQueryBuilder.java:152) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.TermQueryBuilder.doWriteTo(TermQueryBuilder.java:122) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.AbstractQueryBuilder.writeTo(AbstractQueryBuilder.java:93) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1126) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.AbstractQueryBuilder.writeQueries(AbstractQueryBuilder.java:267) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.BoolQueryBuilder.doWriteTo(BoolQueryBuilder.java:109) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.AbstractQueryBuilder.writeTo(AbstractQueryBuilder.java:93) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1126) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeOptionalNamedWriteable(StreamOutput.java:1137) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.search.builder.SearchSourceBuilder.writeTo(SearchSourceBuilder.java:303) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:969) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.search.internal.ShardSearchRequest.innerWriteTo(ShardSearchRequest.java:322) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.search.internal.ShardSearchRequest.writeTo(ShardSearchRequest.java:311) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:104) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundMessage.serialize(OutboundMessage.java:81) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:235) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:221) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:275) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler.internalSend(OutboundHandler.java:197) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:192) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler.sendRequest(OutboundHandler.java:129) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:320) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1038) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:265) [opensearch-security-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:793) [opensearch-security-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.transport.TransportService.sendRequest(TransportService.java:924) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:1000) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:988) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:248) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:135) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:288) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:331) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:259) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:59) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:446) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:225) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:816) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$4(TransportSearchAction.java:457) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:138) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:546) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:310) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:128) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324) [opensearch-security-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.support.AbstractClient.search(AbstractClient.java:607) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner$fetchAlertsGeneratedInCurrentExecution$searchResponse$1.invoke(CompositeWorkflowRunner.kt:351) [opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner$fetchAlertsGeneratedInCurrentExecution$searchResponse$1.invoke(CompositeWorkflowRunner.kt:351) [opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:152) [alerting-core-2.11.1.0.jar:?]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner.fetchAlertsGeneratedInCurrentExecution(CompositeWorkflowRunner.kt:351) [opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner.runWorkflow(CompositeWorkflowRunner.kt:178) [opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner$runWorkflow$1.invokeSuspend(CompositeWorkflowRunner.kt) [opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-08T20:21:26,608][ERROR][o.o.a.w.CompositeWorkflowRunner] [os03] failed to get alerts generated by delegate monitors in current execution u4kfM5EBNMTkGZssiXlq_2024-08-08T20:21:24.342748575_383e488d-38ac-4877-90c2-9401ef33283e
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:706) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:379) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:745) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:503) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:301) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:75) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:755) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$6.handleException(TransportService.java:903) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:402) ~[?:?]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1526) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1640) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1614) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:80) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:72) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.ActionRunnable.onFailure(ActionRunnable.java:104) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.11.1.jar:2.11.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.opensearch.OpenSearchException$3: can not write type [class org.opensearch.commons.alerting.model.Alert$State]
	at org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:708) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:377) ~[opensearch-2.11.1.jar:2.11.1]
	... 24 more
Caused by: java.lang.IllegalArgumentException: can not write type [class org.opensearch.commons.alerting.model.Alert$State]
	at org.opensearch.core.common.io.stream.StreamOutput.getWriter(StreamOutput.java:821) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeGenericValue(StreamOutput.java:838) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.BaseTermQueryBuilder.doWriteTo(BaseTermQueryBuilder.java:152) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.TermQueryBuilder.doWriteTo(TermQueryBuilder.java:122) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.AbstractQueryBuilder.writeTo(AbstractQueryBuilder.java:93) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1126) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.AbstractQueryBuilder.writeQueries(AbstractQueryBuilder.java:267) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.BoolQueryBuilder.doWriteTo(BoolQueryBuilder.java:109) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.AbstractQueryBuilder.writeTo(AbstractQueryBuilder.java:93) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1126) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeOptionalNamedWriteable(StreamOutput.java:1137) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.search.builder.SearchSourceBuilder.writeTo(SearchSourceBuilder.java:303) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:969) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.search.internal.ShardSearchRequest.innerWriteTo(ShardSearchRequest.java:322) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.search.internal.ShardSearchRequest.writeTo(ShardSearchRequest.java:311) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:104) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundMessage.serialize(OutboundMessage.java:81) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:235) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:221) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:275) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler.internalSend(OutboundHandler.java:197) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:192) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.OutboundHandler.sendRequest(OutboundHandler.java:129) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:320) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1038) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:265) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:793) ~[?:?]
	at org.opensearch.transport.TransportService.sendRequest(TransportService.java:924) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:1000) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:988) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:248) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:135) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:288) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:331) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:259) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:59) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:446) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:225) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:816) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$4(TransportSearchAction.java:457) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:138) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:546) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:310) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:128) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) ~[?:?]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) ~[?:?]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324) ~[?:?]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[?:?]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.support.AbstractClient.search(AbstractClient.java:607) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner$fetchAlertsGeneratedInCurrentExecution$searchResponse$1.invoke(CompositeWorkflowRunner.kt:351) ~[opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner$fetchAlertsGeneratedInCurrentExecution$searchResponse$1.invoke(CompositeWorkflowRunner.kt:351) ~[opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.alerting.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:152) ~[alerting-core-2.11.1.0.jar:?]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner.fetchAlertsGeneratedInCurrentExecution(CompositeWorkflowRunner.kt:351) ~[opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner.runWorkflow(CompositeWorkflowRunner.kt:178) ~[opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.alerting.workflow.CompositeWorkflowRunner$runWorkflow$1.invokeSuspend(CompositeWorkflowRunner.kt) ~[opensearch-alerting-2.11.1.0.jar:2.11.1.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-08T20:21:27,099][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [os03] Detected cluster change event for destination migration
@duzvik duzvik added bug Something isn't working untriaged labels Aug 8, 2024
@duzvik duzvik changed the title [BUG] In custom detection rule [BUG] Alerts not working with custom detection rule Aug 8, 2024
@duzvik
Copy link
Author

duzvik commented Aug 9, 2024

updates cluster + plugin to latest(2.16.0)

error message in logs:

[2024-08-09T10:33:51,196][INFO ][o.o.a.TriggerService     ] [os03] Error running script for monitor h4GxNpEBewQZ82gO24aB, trigger: hoGxNpEBewQZ82gO2YYh
java.util.EmptyStackException: null
	at java.base/java.util.Stack.peek(Stack.java:103) ~[?:?]
	at java.base/java.util.Stack.pop(Stack.java:85) ~[?:?]
	at org.opensearch.alerting.triggercondition.resolvers.TriggerExpressionRPNResolver.evaluate(TriggerExpressionRPNResolver.kt:68) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.TriggerService.runDocLevelTrigger(TriggerService.kt:147) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.runForEachDocTrigger(TransportDocLevelMonitorFanOutAction.kt:360) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.executeMonitor(TransportDocLevelMonitorFanOutAction.kt:300) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.access$executeMonitor(TransportDocLevelMonitorFanOutAction.kt:132) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction$executeMonitor$1.invokeSuspend(TransportDocLevelMonitorFanOutAction.kt) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-09T10:33:51,196][INFO ][o.o.c.a.m.MonitorRunResult] [os03] Unknown Internal error. See the OpenSearch log for details.
java.util.EmptyStackException: null
	at java.base/java.util.Stack.peek(Stack.java:103) ~[?:?]
	at java.base/java.util.Stack.pop(Stack.java:85) ~[?:?]
	at org.opensearch.alerting.triggercondition.resolvers.TriggerExpressionRPNResolver.evaluate(TriggerExpressionRPNResolver.kt:68) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.TriggerService.runDocLevelTrigger(TriggerService.kt:147) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.runForEachDocTrigger(TransportDocLevelMonitorFanOutAction.kt:360) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.executeMonitor(TransportDocLevelMonitorFanOutAction.kt:300) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.access$executeMonitor(TransportDocLevelMonitorFanOutAction.kt:132) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction$executeMonitor$1.invokeSuspend(TransportDocLevelMonitorFanOutAction.kt) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-09T10:33:51,196][INFO ][o.o.a.t.TransportDocLevelMonitorFanOutAction] [os03] Trigger errors: [hoGxNpEBewQZ82gO2YYh]: [Unknown Internal error. See the OpenSearch log for details.] | 
[2024-08-09T10:33:51,327][DEBUG][o.o.a.AlertService       ] [os03] Monitor error Alert successfully upserted. Op result: UPDATED
[2024-08-09T10:33:51,328][DEBUG][o.o.a.t.TransportDocLevelMonitorFanOutAction] [os03] Monitor h4GxNpEBewQZ82gO24aB Querying only fields winlog.event_data.Image, OriginalFileName, action, process_command_line instead of entire _source of documents
[2024-08-09T10:33:51,427][DEBUG][o.o.a.t.TransportDocLevelMonitorFanOutAction] [os03] Monitor h4GxNpEBewQZ82gO24aB: Executing percolate query for docs from source indices [logs-endpoint-winevent-sysmon-_] against query index [.opensearch-sap-windows-detectors-queries-000001]
[2024-08-09T10:33:51,460][DEBUG][o.o.a.t.TransportDocLevelMonitorFanOutAction] [os03] Monitor h4GxNpEBewQZ82gO24aB PERF_DEBUG: Percolate query time taken millis = 33ms
[2024-08-09T10:33:51,461][INFO ][o.o.a.TriggerService     ] [os03] Error running script for monitor h4GxNpEBewQZ82gO24aB, trigger: hoGxNpEBewQZ82gO2YYh
java.util.EmptyStackException: null
	at java.base/java.util.Stack.peek(Stack.java:103) ~[?:?]
	at java.base/java.util.Stack.pop(Stack.java:85) ~[?:?]
	at org.opensearch.alerting.triggercondition.resolvers.TriggerExpressionRPNResolver.evaluate(TriggerExpressionRPNResolver.kt:68) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.TriggerService.runDocLevelTrigger(TriggerService.kt:147) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.runForEachDocTrigger(TransportDocLevelMonitorFanOutAction.kt:360) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.executeMonitor(TransportDocLevelMonitorFanOutAction.kt:300) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.access$executeMonitor(TransportDocLevelMonitorFanOutAction.kt:132) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction$executeMonitor$1.invokeSuspend(TransportDocLevelMonitorFanOutAction.kt) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-09T10:33:51,461][INFO ][o.o.c.a.m.MonitorRunResult] [os03] Unknown Internal error. See the OpenSearch log for details.
java.util.EmptyStackException: null
	at java.base/java.util.Stack.peek(Stack.java:103) ~[?:?]
	at java.base/java.util.Stack.pop(Stack.java:85) ~[?:?]
	at org.opensearch.alerting.triggercondition.resolvers.TriggerExpressionRPNResolver.evaluate(TriggerExpressionRPNResolver.kt:68) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.TriggerService.runDocLevelTrigger(TriggerService.kt:147) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.runForEachDocTrigger(TransportDocLevelMonitorFanOutAction.kt:360) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.executeMonitor(TransportDocLevelMonitorFanOutAction.kt:300) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.access$executeMonitor(TransportDocLevelMonitorFanOutAction.kt:132) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction$executeMonitor$1.invokeSuspend(TransportDocLevelMonitorFanOutAction.kt) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-09T10:33:51,461][INFO ][o.o.a.t.TransportDocLevelMonitorFanOutAction] [os03] Trigger errors: [hoGxNpEBewQZ82gO2YYh]: [Unknown Internal error. See the OpenSearch log for details.] | 
[2024-08-09T10:33:51,592][DEBUG][o.o.a.AlertService       ] [os03] Monitor error Alert successfully upserted. Op result: UPDATED
[2024-08-09T10:33:51,593][DEBUG][o.o.a.t.TransportDocLevelMonitorFanOutAction] [os03] Monitor h4GxNpEBewQZ82gO24aB Querying only fields winlog.event_data.Image, OriginalFileName, action, process_command_line instead of entire _source of documents
[2024-08-09T10:33:51,594][INFO ][o.o.a.TriggerService     ] [os03] Error running script for monitor h4GxNpEBewQZ82gO24aB, trigger: hoGxNpEBewQZ82gO2YYh
java.util.EmptyStackException: null
	at java.base/java.util.Stack.peek(Stack.java:103) ~[?:?]
	at java.base/java.util.Stack.pop(Stack.java:85) ~[?:?]
	at org.opensearch.alerting.triggercondition.resolvers.TriggerExpressionRPNResolver.evaluate(TriggerExpressionRPNResolver.kt:68) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.TriggerService.runDocLevelTrigger(TriggerService.kt:147) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.runForEachDocTrigger(TransportDocLevelMonitorFanOutAction.kt:360) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.executeMonitor(TransportDocLevelMonitorFanOutAction.kt:300) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.access$executeMonitor(TransportDocLevelMonitorFanOutAction.kt:132) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction$executeMonitor$1.invokeSuspend(TransportDocLevelMonitorFanOutAction.kt) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-09T10:33:51,595][INFO ][o.o.c.a.m.MonitorRunResult] [os03] Unknown Internal error. See the OpenSearch log for details.
java.util.EmptyStackException: null
	at java.base/java.util.Stack.peek(Stack.java:103) ~[?:?]
	at java.base/java.util.Stack.pop(Stack.java:85) ~[?:?]
	at org.opensearch.alerting.triggercondition.resolvers.TriggerExpressionRPNResolver.evaluate(TriggerExpressionRPNResolver.kt:68) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.TriggerService.runDocLevelTrigger(TriggerService.kt:147) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.runForEachDocTrigger(TransportDocLevelMonitorFanOutAction.kt:360) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.executeMonitor(TransportDocLevelMonitorFanOutAction.kt:300) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.access$executeMonitor(TransportDocLevelMonitorFanOutAction.kt:132) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction$executeMonitor$1.invokeSuspend(TransportDocLevelMonitorFanOutAction.kt) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-09T10:33:51,595][INFO ][o.o.a.t.TransportDocLevelMonitorFanOutAction] [os03] Trigger errors: [hoGxNpEBewQZ82gO2YYh]: [Unknown Internal error. See the OpenSearch log for details.] | 
[2024-08-09T10:33:51,747][DEBUG][o.o.a.AlertService       ] [os03] Monitor error Alert successfully upserted. Op result: UPDATED
[2024-08-09T10:33:51,747][ERROR][o.o.c.a.u.AlertingException] [os03] Alerting error: java.util.EmptyStackException
[2024-08-09T10:33:51,901][DEBUG][o.o.a.AlertService       ] [os03] Monitor error Alert successfully upserted. Op result: UPDATED
[2024-08-09T10:33:51,901][ERROR][o.o.a.DocumentLevelMonitorRunner] [os03] Failed running Document-level-monitor DetectorWin3
java.lang.ClassCastException: class java.util.EmptyStackException cannot be cast to class org.opensearch.commons.alerting.util.AlertingException (java.util.EmptyStackException is in module java.base of loader 'bootstrap'; org.opensearch.commons.alerting.util.AlertingException is in unnamed module of loader java.net.FactoryURLClassLoader @328e4ec2)
	at org.opensearch.alerting.DocumentLevelMonitorRunner.buildTriggerResults(DocumentLevelMonitorRunner.kt:481) ~[opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.DocumentLevelMonitorRunner.runMonitor(DocumentLevelMonitorRunner.kt:366) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at org.opensearch.alerting.DocumentLevelMonitorRunner$runMonitor$1.invokeSuspend(DocumentLevelMonitorRunner.kt) [opensearch-alerting-2.16.0.0.jar:2.16.0.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
[2024-08-09T10:33:51,902][DEBUG][o.o.a.DocumentLevelMonitorRunner] [os03] Monitor h4GxNpEBewQZ82gO24aB Time spent on monitor run: 968
[2024-08-09T10:33:51,902][DEBUG][o.o.a.w.CompositeWorkflowRunner] [os03] Workflow ioGxNpEBewQZ82gO5ob3 delegate monitors in execution ioGxNpEBewQZ82gO5ob3_2024-08-09T10:33:50.931116823_a8a06430-0699-48f0-8cc4-ce140cf54be0 completed
[2024-08-09T10:33:52,056][DEBUG][o.o.a.WorkflowMetadataService] [os03] Successfully upserted WorkflowMetadata:ioGxNpEBewQZ82gO5ob3-metadata 
[2024-08-09T10:33:52,063][INFO ][o.o.a.a.AlertIndices     ] [os03] Index mapping of .opensearch-sap-windows-alerts is updated
[2024-08-09T10:33:52,069][INFO ][o.o.a.a.AlertIndices     ] [os03] Index mapping of .opensearch-sap-windows-alerts-history-2024.08.08-1 is updated
[2024-08-09T10:33:52,074][INFO ][o.o.a.a.AlertIndices     ] [os03] Index mapping of .opensearch-sap-windows-alerts is updated
[2024-08-09T10:33:52,077][DEBUG][o.o.a.c.l.LockService    ] [os03] Releasing lock: org.opensearch.alerting.core.lock.LockModel@4b7eb352
[2024-08-09T10:33:52,256][DEBUG][o.o.a.MonitorRunnerService] [os03] lock ioGxNpEBewQZ82gO5ob3-lock released

@duzvik
Copy link
Author

duzvik commented Aug 9, 2024

Also alert with strange name "NoOp trigger" with status Error has beed created.
Screenshot 2024-08-09 at 1 44 30 PM

@dblock dblock removed the untriaged label Aug 26, 2024
@dblock
Copy link
Member

dblock commented Aug 26, 2024

[Catch All Triage - 1, 2, 3, 4, 5]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dblock @duzvik