[Security Solution] [Attack discovery] Alerts filtering (elastic#205070)

## [Security Solution] [Attack discovery] Alerts filtering

![00_alerts_filtering](https://github.com/user-attachments/assets/1a81413b-b8f4-4965-a006-25fb529668a6)

This PR enhances _Attack discovery_ by providing users additional control over which alerts are included as context to the large language model (LLM).

Using the new resizeable _Attack discovery settings flyout_, users may:

- Filter alerts via a search bar and filters
- Control the time window (previously fixed to `Last 24 hrs`)

### Before (feature flag disabled)

Previously, users could only set the number of alerts sent as context to the LLM via a modal:

![01_before_full_page](https://github.com/user-attachments/assets/65eaf604-3bdf-41bd-a726-f03ba5d630d5)

### After (feature flag enabled)

The new Attack discovery settings flyout replaces the modal:

![02_alert_summary_full_page](https://github.com/user-attachments/assets/613c292b-c6ec-4dc6-aea3-b2eddbacd614)

It has two tabs, _Alert summary_ and _Alerts preview_.

### Alert summary

The _Alert summary_ Lens embeddable counts the selected field name via an ES|QL query:

![03_alert_summary_cropped](https://github.com/user-attachments/assets/6f5de0e4-3da6-4937-a3cd-9a0f80df16b6)

The Alert summary query is an aggregation. It does NOT display the details of individual alerts.

### Alerts preview

The _Alerts preview_ Lens embeddable shows a preview of the actual alerts that will be sent as context via an ES|QL query:

![05_alerts_preview_cropped](https://github.com/user-attachments/assets/6db23931-3fe6-46d2-8b9a-6cc7a9d8720c)

Users may resize the settings flyout to view all the fields in the Alerts preview.

### Feature flag

Enable the `attackDiscoveryAlertFiltering` feature flag via the following setting in `kibana.dev.yml`:

```yaml
xpack.securitySolution.enableExperimental:
  - 'attackDiscoveryAlertFiltering'
```

Enabling the feature flag:

- Replaces the `Settings` modal with the `Attack discovery settings` flyout
- Includes additional `start`, `end`, and `filters` parameters in requests to generate Attack discoveries
- Enables new loading messages

### Details

#### Loading messages

The loading messages displayed when generating Attack discoveries were updated to render three types of date ranges:

1) The default date range (`Last 24 hours`), which displays the same message seen in previous versions:

![06_loading_default_date_range](https://github.com/user-attachments/assets/b376a87c-b4b8-42d8-bcbf-ddf79cc82800)

2) Relative date ranges:

![07_loading_relative_date_range](https://github.com/user-attachments/assets/d0b6bddd-7722-4181-a99c-7450d07a6624)

3) Absolute date ranges:

![08_loading_absolute_date_range](https://github.com/user-attachments/assets/a542a921-eeaa-4ced-9568-25e63a47d42d)

#### Filtering preferences

Alert filtering preferences are stored in local storage.

This PR adds the following new local storage keys:

```
elasticAssistantDefault.attackDiscovery.default.end
elasticAssistantDefault.attackDiscovery.default.filters
elasticAssistantDefault.attackDiscovery.default.query
elasticAssistantDefault.attackDiscovery.default.start
```

Users may use the `Reset` button in the Attack discovery settings flyout to restore the above to their defaults.

#### Known limitations

The following known limitations in this PR may be mitigated in follow-up PRs:

#### Table cell hover actions are disabled

Table cell actions, i.e. `Filter for` and `Filter out` are disabled in the `Alert summary` and `Alerts preview` tables.

The actions are disabled because custom cell hover actions registered in `x-pack/solutions/security/plugins/security_solution/public/app/actions/register.ts` do NOT appear to receive field metadata (i.e. the name of the field being hovered over) when the action is triggered. This limitation also appears to apply to ad hoc ES|QL visualizations created via Lens in Kibana's _Dashboard_ app.

##### Default table sort indicators are hidden

The `Alert summary` and `Alerts preview` tables are sorted descending by Count, and Risk score, respectively, via their ES|QL queries.

The tables _should_ display default sort indicators, as illustrated by the screenshots below:

![09_alert_summary_with_sort_indicator](https://github.com/user-attachments/assets/c4e78144-f516-40f8-b6da-7c8c808841c4)

![10_alerts_preview_with_sort_indicator](https://github.com/user-attachments/assets/c0061134-4734-462f-8eb0-978b2b02fb1e)

The default indicators are hidden in this PR as a workaround for an  error that occurs in `EuiDataGrid` when switching tabs when the column sort indicators are enabled:

```
TypeError: Cannot read properties of undefined (reading 'split')
```

To re-enable the sort indicators, `DEFAULT_ALERT_SUMMARY_SORT` and `DEFAULT_ALERTS_PREVIEW_SORT` must respectively be passed as the `sorting` prop to the `PreviewTab` in `x-pack/solutions/security/plugins/security_solution/public/attack_discovery/pages/settings_flyout/alert_selection/helpers/get_tabs/index.tsx`, as illustrated by the following code:

```typescript
  <PreviewTab
    dataTestSubj={ALERT_SUMMARY_TEST_SUBJ}
    embeddableId={SUMMARY_TAB_EMBEDDABLE_ID}
    end={end}
    filters={filters}
    getLensAttributes={getAlertSummaryLensAttributes}
    getPreviewEsqlQuery={getAlertSummaryEsqlQuery}
    maxAlerts={maxAlerts}
    query={query}
    setTableStackBy0={setAlertSummaryStackBy0}
    start={start}
    sorting={DEFAULT_ALERT_SUMMARY_SORT} // <-- enables the sort indicator
    tableStackBy0={alertSummaryStackBy0}
  />
```

##### Selected date range not persisted

The `start` and `end` date range selected when a user starts generation are not (yet) persisted in Elasticsearch. As a result, the loading message always displays the currently configured range, rather than the range selected at the start of generation.

(cherry picked from commit 681d40e)

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/alerts/get_open_and_acknowledged_alerts_query/index.test.ts

@@ -13,6 +13,7 @@ describe('getOpenAndAcknowledgedAlertsQuery', () => {
     const anonymizationFields = [
       { id: 'field1', field: 'field1', allowed: true, anonymized: false },
       { id: 'field2', field: 'field2', allowed: true, anonymized: false },
+      { id: 'field3', field: 'field3', allowed: false, anonymized: false },
     ];
     const size = 10;
 

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/alerts/get_open_and_acknowledged_alerts_query/index.ts

@@ -7,6 +7,34 @@
 
 import type { AnonymizationFieldResponse } from '../../schemas/anonymization_fields/bulk_crud_anonymization_fields_route.gen';
 
+export const DEFAULT_END = 'now';
+export const DEFAULT_START = 'now-24h';
+
+interface GetOpenAndAcknowledgedAlertsQuery {
+  allow_no_indices: boolean;
+  body: {
+    fields: Array<{
+      field: string;
+      include_unmapped: boolean;
+    }>;
+    query: {
+      bool: {
+        filter: Array<Record<string, unknown>>;
+      };
+    };
+    runtime_mappings: Record<string, unknown>;
+    size: number;
+    sort: Array<{
+      [key: string]: {
+        order: string;
+      };
+    }>;
+    _source: boolean;
+  };
+  ignore_unavailable: boolean;
+  index: string[];
+}
+
 /**
  * This query returns open and acknowledged (non-building block) alerts in the last 24 hours.
  *
@@ -15,12 +43,18 @@ import type { AnonymizationFieldResponse } from '../../schemas/anonymization_fie
 export const getOpenAndAcknowledgedAlertsQuery = ({
   alertsIndexPattern,
   anonymizationFields,
+  end,
+  filter,
   size,
+  start,
 }: {
   alertsIndexPattern: string;
   anonymizationFields: AnonymizationFieldResponse[];
+  end?: string | null;
+  filter?: Record<string, unknown> | null;
   size: number;
-}) => ({
+  start?: string | null;
+}): GetOpenAndAcknowledgedAlertsQuery => ({
   allow_no_indices: true,
   body: {
     fields: anonymizationFields
@@ -53,11 +87,12 @@ export const getOpenAndAcknowledgedAlertsQuery = ({
                     minimum_should_match: 1,
                   },
                 },
+                ...(filter != null ? [filter] : []),
                 {
                   range: {
                     '@timestamp': {
-                      gte: 'now-24h',
-                      lte: 'now',
+                      gte: start != null ? start : DEFAULT_START,
+                      lte: end != null ? end : DEFAULT_END,
                       format: 'strict_date_optional_time',
                     },
                   },

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/capabilities/index.ts

@@ -20,5 +20,6 @@ export type AssistantFeatureKey = keyof AssistantFeatures;
  */
 export const defaultAssistantFeatures = Object.freeze({
   assistantModelEvaluation: false,
+  attackDiscoveryAlertFiltering: false,
   defendInsights: false,
 });

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/attack_discovery/post_attack_discovery_route.gen.ts

@@ -28,11 +28,14 @@ export const AttackDiscoveryPostRequestBody = z.object({
    * LLM API configuration.
    */
   apiConfig: ApiConfig,
+  end: z.string().optional(),
+  filter: z.object({}).catchall(z.unknown()).optional(),
   langSmithProject: z.string().optional(),
   langSmithApiKey: z.string().optional(),
   model: z.string().optional(),
   replacements: Replacements.optional(),
   size: z.number(),
+  start: z.string().optional(),
   subAction: z.enum(['invokeAI', 'invokeStream']),
 });
 export type AttackDiscoveryPostRequestBodyInput = z.input<typeof AttackDiscoveryPostRequestBody>;

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/attack_discovery/post_attack_discovery_route.schema.yaml

@@ -38,6 +38,11 @@ paths:
                 apiConfig:
                   $ref: '../conversations/common_attributes.schema.yaml#/components/schemas/ApiConfig'
                   description: LLM API configuration.
+                end:
+                  type: string
+                filter:
+                  type: object
+                  additionalProperties: true
                 langSmithProject:
                   type: string
                 langSmithApiKey:
@@ -48,6 +53,8 @@ paths:
                   $ref: '../conversations/common_attributes.schema.yaml#/components/schemas/Replacements'
                 size:
                   type: number
+                start:
+                  type: string
                 subAction:
                   type: string
                   enum:

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/capabilities/get_capabilities_route.gen.ts

@@ -19,5 +19,6 @@ import { z } from '@kbn/zod';
 export type GetCapabilitiesResponse = z.infer<typeof GetCapabilitiesResponse>;
 export const GetCapabilitiesResponse = z.object({
   assistantModelEvaluation: z.boolean(),
+  attackDiscoveryAlertFiltering: z.boolean(),
   defendInsights: z.boolean(),
 });

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/capabilities/get_capabilities_route.schema.yaml

@@ -22,10 +22,13 @@ paths:
                 properties:
                   assistantModelEvaluation:
                     type: boolean
+                  attackDiscoveryAlertFiltering:
+                    type: boolean
                   defendInsights:
                     type: boolean
                 required:
                   - assistantModelEvaluation
+                  - attackDiscoveryAlertFiltering
                   - defendInsights
         '400':
           description: Generic Error

x-pack/platform/packages/shared/kbn-elastic-assistant-common/index.ts

@@ -41,3 +41,10 @@ export { getRawDataOrDefault } from './impl/alerts/helpers/get_raw_data_or_defau
 
 /** Return true if the provided size is out of range */
 export { sizeIsOutOfRange } from './impl/alerts/helpers/size_is_out_of_range';
+
+export {
+  /** The default (relative) end of the date range (i.e. `now`) */
+  DEFAULT_END,
+  /** The default (relative) start of the date range (i.e. `now-24h`) */
+  DEFAULT_START,
+} from './impl/alerts/get_open_and_acknowledged_alerts_query';

x-pack/platform/packages/shared/kbn-elastic-assistant/impl/assistant_context/constants.tsx

@@ -10,10 +10,14 @@ import { KnowledgeBaseConfig } from '../assistant/types';
 export const ATTACK_DISCOVERY_STORAGE_KEY = 'attackDiscovery';
 export const DEFEND_INSIGHTS_STORAGE_KEY = 'defendInsights';
 export const DEFAULT_ASSISTANT_NAMESPACE = 'elasticAssistantDefault';
+export const END_LOCAL_STORAGE_KEY = 'end';
 export const LAST_CONVERSATION_ID_LOCAL_STORAGE_KEY = 'lastConversationId';
+export const FILTERS_LOCAL_STORAGE_KEY = 'filters';
 export const MAX_ALERTS_LOCAL_STORAGE_KEY = 'maxAlerts';
 export const KNOWLEDGE_BASE_LOCAL_STORAGE_KEY = 'knowledgeBase';
+export const QUERY_LOCAL_STORAGE_KEY = 'query';
 export const SHOW_SETTINGS_TOUR_LOCAL_STORAGE_KEY = 'showSettingsTour';
+export const START_LOCAL_STORAGE_KEY = 'start';
 export const STREAMING_LOCAL_STORAGE_KEY = 'streaming';
 export const TRACE_OPTIONS_SESSION_STORAGE_KEY = 'traceOptions';
 export const CONVERSATION_TABLE_SESSION_STORAGE_KEY = 'conversationTable';

x-pack/platform/packages/shared/kbn-elastic-assistant/index.ts

@@ -84,11 +84,19 @@ export {
   DEFAULT_ATTACK_DISCOVERY_MAX_ALERTS,
   DEFAULT_LATEST_ALERTS,
   DEFEND_INSIGHTS_STORAGE_KEY,
+  /** The end of the date range of alerts, sent as context to the LLM */
+  END_LOCAL_STORAGE_KEY,
+  /** Search bar filters that apply to the alerts sent as context to the LLM */
+  FILTERS_LOCAL_STORAGE_KEY,
   KNOWLEDGE_BASE_LOCAL_STORAGE_KEY,
   /** The local storage key that specifies the maximum number of alerts to send as context */
   MAX_ALERTS_LOCAL_STORAGE_KEY,
+  /** Search bar query that apply to the alerts sent as context to the LLM */
+  QUERY_LOCAL_STORAGE_KEY,
   /** The local storage key that specifies whether the settings tour should be shown */
   SHOW_SETTINGS_TOUR_LOCAL_STORAGE_KEY,
+  /** The start of the date range of alerts, sent as context to the LLM */
+  START_LOCAL_STORAGE_KEY,
 } from './impl/assistant_context/constants';
 
 export { useLoadConnectors } from './impl/connectorland/use_load_connectors';

x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/index.ts

@@ -26,12 +26,15 @@ import type { GraphState } from './types';
 export interface GetDefaultAttackDiscoveryGraphParams {
   alertsIndexPattern?: string;
   anonymizationFields: AnonymizationFieldResponse[];
+  end?: string;
   esClient: ElasticsearchClient;
+  filter?: Record<string, unknown>;
   llm: ActionsClientLlm;
   logger?: Logger;
   onNewReplacements?: (replacements: Replacements) => void;
   replacements?: Replacements;
   size: number;
+  start?: string;
 }
 
 export type DefaultAttackDiscoveryGraph = ReturnType<typeof getDefaultAttackDiscoveryGraph>;
@@ -46,19 +49,22 @@ export type DefaultAttackDiscoveryGraph = ReturnType<typeof getDefaultAttackDisc
 export const getDefaultAttackDiscoveryGraph = ({
   alertsIndexPattern,
   anonymizationFields,
+  end,
   esClient,
+  filter,
   llm,
   logger,
   onNewReplacements,
   replacements,
   size,
+  start,
 }: GetDefaultAttackDiscoveryGraphParams): CompiledStateGraph<
   GraphState,
   Partial<GraphState>,
   'generate' | 'refine' | 'retrieve_anonymized_alerts' | '__start__'
 > => {
   try {
-    const graphState = getDefaultGraphState();
+    const graphState = getDefaultGraphState({ end, filter, start });
 
     // get nodes:
     const retrieveAnonymizedAlertsNode = getRetrieveAnonymizedAlertsNode({

x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/nodes/retriever/anonymized_alerts_retriever/index.ts

@@ -21,36 +21,48 @@ export class AnonymizedAlertsRetriever extends BaseRetriever {
 
   #alertsIndexPattern?: string;
   #anonymizationFields?: AnonymizationFieldResponse[];
+  #end?: string | null;
   #esClient: ElasticsearchClient;
+  #filter?: Record<string, unknown> | null;
   #onNewReplacements?: (newReplacements: Replacements) => void;
   #replacements?: Replacements;
   #size?: number;
+  #start?: string | null;
 
   constructor({
     alertsIndexPattern,
     anonymizationFields,
     fields,
+    end,
     esClient,
+    filter,
     onNewReplacements,
     replacements,
     size,
+    start,
   }: {
     alertsIndexPattern?: string;
     anonymizationFields?: AnonymizationFieldResponse[];
-    fields?: CustomRetrieverInput;
+    end?: string | null;
     esClient: ElasticsearchClient;
+    fields?: CustomRetrieverInput;
+    filter?: Record<string, unknown> | null;
     onNewReplacements?: (newReplacements: Replacements) => void;
     replacements?: Replacements;
     size?: number;
+    start?: string | null;
   }) {
     super(fields);
 
     this.#alertsIndexPattern = alertsIndexPattern;
     this.#anonymizationFields = anonymizationFields;
+    this.#end = end;
     this.#esClient = esClient;
+    this.#filter = filter;
     this.#onNewReplacements = onNewReplacements;
     this.#replacements = replacements;
     this.#size = size;
+    this.#start = start;
   }
 
   async _getRelevantDocuments(
@@ -60,10 +72,13 @@ export class AnonymizedAlertsRetriever extends BaseRetriever {
     const anonymizedAlerts = await getAnonymizedAlerts({
       alertsIndexPattern: this.#alertsIndexPattern,
       anonymizationFields: this.#anonymizationFields,
+      end: this.#end,
       esClient: this.#esClient,
+      filter: this.#filter,
       onNewReplacements: this.#onNewReplacements,
       replacements: this.#replacements,
       size: this.#size,
+      start: this.#start,
     });
 
     return anonymizedAlerts.map((alert) => ({

x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/nodes/retriever/helpers/get_anonymized_alerts/index.ts

@@ -21,17 +21,23 @@ import { AnonymizationFieldResponse } from '@kbn/elastic-assistant-common/impl/s
 export const getAnonymizedAlerts = async ({
   alertsIndexPattern,
   anonymizationFields,
+  end,
   esClient,
+  filter,
   onNewReplacements,
   replacements,
   size,
+  start,
 }: {
   alertsIndexPattern?: string;
   anonymizationFields?: AnonymizationFieldResponse[];
+  end?: string | null;
   esClient: ElasticsearchClient;
+  filter?: Record<string, unknown> | null;
   onNewReplacements?: (replacements: Replacements) => void;
   replacements?: Replacements;
   size?: number;
+  start?: string | null;
 }): Promise<string[]> => {
   if (alertsIndexPattern == null || size == null || sizeIsOutOfRange(size)) {
     return [];
@@ -40,7 +46,10 @@ export const getAnonymizedAlerts = async ({
   const query = getOpenAndAcknowledgedAlertsQuery({
     alertsIndexPattern,
     anonymizationFields: anonymizationFields ?? [],
+    end,
+    filter,
     size,
+    start,
   });
 
   const result = await esClient.search<SearchResponse>(query);

x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/graphs/default_attack_discovery_graph/nodes/retriever/index.ts

@@ -36,17 +36,23 @@ export const getRetrieveAnonymizedAlertsNode = ({
     onNewReplacements?.(localReplacements); // invoke the callback with the latest replacements
   };
 
-  const retriever = new AnonymizedAlertsRetriever({
-    alertsIndexPattern,
-    anonymizationFields,
-    esClient,
-    onNewReplacements: localOnNewReplacements,
-    replacements,
-    size,
-  });
-
   const retrieveAnonymizedAlerts = async (state: GraphState): Promise<GraphState> => {
     logger?.debug(() => '---RETRIEVE ANONYMIZED ALERTS---');
+
+    const { end, filter, start } = state;
+
+    const retriever = new AnonymizedAlertsRetriever({
+      alertsIndexPattern,
+      anonymizationFields,
+      end,
+      esClient,
+      filter,
+      onNewReplacements: localOnNewReplacements,
+      replacements,
+      size,
+      start,
+    });
+
     const documents = await retriever
       .withConfig({ runName: 'runAnonymizedAlertsRetriever' })
       .invoke('');