Skip to content

Commit

Permalink
[8.17] [Security Solution] Unskip tests for the prebuilt rules bootst…
Browse files Browse the repository at this point in the history
…rap endpoint (#205106) (#205134)

# Backport

This will backport the following commits from `main` to `8.17`:
- [[Security Solution] Unskip tests for the prebuilt rules bootstrap
endpoint (#205106)](#205106)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Georgii
Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2024-12-24T09:58:37Z","message":"[Security
Solution] Unskip tests for the prebuilt rules bootstrap endpoint
(#205106)\n\n**Fixes:
https://github.com/elastic/kibana/issues/197108**\r\n**Fixes:
https://github.com/elastic/kibana/issues/202037**\r\n**Fixes:
https://github.com/elastic/kibana/issues/203632**\r\n\r\n##
Summary\r\n\r\nBuilds on top of
#203799 and:\r\n\r\n- simplifies
the package deletion code (1
[API\r\ncall](https://www.elastic.co/docs/api/doc/kibana/operation/operation-delete-fleet-epm-packages-pkgname-pkgversion)\r\ninstead
of 2, per package)\r\n- adds retry logic\r\n- refactors the testing
utils a bit\r\n\r\nFlaky test
runs:\r\n\r\n-\r\n[ESS](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7632)\r\n-
190
runs\r\n-\r\n[Serverless](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7633)\r\n-
100 runs\r\n\r\n## Details\r\n\r\nHere's how the test logs look
like:\r\n\r\n```\r\n info Starting tests\r\n\r\n └-: Rules Management -
Prebuilt Rules - Prebuilt Rules Management\r\n └-> \"before all\" hook:
beforeTestSuite.trigger in \"Rules Management - Prebuilt Rules -
Prebuilt Rules Management\"\r\n └-: @ess @serverless
@skipInServerlessMKI Bootstrap Prebuilt Rules\r\n └-> \"before all\"
hook: beforeTestSuite.trigger for \"should install fleet packages
required for detection engine to function\"\r\n └-> should install fleet
packages required for detection engine to function\r\n └-> \"before
each\" hook: global before each for \"should install fleet packages
required for detection engine to function\"\r\n └-> \"before each\" hook
for \"should install fleet packages required for detection engine to
function\"\r\n │ debg Deleting security_detection_engine package\r\n │
debg Deleted security_detection_engine package (was not installed) {\r\n
│ statusCode: 400,\r\n │ error: 'Bad Request',\r\n │ message:
'security_detection_engine is not installed'\r\n │ }\r\n │ debg Deleting
endpoint package\r\n │ debg Deleted endpoint package (was not installed)
{\r\n │ statusCode: 400,\r\n │ error: 'Bad Request',\r\n │ message:
'endpoint is not installed'\r\n │ }\r\n └- ✓ pass (21.6s)\r\n └-> should
skip installing fleet packages if they are already installed\r\n └->
\"before each\" hook: global before each for \"should skip installing
fleet packages if they are already installed\"\r\n └-> \"before each\"
hook for \"should skip installing fleet packages if they are already
installed\"\r\n │ debg Deleting security_detection_engine package\r\n │
debg Deleted security_detection_engine package (was installed)\r\n │
debg Deleting endpoint package\r\n │ debg Deleted endpoint package (was
installed)\r\n └- ✓ pass (22.4s)\r\n └-> \"after all\" hook:
afterTestSuite.trigger for \"should skip installing fleet packages if
they are already installed\"\r\n └-> \"after all\" hook:
afterTestSuite.trigger in \"Rules Management - Prebuilt Rules - Prebuilt
Rules Management\"\r\n\r\n2 passing (1.0m)\r\n```\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests
changed","sha":"71144eded7f56705e54c768c3741e5fcf1b62f9a","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0","v8.16.3","v8.17.1"],"number":205106,"url":"https://github.com/elastic/kibana/pull/205106","mergeCommit":{"message":"[Security
Solution] Unskip tests for the prebuilt rules bootstrap endpoint
(#205106)\n\n**Fixes:
https://github.com/elastic/kibana/issues/197108**\r\n**Fixes:
https://github.com/elastic/kibana/issues/202037**\r\n**Fixes:
https://github.com/elastic/kibana/issues/203632**\r\n\r\n##
Summary\r\n\r\nBuilds on top of
#203799 and:\r\n\r\n- simplifies
the package deletion code (1
[API\r\ncall](https://www.elastic.co/docs/api/doc/kibana/operation/operation-delete-fleet-epm-packages-pkgname-pkgversion)\r\ninstead
of 2, per package)\r\n- adds retry logic\r\n- refactors the testing
utils a bit\r\n\r\nFlaky test
runs:\r\n\r\n-\r\n[ESS](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7632)\r\n-
190
runs\r\n-\r\n[Serverless](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7633)\r\n-
100 runs\r\n\r\n## Details\r\n\r\nHere's how the test logs look
like:\r\n\r\n```\r\n info Starting tests\r\n\r\n └-: Rules Management -
Prebuilt Rules - Prebuilt Rules Management\r\n └-> \"before all\" hook:
beforeTestSuite.trigger in \"Rules Management - Prebuilt Rules -
Prebuilt Rules Management\"\r\n └-: @ess @serverless
@skipInServerlessMKI Bootstrap Prebuilt Rules\r\n └-> \"before all\"
hook: beforeTestSuite.trigger for \"should install fleet packages
required for detection engine to function\"\r\n └-> should install fleet
packages required for detection engine to function\r\n └-> \"before
each\" hook: global before each for \"should install fleet packages
required for detection engine to function\"\r\n └-> \"before each\" hook
for \"should install fleet packages required for detection engine to
function\"\r\n │ debg Deleting security_detection_engine package\r\n │
debg Deleted security_detection_engine package (was not installed) {\r\n
│ statusCode: 400,\r\n │ error: 'Bad Request',\r\n │ message:
'security_detection_engine is not installed'\r\n │ }\r\n │ debg Deleting
endpoint package\r\n │ debg Deleted endpoint package (was not installed)
{\r\n │ statusCode: 400,\r\n │ error: 'Bad Request',\r\n │ message:
'endpoint is not installed'\r\n │ }\r\n └- ✓ pass (21.6s)\r\n └-> should
skip installing fleet packages if they are already installed\r\n └->
\"before each\" hook: global before each for \"should skip installing
fleet packages if they are already installed\"\r\n └-> \"before each\"
hook for \"should skip installing fleet packages if they are already
installed\"\r\n │ debg Deleting security_detection_engine package\r\n │
debg Deleted security_detection_engine package (was installed)\r\n │
debg Deleting endpoint package\r\n │ debg Deleted endpoint package (was
installed)\r\n └- ✓ pass (22.4s)\r\n └-> \"after all\" hook:
afterTestSuite.trigger for \"should skip installing fleet packages if
they are already installed\"\r\n └-> \"after all\" hook:
afterTestSuite.trigger in \"Rules Management - Prebuilt Rules - Prebuilt
Rules Management\"\r\n\r\n2 passing (1.0m)\r\n```\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests
changed","sha":"71144eded7f56705e54c768c3741e5fcf1b62f9a"}},"sourceBranch":"main","suggestedTargetBranches":["8.x","8.16","8.17"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205106","number":205106,"mergeCommit":{"message":"[Security
Solution] Unskip tests for the prebuilt rules bootstrap endpoint
(#205106)\n\n**Fixes:
https://github.com/elastic/kibana/issues/197108**\r\n**Fixes:
https://github.com/elastic/kibana/issues/202037**\r\n**Fixes:
https://github.com/elastic/kibana/issues/203632**\r\n\r\n##
Summary\r\n\r\nBuilds on top of
#203799 and:\r\n\r\n- simplifies
the package deletion code (1
[API\r\ncall](https://www.elastic.co/docs/api/doc/kibana/operation/operation-delete-fleet-epm-packages-pkgname-pkgversion)\r\ninstead
of 2, per package)\r\n- adds retry logic\r\n- refactors the testing
utils a bit\r\n\r\nFlaky test
runs:\r\n\r\n-\r\n[ESS](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7632)\r\n-
190
runs\r\n-\r\n[Serverless](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7633)\r\n-
100 runs\r\n\r\n## Details\r\n\r\nHere's how the test logs look
like:\r\n\r\n```\r\n info Starting tests\r\n\r\n └-: Rules Management -
Prebuilt Rules - Prebuilt Rules Management\r\n └-> \"before all\" hook:
beforeTestSuite.trigger in \"Rules Management - Prebuilt Rules -
Prebuilt Rules Management\"\r\n └-: @ess @serverless
@skipInServerlessMKI Bootstrap Prebuilt Rules\r\n └-> \"before all\"
hook: beforeTestSuite.trigger for \"should install fleet packages
required for detection engine to function\"\r\n └-> should install fleet
packages required for detection engine to function\r\n └-> \"before
each\" hook: global before each for \"should install fleet packages
required for detection engine to function\"\r\n └-> \"before each\" hook
for \"should install fleet packages required for detection engine to
function\"\r\n │ debg Deleting security_detection_engine package\r\n │
debg Deleted security_detection_engine package (was not installed) {\r\n
│ statusCode: 400,\r\n │ error: 'Bad Request',\r\n │ message:
'security_detection_engine is not installed'\r\n │ }\r\n │ debg Deleting
endpoint package\r\n │ debg Deleted endpoint package (was not installed)
{\r\n │ statusCode: 400,\r\n │ error: 'Bad Request',\r\n │ message:
'endpoint is not installed'\r\n │ }\r\n └- ✓ pass (21.6s)\r\n └-> should
skip installing fleet packages if they are already installed\r\n └->
\"before each\" hook: global before each for \"should skip installing
fleet packages if they are already installed\"\r\n └-> \"before each\"
hook for \"should skip installing fleet packages if they are already
installed\"\r\n │ debg Deleting security_detection_engine package\r\n │
debg Deleted security_detection_engine package (was installed)\r\n │
debg Deleting endpoint package\r\n │ debg Deleted endpoint package (was
installed)\r\n └- ✓ pass (22.4s)\r\n └-> \"after all\" hook:
afterTestSuite.trigger for \"should skip installing fleet packages if
they are already installed\"\r\n └-> \"after all\" hook:
afterTestSuite.trigger in \"Rules Management - Prebuilt Rules - Prebuilt
Rules Management\"\r\n\r\n2 passing (1.0m)\r\n```\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests
changed","sha":"71144eded7f56705e54c768c3741e5fcf1b62f9a"}},{"branch":"8.x","label":"v8.18.0","labelRegex":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.16","label":"v8.16.3","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
  • Loading branch information
banderror authored Dec 24, 2024
1 parent f7b47d8 commit e2848c4
Show file tree
Hide file tree
Showing 7 changed files with 107 additions and 76 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ export default ({ getService }: FtrProviderContext): void => {
const es = getService('es');
const supertest = getService('supertest');
const log = getService('log');
const retry = getService('retry');
const retryService = getService('retry');

/* This test makes use of the mock packages created in the '/fleet_bundled_packages' folder,
/* in order to assert that, in production environments, the latest stable version of the package
Expand All @@ -35,7 +35,7 @@ export default ({ getService }: FtrProviderContext): void => {
beforeEach(async () => {
await deleteAllRules(supertest, log);
await deleteAllPrebuiltRuleAssets(es, log);
await deletePrebuiltRulesFleetPackage(supertest);
await deletePrebuiltRulesFleetPackage({ supertest, es, log, retryService });
});

it('should install latest stable version and ignore prerelease packages', async () => {
Expand All @@ -49,7 +49,7 @@ export default ({ getService }: FtrProviderContext): void => {
const fleetPackageInstallationResponse = await installPrebuiltRulesPackageViaFleetAPI(
es,
supertest,
retry
retryService
);

expect(fleetPackageInstallationResponse.items.length).toBe(1);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,24 @@ import {
} from '@kbn/security-solution-plugin/common/detection_engine/constants';
import expect from 'expect';
import { FtrProviderContext } from '../../../../../../ftr_provider_context';
import { deletePrebuiltRulesFleetPackage } from '../../../../utils';
import { deleteEndpointFleetPackage } from '../../../../utils/rules/prebuilt_rules/delete_endpoint_fleet_package';
import {
deleteAllPrebuiltRuleAssets,
deleteEndpointFleetPackage,
deletePrebuiltRulesFleetPackage,
} from '../../../../utils';

export default ({ getService }: FtrProviderContext): void => {
const es = getService('es');
const log = getService('log');
const supertest = getService('supertest');
const retryService = getService('retry');
const securitySolutionApi = getService('securitySolutionApi');

// Failing: See https://github.com/elastic/kibana/issues/197108
describe.skip('@ess @serverless @skipInServerlessMKI Bootstrap Prebuilt Rules', () => {
describe('@ess @serverless @skipInServerlessMKI Bootstrap Prebuilt Rules', () => {
beforeEach(async () => {
await deletePrebuiltRulesFleetPackage(supertest);
await deleteEndpointFleetPackage(supertest);
await deleteAllPrebuiltRuleAssets(es, log);
await deletePrebuiltRulesFleetPackage({ supertest, es, log, retryService });
await deleteEndpointFleetPackage({ supertest, es, log, retryService });
});

it('should install fleet packages required for detection engine to function', async () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,18 @@ import {
import { deleteAllRules } from '../../../../../../../common/utils/security_solution';
import { deleteAllPrebuiltRuleAssets } from '../../../../utils/rules/prebuilt_rules/delete_all_prebuilt_rule_assets';
import { deleteAllTimelines } from '../../../../utils/rules/prebuilt_rules/delete_all_timelines';
import { deletePrebuiltRulesFleetPackage } from '../../../../utils/rules/prebuilt_rules/delete_prebuilt_rules_fleet_package';
import { deletePrebuiltRulesFleetPackage } from '../../../../utils/rules/prebuilt_rules/delete_fleet_packages';
import { installPrebuiltRulesFleetPackage } from '../../../../utils/rules/prebuilt_rules/install_prebuilt_rules_fleet_package';

export default ({ getService }: FtrProviderContext): void => {
const es = getService('es');
const supertest = getService('supertest');
const log = getService('log');
const retry = getService('retry');
const retryService = getService('retry');

describe('@ess @serverless @skipInServerlessMKI install_prebuilt_rules_from_real_package', () => {
beforeEach(async () => {
await deletePrebuiltRulesFleetPackage(supertest);
await deletePrebuiltRulesFleetPackage({ supertest, es, log, retryService });
await deleteAllRules(supertest, log);
await deleteAllTimelines(es, log);
await deleteAllPrebuiltRuleAssets(es, log);
Expand All @@ -48,7 +48,7 @@ export default ({ getService }: FtrProviderContext): void => {
es,
supertest,
overrideExistingPackage: true,
retryService: retry,
retryService,
});

// Verify that status is updated after package installation
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import type { Client } from '@elastic/elasticsearch';
import { epmRouteService } from '@kbn/fleet-plugin/common';
import { RetryService } from '@kbn/ftr-common-functional-services';
import {
ENDPOINT_PACKAGE_NAME,
PREBUILT_RULES_PACKAGE_NAME,
} from '@kbn/security-solution-plugin/common/detection_engine/constants';
import { ToolingLog } from '@kbn/tooling-log';
import type SuperTest from 'supertest';
import { refreshSavedObjectIndices } from '../../refresh_index';

interface DeleteFleetPackageDeps {
supertest: SuperTest.Agent;
retryService: RetryService;
log: ToolingLog;
es: Client;
}

interface DeleteFleetPackageArgs {
packageName: string;
dependencies: DeleteFleetPackageDeps;
}

/**
* Delete the security_detection_engine package using fleet API.
*/
export async function deletePrebuiltRulesFleetPackage(params: DeleteFleetPackageDeps) {
await deleteFleetPackage({
packageName: PREBUILT_RULES_PACKAGE_NAME,
dependencies: params,
});
}

/**
* Delete the endpoint package using fleet API.
*/
export async function deleteEndpointFleetPackage(params: DeleteFleetPackageDeps) {
await deleteFleetPackage({
packageName: ENDPOINT_PACKAGE_NAME,
dependencies: params,
});
}

async function deleteFleetPackage(params: DeleteFleetPackageArgs): Promise<void> {
const { packageName, dependencies } = params;
const { supertest, retryService, log, es } = dependencies;

await retryService.tryWithRetries(
'deleteFleetPackage',
async () => {
log.debug(`Deleting ${packageName} package`);

const response = await supertest
.delete(epmRouteService.getRemovePath(packageName, ''))
.set('kbn-xsrf', 'true')
.set('elastic-api-version', '2023-10-31')
.send({ force: true });

if (response.status === 200) {
log.debug(`Deleted ${packageName} package (was installed)`);
return;
} else if (
response.status === 400 &&
response.body.message === `${packageName} is not installed`
) {
log.debug(`Deleted ${packageName} package (was not installed)`, response.body);
return;
} else {
log.warning(`Error deleting ${packageName} package`, response.body);
throw response.error;
}
},
{
retryCount: 2, // overall max 3 attempts
timeout: 3 * 60000, // total timeout applied to all attempts altogether
}
);

await refreshSavedObjectIndices(es);
}

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
export * from './create_prebuilt_rule_saved_objects';
export * from './delete_all_prebuilt_rule_assets';
export * from './delete_all_timelines';
export * from './delete_prebuilt_rules_fleet_package';
export * from './delete_fleet_packages';
export * from './get_installed_rules';
export * from './get_prebuilt_rules_and_timelines_status';
export * from './get_prebuilt_rules_status';
Expand Down

0 comments on commit e2848c4

Please sign in to comment.