Benchmarking tools and techniques investigation #5502
Comments
juliamagan
changed the title
Tools for Different PurposesTools SummaryThe following is a summary table of the tools analyzed with the main characteristics of each one
The information on each tool explains the advantages and disadvantages of each one.
Tools InformationArtillery (Performance Testing)InformationArtillery is an open source load testing tool. It allows you to simulate multiple concurrent users, making it an option for performance testing. Artillery supports HTTP, WebSocket and Server-Sent Events protocols. It is free and open source. You can use it to evaluate the performance and scalability of web services, APIs and other networked systems. Artillery test scripts are usually written as YAML, but they can also be written in JavaScript. Artillery scripts have two parts:
Advantages and disadvantages
Installationroot@ubuntu2204:/home/vagrant# apt update
root@ubuntu2204:/home/vagrant# curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
root@ubuntu2204:/home/vagrant# nvm install 20
root@ubuntu2204:/home/vagrant# node -v
root@ubuntu2204:/home/vagrant# npm -v
root@ubuntu2204:/home/vagrant# npm install -g artillery@latestIt is also possible to install the VS Code extension for Artillery, making it easier to create tests. Artillery Test
root@ubuntu2204:/home/vagrant# touch asciiart-load-test.yml
config:
target: http://asciiart.artillery.io:8080
phases:
- duration: 60
arrivalRate: 1
rampTo: 5
name: Warm up phase
- duration: 60
arrivalRate: 5
rampTo: 10
name: Ramp up load
- duration: 30
arrivalRate: 10
rampTo: 30
name: Spike phase
plugins:
ensure: {}
apdex: {}
metrics-by-endpoint: {}
apdex:
threshold: 100
ensure:
thresholds:
- http.response_time.p99: 100
- http.response_time.p95: 75
scenarios:
- flow:
- loop:
- get:
url: '/dino'
- get:
url: '/pony'
- get:
url: '/armadillo'
count: 100
root@ubuntu2204:/home/vagrant# artillery run asciiart-load-test.yml
Test run id: tkrrk_eap897yrfekgtcdemgax3jbqjxzwn_4qth
Phase started: Warm up phase (index: 0, duration: 60s) 07:35:49(+0000)
--------------------------------------
Metrics for period to: 07:36:00(+0000) (width: 9.895s)
--------------------------------------
apdex.frustrated: .............................................................. 1
apdex.satisfied: ............................................................... 1017
apdex.tolerated: ............................................................... 16
http.codes.200: ................................................................ 1034
http.downloaded_bytes: ......................................................... 598768
http.request_rate: ............................................................. 106/sec
http.requests: ................................................................. 1047
http.response_time:
min: ......................................................................... 46
max: ......................................................................... 133
mean: ........................................................................ 53.7
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 80.6
http.responses: ................................................................ 1034
plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 339
plugins.metrics-by-endpoint./dino.codes.200: ................................... 350
plugins.metrics-by-endpoint./pony.codes.200: ................................... 345
plugins.metrics-by-endpoint.response_time./armadillo:
min: ......................................................................... 46
max: ......................................................................... 133
mean: ........................................................................ 53.9
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 85.6
plugins.metrics-by-endpoint.response_time./dino:
min: ......................................................................... 48
max: ......................................................................... 132
mean: ........................................................................ 53.6
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 76
plugins.metrics-by-endpoint.response_time./pony:
min: ......................................................................... 47
max: ......................................................................... 130
mean: ........................................................................ 53.5
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 76
vusers.created: ................................................................ 13
vusers.created_by_name.0: ...................................................... 13
--------------------------------------
Metrics for period to: 07:36:10(+0000) (width: 9.961s)
--------------------------------------
apdex.frustrated: .............................................................. 0
apdex.satisfied: ............................................................... 4153
apdex.tolerated: ............................................................... 20
http.codes.200: ................................................................ 4173
http.downloaded_bytes: ......................................................... 2413754
http.request_rate: ............................................................. 421/sec
http.requests: ................................................................. 4189
http.response_time:
min: ......................................................................... 47
max: ......................................................................... 99
mean: ........................................................................ 53.1
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 77.5
http.responses: ................................................................ 4173
plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 1388
plugins.metrics-by-endpoint./dino.codes.200: ................................... 1394
plugins.metrics-by-endpoint./pony.codes.200: ................................... 1391
plugins.metrics-by-endpoint.response_time./armadillo:
min: ......................................................................... 47
max: ......................................................................... 99
mean: ........................................................................ 52.9
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 77.5
plugins.metrics-by-endpoint.response_time./dino:
min: ......................................................................... 47
max: ......................................................................... 93
mean: ........................................................................ 53.2
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 77.5
plugins.metrics-by-endpoint.response_time./pony:
min: ......................................................................... 47
max: ......................................................................... 97
mean: ........................................................................ 53.2
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 77.5
vusers.completed: .............................................................. 4
vusers.created: ................................................................ 20
vusers.created_by_name.0: ...................................................... 20
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 16380.2
max: ......................................................................... 17135.7
mean: ........................................................................ 16639
median: ...................................................................... 16486.1
p95: ......................................................................... 16486.1
p99: ......................................................................... 16486.1
--------------------------------------
Metrics for period to: 07:36:20(+0000) (width: 9.994s)
--------------------------------------
apdex.frustrated: .............................................................. 0
apdex.satisfied: ............................................................... 6390
apdex.tolerated: ............................................................... 28
http.codes.200: ................................................................ 6418
http.downloaded_bytes: ......................................................... 3711633
http.request_rate: ............................................................. 643/sec
http.requests: ................................................................. 6427
http.response_time:
min: ......................................................................... 47
max: ......................................................................... 97
mean: ........................................................................ 53.5
median: ...................................................................... 51.9
p95: ......................................................................... 62.2
p99: ......................................................................... 85.6
http.responses: ................................................................ 6418
plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 2136
plugins.metrics-by-endpoint./dino.codes.200: ................................... 2145
plugins.metrics-by-endpoint./pony.codes.200: ................................... 2137
plugins.metrics-by-endpoint.response_time./armadillo:
min: ......................................................................... 47
max: ......................................................................... 97
mean: ........................................................................ 53.6
median: ...................................................................... 51.9
p95: ......................................................................... 62.2
p99: ......................................................................... 85.6
plugins.metrics-by-endpoint.response_time./dino:
min: ......................................................................... 47
max: ......................................................................... 96
mean: ........................................................................ 53.5
median: ...................................................................... 51.9
p95: ......................................................................... 62.2
p99: ......................................................................... 87.4
plugins.metrics-by-endpoint.response_time./pony:
min: ......................................................................... 47
max: ......................................................................... 96
mean: ........................................................................ 53.4
median: ...................................................................... 51.9
p95: ......................................................................... 63.4
p99: ......................................................................... 85.6
vusers.completed: .............................................................. 18
vusers.created: ................................................................ 28
vusers.created_by_name.0: ...................................................... 28
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 15695.4
max: ......................................................................... 16830.3
mean: ........................................................................ 16144.8
median: ...................................................................... 16159.7
p95: ......................................................................... 16819.2
p99: ......................................................................... 16819.2
--------------------------------------
Metrics for period to: 07:36:30(+0000) (width: 9.999s)
--------------------------------------
apdex.frustrated: .............................................................. 0
apdex.satisfied: ............................................................... 8548
apdex.tolerated: ............................................................... 30
http.codes.200: ................................................................ 8578
http.downloaded_bytes: ......................................................... 4962440
http.request_rate: ............................................................. 859/sec
http.requests: ................................................................. 8592
http.response_time:
min: ......................................................................... 46
max: ......................................................................... 95
mean: ........................................................................ 52.9
median: ...................................................................... 51.9
p95: ......................................................................... 58.6
p99: ......................................................................... 74.4
http.responses: ................................................................ 8578
plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 2852
plugins.metrics-by-endpoint./dino.codes.200: ................................... 2864
plugins.metrics-by-endpoint./pony.codes.200: ................................... 2862
plugins.metrics-by-endpoint.response_time./armadillo:
min: ......................................................................... 47
max: ......................................................................... 92
mean: ........................................................................ 52.9
median: ...................................................................... 51.9
p95: ......................................................................... 58.6
p99: ......................................................................... 73
plugins.metrics-by-endpoint.response_time./dino:
min: ......................................................................... 47
max: ......................................................................... 95
mean: ........................................................................ 53
median: ...................................................................... 51.9
p95: ......................................................................... 59.7
p99: ......................................................................... 74.4
plugins.metrics-by-endpoint.response_time./pony:
min: ......................................................................... 46
max: ......................................................................... 95
mean: ........................................................................ 52.9
median: ...................................................................... 51.9
p95: ......................................................................... 59.7
p99: ......................................................................... 73
vusers.completed: .............................................................. 20
vusers.created: ................................................................ 34
vusers.created_by_name.0: ...................................................... 34
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 15811
max: ......................................................................... 16706.3
mean: ........................................................................ 16274.5
median: ...................................................................... 16159.7
p95: ......................................................................... 16819.2
p99: ......................................................................... 16819.2
--------------------------------------
Metrics for period to: 07:36:40(+0000) (width: 9.997s)
--------------------------------------
apdex.frustrated: .............................................................. 0
apdex.satisfied: ............................................................... 10436
apdex.tolerated: ............................................................... 38
http.codes.200: ................................................................ 10474
http.downloaded_bytes: ......................................................... 6057553
http.request_rate: ............................................................. 1049/sec
http.requests: ................................................................. 10485
http.response_time:
min: ......................................................................... 46
max: ......................................................................... 96
mean: ........................................................................ 54.4
median: ...................................................................... 53
p95: ......................................................................... 67.4
p99: ......................................................................... 82.3
http.responses: ................................................................ 10474
plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 3490
plugins.metrics-by-endpoint./dino.codes.200: ................................... 3493
plugins.metrics-by-endpoint./pony.codes.200: ................................... 3491
plugins.metrics-by-endpoint.response_time./armadillo:
min: ......................................................................... 46
max: ......................................................................... 96
mean: ........................................................................ 54.5
median: ...................................................................... 53
p95: ......................................................................... 67.4
p99: ......................................................................... 82.3
plugins.metrics-by-endpoint.response_time./dino:
min: ......................................................................... 46
max: ......................................................................... 96
mean: ........................................................................ 54.4
median: ...................................................................... 51.9
p95: ......................................................................... 67.4
p99: ......................................................................... 82.3
plugins.metrics-by-endpoint.response_time./pony:
min: ......................................................................... 47
max: ......................................................................... 95
mean: ........................................................................ 54.4
median: ...................................................................... 53
p95: ......................................................................... 67.4
p99: ......................................................................... 82.3
vusers.completed: .............................................................. 30
vusers.created: ................................................................ 40
vusers.created_by_name.0: ...................................................... 40
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 15758.7
max: ......................................................................... 16982.9
mean: ........................................................................ 16260.6
median: ...................................................................... 16159.7
p95: ......................................................................... 16819.2
p99: ......................................................................... 16819.2
Phase completed: Warm up phase (index: 0, duration: 60s) 07:36:49(+0000)Playwright (Test Automation Framework)InformationPlaywright is a test automation framework. Playwright allows you to perform end-to-end testing in web browsers and web scraping. It allows you to write code that interacts with web pages, performs actions like clicking, scrolling, and filling out forms, and captures screenshots or videos of the browser's state. Artillery and Playwright can be used together to create more realistic test scenarios. It is possible to create a script in Playwright and integrate it into Artillery. Since Artillery allows using JavaScript code, it is possible to create a JS function that calls the script in Playwright to simulate user interactions.
Advantages and disadvantages
Installationroot@ubuntu2204:/home/vagrant# apt update
root@ubuntu2204:/home/vagrant# curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
root@ubuntu2204:/home/vagrant# nvm install 20
root@ubuntu2204:/home/vagrant# node -v
root@ubuntu2204:/home/vagrant# npm -v
root@ubuntu2204:/home/vagrant# npm init playwright@latest
root@ubuntu2204:/home/vagrant# npx playwright install-depsIt is also possible to install Playwright in VS Code, making it easier to create tests. Playwright Test
root@ubuntu2204:/home/vagrant# npx playwright test
Running 6 tests using 1 worker
6 passed (8.3s)
To open last HTML report run:
npx playwright show-reportIntegrate Playwright with Artillery
root@ubuntu2204:/home/vagrant# touch hello-world.yml
config:
target: https://www.artillery.io
engines:
playwright: {}
processor: './flows.js'
scenarios:
- engine: playwright
testFunction: 'helloFlow'
root@ubuntu2204:/home/vagrant# touch flows.js
module.exports = { helloFlow };
async function helloFlow(page) {
await page.goto('https://www.artillery.io/');
await page.click('text=Cloud');
}
root@ubuntu2204:/home/vagrant# artillery run hello-world.yml
Test run id: t9yad_6437mptpftpgeqr6cya4egfyk5k3t_rwqw
Phase started: unnamed (index: 0, duration: 1s) 09:16:07(+0000)
Phase completed: unnamed (index: 0, duration: 1s) 09:16:08(+0000)
--------------------------------------
Metrics for period to: 09:16:10(+0000) (width: 1.259s)
--------------------------------------
browser.http_requests: ......................................................... 34
browser.page.codes.200: ........................................................ 35
browser.page.codes.206: ........................................................ 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
Warning: multiple batches of metrics for period 1719306960000 2024-06-25T09:16:00.000Z
--------------------------------------
Metrics for period to: 09:16:20(+0000) (width: 1.465s)
--------------------------------------
browser.http_requests: ......................................................... 28
browser.page.FCP.https://www.artillery.io/:
min: ......................................................................... 1229.2
max: ......................................................................... 1229.2
mean: ........................................................................ 1229.2
median: ...................................................................... 1224.4
p95: ......................................................................... 1224.4
p99: ......................................................................... 1224.4
browser.page.FID.https://www.artillery.io/:
min: ......................................................................... 14.2
max: ......................................................................... 14.2
mean: ........................................................................ 14.2
median: ...................................................................... 14.2
p95: ......................................................................... 14.2
p99: ......................................................................... 14.2
browser.page.LCP.https://www.artillery.io/:
min: ......................................................................... 1229.2
max: ......................................................................... 1229.2
mean: ........................................................................ 1229.2
median: ...................................................................... 1224.4
p95: ......................................................................... 1224.4
p99: ......................................................................... 1224.4
browser.page.TTFB.https://www.artillery.io/:
min: ......................................................................... 614.1
max: ......................................................................... 614.1
mean: ........................................................................ 614.1
median: ...................................................................... 608
p95: ......................................................................... 608
p99: ......................................................................... 608
browser.page.codes.200: ........................................................ 23
browser.page.codes.206: ........................................................ 1
browser.page.codes.302: ........................................................ 1
browser.page.codes.307: ........................................................ 1
browser.page.codes.308: ........................................................ 1
browser.page.codes.404: ........................................................ 1
vusers.completed: .............................................................. 1
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 2874.4
max: ......................................................................... 2874.4
mean: ........................................................................ 2874.4
median: ...................................................................... 2893.5
p95: ......................................................................... 2893.5
p99: ......................................................................... 2893.5
All VUs finished. Total time: 4 seconds
--------------------------------
Summary report @ 09:16:13(+0000)
--------------------------------
browser.http_requests: ......................................................... 62
browser.page.FCP.https://www.artillery.io/:
min: ......................................................................... 1229.2
max: ......................................................................... 1229.2
mean: ........................................................................ 1229.2
median: ...................................................................... 1224.4
p95: ......................................................................... 1224.4
p99: ......................................................................... 1224.4
browser.page.FID.https://www.artillery.io/:
min: ......................................................................... 14.2
max: ......................................................................... 14.2
mean: ........................................................................ 14.2
median: ...................................................................... 14.2
p95: ......................................................................... 14.2
p99: ......................................................................... 14.2
browser.page.LCP.https://www.artillery.io/:
min: ......................................................................... 1229.2
max: ......................................................................... 1229.2
mean: ........................................................................ 1229.2
median: ...................................................................... 1224.4
p95: ......................................................................... 1224.4
p99: ......................................................................... 1224.4
browser.page.TTFB.https://www.artillery.io/:
min: ......................................................................... 614.1
max: ......................................................................... 614.1
mean: ........................................................................ 614.1
median: ...................................................................... 608
p95: ......................................................................... 608
p99: ......................................................................... 608
browser.page.codes.200: ........................................................ 58
browser.page.codes.206: ........................................................ 2
browser.page.codes.302: ........................................................ 1
browser.page.codes.307: ........................................................ 1
browser.page.codes.308: ........................................................ 1
browser.page.codes.404: ........................................................ 1
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 2874.4
max: ......................................................................... 2874.4
mean: ........................................................................ 2874.4
median: ...................................................................... 2893.5
p95: ......................................................................... 2893.5
p99: ......................................................................... 2893.5Opensearch Benchmark (Performance Test for OpenSearch Clusters)InformationOpenSearch Benchmark is a macrobenchmark utility provided by the OpenSearch Project. It allows you to collect performance metrics from an OpenSearch cluster for a variety of purposes. It is possible to track the overall performance of the cluster, report on when to update the cluster, and evaluate how changes in workflow may affect the cluster.
Advantages and disadvantages
Installationroot@ubuntu2204:/home/vagrant# apt install python3-pip
root@ubuntu2204:/home/vagrant# pip install opensearch-benchmarkInstallation (Docker)root@ubuntu2204:/home/vagrant# docker pull opensearchproject/opensearch-benchmark:latest
root@ubuntu2204:/home/vagrant# docker run opensearchproject/opensearch-benchmark -hOpensearch Benchmark Test (With Wazuh)
root@ubuntu2204:/home/vagrant# opensearch-benchmark list workloads
root@ubuntu2204:/home/vagrant# opensearch-benchmark execute-test --pipeline=benchmark-only --workload=nested --target-host=https://localhost:9200 --client-options=basic_auth_user:admin,basic_auth_password:admin,verify_certs:false
____ _____ __ ____ __ __
/ __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__
/ / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_|
/_/
[INFO] [Test Execution ID]: 73bdc966-abc6-4c84-a4d5-1be749c8062f
[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Downloading workload data: documents.json.bz2 (663.3 MB total size) [100.0%]
[INFO] Decompressing workload data from [/root/.benchmark/benchmarks/data/nested/documents.json.bz2] to [/root/.benchmark/benchmarks/data/nested/documents.json] (resulting size: [3.39] GB) ... [OK]
[INFO] Preparing file offset table for [/root/.benchmark/benchmarks/data/nested/documents.json] ... [OK]
[INFO] Executing test with workload [nested], test_procedure [nested-search-test_procedure] and provision_config_instance ['external'] with version [7.10.2].
[WARNING] merges_total_time is 102 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 536 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 954 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 138 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index [100% done]
Running create-index [100% done]
Running check-cluster-health [100% done]
Running index-append [100% done]
Running refresh-after-index [100% done]
Running force-merge [100% done]
Running refresh-after-force-merge [100% done]
Running wait-until-merges-finish [100% done]
Running randomized-nested-queries [100% done]
Running randomized-term-queries [100% done]
Running randomized-sorted-term-queries [100% done]
Running match-all [100% done]
Running nested-date-histo [100% done]
Running randomized-nested-queries-with-inner-hits_default [100% done]
Running randomized-nested-queries-with-inner-hits_default_big_size [100% done]
------------------------------------------------------
_______ __ _____
/ ____(_)___ ____ _/ / / ___/_________ ________
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------
| Metric | Task | Value | Unit |
|---------------------------------------------------------------:|-----------------------------------------------------------:|------------:|-------:|
| Cumulative indexing time of primary shards | | 8.58217 | min |
| Min cumulative indexing time across primary shards | | 0 | min |
| Median cumulative indexing time across primary shards | | 0.000258333 | min |
| Max cumulative indexing time across primary shards | | 8.57323 | min |
| Cumulative indexing throttle time of primary shards | | 0 | min |
| Min cumulative indexing throttle time across primary shards | | 0 | min |
| Median cumulative indexing throttle time across primary shards | | 0 | min |
| Max cumulative indexing throttle time across primary shards | | 0 | min |
| Cumulative merge time of primary shards | | 4.00155 | min |
| Cumulative merge count of primary shards | | 6 | |
| Min cumulative merge time across primary shards | | 0 | min |
| Median cumulative merge time across primary shards | | 0 | min |
| Max cumulative merge time across primary shards | | 3.99985 | min |
| Cumulative merge throttle time of primary shards | | 1.66413 | min |
| Min cumulative merge throttle time across primary shards | | 0 | min |
| Median cumulative merge throttle time across primary shards | | 0 | min |
| Max cumulative merge throttle time across primary shards | | 1.66413 | min |
| Cumulative refresh time of primary shards | | 0.297567 | min |
| Cumulative refresh count of primary shards | | 141 | |
| Min cumulative refresh time across primary shards | | 0 | min |
| Median cumulative refresh time across primary shards | | 0.00189167 | min |
| Max cumulative refresh time across primary shards | | 0.281667 | min |
| Cumulative flush time of primary shards | | 0.666633 | min |
| Cumulative flush count of primary shards | | 23 | |
| Min cumulative flush time across primary shards | | 0 | min |
| Median cumulative flush time across primary shards | | 0.000275 | min |
| Max cumulative flush time across primary shards | | 0.664333 | min |
| Total Young Gen GC time | | 7.809 | s |
| Total Young Gen GC count | | 651 | |
| Total Old Gen GC time | | 0 | s |
| Total Old Gen GC count | | 0 | |
| Store size | | 3.21809 | GB |
| Translog size | | 5.12227e-07 | GB |
| Heap used for segments | | 0 | MB |
| Heap used for doc values | | 0 | MB |
| Heap used for terms | | 0 | MB |
| Heap used for norms | | 0 | MB |
| Heap used for points | | 0 | MB |
| Heap used for stored fields | | 0 | MB |
| Segment count | | 50 | |
| Min Throughput | index-append | 39833.5 | docs/s |
| Mean Throughput | index-append | 41103.6 | docs/s |
| Median Throughput | index-append | 41187.7 | docs/s |
| Max Throughput | index-append | 42569.8 | docs/s |
| 50th percentile latency | index-append | 422.194 | ms |
| 90th percentile latency | index-append | 667.805 | ms |
| 99th percentile latency | index-append | 2576.67 | ms |
| 99.9th percentile latency | index-append | 6141.64 | ms |
| 100th percentile latency | index-append | 6759.82 | ms |
| 50th percentile service time | index-append | 422.194 | ms |
| 90th percentile service time | index-append | 667.805 | ms |
| 99th percentile service time | index-append | 2576.67 | ms |
| 99.9th percentile service time | index-append | 6141.64 | ms |
| 100th percentile service time | index-append | 6759.82 | ms |
| error rate | index-append | 0 | % |
| Min Throughput | wait-until-merges-finish | 0.06 | ops/s |
| Mean Throughput | wait-until-merges-finish | 0.06 | ops/s |
| Median Throughput | wait-until-merges-finish | 0.06 | ops/s |
| Max Throughput | wait-until-merges-finish | 0.06 | ops/s |
| 100th percentile latency | wait-until-merges-finish | 16252.9 | ms |
| 100th percentile service time | wait-until-merges-finish | 16252.9 | ms |
| error rate | wait-until-merges-finish | 0 | % |
| Min Throughput | randomized-nested-queries | 19.9 | ops/s |
| Mean Throughput | randomized-nested-queries | 19.94 | ops/s |
| Median Throughput | randomized-nested-queries | 19.95 | ops/s |
| Max Throughput | randomized-nested-queries | 19.97 | ops/s |
| 50th percentile latency | randomized-nested-queries | 28.4881 | ms |
| 90th percentile latency | randomized-nested-queries | 43.5573 | ms |
| 99th percentile latency | randomized-nested-queries | 71.1674 | ms |
| 99.9th percentile latency | randomized-nested-queries | 120.874 | ms |
| 100th percentile latency | randomized-nested-queries | 129.945 | ms |
| 50th percentile service time | randomized-nested-queries | 26.4684 | ms |
| 90th percentile service time | randomized-nested-queries | 41.1465 | ms |
| 99th percentile service time | randomized-nested-queries | 68.0754 | ms |
| 99.9th percentile service time | randomized-nested-queries | 106.919 | ms |
| 100th percentile service time | randomized-nested-queries | 125.263 | ms |
| error rate | randomized-nested-queries | 0 | % |
| Min Throughput | randomized-term-queries | 24.99 | ops/s |
| Mean Throughput | randomized-term-queries | 25 | ops/s |
| Median Throughput | randomized-term-queries | 25 | ops/s |
| Max Throughput | randomized-term-queries | 25 | ops/s |
| 50th percentile latency | randomized-term-queries | 7.88739 | ms |
| 90th percentile latency | randomized-term-queries | 11.0647 | ms |
| 99th percentile latency | randomized-term-queries | 15.2192 | ms |
| 100th percentile latency | randomized-term-queries | 22.8419 | ms |
| 50th percentile service time | randomized-term-queries | 6.12509 | ms |
| 90th percentile service time | randomized-term-queries | 8.878 | ms |
| 99th percentile service time | randomized-term-queries | 12.6071 | ms |
| 100th percentile service time | randomized-term-queries | 20.7344 | ms |
| error rate | randomized-term-queries | 0 | % |
| Min Throughput | randomized-sorted-term-queries | 15.99 | ops/s |
| Mean Throughput | randomized-sorted-term-queries | 15.99 | ops/s |
| Median Throughput | randomized-sorted-term-queries | 15.99 | ops/s |
| Max Throughput | randomized-sorted-term-queries | 15.99 | ops/s |
| 50th percentile latency | randomized-sorted-term-queries | 15.7295 | ms |
| 90th percentile latency | randomized-sorted-term-queries | 21.3126 | ms |
| 99th percentile latency | randomized-sorted-term-queries | 37.9242 | ms |
| 100th percentile latency | randomized-sorted-term-queries | 46.5997 | ms |
| 50th percentile service time | randomized-sorted-term-queries | 13.7363 | ms |
| 90th percentile service time | randomized-sorted-term-queries | 19.2022 | ms |
| 99th percentile service time | randomized-sorted-term-queries | 34.7268 | ms |
| 100th percentile service time | randomized-sorted-term-queries | 44.1017 | ms |
| error rate | randomized-sorted-term-queries | 0 | % |
| Min Throughput | match-all | 5 | ops/s |
| Mean Throughput | match-all | 5 | ops/s |
| Median Throughput | match-all | 5 | ops/s |
| Max Throughput | match-all | 5 | ops/s |
| 50th percentile latency | match-all | 6.94698 | ms |
| 90th percentile latency | match-all | 10.5454 | ms |
| 99th percentile latency | match-all | 14.2265 | ms |
| 100th percentile latency | match-all | 20.7373 | ms |
| 50th percentile service time | match-all | 4.8253 | ms |
| 90th percentile service time | match-all | 8.11081 | ms |
| 99th percentile service time | match-all | 10.3598 | ms |
| 100th percentile service time | match-all | 16.6433 | ms |
| error rate | match-all | 0 | % |
| Min Throughput | nested-date-histo | 1 | ops/s |
| Mean Throughput | nested-date-histo | 1 | ops/s |
| Median Throughput | nested-date-histo | 1 | ops/s |
| Max Throughput | nested-date-histo | 1 | ops/s |
| 50th percentile latency | nested-date-histo | 634.563 | ms |
| 90th percentile latency | nested-date-histo | 684.109 | ms |
| 99th percentile latency | nested-date-histo | 738.059 | ms |
| 100th percentile latency | nested-date-histo | 777.098 | ms |
| 50th percentile service time | nested-date-histo | 631.674 | ms |
| 90th percentile service time | nested-date-histo | 680.681 | ms |
| 99th percentile service time | nested-date-histo | 736.705 | ms |
| 100th percentile service time | nested-date-histo | 775.063 | ms |
| error rate | nested-date-histo | 0 | % |
| Min Throughput | randomized-nested-queries-with-inner-hits_default | 17.83 | ops/s |
| Mean Throughput | randomized-nested-queries-with-inner-hits_default | 17.99 | ops/s |
| Median Throughput | randomized-nested-queries-with-inner-hits_default | 17.99 | ops/s |
| Max Throughput | randomized-nested-queries-with-inner-hits_default | 18 | ops/s |
| 50th percentile latency | randomized-nested-queries-with-inner-hits_default | 33.6864 | ms |
| 90th percentile latency | randomized-nested-queries-with-inner-hits_default | 52.4591 | ms |
| 99th percentile latency | randomized-nested-queries-with-inner-hits_default | 81.8842 | ms |
| 99.9th percentile latency | randomized-nested-queries-with-inner-hits_default | 568.065 | ms |
| 100th percentile latency | randomized-nested-queries-with-inner-hits_default | 617.288 | ms |
| 50th percentile service time | randomized-nested-queries-with-inner-hits_default | 30.9282 | ms |
| 90th percentile service time | randomized-nested-queries-with-inner-hits_default | 48.6987 | ms |
| 99th percentile service time | randomized-nested-queries-with-inner-hits_default | 68.033 | ms |
| 99.9th percentile service time | randomized-nested-queries-with-inner-hits_default | 100.44 | ms |
| 100th percentile service time | randomized-nested-queries-with-inner-hits_default | 616.964 | ms |
| error rate | randomized-nested-queries-with-inner-hits_default | 0 | % |
| Min Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 16 | ops/s |
| Mean Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 16 | ops/s |
| Median Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 16 | ops/s |
| Max Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 16 | ops/s |
| 50th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 34.9921 | ms |
| 90th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 51.1011 | ms |
| 99th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 69.6266 | ms |
| 99.9th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 87.5509 | ms |
| 100th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 114.079 | ms |
| 50th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 33.2397 | ms |
| 90th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 49.1744 | ms |
| 99th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 66.9583 | ms |
| 99.9th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 84.0898 | ms |
| 100th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 111.146 | ms |
| error rate | randomized-nested-queries-with-inner-hits_default_big_size | 0 | % |
----------------------------------
[INFO] SUCCESS (took 2042 seconds)
----------------------------------
Locust (Load Testing Framework)InformationLocust is a Python load testing framework. It allows you to define user behavior using Python code and simulate thousands of users performing actions concurrently. You can install Locust from PyPI.
Advantages and disadvantages
Installationroot@ubuntu2204:/home/vagrant# apt install python3-pip
root@ubuntu2204:/home/vagrant# pip install locustLocust Test
root@ubuntu2204:/home/vagrant# touch locustfile.py
from locust import HttpUser, task
class HelloWorldUser(HttpUser):
@task
def hello_world(self):
self.client.get("/hello")
self.client.get("/world")
root@ubuntu2204:/home/vagrant# locust
[2024-06-25 10:47:39,471] ubuntu2204/INFO/locust.main: Starting web interface at http://0.0.0.0:8089
[2024-06-25 10:47:39,477] ubuntu2204/INFO/locust.main: Starting Locust 2.29.0
Fluentd (Data Collector for Unified Logging Layer)InformationFluentd is an open source data collector that allows you to unify the collection and consumption of data from various sources and destinations. Fluentd provides a unified layer of record between data sources and backend systems. This allows you to decouple data sources from target systems. One of Fluentd's most useful features is its ability to filter and enrich records as they are collected. Users can create custom filtering rules to remove unwanted records, add additional fields to records, and restructure data to make it more useful for further analysis. In addition, it can also be used to monitor system status and alert users if a problem occurs. Users can set up alerts based on certain criteria, such as the number of errors in a given time period, and receive real-time notifications if a problem occurs.
Advantages and disadvantages
Installationroot@ubuntu2204:/home/vagrant# curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-fluent-package5-lts.sh | sh
root@ubuntu2204:/home/vagrant# sudo systemctl start fluentd.service
root@ubuntu2204:/home/vagrant# sudo systemctl status fluentd.service
root@ubuntu2204:/home/vagrant# curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test
root@ubuntu2204:/home/vagrant# tail -n 1 /var/log/fluent/fluentd.logPost-Installation
root@ubuntu2204:/home/vagrant# cat /etc/fluent/fluentd.conf
root@ubuntu2204:/home/vagrant# cat /var/log/fluent/fluentd.log
Integrating Wazuh with FluentdFollowing the official guide, it is possible to integrate Wazuh and Fluentd (Wazuh v4.8.0) Tsung (Distributed Load Testing Tool)InformationTsung is an open source distributed load testing tool. It can be used to stress servers that handle multiple protocols. Tsung is not limited to a single protocol and can simulate thousands of concurrent virtual users. It can also be distributed across multiple client machines to increase the testing load.
Advantages and disadvantages
InstallationTo install without using the packages, the following dependencies must be satisfied:
It is also necessary to compile the code (you have to download it beforehand): Another option is to download the packages: Tsung TestsTsung uses XML files to run the tests. The learning curve is steep and does not provide example code to test the tool. Cypress (Testing Frameworks for Javascript) (E2E Tests)InformationCypress is a next-generation front-end testing tool designed for modern web applications. Here are some key points:
In summary, Cypress simplifies end-to-end testing and enhances your testing workflow.
Advantages and disadvantages
Installationroot@ubuntu2204:/home/vagrant# apt update
root@ubuntu2204:/home/vagrant# curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
root@ubuntu2204:/home/vagrant# nvm install 20
root@ubuntu2204:/home/vagrant# node -v
root@ubuntu2204:/home/vagrant# npm -v
root@ubuntu2204:/home/vagrant# npm install cypress --save-devConclusionsOf all the tools analyzed, the most notable are: Artillery, Playwright, Fluentd and Cypress. Artillery and Cypress can also be integrated to work together. Cypress can be used in place of Playwright but not integrated with Artillery. Cypress has in its favor that it was previously used to test the Wazuh Dashboard. Likewise, for Fluentd there is documentation to integrate it and use it with Wazuh. Cypress, from the tests carried out, seems more versatile and powerful than Playwright. It all depends on the specific use you want to give it. Of the rest of the tools, I would highlight OpenSearch Benchmark for Indexer Cluster tests (it is a very specific Benchmark, for OpenSearch). Locust could also be useful if you want to measure loading times in requests. Although it is true that this could be done with other tools. |
Metrics analysis toolsSummary table
Prometheus and Grafana
InformationPrometheus is a Open Source systems monitoring and alerting toolkit. Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. Prometheus consists of multiple components, like the main server which scrapes and stores time series data, client libraries for instrumenting application code, an alert manager to handle alerts, etc. Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana can be used to visualize the collected data. Grafana enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they are stored. It supports querying Prometheus, so both platforms can be used to monitor and visualize the metrics of our systems.
Advantages, Disadvantages and Examples
It is necessary to install and configure an Exporter. In these examples, we are using the Node Exporter, which is used for monitoring Linux hosts metrics, but depending on the needs, we can choose from different options. For example, there is another exporter which is responsible for exporting the information of the system processes (Process Exporter). This will be useful if we want to analyse the different daemons and processes of Wazuh. The integration is very simple, it is done through the Grafana dashboard (UI). In it, Grafana is configured to generate a dashboard through the information that Prometheus sends. There are many dashboards and many different options and configurations to obtain the metrics. It has been tested with a very complete Node Exporter dashboard in which we can see a multitude of graphs about the Linux system metrics. Examples
I think this is a good tool to get metrics such as CPU, memory, file descriptors, disk operations, hardware usage of all Wazuh components: Agent, Manager, Indexer, Dashboard. It is flexible, as we can choose what kind of data we want to plot, and it offers a very complete user interface with many configurable options. Another positive aspect of these tools is that it is possible to configure an alert module in which, if the values obtained by Prometheus differ from a threshold value, an alert will be generated with the corresponding information. This is done by constructing rules in a YML file. The configuration is really simple, you only need to create a YML file with the rule and add its path to the Prometheus configuration file. Therefore, this allows to analyse the data automatically and alert when a discrepancy with the "accepted" values is found. ExamplesI have performed a simple test where if the CPU use exceeds 2%, an alert is generated. The YML file with the rule that triggers the alert is: groups:
- name: example_alert_rules
rules:
- alert: HighCPUUsage
expr: avg(rate(node_cpu_seconds_total{mode="system"}[1m])) > 0.02
labels:
severity: critical
annotations:
summary: "High CPU Usage"
description: "The CPU usage is above 2%."And then specify the path to the file in the configuration file: rule_files:
- "rule.yml"
The result is:
Offline visualizationAs part of the analysis of the data obtained after monitoring the different metrics of the system, it is necessary to be able to export and save this data in order to be able to refer to it when necessary and to make comparisons, serving as a baseline. Once these data have been saved, it will be necessary to visualise them again. The first thing to do is to create a snapshot of the current Prometheus data: # curl -XPOST localhost:9090/api/v1/admin/tsdb/snapshotThen VictoriaMetrics must be installed and launched: # wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.76.1/victoria-metrics-linux-amd64-v1.76.1.tar.gz
# tar -xvf victoria-metrics-linux-amd64-v1.76.1.tar.gz
# ./victoria-metrics-prod &Also is necessary to install # wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.76.1/vmutils-linux-amd64-v1.76.1.tar.gz
# tar -xvf vmutils-linux-amd64-v1.76.1.tar.gzAnd finally, to load the Prometheus snapshot into VictoriaMetrics: # ./vmctl-prod prometheus --prom-snapshot ./prometheus/data/snapshots/20240702T102147Z-11f770969d68dec7/To visualise this data in Grafana, simply add a data source from VictoriaMetrics and create a dashboard with the appropriate graphs. Simultaneous visualization of different testsIn order to be able to make a more exhaustive comparison between the graphs of different tests, we have investigated the possibility of joining all the data in the same graph, so that we can visualise in the same graph both the results of one test and the results of any other test. This combined with the previous point (offline data visualisation) means that we have a tool capable of fulfilling all the needs requested in the issue. This can easily be done with Grafana if the tests were measured in the same time interval. For example: let's imagine that we have two different data sources, one representing the values from an old test for which we have stored its data (and which will act as a baseline), and the other representing the values taken from a current test. The first source is called "prometheus" and the second one is called "prometheus-1". In the image below we can see the CPU usage percentage of the system in two different lines, each one from a different data source:
Now for example, in the same graph, let's compare the percentage of CPU used by user processes, and hide the previous system CPU measurements for better comparison:
It is also possible to manually change to any color for each of the metrics in the configuration panel on the right, making the graphs more configurable and customisable. But, in a real case, we will have some data corresponding to a version of Wazuh taken in a time interval, and other data corresponding to another version that will have been taken in a time interval different from the previous one and possibly quite separated in time. In this case, we will need the times of both tests to be normalised, in order to be able to visualise and compare the data more easily. The problem is that Grafana does not allow this, at least in the Open Source version.
If we zoom in on the graph we can see how the data differs:
Netdata
InformationNetdata is an open-source, real-time performance and health monitoring tool for systems and applications. It provides comprehensive insights into the performance of your entire IT infrastructure, offering detailed metrics and visualizations. There are 3 components in the Netdata structure: Netdata Agents, to monitor the physical or virtual nodes of your infrastructure, including all applications and containers running on them; Netdata Parents, to create observability centralization pointswithin your infrastructure, to offload Netdata Agents functions from your production systems, to provide high-availability of your data, increased data retention and isolation of your nodes; and Netdata Cloud combining all your infrastructure, all your Netdata Agents and Parents, into one uniform, distributed, scalable, monitoring database, offering custom dashboards, troubleshooting tools, etc.
Advantages, Disadvantages and Examples
The installation is completed by logging into Netdata and executing a single command. After this, we can access the Netdata dashboard and get graphs of different system metrics: Examples
In this tool, there are a number of preconfigured alerts with default values to indicate when the metrics obtained exceed these values. It is also possible to create new customised rules. The configuration of the alerts is simple and is done in ExamplesThe file containing the rule is: template: 10min_cpu_usage
on: system.cpu
class: Utilization
type: System
component: CPU
host labels: _os=linux
lookup: average -10m unaligned of user,system,softirq,irq,guest
units: %
every: 1m
warn: $this > (($status >= $WARNING) ? (1) : (2))
crit: $this > (($status == $CRITICAL) ? (3) : (4))
delay: down 15m multiplier 1.5 max 1h
summary: System CPU utilization
info: Average CPU utilization over the last 10 minutes (excluding iowait, nice and steal)
to: sysadmin
Simultaneous visualization of different testsNetdata does not allow the simultaneous visualisation of data from different data sources, which requires the use of platforms such as Grafana, so using Netdata would be a restriction on this point. Nagios
InformationNagios is an open-source monitoring system designed to monitor systems, networks, and infrastructure. It provides monitoring capabilities to ensure the availability and performance of IT systems and services. There are different types of Nagios solutions, but we should focus on Nagios Core and Nagios XI. Nagios XI is capable of monitoring hundreds of different types of applications, services, protocols, and computer hardware through the use of built-in capabilities and third-party extensions and addons. Nagios Core is an Open Source system and network monitoring application. It watches hosts and services that you specify, alerting you when things go bad and when they get better.
Advantages and Disadvantages
In order to obtain system metrics, it is necessary to install a number of plugins and make changes to the Nagios configuration, which is more tedious than the previous tools. Therefore, we will first move forward with further research on the above tools, and if necessary, further research on Nagios will be carried out. Zabbix
InformationZabbix is software that monitors numerous parameters of a network and the health and integrity of servers, virtual machines, applications, services, databases, websites, the cloud, and more. Zabbix uses a flexible notification mechanism that allows users to configure email-based alerts for virtually any event. Zabbix offers reporting and data visualization features based on the stored data. It is composed of different components, like the Zabbix server, DataBase storage, a Web interface and a Zabbix agent that are deployed on monitoring targets to actively monitor local resources and applications and report the gathered data to the Zabbix server.
Advantages and disadvantages
Due to the disadvantages provided by this tool and given that there are other tools that are simpler and offer better features, I do not think that Zabbix is an option to consider. Therefore, we will first move forward with further research on the above tools, and if necessary, further research on Zabbix will be carried out. ConclusionIn conclusion, after analysing the different tools, we can see that using Prometheus as a system monitoring tool and Grafana as a tool for visualising the data collected by Prometheus is one of the most viable and favorable options we have. On the other hand, Netdata also seems to be a good tool, but it has some disadvantages and differences with respect to the previous two, making its use less interesting. As for Nagios and Zabbix, their configuration has been much more complicated than the rest of the tools, so their analysis has been less complete, as the priority has been to invest most of the time in deeply analysing the other tools, and this is a great disadvantage. Prometheus and Grafana seem to be a great option in terms of the objectives required in this issue, but they also have some limitations such as the simultaneous visualisation of different data, which has finally been solved with a Python script. Update after reviewAfter the review that have been performed, it is necessary to investigate Plotly/Dash applications to visualize performance metrics. Plotly/DashInformationThe Plotly Python library is an interactive plotting library that supports a wide range of use-cases. It enables users to create interactive web-based visualizations that can be displayed in Jupyter notebooks, saved to standalone HTML files, or served as part of Python-built web applications using Dash. Dash is a framework for rapidly building data apps in Python. It is a good way to build analytical apps in Python using Plotly figures. Main features:
The installation of these tools is really simple and their documentation is very intuitive and complete. Example of useTo show an example of use, @Rebits has created a script in which in a few lines of code we can demonstrate the potential of this tool, seeing how the comparison between versions is really simple, simplifying the process that we currently have in the footprint tests: poc_graph.webmWith just a small change in the code, we can get more than one demon drawn on the graph at a time:
Now, we want to test another use case. We want to check if real-time loading of CSV files is possible. This is motivated by the fact that, in the future, we will have a nightly that will run the necessary processes and generate the corresponding reports. These reports will be stored somewhere, and Plotly should be able to load this data for display automatically. Example: PoCexample2.avi.mp4In this case, we are using a script that updates the database information (enter the data from the new file), therefore, we only need to run the script for the database to update, and thus, the Dash server is updated. We will now modify the application to add a checklist that allows us to select the versions we want to plot from all those available in the database:
It is now necessary to test the ability of this implementation to support large amounts of data, that is, multiple files representing different versions or stages, as this will be necessary when generating reports daily. PoCexample3.mp4To conclude, this comparison table between Plotly and Grafana shows the advantages and disadvantages of both tools.
Ultimately, using Grafana seems like an overkill option since it offers a rich set of features that are not adapted to what we are looking for and that may complicate the process unnecessarily. Using Plotly we have a very simple way to build custom graphics, very intuitive, with a multitude of options to configure and extensive documentation to support us. This way of visualisation will make our work much easier and will allow us to advance faster. In any case, it will have to be decided whether this tool is suitable for our case and whether it will finally be chosen over Grafana and Prometheus. |
Metrics Analysis Tools
Advantages and Disadvantages (Explained)Prometheus
Grafana
Netdata
Nagios
Zabbix
|
UpdateResearch must be expanded to the tools that are already being used. Specifically, we must review how hardware resources are monitored (tests and footprints). Also, alerts and specific Wazuh measurements are generated that are not possible to cover with an external tool. |
Tools for saturation of wazuh modules (internal) @santipadillaTest stress tier
Test stress
Scripts in charge of module saturation
|
Tools for running processes and system utilization monitoring (external and internal tools) @santipadillaTools usedpsutil
With psutil we can:
AlternativesGlances
SystemTap
DTrace
Tools comparison
|
Locust vs ArtilleryBoth Artillery and Locust are popular tools for load and performance testing.
In short, Locust has improved in performance, while Artillery has decreased in speed. Both tools have their advantages and disadvantages, so the choice depends on your specific needs and preferences. If you are looking for a faster and more distributed tool, Locust might be the better choice. Locust is more widely used than Artillery at the moment. The most notable advantage is that whenever you can write Python code to do something, you can test it with Locust. This makes it much easier to deploy. Also, the integration with Jenkins is very simple and fast since it can even be used as a library. This would allow you to run tests automatically, generate results and even analyze them. In the case of Artillery, it is not yet as widely used as Locust. It uses JavaScript for tests. This not only complicates the creation of tests but, if you want to integrate it with Jenkins, it is more complex (and requires Docker to run the tests). In my tests, it has also turned out to be slower when running tests and does not offer the possibility of distributed load generation. |
Tools for AnalysisAfter a first investigation, I have been able to find the following tools for data analysis. There does not seem to be specific software (although research continues) and, instead, specific programming languages or libraries oriented toward data analysis and graph generation are used.
Both R and Python are widely used and versatile options. Specifically, Python is supported by different libraries such as Numpy or Pandas for data analysis. Below I include some more information about all the tools. Tool InformationR
Python
Apache Kafka
Apache Spark
For Python there are several useful libraries when analyzing data and generating graphs. The advantage is that you can create a script adapted to the data to be used. This is also an inconvenience since for both R and Python you have to manually create the scripts. Below I leave a list of useful Python libraries.
For R there are also several libraries that can be used to support the creation of scripts for data analysis and graph generation. Below is a list of the most popular ones.
|
Grafana Open Source vs Grafana Enterprise
Grafana is a widely used open source visualization and monitoring tool. The Grafana Enterprise version is the commercial edition of Grafana and offers additional features not found in the open source version. Here are the key differences: Premium Plugins
Authentication and Security
According to the official documentation, this is the list of exclusive features of the Enterprise version:
In general, depending on the features you need, this is a summary of which version of Grafana to use: Grafana Open Source
Grafana Enterprise
List of Premium Plugins: Expand List
Regarding prices, Grafana does not clearly show the price of its Enterprise version. Likewise, it does not easily offer a demo. You need to contact Grafana to get a demo and pricing for Grafana Enterprise. Likewise, the information is not very clear since they focus on the sale of their cloud services. To consider using Grafana Enterprise we would have to see if we need any of the extra functions. If they are not necessary, the Open Source version should cover all needs. |
ConclusionsAfter finishing the investigations, it was decided to use a series of tools for different purposes.
|
Dashboard Performance testsIt has been confirmed in a meeting with @davidjiglesias @havidarou and @juliamagan the use of Artillery and Playwright for basic performance testing. LGTM API TestsIt has been confirmed in a meeting with @davidjiglesias @havidarou and @juliamagan the use of Locust for basic performance testing. LGTM Data AnalysisIt has been confirmed in a meeting with @davidjiglesias @havidarou and @juliamagan the use of Python libraries for statistical analysis of performance results. Further research will be carried on at the design phase. LGTM Modules saturationIt has been confirmed in a meeting with @davidjiglesias @havidarou and @juliamagan the use of custom tools for module saturation in the Agent performance test. LGTM Process monitoringIt has been confirmed in a meeting with @davidjiglesias @havidarou and @juliamagan the use of current libraries based on psutils for process monitoring. LGTM Visualization (@MARCOSD4 )Grafana and Prometheus visualization method could maybe not be the best suit due to the following reasons:
For these reasons, I suggest researching Plotly/Dash applications to visualize performance metrics https://plotly.com/python/. I also provided a basic PoC of how metrics could appear in a future performance dashboard: It is requested to investigate these libraries and incorporate them into the analysis to finalize the research Finally, other team members have analyzed the usage of Grafana for an observability module #4524. It is pending to have a meeting to share knowledge in order to determinate the capabilities of the current development. |
|
An in-depth analysis of Plotly and Dash has been added to this commentary, and the final conclusion has been edited according to the research done. |
|
LGTM |
|
LGTM |
Description
The objective of this issue is to investigate and identify the most effective tools, techniques, and methodologies for benchmarking testing. These tests will serve as a baseline for comparison, enabling the tracking of performance improvements or regressions. By doing so, benchmarking will help maintain and enhance the efficiency of the product. The proposed tools and techniques should meet the proposed functional and non-functional requirements. This research will enable us to determine the necessary functionalities to develop a performance test for the product, ensuring that it is comprehensive and effective.
Funcional Requirements
Capabilities
KPTM analysis and data collection
Proposed tools should be capable of monitoring the following KPTM (Key Performance Test Metrics):
Implementation restrictions
Plan
Related issues
Parent issue
The text was updated successfully, but these errors were encountered: