Implement impacted tests detection #8188

nikita-tkachenko-datadog · 2025-01-13T11:29:10Z

What Does This Do

Motivation

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1389

pr-commenter · 2025-01-13T12:00:37Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/impacted-tests
git_commit_date	1737124507	1737129672
git_commit_sha	`fbb36f9`	`41bfd9f`
release_version	1.46.0-SNAPSHOT~fbb36f9b5b	1.46.0-SNAPSHOT~41bfd9f9e3

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1737132084	1737132084
ci_job_id	771025762	771025762
ci_pipeline_id	53329237	53329237
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 59 metrics, 3 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:petclinic:profiling:GlobalTracer	worse [+13.577ms; +22.648ms] or [+3.866%; +6.449%]	369.282ms	351.169ms

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~41bfd9f9e3, baseline=1.46.0-SNAPSHOT~fbb36f9b5b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1057052
Total [baseline] (10.498 s) : 0, 10497751
Agent [candidate] (1.055 s) : 0, 1054950
Total [candidate] (10.464 s) : 0, 10464438
section appsec
Agent [baseline] (1.19 s) : 0, 1189661
Total [baseline] (10.774 s) : 0, 10773502
Agent [candidate] (1.194 s) : 0, 1194232
Total [candidate] (10.73 s) : 0, 10729678
section iast
Agent [baseline] (1.205 s) : 0, 1204797
Total [baseline] (11.015 s) : 0, 11015072
Agent [candidate] (1.185 s) : 0, 1185130
Total [candidate] (11.012 s) : 0, 11012265
section profiling
Agent [baseline] (1.261 s) : 0, 1260614
Total [baseline] (10.861 s) : 0, 10860791
Agent [candidate] (1.273 s) : 0, 1272699
Total [candidate] (10.895 s) : 0, 10894932

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.057 s	-
Agent	appsec	1.19 s	132.609 ms (12.5%)
Agent	iast	1.205 s	147.746 ms (14.0%)
Agent	profiling	1.261 s	203.562 ms (19.3%)
Total	tracing	10.498 s	-
Total	appsec	10.774 s	275.75 ms (2.6%)
Total	iast	11.015 s	517.32 ms (4.9%)
Total	profiling	10.861 s	363.04 ms (3.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	appsec	1.194 s	139.282 ms (13.2%)
Agent	iast	1.185 s	130.18 ms (12.3%)
Agent	profiling	1.273 s	217.749 ms (20.6%)
Total	tracing	10.464 s	-
Total	appsec	10.73 s	265.24 ms (2.5%)
Total	iast	11.012 s	547.827 ms (5.2%)
Total	profiling	10.895 s	430.493 ms (4.1%)

gantt
    title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~41bfd9f9e3, baseline=1.46.0-SNAPSHOT~fbb36f9b5b

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.35 ms) : 0, 713350
BytebuddyAgent [candidate] (713.375 ms) : 0, 713375
GlobalTracer [baseline] (255.807 ms) : 0, 255807
GlobalTracer [candidate] (256.688 ms) : 0, 256688
AppSec [baseline] (56.296 ms) : 0, 56296
AppSec [candidate] (56.747 ms) : 0, 56747
Remote Config [baseline] (743.285 µs) : 0, 743
Remote Config [candidate] (724.157 µs) : 0, 724
Telemetry [baseline] (15.896 ms) : 0, 15896
Telemetry [candidate] (12.356 ms) : 0, 12356
section appsec
BytebuddyAgent [baseline] (731.526 ms) : 0, 731526
BytebuddyAgent [candidate] (734.72 ms) : 0, 734720
GlobalTracer [baseline] (252.991 ms) : 0, 252991
GlobalTracer [candidate] (254.398 ms) : 0, 254398
AppSec [baseline] (171.571 ms) : 0, 171571
AppSec [candidate] (171.084 ms) : 0, 171084
Remote Config [baseline] (672.362 µs) : 0, 672
Remote Config [candidate] (665.563 µs) : 0, 666
Telemetry [baseline] (8.252 ms) : 0, 8252
Telemetry [candidate] (8.227 ms) : 0, 8227
IAST [baseline] (19.331 ms) : 0, 19331
IAST [candidate] (20.124 ms) : 0, 20124
section iast
BytebuddyAgent [baseline] (849.614 ms) : 0, 849614
BytebuddyAgent [candidate] (834.404 ms) : 0, 834404
GlobalTracer [baseline] (249.625 ms) : 0, 249625
GlobalTracer [candidate] (246.405 ms) : 0, 246405
AppSec [baseline] (58.685 ms) : 0, 58685
AppSec [candidate] (58.209 ms) : 0, 58209
Remote Config [baseline] (670.979 µs) : 0, 671
Remote Config [candidate] (679.605 µs) : 0, 680
Telemetry [baseline] (9.005 ms) : 0, 9005
Telemetry [candidate] (8.842 ms) : 0, 8842
IAST [baseline] (21.91 ms) : 0, 21910
IAST [candidate] (21.489 ms) : 0, 21489
section profiling
BytebuddyAgent [baseline] (706.4 ms) : 0, 706400
BytebuddyAgent [candidate] (702.612 ms) : 0, 702612
GlobalTracer [baseline] (351.169 ms) : 0, 351169
GlobalTracer [candidate] (369.282 ms) : 0, 369282
AppSec [baseline] (55.041 ms) : 0, 55041
AppSec [candidate] (54.127 ms) : 0, 54127
Remote Config [baseline] (712.983 µs) : 0, 713
Remote Config [candidate] (707.453 µs) : 0, 707
Telemetry [baseline] (8.864 ms) : 0, 8864
Telemetry [candidate] (8.843 ms) : 0, 8843
ProfilingAgent [baseline] (96.228 ms) : 0, 96228
ProfilingAgent [candidate] (95.149 ms) : 0, 95149
Profiling [baseline] (96.255 ms) : 0, 96255
Profiling [candidate] (95.176 ms) : 0, 95176

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~41bfd9f9e3, baseline=1.46.0-SNAPSHOT~fbb36f9b5b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1054936
Total [baseline] (8.627 s) : 0, 8626746
Agent [candidate] (1.055 s) : 0, 1054636
Total [candidate] (8.623 s) : 0, 8623488
section iast
Agent [baseline] (1.181 s) : 0, 1181339
Total [baseline] (9.175 s) : 0, 9174956
Agent [candidate] (1.183 s) : 0, 1182993
Total [candidate] (9.232 s) : 0, 9232307
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.185 s) : 0, 1184851
Total [baseline] (9.218 s) : 0, 9217792
Agent [candidate] (1.183 s) : 0, 1182646
Total [candidate] (9.172 s) : 0, 9171759
section iast_TELEMETRY_OFF
Agent [baseline] (1.178 s) : 0, 1177989
Total [baseline] (9.197 s) : 0, 9196838
Agent [candidate] (1.177 s) : 0, 1177303
Total [candidate] (9.225 s) : 0, 9225318

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	iast	1.181 s	126.403 ms (12.0%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.185 s	129.915 ms (12.3%)
Agent	iast_TELEMETRY_OFF	1.178 s	123.053 ms (11.7%)
Total	tracing	8.627 s	-
Total	iast	9.175 s	548.21 ms (6.4%)
Total	iast_HARDCODED_SECRET_DISABLED	9.218 s	591.046 ms (6.9%)
Total	iast_TELEMETRY_OFF	9.197 s	570.092 ms (6.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	iast	1.183 s	128.357 ms (12.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.183 s	128.01 ms (12.1%)
Agent	iast_TELEMETRY_OFF	1.177 s	122.667 ms (11.6%)
Total	tracing	8.623 s	-
Total	iast	9.232 s	608.819 ms (7.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.172 s	548.27 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.225 s	601.83 ms (7.0%)

gantt
    title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~41bfd9f9e3, baseline=1.46.0-SNAPSHOT~fbb36f9b5b

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (714.375 ms) : 0, 714375
BytebuddyAgent [candidate] (713.242 ms) : 0, 713242
GlobalTracer [baseline] (255.651 ms) : 0, 255651
GlobalTracer [candidate] (256.323 ms) : 0, 256323
AppSec [baseline] (55.514 ms) : 0, 55514
AppSec [candidate] (56.45 ms) : 0, 56450
Remote Config [baseline] (731.678 µs) : 0, 732
Remote Config [candidate] (718.24 µs) : 0, 718
Telemetry [baseline] (13.666 ms) : 0, 13666
Telemetry [candidate] (12.952 ms) : 0, 12952
section iast
BytebuddyAgent [baseline] (830.946 ms) : 0, 830946
BytebuddyAgent [candidate] (831.258 ms) : 0, 831258
GlobalTracer [baseline] (246.491 ms) : 0, 246491
GlobalTracer [candidate] (247.177 ms) : 0, 247177
AppSec [baseline] (57.831 ms) : 0, 57831
AppSec [candidate] (58.39 ms) : 0, 58390
IAST [baseline] (21.488 ms) : 0, 21488
IAST [candidate] (21.619 ms) : 0, 21619
Remote Config [baseline] (680.552 µs) : 0, 681
Remote Config [candidate] (696.266 µs) : 0, 696
Telemetry [baseline] (8.876 ms) : 0, 8876
Telemetry [candidate] (8.866 ms) : 0, 8866
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (831.819 ms) : 0, 831819
BytebuddyAgent [candidate] (831.357 ms) : 0, 831357
GlobalTracer [baseline] (248.281 ms) : 0, 248281
GlobalTracer [candidate] (247.194 ms) : 0, 247194
AppSec [baseline] (58.464 ms) : 0, 58464
AppSec [candidate] (58.291 ms) : 0, 58291
IAST [baseline] (21.661 ms) : 0, 21661
IAST [candidate] (21.335 ms) : 0, 21335
Remote Config [baseline] (680.864 µs) : 0, 681
Remote Config [candidate] (671.288 µs) : 0, 671
Telemetry [baseline] (8.946 ms) : 0, 8946
Telemetry [candidate] (8.752 ms) : 0, 8752
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (828.364 ms) : 0, 828364
BytebuddyAgent [candidate] (827.916 ms) : 0, 827916
GlobalTracer [baseline] (246.39 ms) : 0, 246390
GlobalTracer [candidate] (246.785 ms) : 0, 246785
AppSec [baseline] (58.033 ms) : 0, 58033
AppSec [candidate] (57.81 ms) : 0, 57810
IAST [baseline] (20.871 ms) : 0, 20871
IAST [candidate] (20.626 ms) : 0, 20626
Remote Config [baseline] (665.325 µs) : 0, 665
Remote Config [candidate] (664.647 µs) : 0, 665
Telemetry [baseline] (8.692 ms) : 0, 8692
Telemetry [candidate] (8.511 ms) : 0, 8511

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-17T16:11:30	2025-01-17T16:18:35
git_branch	master	nikita-tkachenko/impacted-tests
git_commit_date	1737124507	1737129672
git_commit_sha	`fbb36f9`	`41bfd9f`
release_version	1.46.0-SNAPSHOT~fbb36f9b5b	1.46.0-SNAPSHOT~41bfd9f9e3
start_time	2025-01-17T16:11:16	2025-01-17T16:18:21

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1737131069	1737131069
ci_job_id	771025764	771025764
ci_pipeline_id	53329237	53329237
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 2 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 16 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:iast_FULL	better [-107.395µs; -60.777µs] or [-14.410%; -8.155%]	unstable [-526.008op/s; +1859.341op/s] or [-8.767%; +30.989%]	661.198µs	6666.667op/s	745.283µs	6000.000op/s
scenario:load:insecure-bank:iast_GLOBAL	better [-62.764µs; -15.476µs] or [-11.165%; -2.753%]	unstable [-1452.813op/s; +2452.813op/s] or [-19.371%; +32.704%]	523.029µs	8000.000op/s	562.149µs	7500.000op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~41bfd9f9e3, baseline=1.46.0-SNAPSHOT~fbb36f9b5b
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.363 ms) : 1344, 1383
.   : milestone, 1363,
appsec (1.756 ms) : 1733, 1780
.   : milestone, 1756,
appsec_no_iast (1.761 ms) : 1738, 1784
.   : milestone, 1761,
iast (1.509 ms) : 1485, 1534
.   : milestone, 1509,
profiling (1.519 ms) : 1496, 1542
.   : milestone, 1519,
tracing (1.486 ms) : 1461, 1511
.   : milestone, 1486,
section candidate
no_agent (1.366 ms) : 1346, 1386
.   : milestone, 1366,
appsec (1.758 ms) : 1734, 1782
.   : milestone, 1758,
appsec_no_iast (1.773 ms) : 1749, 1798
.   : milestone, 1773,
iast (1.496 ms) : 1473, 1520
.   : milestone, 1496,
profiling (1.519 ms) : 1496, 1542
.   : milestone, 1519,
tracing (1.509 ms) : 1485, 1534
.   : milestone, 1509,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.363 ms [1.344 ms, 1.383 ms]	-
appsec	1.756 ms [1.733 ms, 1.78 ms]	392.87 µs (28.8%)
appsec_no_iast	1.761 ms [1.738 ms, 1.784 ms]	397.553 µs (29.2%)
iast	1.509 ms [1.485 ms, 1.534 ms]	146.134 µs (10.7%)
profiling	1.519 ms [1.496 ms, 1.542 ms]	155.387 µs (11.4%)
tracing	1.486 ms [1.461 ms, 1.511 ms]	122.183 µs (9.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.366 ms [1.346 ms, 1.386 ms]	-
appsec	1.758 ms [1.734 ms, 1.782 ms]	391.37 µs (28.6%)
appsec_no_iast	1.773 ms [1.749 ms, 1.798 ms]	407.025 µs (29.8%)
iast	1.496 ms [1.473 ms, 1.52 ms]	130.226 µs (9.5%)
profiling	1.519 ms [1.496 ms, 1.542 ms]	152.751 µs (11.2%)
tracing	1.509 ms [1.485 ms, 1.534 ms]	143.001 µs (10.5%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~41bfd9f9e3, baseline=1.46.0-SNAPSHOT~fbb36f9b5b
    dateFormat X
    axisFormat %s
section baseline
no_agent (384.863 µs) : 364, 405
.   : milestone, 385,
iast (512.838 µs) : 491, 535
.   : milestone, 513,
iast_FULL (745.283 µs) : 723, 767
.   : milestone, 745,
iast_GLOBAL (562.149 µs) : 540, 585
.   : milestone, 562,
iast_HARDCODED_SECRET_DISABLED (517.247 µs) : 495, 539
.   : milestone, 517,
iast_INACTIVE (468.132 µs) : 446, 490
.   : milestone, 468,
iast_TELEMETRY_OFF (500.046 µs) : 478, 522
.   : milestone, 500,
tracing (452.92 µs) : 432, 474
.   : milestone, 453,
section candidate
no_agent (377.493 µs) : 358, 397
.   : milestone, 377,
iast (504.34 µs) : 483, 526
.   : milestone, 504,
iast_FULL (661.198 µs) : 640, 683
.   : milestone, 661,
iast_GLOBAL (523.029 µs) : 502, 544
.   : milestone, 523,
iast_HARDCODED_SECRET_DISABLED (501.525 µs) : 480, 523
.   : milestone, 502,
iast_INACTIVE (464.244 µs) : 443, 486
.   : milestone, 464,
iast_TELEMETRY_OFF (495.4 µs) : 473, 518
.   : milestone, 495,
tracing (452.552 µs) : 432, 473
.   : milestone, 453,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	384.863 µs [364.259 µs, 405.466 µs]	-
iast	512.838 µs [491.063 µs, 534.613 µs]	127.975 µs (33.3%)
iast_FULL	745.283 µs [723.467 µs, 767.099 µs]	360.421 µs (93.6%)
iast_GLOBAL	562.149 µs [539.648 µs, 584.65 µs]	177.286 µs (46.1%)
iast_HARDCODED_SECRET_DISABLED	517.247 µs [495.129 µs, 539.366 µs]	132.384 µs (34.4%)
iast_INACTIVE	468.132 µs [445.901 µs, 490.364 µs]	83.27 µs (21.6%)
iast_TELEMETRY_OFF	500.046 µs [478.045 µs, 522.048 µs]	115.183 µs (29.9%)
tracing	452.92 µs [432.317 µs, 473.523 µs]	68.058 µs (17.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	377.493 µs [357.952 µs, 397.035 µs]	-
iast	504.34 µs [482.911 µs, 525.769 µs]	126.846 µs (33.6%)
iast_FULL	661.198 µs [639.692 µs, 682.703 µs]	283.704 µs (75.2%)
iast_GLOBAL	523.029 µs [501.598 µs, 544.46 µs]	145.536 µs (38.6%)
iast_HARDCODED_SECRET_DISABLED	501.525 µs [480.236 µs, 522.814 µs]	124.032 µs (32.9%)
iast_INACTIVE	464.244 µs [442.867 µs, 485.621 µs]	86.751 µs (23.0%)
iast_TELEMETRY_OFF	495.4 µs [473.107 µs, 517.692 µs]	117.906 µs (31.2%)
tracing	452.552 µs [432.054 µs, 473.05 µs]	75.058 µs (19.9%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/impacted-tests
git_commit_date	1737124507	1737129672
git_commit_sha	`fbb36f9`	`41bfd9f`
release_version	1.46.0-SNAPSHOT~fbb36f9b5b	1.46.0-SNAPSHOT~41bfd9f9e3

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1737131597	1737131597
ci_job_id	771025766	771025766
ci_pipeline_id	53329237	53329237
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~41bfd9f9e3, baseline=1.46.0-SNAPSHOT~fbb36f9b5b
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.548 s) : 15548000, 15548000
.   : milestone, 15548000,
appsec (15.001 s) : 15001000, 15001000
.   : milestone, 15001000,
iast (19.044 s) : 19044000, 19044000
.   : milestone, 19044000,
iast_GLOBAL (18.034 s) : 18034000, 18034000
.   : milestone, 18034000,
profiling (15.578 s) : 15578000, 15578000
.   : milestone, 15578000,
tracing (14.931 s) : 14931000, 14931000
.   : milestone, 14931000,
section candidate
no_agent (15.161 s) : 15161000, 15161000
.   : milestone, 15161000,
appsec (15.077 s) : 15077000, 15077000
.   : milestone, 15077000,
iast (18.618 s) : 18618000, 18618000
.   : milestone, 18618000,
iast_GLOBAL (18.323 s) : 18323000, 18323000
.   : milestone, 18323000,
profiling (15.033 s) : 15033000, 15033000
.   : milestone, 15033000,
tracing (15.177 s) : 15177000, 15177000
.   : milestone, 15177000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.548 s [15.548 s, 15.548 s]	-
appsec	15.001 s [15.001 s, 15.001 s]	-547.0 ms (-3.5%)
iast	19.044 s [19.044 s, 19.044 s]	3.496 s (22.5%)
iast_GLOBAL	18.034 s [18.034 s, 18.034 s]	2.486 s (16.0%)
profiling	15.578 s [15.578 s, 15.578 s]	30.0 ms (0.2%)
tracing	14.931 s [14.931 s, 14.931 s]	-617.0 ms (-4.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.161 s [15.161 s, 15.161 s]	-
appsec	15.077 s [15.077 s, 15.077 s]	-84.0 ms (-0.6%)
iast	18.618 s [18.618 s, 18.618 s]	3.457 s (22.8%)
iast_GLOBAL	18.323 s [18.323 s, 18.323 s]	3.162 s (20.9%)
profiling	15.033 s [15.033 s, 15.033 s]	-128.0 ms (-0.8%)
tracing	15.177 s [15.177 s, 15.177 s]	16.0 ms (0.1%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~41bfd9f9e3, baseline=1.46.0-SNAPSHOT~fbb36f9b5b
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.475 ms) : 1463, 1486
.   : milestone, 1475,
appsec (2.366 ms) : 2323, 2409
.   : milestone, 2366,
iast (2.113 ms) : 2059, 2167
.   : milestone, 2113,
iast_GLOBAL (2.15 ms) : 2095, 2204
.   : milestone, 2150,
profiling (1.966 ms) : 1923, 2009
.   : milestone, 1966,
tracing (1.956 ms) : 1914, 1997
.   : milestone, 1956,
section candidate
no_agent (1.477 ms) : 1466, 1489
.   : milestone, 1477,
appsec (2.361 ms) : 2318, 2403
.   : milestone, 2361,
iast (2.108 ms) : 2054, 2163
.   : milestone, 2108,
iast_GLOBAL (2.151 ms) : 2097, 2206
.   : milestone, 2151,
profiling (1.974 ms) : 1930, 2019
.   : milestone, 1974,
tracing (1.949 ms) : 1907, 1991
.   : milestone, 1949,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.475 ms [1.463 ms, 1.486 ms]	-
appsec	2.366 ms [2.323 ms, 2.409 ms]	891.311 µs (60.4%)
iast	2.113 ms [2.059 ms, 2.167 ms]	638.025 µs (43.3%)
iast_GLOBAL	2.15 ms [2.095 ms, 2.204 ms]	675.395 µs (45.8%)
profiling	1.966 ms [1.923 ms, 2.009 ms]	491.755 µs (33.3%)
tracing	1.956 ms [1.914 ms, 1.997 ms]	481.23 µs (32.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.466 ms, 1.489 ms]	-
appsec	2.361 ms [2.318 ms, 2.403 ms]	883.262 µs (59.8%)
iast	2.108 ms [2.054 ms, 2.163 ms]	631.071 µs (42.7%)
iast_GLOBAL	2.151 ms [2.097 ms, 2.206 ms]	674.019 µs (45.6%)
profiling	1.974 ms [1.93 ms, 2.019 ms]	497.2 µs (33.7%)
tracing	1.949 ms [1.907 ms, 1.991 ms]	471.579 µs (31.9%)

…tection

…tion

…+ extract a no-op git client implementation to avoid spamming logs with exceptions in envs where git executable is not available

Pass test source data to retry policy factory

b079a82

nikita-tkachenko-datadog added tag: no release notes Changes to exclude from release notes type: refactoring comp: ci visibility Continuous Integration Visibility labels Jan 13, 2025

nikita-tkachenko-datadog added 5 commits January 15, 2025 17:13

Collect pull request info for GitHub and Gitlab

10bec41

Implement git diff fetching and parsing

12ea7fe

Add PR diff to execution settings

f79d619

Implement EFD for modified tests

e7aa623

Implement backend config setting and killswitch for impacted tests de…

8a25380

…tection

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/impacted-tests branch from 0cfd616 to 8a25380 Compare January 16, 2025 17:49

nikita-tkachenko-datadog added 4 commits January 16, 2025 19:23

Implement is_modified telemetry tag

4b8a054

Extract Diff interface and implement polymorphic serialization

78b6b6b

Implement file-granularity diff

4e959f9

Implement fallback to file-level granularity for impacted tests detec…

94804ac

…tion

nikita-tkachenko-datadog changed the title ~~Pass test source data to retry policy factory~~ Implement impacted tests detection Jan 16, 2025

nikita-tkachenko-datadog added type: enhancement and removed tag: no release notes Changes to exclude from release notes type: refactoring labels Jan 16, 2025

nikita-tkachenko-datadog added 3 commits January 17, 2025 12:00

Ensure git repo is not shallow before calculating git diff

a7eb1d6

Use commit SHA from backend diff response if no PR info is available …

3289b88

…+ extract a no-op git client implementation to avoid spamming logs with exceptions in envs where git executable is not available

Fix spotBugs warning

41bfd9f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement impacted tests detection #8188

Implement impacted tests detection #8188

nikita-tkachenko-datadog commented Jan 13, 2025 •

edited

Loading

pr-commenter bot commented Jan 13, 2025 •

edited

Loading

Implement impacted tests detection #8188

Are you sure you want to change the base?

Implement impacted tests detection #8188

Conversation

nikita-tkachenko-datadog commented Jan 13, 2025 • edited Loading

What Does This Do

Motivation

Contributor Checklist

pr-commenter bot commented Jan 13, 2025 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

nikita-tkachenko-datadog commented Jan 13, 2025 •

edited

Loading

pr-commenter bot commented Jan 13, 2025 •

edited

Loading