Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add caught/uncaught exception smoke tests #6191

Merged
merged 1 commit into from
Nov 10, 2023
Merged

Add caught/uncaught exception smoke tests #6191

merged 1 commit into from
Nov 10, 2023

Conversation

jpbempel
Copy link
Member

@jpbempel jpbempel commented Nov 10, 2023 •
edited by jira bot
Loading

What Does This Do

Motivation

Additional Notes

Jira ticket: DEBUG-1462

@jpbempel jpbempel added comp: testing Testing tag: no release notes Changes to exclude from release notes comp: debugger Dynamic Instrumentation labels Nov 10, 2023
@jpbempel jpbempel requested a review from a team as a code owner November 10, 2023 10:47
@jpbempel jpbempel requested review from shatzi and removed request for a team November 10, 2023 10:47
@pr-commenter
Copy link

pr-commenter bot commented Nov 10, 2023

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.24.0-SNAPSHOT~24a60cb17a 1.24.0-SNAPSHOT~fce52e8416
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant iast iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 53 cases.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:insecure-bank:iast:Telemetry better
[-6.536ms; -0.424ms] or [-42.376%; -2.748%]		 11.944ms 15.424ms
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.24.0-SNAPSHOT~fce52e8416, baseline=1.24.0-SNAPSHOT~24a60cb17a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.037 s) : 0, 1037083
Total [baseline] (9.281 s) : 0, 9280802
Agent [candidate] (1.041 s) : 0, 1041271
Total [candidate] (9.285 s) : 0, 9285470
section appsec
Agent [baseline] (1.121 s) : 0, 1120660
Total [baseline] (9.408 s) : 0, 9407885
Agent [candidate] (1.125 s) : 0, 1124729
Total [candidate] (9.431 s) : 0, 9430859
section iast
Agent [baseline] (1.154 s) : 0, 1154271
Total [baseline] (9.511 s) : 0, 9511048
Agent [candidate] (1.161 s) : 0, 1161405
Total [candidate] (9.46 s) : 0, 9460199
section profiling
Agent [baseline] (1.228 s) : 0, 1227912
Total [baseline] (9.537 s) : 0, 9537442
Agent [candidate] (1.225 s) : 0, 1225315
Total [candidate] (9.519 s) : 0, 9519328
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.037 s -
Agent appsec 1.121 s 83.577 ms (8.1%)
Agent iast 1.154 s 117.188 ms (11.3%)
Agent profiling 1.228 s 190.828 ms (18.4%)
Total tracing 9.281 s -
Total appsec 9.408 s 127.082 ms (1.4%)
Total iast 9.511 s 230.245 ms (2.5%)
Total profiling 9.537 s 256.639 ms (2.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.041 s -
Agent appsec 1.125 s 83.458 ms (8.0%)
Agent iast 1.161 s 120.134 ms (11.5%)
Agent profiling 1.225 s 184.044 ms (17.7%)
Total tracing 9.285 s -
Total appsec 9.431 s 145.389 ms (1.6%)
Total iast 9.46 s 174.728 ms (1.9%)
Total profiling 9.519 s 233.858 ms (2.5%) 
gantt
    title petclinic - break down per module: candidate=1.24.0-SNAPSHOT~fce52e8416, baseline=1.24.0-SNAPSHOT~24a60cb17a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (646.922 ms) : 0, 646922
BytebuddyAgent [candidate] (649.919 ms) : 0, 649919
GlobalTracer [baseline] (294.534 ms) : 0, 294534
GlobalTracer [candidate] (295.78 ms) : 0, 295780
AppSec [baseline] (49.068 ms) : 0, 49068
AppSec [candidate] (49.057 ms) : 0, 49057
Remote Config [baseline] (700.295 µs) : 0, 700
Remote Config [candidate] (701.122 µs) : 0, 701
Telemetry [baseline] (11.403 ms) : 0, 11403
Telemetry [candidate] (11.244 ms) : 0, 11244
section appsec
BytebuddyAgent [baseline] (646.511 ms) : 0, 646511
BytebuddyAgent [candidate] (647.039 ms) : 0, 647039
GlobalTracer [baseline] (293.75 ms) : 0, 293750
GlobalTracer [candidate] (296.437 ms) : 0, 296437
AppSec [baseline] (138.42 ms) : 0, 138420
AppSec [candidate] (139.268 ms) : 0, 139268
Remote Config [baseline] (645.231 µs) : 0, 645
Remote Config [candidate] (654.663 µs) : 0, 655
Telemetry [baseline] (6.844 ms) : 0, 6844
Telemetry [candidate] (6.915 ms) : 0, 6915
section iast
BytebuddyAgent [baseline] (768.217 ms) : 0, 768217
BytebuddyAgent [candidate] (771.334 ms) : 0, 771334
GlobalTracer [baseline] (274.364 ms) : 0, 274364
GlobalTracer [candidate] (276.148 ms) : 0, 276148
AppSec [baseline] (46.812 ms) : 0, 46812
AppSec [candidate] (47.078 ms) : 0, 47078
Remote Config [baseline] (570.552 µs) : 0, 571
Remote Config [candidate] (589.659 µs) : 0, 590
Telemetry [baseline] (11.231 ms) : 0, 11231
Telemetry [candidate] (11.789 ms) : 0, 11789
IAST [baseline] (18.369 ms) : 0, 18369
IAST [candidate] (19.846 ms) : 0, 19846
section profiling
BytebuddyAgent [baseline] (661.813 ms) : 0, 661813
BytebuddyAgent [candidate] (660.963 ms) : 0, 660963
GlobalTracer [baseline] (361.007 ms) : 0, 361007
GlobalTracer [candidate] (359.693 ms) : 0, 359693
AppSec [baseline] (49.34 ms) : 0, 49340
AppSec [candidate] (49.295 ms) : 0, 49295
Remote Config [baseline] (649.039 µs) : 0, 649
Remote Config [candidate] (661.252 µs) : 0, 661
Telemetry [baseline] (11.418 ms) : 0, 11418
Telemetry [candidate] (11.369 ms) : 0, 11369
ProfilingAgent [baseline] (88.971 ms) : 0, 88971
ProfilingAgent [candidate] (88.6 ms) : 0, 88600
Profiling [baseline] (88.995 ms) : 0, 88995
Profiling [candidate] (88.624 ms) : 0, 88624
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.24.0-SNAPSHOT~fce52e8416, baseline=1.24.0-SNAPSHOT~24a60cb17a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.042 s) : 0, 1042278
Total [baseline] (8.8 s) : 0, 8800009
Agent [candidate] (1.043 s) : 0, 1043169
Total [candidate] (8.832 s) : 0, 8832458
section iast
Agent [baseline] (1.151 s) : 0, 1151456
Total [baseline] (9.33 s) : 0, 9330046
Agent [candidate] (1.159 s) : 0, 1159371
Total [candidate] (9.391 s) : 0, 9390557
section iast_TELEMETRY_OFF
Agent [baseline] (1.143 s) : 0, 1143495
Total [baseline] (9.295 s) : 0, 9295045
Agent [candidate] (1.148 s) : 0, 1148028
Total [candidate] (9.315 s) : 0, 9315006
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.042 s -
Agent iast 1.151 s 109.178 ms (10.5%)
Agent iast_TELEMETRY_OFF 1.143 s 101.217 ms (9.7%)
Total tracing 8.8 s -
Total iast 9.33 s 530.037 ms (6.0%)
Total iast_TELEMETRY_OFF 9.295 s 495.036 ms (5.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.043 s -
Agent iast 1.159 s 116.202 ms (11.1%)
Agent iast_TELEMETRY_OFF 1.148 s 104.859 ms (10.1%)
Total tracing 8.832 s -
Total iast 9.391 s 558.099 ms (6.3%)
Total iast_TELEMETRY_OFF 9.315 s 482.549 ms (5.5%) 
gantt
    title insecure-bank - break down per module: candidate=1.24.0-SNAPSHOT~fce52e8416, baseline=1.24.0-SNAPSHOT~24a60cb17a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (650.536 ms) : 0, 650536
BytebuddyAgent [candidate] (650.551 ms) : 0, 650551
GlobalTracer [baseline] (295.599 ms) : 0, 295599
GlobalTracer [candidate] (296.204 ms) : 0, 296204
AppSec [baseline] (49.035 ms) : 0, 49035
AppSec [candidate] (49.485 ms) : 0, 49485
Remote Config [baseline] (709.867 µs) : 0, 710
Remote Config [candidate] (707.025 µs) : 0, 707
Telemetry [baseline] (11.524 ms) : 0, 11524
Telemetry [candidate] (11.549 ms) : 0, 11549
section iast
BytebuddyAgent [baseline] (764.397 ms) : 0, 764397
BytebuddyAgent [candidate] (771.531 ms) : 0, 771531
GlobalTracer [baseline] (273.755 ms) : 0, 273755
GlobalTracer [candidate] (276.128 ms) : 0, 276128
AppSec [baseline] (46.684 ms) : 0, 46684
AppSec [candidate] (46.863 ms) : 0, 46863
IAST [baseline] (16.174 ms) : 0, 16174
IAST [candidate] (17.651 ms) : 0, 17651
Remote Config [baseline] (572.505 µs) : 0, 573
Remote Config [candidate] (577.549 µs) : 0, 578
Telemetry [baseline] (15.424 ms) : 0, 15424
Telemetry [candidate] (11.944 ms) : 0, 11944
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (758.183 ms) : 0, 758183
BytebuddyAgent [candidate] (759.706 ms) : 0, 759706
GlobalTracer [baseline] (273.73 ms) : 0, 273730
GlobalTracer [candidate] (276.391 ms) : 0, 276391
AppSec [baseline] (46.448 ms) : 0, 46448
AppSec [candidate] (47.16 ms) : 0, 47160
IAST [baseline] (16.887 ms) : 0, 16887
IAST [candidate] (18.385 ms) : 0, 18385
Remote Config [baseline] (570.809 µs) : 0, 571
Remote Config [candidate] (576.024 µs) : 0, 576
Telemetry [baseline] (13.277 ms) : 0, 13277
Telemetry [candidate] (11.303 ms) : 0, 11303
Loading

Load

Parameters

Baseline Candidate
commit 1.24.0-SNAPSHOT~24a60cb17a 1.24.0-SNAPSHOT~fce52e8416
config baseline candidate
end_time 2023-11-10T11:08:13 2023-11-10T11:24:40
start_time 2023-11-10T11:08:00 2023-11-10T11:24:28
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 22 cases.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.24.0-SNAPSHOT~fce52e8416, baseline=1.24.0-SNAPSHOT~24a60cb17a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.334 ms) : 1314, 1353
.   : milestone, 1334,
appsec (1.714 ms) : 1689, 1739
.   : milestone, 1714,
iast (1.483 ms) : 1459, 1507
.   : milestone, 1483,
profiling (1.439 ms) : 1414, 1464
.   : milestone, 1439,
tracing (1.469 ms) : 1445, 1494
.   : milestone, 1469,
section candidate
no_agent (1.335 ms) : 1316, 1354
.   : milestone, 1335,
appsec (1.738 ms) : 1713, 1763
.   : milestone, 1738,
iast (1.478 ms) : 1454, 1503
.   : milestone, 1478,
profiling (1.445 ms) : 1420, 1470
.   : milestone, 1445,
tracing (1.458 ms) : 1433, 1482
.   : milestone, 1458,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.334 ms [1.314 ms, 1.353 ms] -
appsec 1.714 ms [1.689 ms, 1.739 ms] 380.354 µs (28.5%)
iast 1.483 ms [1.459 ms, 1.507 ms] 149.444 µs (11.2%)
profiling 1.439 ms [1.414 ms, 1.464 ms] 105.259 µs (7.9%)
tracing 1.469 ms [1.445 ms, 1.494 ms] 135.477 µs (10.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.335 ms [1.316 ms, 1.354 ms] -
appsec 1.738 ms [1.713 ms, 1.763 ms] 402.847 µs (30.2%)
iast 1.478 ms [1.454 ms, 1.503 ms] 143.527 µs (10.8%)
profiling 1.445 ms [1.42 ms, 1.47 ms] 109.795 µs (8.2%)
tracing 1.458 ms [1.433 ms, 1.482 ms] 122.782 µs (9.2%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.24.0-SNAPSHOT~fce52e8416, baseline=1.24.0-SNAPSHOT~24a60cb17a
    dateFormat X
    axisFormat %s
section baseline
no_agent (361.629 µs) : 340, 383
.   : milestone, 362,
iast (458.119 µs) : 437, 479
.   : milestone, 458,
iast_FULL (517.406 µs) : 497, 538
.   : milestone, 517,
iast_INACTIVE (437.773 µs) : 416, 459
.   : milestone, 438,
iast_TELEMETRY_OFF (458.303 µs) : 436, 480
.   : milestone, 458,
tracing (436.275 µs) : 415, 458
.   : milestone, 436,
section candidate
no_agent (365.043 µs) : 345, 385
.   : milestone, 365,
iast (457.039 µs) : 436, 478
.   : milestone, 457,
iast_FULL (522.874 µs) : 502, 544
.   : milestone, 523,
iast_INACTIVE (430.755 µs) : 410, 452
.   : milestone, 431,
iast_TELEMETRY_OFF (454.246 µs) : 433, 476
.   : milestone, 454,
tracing (430.945 µs) : 410, 452
.   : milestone, 431,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 361.629 µs [340.093 µs, 383.165 µs] -
iast 458.119 µs [437.382 µs, 478.856 µs] 96.49 µs (26.7%)
iast_FULL 517.406 µs [496.711 µs, 538.101 µs] 155.777 µs (43.1%)
iast_INACTIVE 437.773 µs [416.078 µs, 459.467 µs] 76.144 µs (21.1%)
iast_TELEMETRY_OFF 458.303 µs [436.411 µs, 480.195 µs] 96.674 µs (26.7%)
tracing 436.275 µs [414.777 µs, 457.773 µs] 74.646 µs (20.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 365.043 µs [344.802 µs, 385.284 µs] -
iast 457.039 µs [436.145 µs, 477.933 µs] 91.996 µs (25.2%)
iast_FULL 522.874 µs [502.162 µs, 543.586 µs] 157.831 µs (43.2%)
iast_INACTIVE 430.755 µs [409.926 µs, 451.584 µs] 65.712 µs (18.0%)
iast_TELEMETRY_OFF 454.246 µs [432.867 µs, 475.625 µs] 89.203 µs (24.4%)
tracing 430.945 µs [410.018 µs, 451.872 µs] 65.902 µs (18.1%)

shatzi
shatzi approved these changes Nov 10, 2023
View reviewed changes
Copy link
Contributor

@shatzi shatzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good

...ts/debugger-integration-tests/src/test/java/datadog/smoketest/SpanProbesIntegrationTest.java
.probeId(PROBE_ID)
// from line: System.out.println("fullMethod");
// to line: + String.join(",", argVar);
.where(MAIN_CLASS_NAME, 88, 97)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit - would be cool if we could have a utility function that read the ".java" file (can we put it as a project resource?) and find the line number based on a regex.

from = Util.findLineInFile('DebuggerTestApplicaiton.java', RegEx('System.out.println("fullMethod")'))

@jpbempel jpbempel merged commit d7a0af0 into master Nov 10, 2023
12 of 13 checks passed
@jpbempel jpbempel deleted the jpbempel/exceptions-smoke-tests branch November 10, 2023 19:45
@github-actions github-actions bot added this to the 1.24.0 milestone Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shatzi shatzi shatzi approved these changes

Assignees
No one assigned
Labels
comp: debugger Dynamic Instrumentation comp: testing Testing tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.24.0
Development

Successfully merging this pull request may close these issues.
2 participants
@jpbempel @shatzi