Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix CI Visibility Manual API to not fail when there is no tracer injected #6192

Merged
merged 2 commits into from
Nov 14, 2023
Merged

Fix CI Visibility Manual API to not fail when there is no tracer injected #6192

merged 2 commits into from
Nov 14, 2023

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Nov 10, 2023 •
edited by jira bot
Loading

What Does This Do

Adds "no-op" implementations of CI Visibility Manual API interfaces, and ensures that they're used by default if no tracer is injected (or if CI Visibility is disabled).

Motivation

Customers use the Manual API interfaces in their codebase to manually instrument testing frameworks that we do not support.
The Manual API code is going to be in their codebase regardless of whether the tracer is present or not.
If there's no tracer present (e.g. this is a local run), customer's code should not be failing.

Jira ticket: CIVIS-7974

@nikita-tkachenko-datadog nikita-tkachenko-datadog added type: bug comp: ci visibility Continuous Integration Visibility labels Nov 10, 2023
@pr-commenter
Copy link

pr-commenter bot commented Nov 10, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.24.0-SNAPSHOT~5dce4fdae6 1.24.0-SNAPSHOT~044def1469
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 cases.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.24.0-SNAPSHOT~044def1469, baseline=1.24.0-SNAPSHOT~5dce4fdae6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.038 s) : 0, 1038296
Total [baseline] (8.806 s) : 0, 8805694
Agent [candidate] (1.043 s) : 0, 1043108
Total [candidate] (8.821 s) : 0, 8820775
section iast
Agent [baseline] (1.151 s) : 0, 1150818
Total [baseline] (9.352 s) : 0, 9351902
Agent [candidate] (1.162 s) : 0, 1161723
Total [candidate] (9.332 s) : 0, 9331565
section iast_TELEMETRY_OFF
Agent [baseline] (1.149 s) : 0, 1149315
Total [baseline] (9.347 s) : 0, 9346841
Agent [candidate] (1.151 s) : 0, 1151390
Total [candidate] (9.318 s) : 0, 9318211
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.038 s -
Agent iast 1.151 s 112.523 ms (10.8%)
Agent iast_TELEMETRY_OFF 1.149 s 111.019 ms (10.7%)
Total tracing 8.806 s -
Total iast 9.352 s 546.207 ms (6.2%)
Total iast_TELEMETRY_OFF 9.347 s 541.146 ms (6.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.043 s -
Agent iast 1.162 s 118.615 ms (11.4%)
Agent iast_TELEMETRY_OFF 1.151 s 108.282 ms (10.4%)
Total tracing 8.821 s -
Total iast 9.332 s 510.79 ms (5.8%)
Total iast_TELEMETRY_OFF 9.318 s 497.436 ms (5.6%) 
gantt
    title insecure-bank - break down per module: candidate=1.24.0-SNAPSHOT~044def1469, baseline=1.24.0-SNAPSHOT~5dce4fdae6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (647.983 ms) : 0, 647983
BytebuddyAgent [candidate] (650.313 ms) : 0, 650313
GlobalTracer [baseline] (294.774 ms) : 0, 294774
GlobalTracer [candidate] (296.474 ms) : 0, 296474
AppSec [baseline] (48.888 ms) : 0, 48888
AppSec [candidate] (49.39 ms) : 0, 49390
Remote Config [baseline] (695.309 µs) : 0, 695
Remote Config [candidate] (696.496 µs) : 0, 696
Telemetry [baseline] (11.334 ms) : 0, 11334
Telemetry [candidate] (11.515 ms) : 0, 11515
section iast
BytebuddyAgent [baseline] (765.048 ms) : 0, 765048
BytebuddyAgent [candidate] (772.378 ms) : 0, 772378
GlobalTracer [baseline] (274.09 ms) : 0, 274090
GlobalTracer [candidate] (276.521 ms) : 0, 276521
AppSec [baseline] (46.548 ms) : 0, 46548
AppSec [candidate] (47.067 ms) : 0, 47067
IAST [baseline] (17.633 ms) : 0, 17633
IAST [candidate] (17.353 ms) : 0, 17353
Remote Config [baseline] (574.387 µs) : 0, 574
Remote Config [candidate] (566.802 µs) : 0, 567
Telemetry [baseline] (12.516 ms) : 0, 12516
Telemetry [candidate] (13.083 ms) : 0, 13083
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (762.004 ms) : 0, 762004
BytebuddyAgent [candidate] (763.366 ms) : 0, 763366
GlobalTracer [baseline] (275.215 ms) : 0, 275215
GlobalTracer [candidate] (276.17 ms) : 0, 276170
AppSec [baseline] (46.331 ms) : 0, 46331
AppSec [candidate] (46.459 ms) : 0, 46459
IAST [baseline] (18.762 ms) : 0, 18762
IAST [candidate] (15.517 ms) : 0, 15517
Remote Config [baseline] (581.072 µs) : 0, 581
Remote Config [candidate] (571.691 µs) : 0, 572
Telemetry [baseline] (11.834 ms) : 0, 11834
Telemetry [candidate] (14.719 ms) : 0, 14719
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.24.0-SNAPSHOT~044def1469, baseline=1.24.0-SNAPSHOT~5dce4fdae6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.037 s) : 0, 1037034
Total [baseline] (9.392 s) : 0, 9391804
Agent [candidate] (1.046 s) : 0, 1046195
Total [candidate] (9.334 s) : 0, 9333671
section appsec
Agent [baseline] (1.121 s) : 0, 1121444
Total [baseline] (9.427 s) : 0, 9426925
Agent [candidate] (1.126 s) : 0, 1126387
Total [candidate] (9.448 s) : 0, 9447872
section iast
Agent [baseline] (1.159 s) : 0, 1159486
Total [baseline] (9.493 s) : 0, 9493077
Agent [candidate] (1.163 s) : 0, 1163176
Total [candidate] (9.555 s) : 0, 9554530
section profiling
Agent [baseline] (1.22 s) : 0, 1220169
Total [baseline] (9.56 s) : 0, 9560476
Agent [candidate] (1.231 s) : 0, 1230997
Total [candidate] (9.607 s) : 0, 9606596
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.037 s -
Agent appsec 1.121 s 84.411 ms (8.1%)
Agent iast 1.159 s 122.452 ms (11.8%)
Agent profiling 1.22 s 183.136 ms (17.7%)
Total tracing 9.392 s -
Total appsec 9.427 s 35.121 ms (0.4%)
Total iast 9.493 s 101.273 ms (1.1%)
Total profiling 9.56 s 168.672 ms (1.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.046 s -
Agent appsec 1.126 s 80.192 ms (7.7%)
Agent iast 1.163 s 116.982 ms (11.2%)
Agent profiling 1.231 s 184.802 ms (17.7%)
Total tracing 9.334 s -
Total appsec 9.448 s 114.202 ms (1.2%)
Total iast 9.555 s 220.859 ms (2.4%)
Total profiling 9.607 s 272.925 ms (2.9%) 
gantt
    title petclinic - break down per module: candidate=1.24.0-SNAPSHOT~044def1469, baseline=1.24.0-SNAPSHOT~5dce4fdae6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (646.87 ms) : 0, 646870
BytebuddyAgent [candidate] (652.098 ms) : 0, 652098
GlobalTracer [baseline] (294.849 ms) : 0, 294849
GlobalTracer [candidate] (298.072 ms) : 0, 298072
AppSec [baseline] (48.798 ms) : 0, 48798
AppSec [candidate] (49.233 ms) : 0, 49233
Remote Config [baseline] (690.745 µs) : 0, 691
Remote Config [candidate] (705.952 µs) : 0, 706
Telemetry [baseline] (11.273 ms) : 0, 11273
Telemetry [candidate] (11.413 ms) : 0, 11413
section appsec
BytebuddyAgent [baseline] (646.767 ms) : 0, 646767
BytebuddyAgent [candidate] (648.578 ms) : 0, 648578
GlobalTracer [baseline] (293.923 ms) : 0, 293923
GlobalTracer [candidate] (296.338 ms) : 0, 296338
AppSec [baseline] (138.917 ms) : 0, 138917
AppSec [candidate] (139.387 ms) : 0, 139387
Remote Config [baseline] (646.152 µs) : 0, 646
Remote Config [candidate] (649.712 µs) : 0, 650
Telemetry [baseline] (6.853 ms) : 0, 6853
Telemetry [candidate] (6.935 ms) : 0, 6935
section iast
BytebuddyAgent [baseline] (772.174 ms) : 0, 772174
BytebuddyAgent [candidate] (773.848 ms) : 0, 773848
GlobalTracer [baseline] (276.257 ms) : 0, 276257
GlobalTracer [candidate] (277.321 ms) : 0, 277321
AppSec [baseline] (46.848 ms) : 0, 46848
AppSec [candidate] (47.145 ms) : 0, 47145
Remote Config [baseline] (576.959 µs) : 0, 577
Remote Config [candidate] (581.527 µs) : 0, 582
Telemetry [baseline] (12.584 ms) : 0, 12584
Telemetry [candidate] (11.294 ms) : 0, 11294
IAST [baseline] (16.441 ms) : 0, 16441
IAST [candidate] (18.314 ms) : 0, 18314
section profiling
ProfilingAgent [baseline] (88.615 ms) : 0, 88615
ProfilingAgent [candidate] (89.106 ms) : 0, 89106
BytebuddyAgent [baseline] (657.199 ms) : 0, 657199
BytebuddyAgent [candidate] (663.922 ms) : 0, 663922
GlobalTracer [baseline] (359.286 ms) : 0, 359286
GlobalTracer [candidate] (361.774 ms) : 0, 361774
AppSec [baseline] (48.844 ms) : 0, 48844
AppSec [candidate] (49.299 ms) : 0, 49299
Remote Config [baseline] (638.918 µs) : 0, 639
Remote Config [candidate] (650.229 µs) : 0, 650
Telemetry [baseline] (11.265 ms) : 0, 11265
Telemetry [candidate] (11.467 ms) : 0, 11467
Profiling [baseline] (88.64 ms) : 0, 88640
Profiling [candidate] (89.13 ms) : 0, 89130
Loading

Load

Parameters

Baseline Candidate
commit 1.24.0-SNAPSHOT~5dce4fdae6 1.24.0-SNAPSHOT~044def1469
config baseline candidate
end_time 2023-11-13T15:51:54 2023-11-13T16:08:24
start_time 2023-11-13T15:51:41 2023-11-13T16:08:11
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 22 cases.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.24.0-SNAPSHOT~044def1469, baseline=1.24.0-SNAPSHOT~5dce4fdae6
    dateFormat X
    axisFormat %s
section baseline
no_agent (363.703 µs) : 343, 385
.   : milestone, 364,
iast (462.322 µs) : 441, 483
.   : milestone, 462,
iast_FULL (530.013 µs) : 509, 551
.   : milestone, 530,
iast_INACTIVE (439.852 µs) : 419, 461
.   : milestone, 440,
iast_TELEMETRY_OFF (460.281 µs) : 439, 481
.   : milestone, 460,
tracing (437.946 µs) : 417, 459
.   : milestone, 438,
section candidate
no_agent (369.345 µs) : 349, 390
.   : milestone, 369,
iast (469.688 µs) : 448, 491
.   : milestone, 470,
iast_FULL (531.554 µs) : 511, 552
.   : milestone, 532,
iast_INACTIVE (435.41 µs) : 414, 456
.   : milestone, 435,
iast_TELEMETRY_OFF (462.864 µs) : 441, 484
.   : milestone, 463,
tracing (435.061 µs) : 414, 456
.   : milestone, 435,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 363.703 µs [342.819 µs, 384.588 µs] -
iast 462.322 µs [441.434 µs, 483.21 µs] 98.619 µs (27.1%)
iast_FULL 530.013 µs [509.272 µs, 550.754 µs] 166.31 µs (45.7%)
iast_INACTIVE 439.852 µs [418.707 µs, 460.997 µs] 76.149 µs (20.9%)
iast_TELEMETRY_OFF 460.281 µs [439.299 µs, 481.263 µs] 96.578 µs (26.6%)
tracing 437.946 µs [416.827 µs, 459.066 µs] 74.243 µs (20.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 369.345 µs [348.744 µs, 389.946 µs] -
iast 469.688 µs [448.438 µs, 490.938 µs] 100.343 µs (27.2%)
iast_FULL 531.554 µs [510.898 µs, 552.21 µs] 162.209 µs (43.9%)
iast_INACTIVE 435.41 µs [414.322 µs, 456.499 µs] 66.066 µs (17.9%)
iast_TELEMETRY_OFF 462.864 µs [441.499 µs, 484.23 µs] 93.519 µs (25.3%)
tracing 435.061 µs [414.206 µs, 455.916 µs] 65.716 µs (17.8%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.24.0-SNAPSHOT~044def1469, baseline=1.24.0-SNAPSHOT~5dce4fdae6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.343 ms) : 1324, 1362
.   : milestone, 1343,
appsec (1.734 ms) : 1709, 1759
.   : milestone, 1734,
iast (1.491 ms) : 1467, 1515
.   : milestone, 1491,
profiling (1.463 ms) : 1438, 1488
.   : milestone, 1463,
tracing (1.465 ms) : 1441, 1490
.   : milestone, 1465,
section candidate
no_agent (1.37 ms) : 1350, 1389
.   : milestone, 1370,
appsec (1.7 ms) : 1675, 1725
.   : milestone, 1700,
iast (1.463 ms) : 1439, 1487
.   : milestone, 1463,
profiling (1.477 ms) : 1451, 1503
.   : milestone, 1477,
tracing (1.472 ms) : 1448, 1497
.   : milestone, 1472,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.343 ms [1.324 ms, 1.362 ms] -
appsec 1.734 ms [1.709 ms, 1.759 ms] 390.956 µs (29.1%)
iast 1.491 ms [1.467 ms, 1.515 ms] 148.188 µs (11.0%)
profiling 1.463 ms [1.438 ms, 1.488 ms] 119.95 µs (8.9%)
tracing 1.465 ms [1.441 ms, 1.49 ms] 122.352 µs (9.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.37 ms [1.35 ms, 1.389 ms] -
appsec 1.7 ms [1.675 ms, 1.725 ms] 330.813 µs (24.2%)
iast 1.463 ms [1.439 ms, 1.487 ms] 93.495 µs (6.8%)
profiling 1.477 ms [1.451 ms, 1.503 ms] 107.427 µs (7.8%)
tracing 1.472 ms [1.448 ms, 1.497 ms] 102.765 µs (7.5%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review November 13, 2023 16:54
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit 712289c into master Nov 14, 2023
10 of 11 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/manual-api-without-tracer branch November 14, 2023 09:43
@github-actions github-actions bot added this to the 1.24.0 milestone Nov 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@romainkomorn-exdatadog romainkomorn-exdatadog romainkomorn-exdatadog approved these changes

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

@am312 am312 Awaiting requested review from am312 am312 is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility type: bug
Projects
None yet
Milestone
1.24.0
Development

Successfully merging this pull request may close these issues.
2 participants
@nikita-tkachenko-datadog @romainkomorn-exdatadog