Fail fast when tracer versions do not match in parent and child processes #6168

nikita-tkachenko-datadog · 2023-11-08T11:18:01Z

What Does This Do

Adds a check to ensure that the tracer injected into the parent CI Vis process (build system) has the same version as the tracer injected into the child CI Vis process (JVM forked for tests execution).

Motivation

Whenever a build is executed with the tracer injected, every JVM forked by that build will be configured to run with the same tracer. The configuration is done automatically.
However it is also possible to have manual tracer injection configured in a project (this typically happens in the projects that adopted CI Visibility before build system instrumentation was implemented, or in the projects that add the tracer to the classpath for using other APM products).
When both manual and automatic injections are in place, it is possible to end up having different tracer versions in the parent process and in the child processes.
This can cause compatibility problem since the parent and the children exchange data, whose format changes over time.
Therefore we need to signal these problems by failing fast.

Jira ticket: CIVIS-7959

…sses

pr-commenter · 2023-11-08T11:56:51Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.23.0-SNAPSHOT~2e6162bf24	1.23.0-SNAPSHOT~2731f448d3
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 cases.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.23.0-SNAPSHOT~2731f448d3, baseline=1.23.0-SNAPSHOT~2e6162bf24

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.044 s) : 0, 1044258
Total [baseline] (8.801 s) : 0, 8800699
Agent [candidate] (1.043 s) : 0, 1042541
Total [candidate] (8.832 s) : 0, 8831776
section iast
Agent [baseline] (1.16 s) : 0, 1159761
Total [baseline] (9.331 s) : 0, 9330549
Agent [candidate] (1.16 s) : 0, 1160036
Total [candidate] (9.376 s) : 0, 9376107
section iast_TELEMETRY_OFF
Agent [baseline] (1.145 s) : 0, 1145398
Total [baseline] (9.307 s) : 0, 9306877
Agent [candidate] (1.156 s) : 0, 1156496
Total [candidate] (9.31 s) : 0, 9310385

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.044 s	-
Agent	iast	1.16 s	115.502 ms (11.1%)
Agent	iast_TELEMETRY_OFF	1.145 s	101.14 ms (9.7%)
Total	tracing	8.801 s	-
Total	iast	9.331 s	529.851 ms (6.0%)
Total	iast_TELEMETRY_OFF	9.307 s	506.178 ms (5.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	iast	1.16 s	117.495 ms (11.3%)
Agent	iast_TELEMETRY_OFF	1.156 s	113.955 ms (10.9%)
Total	tracing	8.832 s	-
Total	iast	9.376 s	544.33 ms (6.2%)
Total	iast_TELEMETRY_OFF	9.31 s	478.609 ms (5.4%)

gantt
    title insecure-bank - break down per module: candidate=1.23.0-SNAPSHOT~2731f448d3, baseline=1.23.0-SNAPSHOT~2e6162bf24

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (651.723 ms) : 0, 651723
BytebuddyAgent [candidate] (650.539 ms) : 0, 650539
GlobalTracer [baseline] (296.301 ms) : 0, 296301
GlobalTracer [candidate] (296.121 ms) : 0, 296121
AppSec [baseline] (49.264 ms) : 0, 49264
AppSec [candidate] (49.016 ms) : 0, 49016
Remote Config [baseline] (717.397 µs) : 0, 717
Remote Config [candidate] (703.224 µs) : 0, 703
Telemetry [baseline] (11.544 ms) : 0, 11544
Telemetry [candidate] (11.349 ms) : 0, 11349
section iast
BytebuddyAgent [baseline] (770.983 ms) : 0, 770983
BytebuddyAgent [candidate] (771.614 ms) : 0, 771614
GlobalTracer [baseline] (275.9 ms) : 0, 275900
GlobalTracer [candidate] (276.632 ms) : 0, 276632
AppSec [baseline] (46.983 ms) : 0, 46983
AppSec [candidate] (47.199 ms) : 0, 47199
Remote Config [baseline] (595.881 µs) : 0, 596
Remote Config [candidate] (567.286 µs) : 0, 567
Telemetry [baseline] (14.195 ms) : 0, 14195
Telemetry [candidate] (13.196 ms) : 0, 13196
IAST [baseline] (16.422 ms) : 0, 16422
IAST [candidate] (16.212 ms) : 0, 16212
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (760.057 ms) : 0, 760057
BytebuddyAgent [candidate] (766.509 ms) : 0, 766509
GlobalTracer [baseline] (274.529 ms) : 0, 274529
GlobalTracer [candidate] (276.925 ms) : 0, 276925
AppSec [baseline] (46.454 ms) : 0, 46454
AppSec [candidate] (46.941 ms) : 0, 46941
Remote Config [baseline] (570.832 µs) : 0, 571
Remote Config [candidate] (572.101 µs) : 0, 572
Telemetry [baseline] (13.359 ms) : 0, 13359
Telemetry [candidate] (13.982 ms) : 0, 13982
IAST [baseline] (15.995 ms) : 0, 15995
IAST [candidate] (16.768 ms) : 0, 16768

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.23.0-SNAPSHOT~2731f448d3, baseline=1.23.0-SNAPSHOT~2e6162bf24

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.037 s) : 0, 1036830
Total [baseline] (9.312 s) : 0, 9311605
Agent [candidate] (1.037 s) : 0, 1037391
Total [candidate] (9.28 s) : 0, 9280426
section appsec
Agent [baseline] (1.136 s) : 0, 1135760
Total [baseline] (9.479 s) : 0, 9478554
Agent [candidate] (1.121 s) : 0, 1120985
Total [candidate] (9.426 s) : 0, 9426403
section iast
Agent [baseline] (1.161 s) : 0, 1160934
Total [baseline] (9.554 s) : 0, 9554285
Agent [candidate] (1.161 s) : 0, 1161301
Total [candidate] (9.493 s) : 0, 9492596
section profiling
Agent [baseline] (1.222 s) : 0, 1221935
Total [baseline] (9.523 s) : 0, 9523327
Agent [candidate] (1.221 s) : 0, 1220501
Total [candidate] (9.558 s) : 0, 9558234

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.037 s	-
Agent	appsec	1.136 s	98.93 ms (9.5%)
Agent	iast	1.161 s	124.104 ms (12.0%)
Agent	profiling	1.222 s	185.105 ms (17.9%)
Total	tracing	9.312 s	-
Total	appsec	9.479 s	166.949 ms (1.8%)
Total	iast	9.554 s	242.68 ms (2.6%)
Total	profiling	9.523 s	211.722 ms (2.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.037 s	-
Agent	appsec	1.121 s	83.593 ms (8.1%)
Agent	iast	1.161 s	123.91 ms (11.9%)
Agent	profiling	1.221 s	183.11 ms (17.7%)
Total	tracing	9.28 s	-
Total	appsec	9.426 s	145.977 ms (1.6%)
Total	iast	9.493 s	212.17 ms (2.3%)
Total	profiling	9.558 s	277.807 ms (3.0%)

gantt
    title petclinic - break down per module: candidate=1.23.0-SNAPSHOT~2731f448d3, baseline=1.23.0-SNAPSHOT~2e6162bf24

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (646.639 ms) : 0, 646639
BytebuddyAgent [candidate] (647.176 ms) : 0, 647176
GlobalTracer [baseline] (294.797 ms) : 0, 294797
GlobalTracer [candidate] (295.011 ms) : 0, 295011
AppSec [baseline] (48.789 ms) : 0, 48789
AppSec [candidate] (48.658 ms) : 0, 48658
Remote Config [baseline] (705.714 µs) : 0, 706
Remote Config [candidate] (695.602 µs) : 0, 696
Telemetry [baseline] (11.391 ms) : 0, 11391
Telemetry [candidate] (11.344 ms) : 0, 11344
section appsec
BytebuddyAgent [baseline] (655.651 ms) : 0, 655651
BytebuddyAgent [candidate] (646.331 ms) : 0, 646331
GlobalTracer [baseline] (298.126 ms) : 0, 298126
GlobalTracer [candidate] (294.447 ms) : 0, 294447
AppSec [baseline] (139.46 ms) : 0, 139460
AppSec [candidate] (138.216 ms) : 0, 138216
Remote Config [baseline] (668.18 µs) : 0, 668
Remote Config [candidate] (644.931 µs) : 0, 645
Telemetry [baseline] (6.954 ms) : 0, 6954
Telemetry [candidate] (6.855 ms) : 0, 6855
section iast
BytebuddyAgent [baseline] (772.357 ms) : 0, 772357
BytebuddyAgent [candidate] (772.49 ms) : 0, 772490
GlobalTracer [baseline] (276.098 ms) : 0, 276098
GlobalTracer [candidate] (277.177 ms) : 0, 277177
AppSec [baseline] (47.038 ms) : 0, 47038
AppSec [candidate] (47.258 ms) : 0, 47258
Remote Config [baseline] (577.809 µs) : 0, 578
Remote Config [candidate] (586.704 µs) : 0, 587
Telemetry [baseline] (12.582 ms) : 0, 12582
Telemetry [candidate] (10.582 ms) : 0, 10582
IAST [baseline] (17.567 ms) : 0, 17567
IAST [candidate] (18.464 ms) : 0, 18464
section profiling
BytebuddyAgent [baseline] (659.692 ms) : 0, 659692
BytebuddyAgent [candidate] (657.22 ms) : 0, 657220
GlobalTracer [baseline] (358.021 ms) : 0, 358021
GlobalTracer [candidate] (358.798 ms) : 0, 358798
AppSec [baseline] (49.372 ms) : 0, 49372
AppSec [candidate] (49.186 ms) : 0, 49186
Remote Config [baseline] (645.582 µs) : 0, 646
Remote Config [candidate] (650.514 µs) : 0, 651
Telemetry [baseline] (11.223 ms) : 0, 11223
Telemetry [candidate] (11.408 ms) : 0, 11408
ProfilingAgent [baseline] (88.433 ms) : 0, 88433
ProfilingAgent [candidate] (88.957 ms) : 0, 88957
Profiling [baseline] (88.456 ms) : 0, 88456
Profiling [candidate] (88.981 ms) : 0, 88981

Load

Parameters

	Baseline	Candidate
commit	1.23.0-SNAPSHOT~2e6162bf24	1.23.0-SNAPSHOT~2731f448d3
config	baseline	candidate
end_time	2023-11-09T12:24:56	2023-11-09T12:41:23
start_time	2023-11-09T12:24:43	2023-11-09T12:41:10

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 22 cases.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.23.0-SNAPSHOT~2731f448d3, baseline=1.23.0-SNAPSHOT~2e6162bf24
    dateFormat X
    axisFormat %s
section baseline
no_agent (360.407 µs) : 340, 381
.   : milestone, 360,
iast (453.942 µs) : 433, 475
.   : milestone, 454,
iast_FULL (522.79 µs) : 502, 543
.   : milestone, 523,
iast_INACTIVE (434.265 µs) : 413, 455
.   : milestone, 434,
iast_TELEMETRY_OFF (451.358 µs) : 431, 472
.   : milestone, 451,
tracing (443.769 µs) : 422, 465
.   : milestone, 444,
section candidate
no_agent (360.748 µs) : 341, 381
.   : milestone, 361,
iast (461.343 µs) : 441, 482
.   : milestone, 461,
iast_FULL (523.999 µs) : 503, 545
.   : milestone, 524,
iast_INACTIVE (432.719 µs) : 412, 453
.   : milestone, 433,
iast_TELEMETRY_OFF (456.555 µs) : 436, 477
.   : milestone, 457,
tracing (431.656 µs) : 411, 452
.   : milestone, 432,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	360.407 µs [340.074 µs, 380.739 µs]	-
iast	453.942 µs [432.996 µs, 474.889 µs]	93.536 µs (26.0%)
iast_FULL	522.79 µs [502.195 µs, 543.385 µs]	162.384 µs (45.1%)
iast_INACTIVE	434.265 µs [413.228 µs, 455.303 µs]	73.859 µs (20.5%)
iast_TELEMETRY_OFF	451.358 µs [430.842 µs, 471.873 µs]	90.951 µs (25.2%)
tracing	443.769 µs [422.186 µs, 465.352 µs]	83.362 µs (23.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	360.748 µs [340.653 µs, 380.843 µs]	-
iast	461.343 µs [440.965 µs, 481.721 µs]	100.595 µs (27.9%)
iast_FULL	523.999 µs [503.142 µs, 544.855 µs]	163.251 µs (45.3%)
iast_INACTIVE	432.719 µs [412.291 µs, 453.147 µs]	71.971 µs (20.0%)
iast_TELEMETRY_OFF	456.555 µs [435.755 µs, 477.355 µs]	95.807 µs (26.6%)
tracing	431.656 µs [410.94 µs, 452.372 µs]	70.908 µs (19.7%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.23.0-SNAPSHOT~2731f448d3, baseline=1.23.0-SNAPSHOT~2e6162bf24
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.341 ms) : 1322, 1360
.   : milestone, 1341,
appsec (1.706 ms) : 1681, 1732
.   : milestone, 1706,
iast (1.477 ms) : 1453, 1502
.   : milestone, 1477,
profiling (1.455 ms) : 1429, 1480
.   : milestone, 1455,
tracing (1.446 ms) : 1422, 1471
.   : milestone, 1446,
section candidate
no_agent (1.353 ms) : 1334, 1372
.   : milestone, 1353,
appsec (1.7 ms) : 1675, 1724
.   : milestone, 1700,
iast (1.489 ms) : 1465, 1513
.   : milestone, 1489,
profiling (1.453 ms) : 1428, 1478
.   : milestone, 1453,
tracing (1.456 ms) : 1431, 1481
.   : milestone, 1456,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.341 ms [1.322 ms, 1.36 ms]	-
appsec	1.706 ms [1.681 ms, 1.732 ms]	365.452 µs (27.3%)
iast	1.477 ms [1.453 ms, 1.502 ms]	136.325 µs (10.2%)
profiling	1.455 ms [1.429 ms, 1.48 ms]	113.666 µs (8.5%)
tracing	1.446 ms [1.422 ms, 1.471 ms]	105.15 µs (7.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.353 ms [1.334 ms, 1.372 ms]	-
appsec	1.7 ms [1.675 ms, 1.724 ms]	346.418 µs (25.6%)
iast	1.489 ms [1.465 ms, 1.513 ms]	135.849 µs (10.0%)
profiling	1.453 ms [1.428 ms, 1.478 ms]	99.861 µs (7.4%)
tracing	1.456 ms [1.431 ms, 1.481 ms]	103.087 µs (7.6%)

…on-mismatch

Fix error when trying to get class stream for Mockito mocks (#6183) Fix error when serializing skippable tests whose names contain non-ASCII characters (#6182) Fail fast when tracer versions do not match in parent and child processes (#6168) initial instrumentation

…sses (#6168)

Fail fast when tracer versions do not match in parent and child proce…

7893b87

…sses

nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Nov 8, 2023

nikita-tkachenko-datadog marked this pull request as ready for review November 8, 2023 13:16

nikita-tkachenko-datadog requested review from a team as code owners November 8, 2023 13:16

nikita-tkachenko-datadog requested review from anmarchenko, mcculls and am312 November 8, 2023 13:16

Merge branch 'master' into nikita-tkachenko/fail-fast-on-tracer-versi…

2731f44

…on-mismatch

juan-fernandez approved these changes Nov 13, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit 3fb888b into master Nov 13, 2023
11 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/fail-fast-on-tracer-version-mismatch branch November 13, 2023 11:56

github-actions bot added this to the 1.24.0 milestone Nov 13, 2023

jandro996 pushed a commit that referenced this pull request Nov 14, 2023

Fail fast when tracer versions do not match in parent and child proce…

f0f76c8

…sses (#6168)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fail fast when tracer versions do not match in parent and child processes #6168

Fail fast when tracer versions do not match in parent and child processes #6168

nikita-tkachenko-datadog commented Nov 8, 2023 •

edited by jira bot

Loading

pr-commenter bot commented Nov 8, 2023 •

edited

Loading

Fail fast when tracer versions do not match in parent and child processes #6168

Fail fast when tracer versions do not match in parent and child processes #6168

Conversation

nikita-tkachenko-datadog commented Nov 8, 2023 • edited by jira bot Loading

What Does This Do

Motivation

pr-commenter bot commented Nov 8, 2023 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

nikita-tkachenko-datadog commented Nov 8, 2023 •

edited by jira bot

Loading

pr-commenter bot commented Nov 8, 2023 •

edited

Loading