Add pending traces to tracer flare #8053
Draft
+196
−8
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 11 performance improvements and 14 performance regressions! Performance is the same for 31 metrics, 7 unstable metrics.
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~e6ad777dc5, baseline=1.46.0-SNAPSHOT~148aef92ab
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1053643
Total [baseline] (10.432 s) : 0, 10431520
Agent [candidate] (1.098 s) : 0, 1098062
Total [candidate] (10.439 s) : 0, 10438897
section appsec
Agent [baseline] (1.189 s) : 0, 1189224
Total [baseline] (10.702 s) : 0, 10702024
Agent [candidate] (1.237 s) : 0, 1236540
Total [candidate] (10.729 s) : 0, 10729477
section iast
Agent [baseline] (1.185 s) : 0, 1185183
Total [baseline] (10.969 s) : 0, 10968584
Agent [candidate] (1.224 s) : 0, 1223747
Total [candidate] (10.985 s) : 0, 10985166
section profiling
Agent [baseline] (1.265 s) : 0, 1264559
Total [baseline] (10.882 s) : 0, 10881718
Agent [candidate] (1.324 s) : 0, 1324139
Total [candidate] (10.869 s) : 0, 10869096
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~e6ad777dc5, baseline=1.46.0-SNAPSHOT~148aef92ab
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.545 ms) : 0, 713545
BytebuddyAgent [candidate] (697.402 ms) : 0, 697402
GlobalTracer [baseline] (255.183 ms) : 0, 255183
GlobalTracer [candidate] (319.768 ms) : 0, 319768
AppSec [baseline] (56.189 ms) : 0, 56189
AppSec [candidate] (55.026 ms) : 0, 55026
Remote Config [baseline] (726.973 µs) : 0, 727
Remote Config [candidate] (687.142 µs) : 0, 687
Telemetry [baseline] (13.096 ms) : 0, 13096
Telemetry [candidate] (11.39 ms) : 0, 11390
section appsec
BytebuddyAgent [baseline] (731.857 ms) : 0, 731857
BytebuddyAgent [candidate] (718.485 ms) : 0, 718485
GlobalTracer [baseline] (253.098 ms) : 0, 253098
GlobalTracer [candidate] (318.416 ms) : 0, 318416
AppSec [baseline] (170.691 ms) : 0, 170691
AppSec [candidate] (167.436 ms) : 0, 167436
IAST [baseline] (19.381 ms) : 0, 19381
IAST [candidate] (19.152 ms) : 0, 19152
Remote Config [baseline] (666.698 µs) : 0, 667
Remote Config [candidate] (644.316 µs) : 0, 644
Telemetry [baseline] (8.235 ms) : 0, 8235
Telemetry [candidate] (8.126 ms) : 0, 8126
section iast
BytebuddyAgent [baseline] (833.672 ms) : 0, 833672
BytebuddyAgent [candidate] (814.875 ms) : 0, 814875
GlobalTracer [baseline] (247.103 ms) : 0, 247103
GlobalTracer [candidate] (308.182 ms) : 0, 308182
AppSec [baseline] (58.157 ms) : 0, 58157
AppSec [candidate] (57.056 ms) : 0, 57056
IAST [baseline] (21.653 ms) : 0, 21653
IAST [candidate] (21.746 ms) : 0, 21746
Remote Config [baseline] (685.301 µs) : 0, 685
Remote Config [candidate] (624.134 µs) : 0, 624
Telemetry [baseline] (8.872 ms) : 0, 8872
Telemetry [candidate] (7.447 ms) : 0, 7447
section profiling
BytebuddyAgent [baseline] (709.07 ms) : 0, 709070
BytebuddyAgent [candidate] (693.075 ms) : 0, 693075
GlobalTracer [baseline] (350.848 ms) : 0, 350848
GlobalTracer [candidate] (435.796 ms) : 0, 435796
AppSec [baseline] (55.6 ms) : 0, 55600
AppSec [candidate] (53.823 ms) : 0, 53823
Remote Config [baseline] (669.373 µs) : 0, 669
Remote Config [candidate] (651.81 µs) : 0, 652
Telemetry [baseline] (9.001 ms) : 0, 9001
Telemetry [candidate] (7.747 ms) : 0, 7747
ProfilingAgent [baseline] (96.984 ms) : 0, 96984
ProfilingAgent [candidate] (93.666 ms) : 0, 93666
Profiling [baseline] (97.008 ms) : 0, 97008
Profiling [candidate] (93.69 ms) : 0, 93690
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~e6ad777dc5, baseline=1.46.0-SNAPSHOT~148aef92ab
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1057499
Total [baseline] (8.652 s) : 0, 8652278
Agent [candidate] (1.092 s) : 0, 1091772
Total [candidate] (8.634 s) : 0, 8634152
section iast
Agent [baseline] (1.184 s) : 0, 1184212
Total [baseline] (9.265 s) : 0, 9264867
Agent [candidate] (1.222 s) : 0, 1222344
Total [candidate] (9.181 s) : 0, 9181325
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.191 s) : 0, 1191182
Total [baseline] (9.189 s) : 0, 9189252
Agent [candidate] (1.224 s) : 0, 1223792
Total [candidate] (9.179 s) : 0, 9178628
section iast_TELEMETRY_OFF
Agent [baseline] (1.186 s) : 0, 1185742
Total [baseline] (9.191 s) : 0, 9191066
Agent [candidate] (1.218 s) : 0, 1218380
Total [candidate] (9.194 s) : 0, 9194085
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~e6ad777dc5, baseline=1.46.0-SNAPSHOT~148aef92ab
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (714.744 ms) : 0, 714744
BytebuddyAgent [candidate] (695.621 ms) : 0, 695621
GlobalTracer [baseline] (255.903 ms) : 0, 255903
GlobalTracer [candidate] (319.05 ms) : 0, 319050
AppSec [baseline] (56.171 ms) : 0, 56171
AppSec [candidate] (54.249 ms) : 0, 54249
Remote Config [baseline] (717.758 µs) : 0, 718
Remote Config [candidate] (680.006 µs) : 0, 680
Telemetry [baseline] (15.071 ms) : 0, 15071
Telemetry [candidate] (8.434 ms) : 0, 8434
section iast
BytebuddyAgent [baseline] (833.027 ms) : 0, 833027
BytebuddyAgent [candidate] (814.024 ms) : 0, 814024
GlobalTracer [baseline] (246.978 ms) : 0, 246978
GlobalTracer [candidate] (307.864 ms) : 0, 307864
AppSec [baseline] (58.14 ms) : 0, 58140
AppSec [candidate] (56.861 ms) : 0, 56861
IAST [baseline] (21.514 ms) : 0, 21514
IAST [candidate] (21.736 ms) : 0, 21736
Remote Config [baseline] (675.006 µs) : 0, 675
Remote Config [candidate] (616.352 µs) : 0, 616
Telemetry [baseline] (8.836 ms) : 0, 8836
Telemetry [candidate] (7.487 ms) : 0, 7487
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.611 ms) : 0, 837611
BytebuddyAgent [candidate] (814.522 ms) : 0, 814522
GlobalTracer [baseline] (248.391 ms) : 0, 248391
GlobalTracer [candidate] (308.048 ms) : 0, 308048
AppSec [baseline] (58.581 ms) : 0, 58581
AppSec [candidate] (57.305 ms) : 0, 57305
IAST [baseline] (22.002 ms) : 0, 22002
IAST [candidate] (22.057 ms) : 0, 22057
Remote Config [baseline] (666.49 µs) : 0, 666
Remote Config [candidate] (614.455 µs) : 0, 614
Telemetry [baseline] (8.856 ms) : 0, 8856
Telemetry [candidate] (7.485 ms) : 0, 7485
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (835.622 ms) : 0, 835622
BytebuddyAgent [candidate] (810.748 ms) : 0, 810748
GlobalTracer [baseline] (246.978 ms) : 0, 246978
GlobalTracer [candidate] (307.402 ms) : 0, 307402
AppSec [baseline] (57.899 ms) : 0, 57899
AppSec [candidate] (57.818 ms) : 0, 57818
IAST [baseline] (20.849 ms) : 0, 20849
IAST [candidate] (20.612 ms) : 0, 20612
Remote Config [baseline] (659.19 µs) : 0, 659
Remote Config [candidate] (630.319 µs) : 0, 630
Telemetry [baseline] (8.606 ms) : 0, 8606
Telemetry [candidate] (7.407 ms) : 0, 7407
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 16 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~e6ad777dc5, baseline=1.46.0-SNAPSHOT~148aef92ab
dateFormat X
axisFormat %s
section baseline
no_agent (380.402 µs) : 361, 400
. : milestone, 380,
iast (508.747 µs) : 487, 530
. : milestone, 509,
iast_FULL (744.325 µs) : 722, 766
. : milestone, 744,
iast_GLOBAL (560.425 µs) : 537, 583
. : milestone, 560,
iast_HARDCODED_SECRET_DISABLED (505.03 µs) : 484, 526
. : milestone, 505,
iast_INACTIVE (461.732 µs) : 440, 483
. : milestone, 462,
iast_TELEMETRY_OFF (505.801 µs) : 484, 527
. : milestone, 506,
tracing (451.692 µs) : 431, 473
. : milestone, 452,
section candidate
no_agent (371.751 µs) : 352, 391
. : milestone, 372,
iast (494.589 µs) : 473, 516
. : milestone, 495,
iast_FULL (663.409 µs) : 642, 685
. : milestone, 663,
iast_GLOBAL (526.964 µs) : 505, 549
. : milestone, 527,
iast_HARDCODED_SECRET_DISABLED (497.746 µs) : 476, 519
. : milestone, 498,
iast_INACTIVE (457.976 µs) : 437, 479
. : milestone, 458,
iast_TELEMETRY_OFF (490.671 µs) : 469, 512
. : milestone, 491,
tracing (457.474 µs) : 436, 479
. : milestone, 457,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~e6ad777dc5, baseline=1.46.0-SNAPSHOT~148aef92ab
dateFormat X
axisFormat %s
section baseline
no_agent (1.367 ms) : 1347, 1387
. : milestone, 1367,
appsec (1.76 ms) : 1736, 1784
. : milestone, 1760,
appsec_no_iast (1.761 ms) : 1735, 1786
. : milestone, 1761,
iast (1.512 ms) : 1488, 1537
. : milestone, 1512,
profiling (1.551 ms) : 1526, 1576
. : milestone, 1551,
tracing (1.499 ms) : 1474, 1524
. : milestone, 1499,
section candidate
no_agent (1.352 ms) : 1332, 1371
. : milestone, 1352,
appsec (1.77 ms) : 1747, 1794
. : milestone, 1770,
appsec_no_iast (1.741 ms) : 1717, 1766
. : milestone, 1741,
iast (1.509 ms) : 1486, 1533
. : milestone, 1509,
profiling (1.533 ms) : 1509, 1556
. : milestone, 1533,
tracing (1.496 ms) : 1470, 1521
. : milestone, 1496,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~e6ad777dc5, baseline=1.46.0-SNAPSHOT~148aef92ab
dateFormat X
axisFormat %s
section baseline
no_agent (1.467 ms) : 1455, 1478
. : milestone, 1467,
appsec (2.347 ms) : 2304, 2389
. : milestone, 2347,
iast (2.094 ms) : 2040, 2148
. : milestone, 2094,
iast_GLOBAL (2.142 ms) : 2087, 2197
. : milestone, 2142,
profiling (1.975 ms) : 1931, 2019
. : milestone, 1975,
tracing (1.932 ms) : 1890, 1974
. : milestone, 1932,
section candidate
no_agent (1.465 ms) : 1453, 1476
. : milestone, 1465,
appsec (2.335 ms) : 2293, 2376
. : milestone, 2335,
iast (2.08 ms) : 2028, 2133
. : milestone, 2080,
iast_GLOBAL (2.129 ms) : 2076, 2182
. : milestone, 2129,
profiling (1.959 ms) : 1917, 2002
. : milestone, 1959,
tracing (1.924 ms) : 1884, 1964
. : milestone, 1924,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~e6ad777dc5, baseline=1.46.0-SNAPSHOT~148aef92ab
dateFormat X
axisFormat %s
section baseline
no_agent (14.977 s) : 14977000, 14977000
. : milestone, 14977000,
appsec (28.576 s) : 28576000, 28576000
. : milestone, 28576000,
iast (19.047 s) : 19047000, 19047000
. : milestone, 19047000,
iast_GLOBAL (18.042 s) : 18042000, 18042000
. : milestone, 18042000,
profiling (15.218 s) : 15218000, 15218000
. : milestone, 15218000,
tracing (14.888 s) : 14888000, 14888000
. : milestone, 14888000,
section candidate
no_agent (15.257 s) : 15257000, 15257000
. : milestone, 15257000,
appsec (15.214 s) : 15214000, 15214000
. : milestone, 15214000,
iast (18.809 s) : 18809000, 18809000
. : milestone, 18809000,
iast_GLOBAL (18.191 s) : 18191000, 18191000
. : milestone, 18191000,
profiling (15.15 s) : 15150000, 15150000
. : milestone, 15150000,
tracing (14.999 s) : 14999000, 14999000
. : milestone, 14999000,
|
dd-trace-core/src/main/java/datadog/trace/core/flare/TracerFlareService.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
PerfectSlayer
changed the title
mcculls
reviewed
Dec 12, 2024
mcculls
reviewed
Dec 12, 2024
mcculls
reviewed
Dec 12, 2024
dd-trace-core/src/main/java/datadog/trace/core/PendingTrace.java
Outdated
Show resolved
Hide resolved
mcculls
reviewed
Dec 12, 2024
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
|
Have you considered tackling this another way, by adding the reporting to |
mhlidd
force-pushed
the
mhlidd/pending_traces
branch
from
4be2999 to
3e4955a
Compare
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| DumpSupplier supplier = new DumpSupplier(DumpDrain.data); | ||
| buffer.queue.fill(supplier, supplier.data.size()); |
There was a problem hiding this comment.
I wonder if filling the queue back with the original element can't be done when handling the dump element.
It would limit having new element inserted in the same time and messing with the queue order. WDYT?
There was a problem hiding this comment.
If so, does it make sense having the same object implement both Consummer and Supplier?
There was a problem hiding this comment.
Will add this in the new commit!
| if (e instanceof PendingTrace) { | ||
| PendingTrace trace = (PendingTrace) e; | ||
| for (DDSpan span : trace.getSpans()) { | ||
| dumpText.append(span.toString()).append("\n"); |
There was a problem hiding this comment.
'\n' for single char (instead of using a String object)
There was a problem hiding this comment.
Have drain and fill together feels way better! 🙌
Now, about using the collected elements:
- You can declare your comparator as a
private final Comparator<List<Span>> TRACE_BY_START_DATEfield so its meaning will be clear and it will only be allocated once - You will need to figure out how to filter, limit, and map your elements. I would recommend having a look at the Java
StreamAPI - Now you get most of the part in place, try building some tests / triggering your code. You should be able to find some NPE by yourself 😉
| @@ -182,11 +242,18 @@ public void run() { | |||
|
|
|||
| if (pendingTrace instanceof FlushElement) { | |||
| // Since this is an MPSC queue, the drain needs to be called on the consumer thread | |||
| queue.drain(WriteDrain.WRITE_DRAIN); | |||
| queue.drain(WriteDrain.WRITE_DRAIN, 50); | |||
There was a problem hiding this comment.
Hmm I was trying to use the MPSC drain method that takes in a limit to the number of elements that get drained. Do you think that I should avoid doing this and using the Java stream API instead?
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: APMAPI-906