[MLOB] add LLM obs configs #8076
Open
+213
−12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
from
3863387 to
19d2dc3
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 2 performance improvements and 1 performance regressions! Performance is the same for 54 metrics, 6 unstable metrics.
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~5259c28871, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1053783
Total [baseline] (10.455 s) : 0, 10454638
Agent [candidate] (1.053 s) : 0, 1052781
Total [candidate] (10.436 s) : 0, 10435736
section appsec
Agent [baseline] (1.185 s) : 0, 1184755
Total [baseline] (10.722 s) : 0, 10722266
Agent [candidate] (1.185 s) : 0, 1185398
Total [candidate] (10.755 s) : 0, 10755448
section iast
Agent [baseline] (1.186 s) : 0, 1185705
Total [baseline] (11.009 s) : 0, 11009239
Agent [candidate] (1.178 s) : 0, 1177864
Total [candidate] (11.065 s) : 0, 11064912
section profiling
Agent [baseline] (1.281 s) : 0, 1281381
Total [baseline] (10.861 s) : 0, 10860765
Agent [candidate] (1.275 s) : 0, 1274626
Total [candidate] (10.825 s) : 0, 10824824
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~5259c28871, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (715.567 ms) : 0, 715567
BytebuddyAgent [candidate] (710.681 ms) : 0, 710681
GlobalTracer [baseline] (255.954 ms) : 0, 255954
GlobalTracer [candidate] (255.875 ms) : 0, 255875
AppSec [baseline] (55.047 ms) : 0, 55047
AppSec [candidate] (59.271 ms) : 0, 59271
Remote Config [baseline] (708.823 µs) : 0, 709
Remote Config [candidate] (682.038 µs) : 0, 682
Telemetry [baseline] (11.389 ms) : 0, 11389
Telemetry [candidate] (11.356 ms) : 0, 11356
section appsec
BytebuddyAgent [baseline] (727.636 ms) : 0, 727636
BytebuddyAgent [candidate] (728.584 ms) : 0, 728584
GlobalTracer [baseline] (253.171 ms) : 0, 253171
GlobalTracer [candidate] (252.512 ms) : 0, 252512
AppSec [baseline] (170.66 ms) : 0, 170660
AppSec [candidate] (170.869 ms) : 0, 170869
IAST [baseline] (19.303 ms) : 0, 19303
IAST [candidate] (19.137 ms) : 0, 19137
Remote Config [baseline] (666.202 µs) : 0, 666
Remote Config [candidate] (651.25 µs) : 0, 651
Telemetry [baseline] (8.033 ms) : 0, 8033
Telemetry [candidate] (8.276 ms) : 0, 8276
section iast
BytebuddyAgent [baseline] (833.497 ms) : 0, 833497
BytebuddyAgent [candidate] (827.699 ms) : 0, 827699
GlobalTracer [baseline] (247.657 ms) : 0, 247657
GlobalTracer [candidate] (246.227 ms) : 0, 246227
AppSec [baseline] (58.462 ms) : 0, 58462
AppSec [candidate] (58.502 ms) : 0, 58502
IAST [baseline] (21.684 ms) : 0, 21684
IAST [candidate] (21.272 ms) : 0, 21272
Remote Config [baseline] (676.565 µs) : 0, 677
Remote Config [candidate] (657.576 µs) : 0, 658
Telemetry [baseline] (8.658 ms) : 0, 8658
Telemetry [candidate] (8.554 ms) : 0, 8554
section profiling
BytebuddyAgent [baseline] (706.792 ms) : 0, 706792
BytebuddyAgent [candidate] (701.781 ms) : 0, 701781
GlobalTracer [baseline] (373.262 ms) : 0, 373262
GlobalTracer [candidate] (372.48 ms) : 0, 372480
AppSec [baseline] (54.56 ms) : 0, 54560
AppSec [candidate] (54.103 ms) : 0, 54103
Remote Config [baseline] (678.307 µs) : 0, 678
Remote Config [candidate] (684.85 µs) : 0, 685
Telemetry [baseline] (7.811 ms) : 0, 7811
Telemetry [candidate] (7.884 ms) : 0, 7884
ProfilingAgent [baseline] (96.236 ms) : 0, 96236
ProfilingAgent [candidate] (95.874 ms) : 0, 95874
Profiling [baseline] (96.262 ms) : 0, 96262
Profiling [candidate] (95.899 ms) : 0, 95899
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~5259c28871, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1050831
Total [baseline] (8.576 s) : 0, 8575839
Agent [candidate] (1.056 s) : 0, 1055583
Total [candidate] (8.589 s) : 0, 8588814
section iast
Agent [baseline] (1.178 s) : 0, 1177614
Total [baseline] (9.188 s) : 0, 9187710
Agent [candidate] (1.181 s) : 0, 1181135
Total [candidate] (9.194 s) : 0, 9193671
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.179 s) : 0, 1179446
Total [baseline] (9.167 s) : 0, 9167050
Agent [candidate] (1.189 s) : 0, 1189188
Total [candidate] (9.236 s) : 0, 9236047
section iast_TELEMETRY_OFF
Agent [baseline] (1.182 s) : 0, 1181892
Total [baseline] (9.211 s) : 0, 9211499
Agent [candidate] (1.173 s) : 0, 1172918
Total [candidate] (9.191 s) : 0, 9191217
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~5259c28871, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (711.213 ms) : 0, 711213
BytebuddyAgent [candidate] (714.543 ms) : 0, 714543
GlobalTracer [baseline] (256.312 ms) : 0, 256312
GlobalTracer [candidate] (256.358 ms) : 0, 256358
AppSec [baseline] (54.929 ms) : 0, 54929
AppSec [candidate] (58.002 ms) : 0, 58002
Remote Config [baseline] (708.827 µs) : 0, 709
Remote Config [candidate] (679.008 µs) : 0, 679
Telemetry [baseline] (12.688 ms) : 0, 12688
Telemetry [candidate] (11.118 ms) : 0, 11118
section iast
BytebuddyAgent [baseline] (827.817 ms) : 0, 827817
BytebuddyAgent [candidate] (830.916 ms) : 0, 830916
GlobalTracer [baseline] (246.25 ms) : 0, 246250
GlobalTracer [candidate] (247.143 ms) : 0, 247143
AppSec [baseline] (57.753 ms) : 0, 57753
AppSec [candidate] (57.803 ms) : 0, 57803
IAST [baseline] (21.54 ms) : 0, 21540
IAST [candidate] (21.155 ms) : 0, 21155
Remote Config [baseline] (665.828 µs) : 0, 666
Remote Config [candidate] (648.964 µs) : 0, 649
Telemetry [baseline] (8.633 ms) : 0, 8633
Telemetry [candidate] (8.533 ms) : 0, 8533
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (828.497 ms) : 0, 828497
BytebuddyAgent [candidate] (835.161 ms) : 0, 835161
GlobalTracer [baseline] (247.076 ms) : 0, 247076
GlobalTracer [candidate] (249.925 ms) : 0, 249925
AppSec [baseline] (58.206 ms) : 0, 58206
AppSec [candidate] (58.253 ms) : 0, 58253
IAST [baseline] (21.423 ms) : 0, 21423
IAST [candidate] (21.436 ms) : 0, 21436
Remote Config [baseline] (654.297 µs) : 0, 654
Remote Config [candidate] (666.853 µs) : 0, 667
Telemetry [baseline] (8.585 ms) : 0, 8585
Telemetry [candidate] (8.701 ms) : 0, 8701
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (830.149 ms) : 0, 830149
BytebuddyAgent [candidate] (824.236 ms) : 0, 824236
GlobalTracer [baseline] (247.922 ms) : 0, 247922
GlobalTracer [candidate] (246.67 ms) : 0, 246670
AppSec [baseline] (58.388 ms) : 0, 58388
AppSec [candidate] (57.392 ms) : 0, 57392
IAST [baseline] (21.114 ms) : 0, 21114
IAST [candidate] (20.666 ms) : 0, 20666
Remote Config [baseline] (682.807 µs) : 0, 683
Remote Config [candidate] (642.893 µs) : 0, 643
Telemetry [baseline] (8.587 ms) : 0, 8587
Telemetry [candidate] (8.42 ms) : 0, 8420
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~5259c28871, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section baseline
no_agent (1.347 ms) : 1326, 1368
. : milestone, 1347,
appsec (1.772 ms) : 1749, 1795
. : milestone, 1772,
appsec_no_iast (1.739 ms) : 1714, 1764
. : milestone, 1739,
iast (1.51 ms) : 1488, 1532
. : milestone, 1510,
profiling (1.521 ms) : 1496, 1546
. : milestone, 1521,
tracing (1.477 ms) : 1452, 1501
. : milestone, 1477,
section candidate
no_agent (1.361 ms) : 1342, 1381
. : milestone, 1361,
appsec (1.778 ms) : 1753, 1803
. : milestone, 1778,
appsec_no_iast (1.756 ms) : 1731, 1782
. : milestone, 1756,
iast (1.49 ms) : 1467, 1513
. : milestone, 1490,
profiling (1.536 ms) : 1511, 1561
. : milestone, 1536,
tracing (1.494 ms) : 1469, 1518
. : milestone, 1494,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~5259c28871, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section baseline
no_agent (373.424 µs) : 354, 393
. : milestone, 373,
iast (497.255 µs) : 475, 520
. : milestone, 497,
iast_FULL (656.571 µs) : 635, 678
. : milestone, 657,
iast_GLOBAL (524.439 µs) : 502, 547
. : milestone, 524,
iast_HARDCODED_SECRET_DISABLED (489.322 µs) : 468, 511
. : milestone, 489,
iast_INACTIVE (446.81 µs) : 426, 467
. : milestone, 447,
iast_TELEMETRY_OFF (478.528 µs) : 457, 500
. : milestone, 479,
tracing (455.074 µs) : 433, 477
. : milestone, 455,
section candidate
no_agent (377.297 µs) : 358, 397
. : milestone, 377,
iast (494.355 µs) : 473, 516
. : milestone, 494,
iast_FULL (654.48 µs) : 633, 676
. : milestone, 654,
iast_GLOBAL (527.628 µs) : 505, 550
. : milestone, 528,
iast_HARDCODED_SECRET_DISABLED (491.168 µs) : 470, 513
. : milestone, 491,
iast_INACTIVE (453.241 µs) : 432, 475
. : milestone, 453,
iast_TELEMETRY_OFF (480.283 µs) : 459, 502
. : milestone, 480,
tracing (456.416 µs) : 436, 477
. : milestone, 456,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~5259c28871, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section baseline
no_agent (1.471 ms) : 1459, 1482
. : milestone, 1471,
appsec (2.346 ms) : 2304, 2388
. : milestone, 2346,
iast (2.082 ms) : 2029, 2135
. : milestone, 2082,
iast_GLOBAL (2.13 ms) : 2077, 2183
. : milestone, 2130,
profiling (1.946 ms) : 1904, 1988
. : milestone, 1946,
tracing (1.931 ms) : 1891, 1971
. : milestone, 1931,
section candidate
no_agent (1.468 ms) : 1457, 1480
. : milestone, 1468,
appsec (2.342 ms) : 2301, 2384
. : milestone, 2342,
iast (2.085 ms) : 2032, 2138
. : milestone, 2085,
iast_GLOBAL (2.126 ms) : 2073, 2179
. : milestone, 2126,
profiling (1.96 ms) : 1917, 2002
. : milestone, 1960,
tracing (1.925 ms) : 1884, 1965
. : milestone, 1925,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~5259c28871, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section baseline
no_agent (15.19 s) : 15190000, 15190000
. : milestone, 15190000,
appsec (15.145 s) : 15145000, 15145000
. : milestone, 15145000,
iast (18.834 s) : 18834000, 18834000
. : milestone, 18834000,
iast_GLOBAL (17.956 s) : 17956000, 17956000
. : milestone, 17956000,
profiling (15.749 s) : 15749000, 15749000
. : milestone, 15749000,
tracing (15.17 s) : 15170000, 15170000
. : milestone, 15170000,
section candidate
no_agent (15.305 s) : 15305000, 15305000
. : milestone, 15305000,
appsec (15.014 s) : 15014000, 15014000
. : milestone, 15014000,
iast (18.743 s) : 18743000, 18743000
. : milestone, 18743000,
iast_GLOBAL (17.989 s) : 17989000, 17989000
. : milestone, 17989000,
profiling (15.05 s) : 15050000, 15050000
. : milestone, 15050000,
tracing (14.858 s) : 14858000, 14858000
. : milestone, 14858000,
|
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
2 times, most recently
from
fd136e0 to
ba03bdd
Compare
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
from
ba03bdd to
a8108c6
Compare
|
Hi! 👋 Looks like you updated a Git Submodule.
|
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
Yun-Kim
approved these changes
Dec 30, 2024
Comment on lines
+2673
to
+2678
| public boolean isLlmObsAgentlessEnabled() { | ||
| return llmObsAgentlessEnabled; | ||
| } | ||
|
|
||
| public String getLlmObsMlApp() { | ||
| return llmObsMlApp; |
There was a problem hiding this comment.
Are these two options not set on the instrumenterConfig object? We're fine to just store this directly on the Config instance?
There was a problem hiding this comment.
i think it is fine to store these in the config instance for now, i am not sure but it is possible we may need them, instrumenterConfig seems to be about starting actual implementations classes if a product is enabled, i think we can add them later as needed
There was a problem hiding this comment.
Yup that's fine with me to add them later when needed. Thanks for clarifying
| */ | ||
| public final class LlmObsConfig { | ||
|
|
||
| public static final String LLM_OBS_ENABLED = "llmobs.enabled"; |
There was a problem hiding this comment.
Tiny nit, I think the other two language SDKs use the naming pattern LLMOBS_ instead of LLM_OBS_, would be nice to stay consistent but not that serious so up to you
|
Hi! 👋 Looks like you updated a Git Submodule.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
adds llm observability related configs and some start up behavior that are related to the enablement of llm observability
dd.llmobs.enabledis set to true and there is no ml app defined - throw errordd.llmobs.agentless.enabledis set to true and there is no API key supplied - throw errorjava system property success cases
agent mode configuration
agentless mode configuration with API key env var
env variable success cases
agent mode configuration
agentless mode configuration
Motivation
Additional Notes
tested with a sample spring app
the following stack satisfies 1
the follow stack satisfies 2
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]