feat: add capped sampling rate increases#10715
Draft
raphaelgavache wants to merge 1 commit intomasterfrom
Draft
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 62 metrics, 9 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~c95286e6a5, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1062826
Total [baseline] (10.919 s) : 0, 10918753
Agent [candidate] (1.07 s) : 0, 1070210
Total [candidate] (10.932 s) : 0, 10931866
section appsec
Agent [baseline] (1.241 s) : 0, 1241195
Total [baseline] (11.046 s) : 0, 11046435
Agent [candidate] (1.245 s) : 0, 1244610
Total [candidate] (11.037 s) : 0, 11037178
section iast
Agent [baseline] (1.23 s) : 0, 1230474
Total [baseline] (11.139 s) : 0, 11138941
Agent [candidate] (1.234 s) : 0, 1233930
Total [candidate] (11.183 s) : 0, 11183401
section profiling
Agent [baseline] (1.193 s) : 0, 1193453
Total [baseline] (10.92 s) : 0, 10919505
Agent [candidate] (1.197 s) : 0, 1196910
Total [candidate] (11.034 s) : 0, 11033651
gantt
title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~c95286e6a5, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.2 ms) : 0, 1200
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (627.763 ms) : 0, 627763
BytebuddyAgent [candidate] (628.567 ms) : 0, 628567
AgentMeter [baseline] (29.015 ms) : 0, 29015
AgentMeter [candidate] (29.37 ms) : 0, 29370
GlobalTracer [baseline] (257.282 ms) : 0, 257282
GlobalTracer [candidate] (259.887 ms) : 0, 259887
AppSec [baseline] (32.814 ms) : 0, 32814
AppSec [candidate] (33.262 ms) : 0, 33262
Debugger [baseline] (64.973 ms) : 0, 64973
Debugger [candidate] (65.204 ms) : 0, 65204
Remote Config [baseline] (599.523 µs) : 0, 600
Remote Config [candidate] (616.749 µs) : 0, 617
Telemetry [baseline] (9.063 ms) : 0, 9063
Telemetry [candidate] (10.746 ms) : 0, 10746
Flare Poller [baseline] (3.926 ms) : 0, 3926
Flare Poller [candidate] (5.249 ms) : 0, 5249
section appsec
crashtracking [baseline] (1.195 ms) : 0, 1195
crashtracking [candidate] (1.194 ms) : 0, 1194
BytebuddyAgent [baseline] (658.951 ms) : 0, 658951
BytebuddyAgent [candidate] (662.471 ms) : 0, 662471
AgentMeter [baseline] (12.063 ms) : 0, 12063
AgentMeter [candidate] (12.109 ms) : 0, 12109
GlobalTracer [baseline] (259.396 ms) : 0, 259396
GlobalTracer [candidate] (260.206 ms) : 0, 260206
AppSec [baseline] (167.739 ms) : 0, 167739
AppSec [candidate] (166.985 ms) : 0, 166985
Debugger [baseline] (66.743 ms) : 0, 66743
Debugger [candidate] (66.759 ms) : 0, 66759
Remote Config [baseline] (686.472 µs) : 0, 686
Remote Config [candidate] (664.246 µs) : 0, 664
Telemetry [baseline] (9.369 ms) : 0, 9369
Telemetry [candidate] (9.079 ms) : 0, 9079
Flare Poller [baseline] (3.617 ms) : 0, 3617
Flare Poller [candidate] (3.596 ms) : 0, 3596
IAST [baseline] (25.306 ms) : 0, 25306
IAST [candidate] (25.441 ms) : 0, 25441
section iast
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.183 ms) : 0, 1183
BytebuddyAgent [baseline] (794.639 ms) : 0, 794639
BytebuddyAgent [candidate] (796.512 ms) : 0, 796512
AgentMeter [baseline] (11.313 ms) : 0, 11313
AgentMeter [candidate] (11.327 ms) : 0, 11327
GlobalTracer [baseline] (247.262 ms) : 0, 247262
GlobalTracer [candidate] (248.627 ms) : 0, 248627
AppSec [baseline] (35.606 ms) : 0, 35606
AppSec [candidate] (32.404 ms) : 0, 32404
Debugger [baseline] (64.862 ms) : 0, 64862
Debugger [candidate] (68.411 ms) : 0, 68411
Remote Config [baseline] (532.842 µs) : 0, 533
Remote Config [candidate] (536.471 µs) : 0, 536
Telemetry [baseline] (8.635 ms) : 0, 8635
Telemetry [candidate] (8.531 ms) : 0, 8531
Flare Poller [baseline] (3.432 ms) : 0, 3432
Flare Poller [candidate] (3.439 ms) : 0, 3439
IAST [baseline] (27.098 ms) : 0, 27098
IAST [candidate] (27.101 ms) : 0, 27101
section profiling
ProfilingAgent [baseline] (99.722 ms) : 0, 99722
ProfilingAgent [candidate] (98.803 ms) : 0, 98803
crashtracking [baseline] (1.165 ms) : 0, 1165
crashtracking [candidate] (1.163 ms) : 0, 1163
BytebuddyAgent [baseline] (683.488 ms) : 0, 683488
BytebuddyAgent [candidate] (685.435 ms) : 0, 685435
AgentMeter [baseline] (8.575 ms) : 0, 8575
AgentMeter [candidate] (8.631 ms) : 0, 8631
GlobalTracer [baseline] (216.407 ms) : 0, 216407
GlobalTracer [candidate] (218.182 ms) : 0, 218182
AppSec [baseline] (32.452 ms) : 0, 32452
AppSec [candidate] (32.947 ms) : 0, 32947
Debugger [baseline] (67.406 ms) : 0, 67406
Debugger [candidate] (67.476 ms) : 0, 67476
Remote Config [baseline] (634.877 µs) : 0, 635
Remote Config [candidate] (606.524 µs) : 0, 607
Telemetry [baseline] (8.984 ms) : 0, 8984
Telemetry [candidate] (8.919 ms) : 0, 8919
Flare Poller [baseline] (3.78 ms) : 0, 3780
Flare Poller [candidate] (3.781 ms) : 0, 3781
Profiling [baseline] (100.3 ms) : 0, 100300
Profiling [candidate] (99.383 ms) : 0, 99383
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~c95286e6a5, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.079 s) : 0, 1079436
Total [baseline] (8.783 s) : 0, 8782640
Agent [candidate] (1.068 s) : 0, 1068498
Total [candidate] (8.745 s) : 0, 8745257
section iast
Agent [baseline] (1.246 s) : 0, 1245782
Total [baseline] (9.402 s) : 0, 9402375
Agent [candidate] (1.232 s) : 0, 1232412
Total [candidate] (9.375 s) : 0, 9374878
gantt
title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~c95286e6a5, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.209 ms) : 0, 1209
crashtracking [candidate] (1.183 ms) : 0, 1183
BytebuddyAgent [baseline] (637.368 ms) : 0, 637368
BytebuddyAgent [candidate] (629.409 ms) : 0, 629409
AgentMeter [baseline] (29.57 ms) : 0, 29570
AgentMeter [candidate] (29.083 ms) : 0, 29083
GlobalTracer [baseline] (260.951 ms) : 0, 260951
GlobalTracer [candidate] (258.751 ms) : 0, 258751
AppSec [baseline] (33.542 ms) : 0, 33542
AppSec [candidate] (33.186 ms) : 0, 33186
Debugger [baseline] (64.464 ms) : 0, 64464
Debugger [candidate] (65.257 ms) : 0, 65257
Remote Config [baseline] (634.972 µs) : 0, 635
Remote Config [candidate] (604.324 µs) : 0, 604
Telemetry [baseline] (11.505 ms) : 0, 11505
Telemetry [candidate] (10.532 ms) : 0, 10532
Flare Poller [baseline] (3.696 ms) : 0, 3696
Flare Poller [candidate] (4.396 ms) : 0, 4396
section iast
crashtracking [baseline] (1.204 ms) : 0, 1204
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (806.901 ms) : 0, 806901
BytebuddyAgent [candidate] (796.11 ms) : 0, 796110
AgentMeter [baseline] (11.819 ms) : 0, 11819
AgentMeter [candidate] (11.304 ms) : 0, 11304
GlobalTracer [baseline] (249.504 ms) : 0, 249504
GlobalTracer [candidate] (248.082 ms) : 0, 248082
AppSec [baseline] (33.535 ms) : 0, 33535
AppSec [candidate] (32.489 ms) : 0, 32489
Debugger [baseline] (66.499 ms) : 0, 66499
Debugger [candidate] (67.674 ms) : 0, 67674
Remote Config [baseline] (534.837 µs) : 0, 535
Remote Config [candidate] (529.209 µs) : 0, 529
Telemetry [baseline] (8.592 ms) : 0, 8592
Telemetry [candidate] (8.565 ms) : 0, 8565
Flare Poller [baseline] (3.447 ms) : 0, 3447
Flare Poller [candidate] (3.436 ms) : 0, 3436
IAST [baseline] (27.495 ms) : 0, 27495
IAST [candidate] (26.956 ms) : 0, 26956
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 1 performance regressions! Performance is the same for 15 metrics, 18 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~c95286e6a5, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (1.21 ms) : 1199, 1222
. : milestone, 1210,
iast (3.213 ms) : 3172, 3254
. : milestone, 3213,
iast_FULL (5.89 ms) : 5831, 5950
. : milestone, 5890,
iast_GLOBAL (3.504 ms) : 3451, 3557
. : milestone, 3504,
profiling (1.958 ms) : 1940, 1977
. : milestone, 1958,
tracing (1.75 ms) : 1736, 1763
. : milestone, 1750,
section candidate
no_agent (1.175 ms) : 1163, 1186
. : milestone, 1175,
iast (3.186 ms) : 3143, 3228
. : milestone, 3186,
iast_FULL (5.699 ms) : 5642, 5756
. : milestone, 5699,
iast_GLOBAL (3.569 ms) : 3510, 3629
. : milestone, 3569,
profiling (2.014 ms) : 1995, 2034
. : milestone, 2014,
tracing (1.76 ms) : 1746, 1775
. : milestone, 1760,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~c95286e6a5, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (18.781 ms) : 18588, 18974
. : milestone, 18781,
appsec (17.98 ms) : 17798, 18162
. : milestone, 17980,
code_origins (17.41 ms) : 17236, 17585
. : milestone, 17410,
iast (18.885 ms) : 18694, 19076
. : milestone, 18885,
profiling (18.443 ms) : 18261, 18626
. : milestone, 18443,
tracing (17.87 ms) : 17690, 18050
. : milestone, 17870,
section candidate
no_agent (18.068 ms) : 17884, 18251
. : milestone, 18068,
appsec (18.867 ms) : 18674, 19060
. : milestone, 18867,
code_origins (17.33 ms) : 17159, 17501
. : milestone, 17330,
iast (17.572 ms) : 17398, 17746
. : milestone, 17572,
profiling (18.423 ms) : 18241, 18605
. : milestone, 18423,
tracing (17.505 ms) : 17332, 17678
. : milestone, 17505,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~c95286e6a5, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
. : milestone, 1474,
appsec (3.704 ms) : 3488, 3920
. : milestone, 3704,
iast (2.253 ms) : 2184, 2322
. : milestone, 2253,
iast_GLOBAL (2.287 ms) : 2218, 2356
. : milestone, 2287,
profiling (2.108 ms) : 2052, 2164
. : milestone, 2108,
tracing (2.063 ms) : 2009, 2116
. : milestone, 2063,
section candidate
no_agent (1.47 ms) : 1459, 1482
. : milestone, 1470,
appsec (3.793 ms) : 3570, 4016
. : milestone, 3793,
iast (2.247 ms) : 2178, 2316
. : milestone, 2247,
iast_GLOBAL (2.295 ms) : 2225, 2364
. : milestone, 2295,
profiling (2.101 ms) : 2045, 2158
. : milestone, 2101,
tracing (2.048 ms) : 1994, 2101
. : milestone, 2048,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~c95286e6a5, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (15.019 s) : 15019000, 15019000
. : milestone, 15019000,
appsec (14.889 s) : 14889000, 14889000
. : milestone, 14889000,
iast (18.037 s) : 18037000, 18037000
. : milestone, 18037000,
iast_GLOBAL (17.906 s) : 17906000, 17906000
. : milestone, 17906000,
profiling (15.622 s) : 15622000, 15622000
. : milestone, 15622000,
tracing (14.746 s) : 14746000, 14746000
. : milestone, 14746000,
section candidate
no_agent (15.229 s) : 15229000, 15229000
. : milestone, 15229000,
appsec (14.76 s) : 14760000, 14760000
. : milestone, 14760000,
iast (17.919 s) : 17919000, 17919000
. : milestone, 17919000,
iast_GLOBAL (17.71 s) : 17710000, 17710000
. : milestone, 17710000,
profiling (14.975 s) : 14975000, 14975000
. : milestone, 14975000,
tracing (14.954 s) : 14954000, 14954000
. : milestone, 14954000,
|
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the trace-agent is restarted, a rate of 100% is initially provided by the trace-agent, increasing dramatically the number of traces sampled. A rate could go suddenly from 0.1% to 100% and back to 0.1% when the trace-agent eventually computes the new sampling rate.
In particular it is observed that when the agent restarts, the payload buffering that waits for new container tags breaches its memory limit and we send spans without container tags.
This PR applies a limit of sampling rate increases of x2 every 1s resulting in a x10 completed every 3-4s
1->100% takes 7s
0.1 -> 100% takes 10s
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.