fix: keep a strong reference to the processing started latency gauge - #3522
Draft
csviri wants to merge 1 commit into
Draft
fix: keep a strong reference to the processing started latency gauge#3522csviri wants to merge 1 commit into
csviri wants to merge 1 commit into
Conversation
`eventProcessingStarted` registered the gauge by handing the boxed result
of `getRuntimeMXBean().getUptime()` straight to `registry.gauge(...)`:
registry.gauge(PROCESSING_STARTED_LATENCY_GAUGE, tags,
ManagementFactory.getRuntimeMXBean().getUptime());
Micrometer only keeps a weak reference to the gauged object, so it is the
caller's job to hold a strong reference. Nothing here does, which means
the boxed Long is collectable immediately and the gauge reports NaN.
`processing.started.latency` therefore never carried a usable value.
`controllerRegistered` already handles this correctly by keeping its
`AtomicInteger`s in the `gauges` map; this method just did not follow the
same pattern.
Registers the gauge against an `AtomicLong` held in a `longGauges` map
(an `AtomicInteger` would overflow, uptime is in milliseconds) and
updates that holder on subsequent calls, which also stops a duplicate
gauge from being registered every time the event processor starts.
Applies the same fix to the deprecated `MicrometerMetrics`, which has the
identical problem.
Adds regression tests asserting the gauge keeps a non-NaN value across a
GC and that repeated calls reuse a single gauge; the first fails without
this change. Also adds the (already dependency-managed) mockito-core test
dependency to the module, which had no way to stub a Controller before.
16 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a Micrometer gauge registration bug where processing.started.latency could report NaN because the gauge target was only weakly referenced (a boxed Long with no strong owner). It aligns eventProcessingStarted with the existing “keep a strong reference” pattern used for other gauges, and adds regression tests to prevent the issue from returning.
Changes:
- Update
MicrometerMetricsV2.eventProcessingStartedto register the gauge against anAtomicLongheld strongly in alongGaugesmap (and reuse/update it on subsequent calls). - Apply the same strong-reference fix to the deprecated
MicrometerMetrics. - Add regression tests and introduce
mockito-core(test scope) to enable controller stubbing in the module.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
micrometer-support/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java |
Keep a strong AtomicLong reference for processing.started.latency and update it on repeated starts to avoid NaN/duplicate gauges. |
micrometer-support/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetrics.java |
Apply the same strong-reference gauge registration approach to the deprecated metrics implementation. |
micrometer-support/src/test/java/io/javaoperatorsdk/operator/monitoring/micrometer/ProcessingStartedLatencyGaugeTest.java |
Add regression coverage for non-NaN gauge values after GC and reuse on repeated calls. |
micrometer-support/pom.xml |
Add mockito-core as a test dependency for the new tests. |
Comments suppressed due to low confidence (1)
micrometer-support/src/test/java/io/javaoperatorsdk/operator/monitoring/micrometer/ProcessingStartedLatencyGaugeTest.java:47
- Same as above: uptime can be 0ms, so
isPositive()can be flaky here as well; prefer a non-negative assertion.
assertThat(gauge.value()).isNotNaN().isPositive();
|
|
||
| var gauge = registry.find(MicrometerMetricsV2.PROCESSING_STARTED_LATENCY_GAUGE).gauge(); | ||
| assertThat(gauge).isNotNull(); | ||
| assertThat(gauge.value()).isNotNaN().isPositive(); |
Comment on lines
+71
to
+75
| private static void forceGarbageCollection() { | ||
| for (int i = 0; i < 5; i++) { | ||
| System.gc(); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
eventProcessingStartedregistered the gauge by handing the boxed resultof
getRuntimeMXBean().getUptime()straight toregistry.gauge(...):Micrometer only keeps a weak reference to the gauged object, so it is the
caller's job to hold a strong reference. Nothing here does, which means
the boxed Long is collectable immediately and the gauge reports NaN.
processing.started.latencytherefore never carried a usable value.controllerRegisteredalready handles this correctly by keeping itsAtomicIntegers in thegaugesmap; this method just did not follow thesame pattern.
Registers the gauge against an
AtomicLongheld in alongGaugesmap(an
AtomicIntegerwould overflow, uptime is in milliseconds) andupdates that holder on subsequent calls, which also stops a duplicate
gauge from being registered every time the event processor starts.
Applies the same fix to the deprecated
MicrometerMetrics, which has theidentical problem.
Adds regression tests asserting the gauge keeps a non-NaN value across a
GC and that repeated calls reuse a single gauge; the first fails without
this change. Also adds the (already dependency-managed) mockito-core test
dependency to the module, which had no way to stub a Controller before.
Part of #3517