[Watch] Make watch ID unique per member #21069
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: henrybear327 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
henrybear327
requested review from
fuweid and
serathius
and removed request for
fuweid
|
@serathius Is there a reason that we should keep the tracking of watchID per stream? This PR doesn't address the case where the user can allocate the same watchID across different stream now. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 21 files with indirect coverage changes @@ Coverage Diff @@
## main #21069 +/- ##
==========================================
+ Coverage 68.44% 68.57% +0.12%
==========================================
Files 429 429
Lines 35281 35293 +12
==========================================
+ Hits 24149 24201 +52
+ Misses 9732 9699 -33
+ Partials 1400 1393 -7 Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
henrybear327
force-pushed
the
pr-20282
branch
3 times, most recently
from
ab6ee28 to
cd94024
Compare
|
/retest |
henrybear327
changed the title
server/storage/mvcc/watcher.go
Outdated
| // ensure that the nextWatchID check is done in a critical section | ||
| nextWatchIDMutex.Lock() | ||
| for ws.watchers[WatchID(nextWatchID)] != nil { | ||
| nextWatchID++ | ||
| } | ||
| id = ws.nextID | ||
| ws.nextID++ | ||
| id = WatchID(nextWatchID) | ||
| nextWatchID++ | ||
| nextWatchIDMutex.Unlock() |
There was a problem hiding this comment.
The logic is problematic. The nextWatchID is global, but ws.watchers is local to each watchStream. It can't guarantee unique watchID across all watch streams.
There was a problem hiding this comment.
also let alone multiple etcd servers may have same watch IDs
This change makes watch IDs unique per etcd member rather than per stream, which improves debugging by allowing storage to identify clients by watch IDs across connections. The auto-assignment logic now uses a global atomic counter while still checking for conflicts with user-specified IDs on the same stream, avoiding allocating IDs that have been occupied by user requests. Continuation of the work etcd-io#20282. Signed-off-by: Marek Siarkowicz <siarkowicz@google.com> Signed-off-by: Chun-Hung Tseng <henrytseng@google.com>
The previous implementation checked ws.watchers (per-stream) against the global nextWatchID counter, which could not guarantee unique watch IDs across all streams on a member. Bug scenario: - Stream A: User specifies ID=5 → added to ws_A.watchers[5] - Stream B: Auto-assign when counter=5 → checks ws_B.watchers[5] (nil) - Result: Both streams have ID=5, violating per-member uniqueness Fix: Add a global registry (assignedWatchIDs) with reference counting that tracks all watch IDs across all streams. The auto-assignment loop now checks this global registry instead of the per-stream map. Reference counting is needed because user-specified IDs can be duplicated across streams (for backward compatibility), so we only mark an ID as free when no stream is using it. Signed-off-by: Chun-Hung Tseng <henrytseng@google.com>
|
@henrybear327: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
4 participants
This change makes watch IDs unique per etcd member rather than per
stream, which improves debugging by allowing storage to identify
clients by watch IDs across connections.
The auto-assignment logic now uses a global atomic counter while
still checking for conflicts with user-specified IDs on the same
stream, avoiding allocating IDs that have
been occupied by user requests.
Continuation of the work #20282.
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.