http: fix potential FD leak when zombie streams prevent connection close #42859
+161
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I am chasing a case where a client sends POST messages on a HTTP1.1 connection, but is being systematically denied. The connection is immediately reused by the client. In some case we seem to trigger a racy case where the number of open file descriptor explodes. When a stream becomes a zombie (waiting for codec encode completion) and the connection is in DrainState::Closing, the connection is not closed because checkForDeferredClose() is not called after the zombie stream was finally destroyed. This commonly occurs with HTTP/1.1 connections when: - ext_authz (or similar filter) denies a request early, sending a 403 response before the full request body is received - The connection is (potentially) marked DrainState::Closing - The stream becomes a zombie waiting for the codec to finish encoding - When onCodecEncodeComplete() fires, doDeferredStreamDestroy() was called but checkForDeferredClose() was not, leaving the connection open In scenarios where clients try to reuse the same HTTP/1.1 connection and requests are systematically denied (e.g., by ext_authz), this causes rapid FD exhaustion as each denied request leaves an orphaned connection. The fix adds checkForDeferredClose() calls after zombie stream destruction in both onCodecEncodeComplete() and onCodecLowLevelReset(). Signed-off-by: William Dauchy <william.dauchy@datadoghq.com>
Member
|
@KBaichoo, Kevin, do you want also to take a look for this? |
botengyao
reviewed
Jan 7, 2026
Member
botengyao
left a comment
botengyao
left a comment
There was a problem hiding this comment.
Thanks for fixing this, and left a high level question.
/wait
wdauchy
force-pushed
the
fd_leak
branch
6 times, most recently
from
495f026 to
c96a9be
Compare
Signed-off-by: William Dauchy <william.dauchy@datadoghq.com>
Contributor
Author
|
/retest transients |
botengyao
reviewed
Jan 8, 2026
| if (state_.is_zombie_stream_) { | ||
| const bool skip_delay = shouldSkipDeferredCloseDelay(); | ||
| connection_manager_.doDeferredStreamDestroy(*this); | ||
| // After destroying a zombie stream, check if the connection should be |
Member
There was a problem hiding this comment.
Do you mind adding a runtime guard and a release note? I think this will be a high risk change since the release will be next week, does merging it after early next week work for you?
KBaichoo
previously approved these changes
Jan 9, 2026
Contributor
KBaichoo
left a comment
KBaichoo
left a comment
There was a problem hiding this comment.
Nice work, looks good to me
/wait
| if (state_.is_zombie_stream_) { | ||
| const bool skip_delay = shouldSkipDeferredCloseDelay(); | ||
| connection_manager_.doDeferredStreamDestroy(*this); | ||
| // After destroying a zombie stream, check if the connection should be |
Signed-off-by: William Dauchy <william.dauchy@datadoghq.com>
Contributor
Author
|
/retest transients |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Commit Message:
I am chasing a case where a client sends POST messages on a HTTP1.1 connection, but is being systematically denied. The connection is immediately reused by the client. In some case we seem to trigger a racy case where the number of open file descriptor explodes.
When a stream becomes a zombie (waiting for codec encode completion) and the connection is in DrainState::Closing, the connection is not closed because checkForDeferredClose() is not called after the zombie stream was finally destroyed.
This commonly occurs with HTTP/1.1 connections when:
In scenarios where clients try to reuse the same HTTP/1.1 connection and requests are systematically denied (e.g., by ext_authz), this causes rapid FD exhaustion as each denied request leaves an orphaned connection.
The fix adds checkForDeferredClose() calls after zombie stream destruction in both onCodecEncodeComplete() and onCodecLowLevelReset().
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]