input: add thread.ring_buffer.retry_limit option

[jinyongchoi]

When a threaded input plugin's ring buffer is full, the input thread retries writing to the buffer before dropping data. Previously, this retry limit was hardcoded to 10 (1 second total with 100ms sleep between retries).

This patch makes the retry limit configurable via the new 'thread.ring_buffer.retry_limit' option. The default value remains 10 for backward compatibility. Increasing this value allows the system to handle temporary backpressure situations without dropping data.

Fixes #11393

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[SERVICE]
    flush 1
    grace 600
    log_level info

    storage.path /tmp/storage
    storage.metrics on
    storage.max_chunks_up 512
    storage.sync full
    storage.checksum off
    storage.backlog.mem_limit 100M

[INPUT]
    Name tail
    Path /var/log/app/*.log
    Tag app_logs
    Key message

    Read_from_Head true
    Refresh_Interval 3

    Buffer_Chunk_Size 1MB
    Buffer_Max_Size 16MB

    Mem_Buf_Limit 256MB
    storage.type memory
    storage.pause_on_chunks_overlimit true

    DB /tmp/storage/tail.db
    DB.sync normal

    Threaded true
    thread.ring_buffer.capacity  128
    thread.ring_buffer.retry_limit  1000

[OUTPUT]
    Name kafka
    Match app_logs
    Brokers kafka:9092
    Topics logs
    timestamp_format iso8601

    rdkafka.compression.codec gzip
    rdkafka.compression.level 7
    rdkafka.queue.buffering.max.messages 200000
    rdkafka.batch.size 1000000
    rdkafka.linger.ms 100

    workers 4

• [N/A] Debug log output from testing the change
• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Threaded input ring-buffer writes now support a configurable retry limit via thread.ring_buffer.retry_limit (default: 10), applied per input.

• Bug Fixes / Observability

  • The retry setting is validated (>0) and enqueue failures now increment a failure metric for better observability.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a per-input configurable retry limit for threaded input ring buffer writes: new ring_buffer_retry_limit field on flb_input_instance, a default macro and config property, parsing/validation during input setup, and use of the instance field in enqueue retry logic.

Changes

Cohort / File(s)	Summary
Input struct definition include/fluent-bit/flb_input.h	Added int ring_buffer_retry_limit field to struct flb_input_instance.
Configuration & initialization src/flb_input.c	Added FLB_INPUT_RING_BUFFER_RETRY_LIMIT (default 10); registered thread.ring_buffer.retry_limit in input properties; initialize ins->ring_buffer_retry_limit to default; parse and validate property (>0) in flb_input_set_property.
Ring buffer retry logic src/flb_input_chunk.c	Replaced local hardcoded retry limit with ins->ring_buffer_retry_limit in enqueue retry loop; removed local retry variable and incremented ring buffer retry failure counter on enqueue failure.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested labels

backport to v4.0.x

Suggested reviewers

• edsiper

Poem

🐇 I nudged a limit, soft and slight,
Threads may try a little more each night.
Rings wait patient, retries hold true,
Fewer drops fall, carrots for you. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding a new configuration option thread.ring_buffer.retry_limit for input plugins.
Linked Issues check	✅ Passed	The PR implements all coding requirements from #11393: adds the configurable thread.ring_buffer.retry_limit option with default value 10, exposes it in input configuration, and updates retry logic to use the per-instance limit.
Out of Scope Changes check	✅ Passed	All changes are directly related to implementing the thread.ring_buffer.retry_limit feature; no unrelated modifications detected across the three files modified.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR adds a configurable retry limit for threaded input plugins' ring buffer writes. Previously, the retry limit was hardcoded to 10 attempts (1 second total with 100ms sleep between retries). This caused data loss during temporary backpressure situations that lasted longer than 1 second.

Changes:

• Added thread.ring_buffer.retry_limit configuration option with default value of 10 to maintain backward compatibility
• Modified ring buffer write logic to use the configurable retry limit instead of hardcoded value
• Updated documentation to explain the new configuration option

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
include/fluent-bit/flb_input.h	Added ring_buffer_retry_limit field to flb_input_instance struct
src/flb_input.c	Added configuration map entry, initialization, and property parsing for the new retry limit option
src/flb_input_chunk.c	Removed hardcoded retry limit and updated to use instance-level configurable value

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jinyongchoi]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 👍

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[jinyongchoi]

@cosmo0920 @edsiper
Hello,
This PR is ready for review. It adds a new thread.ring_buffer.retry_limit configuration option for threaded input plugins, replacing the hardcoded retry limit (10) with a user-configurable value. This allows users to tune retry attempts for handling temporary backpressure situations without dropping data.
Thanks!