Skip to content

Conversation

@paul1r
Copy link
Collaborator

@paul1r paul1r commented Oct 27, 2025

What this PR does / why we need it:
This PR adds critical WAL cleanup functionality to the ingester to prevent disk space leaks and maintain a healthy checkpoint state.

  1. Stale Temporary Checkpoint Cleanup
  • Now scans and removes all .tmp checkpoint directories (not just the current one)
  • Prevents accumulation of partial checkpoints from failed attempts
  • Runs during ingester startup
  1. Old Checkpoint Cleanup
  • Identifies and removes checkpoints that reference WAL segments no longer present on disk
  • Triggered when closing a checkpoint and at startup
  • Cleans up old prior checkpoints that were not removed for whatever reason
  • Always preserves at least the current checkpoint

Which issue(s) this PR fixes:
Without this cleanup logic, the WAL directory can accumulate:

  • Failed temporary checkpoints that are never removed
  • Orphaned checkpoint references pointing to deleted segments

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@paul1r paul1r requested a review from a team as a code owner October 27, 2025 19:44
@paul1r paul1r changed the title feat(ingester): Add WAL cleanup code fix(ingester): Add WAL cleanup code Oct 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant