Expose more of the lading_capture interface and add parquet rotation support #1662
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
scottopell
commented
Dec 19, 2025
scottopell
commented
- feat(observer): Expose observer module as public API
- Add documentation to public observer APIs
- docs: Add missing doc comments for public modules
- feat(capture): add file rotation support for Parquet format
Make the observer module and its subcomponents public to allow
external crates to reuse the procfs and cgroup v2 parsing logic.
Exposed APIs:
- observer::linux::Sampler - high-level sampler for a process tree
- observer::linux::procfs::{memory, stat, uptime} - procfs parsers
- observer::linux::cgroup::v2::{poll, get_path, cpu, memory} - cgroup v2 parsers
This enables fine-grained-monitor and similar tools to leverage
lading's battle-tested observer implementation without vendoring.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Required for #![deny(missing_docs)] when using lading as a library dependency.
Add rotation API to CaptureManager that allows rotating to new output files without stopping the capture. This enables long-running capture sessions to produce multiple readable Parquet files with valid footers. Changes: - Add RotationRequest/RotationSender types for async rotation requests - Add start_with_rotation() that spawns event loop and returns sender - Add replace_format() to StateMachine for IO-agnostic format swapping - Add rotate() trait method stub to OutputFormat (returns error by default) - Add rotate_to() inherent method on parquet Format<BufWriter<File>> The rotation flow: 1. Caller sends RotationRequest with new file path via RotationSender 2. CaptureManager creates new file and format 3. StateMachine.replace_format() flushes and swaps formats 4. Old format is closed (writing Parquet footer) 5. Response sent back to caller 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The CPU Sampler was initializing prev stats to zeros, causing the first delta calculation to be (cumulative_since_container_start - 0) which produces an enormous spike in total_cpu_usage_millicores. Fix by making prev an Option<Stats>. On first poll, we record baseline stats but skip metric emission. Subsequent polls compute proper deltas. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Same fix as cgroup/v2/cpu.rs: make prev an Option<Stats> and skip metric emission on first poll to avoid computing delta from cumulative-since-process-start minus zero. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace MapArray-based label storage with flat l_<key> columns in Parquet output. This enables predicate pushdown for filtering by container_id and other labels, avoiding full file scans. Key changes: - Dynamic schema generation based on discovered label keys - Dictionary encoding for low-cardinality label columns - Lazy ArrowWriter initialization (schema determined at first flush) - Updated validation and round-trip tests for new schema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add BloomFilterConfig and BloomFilterColumn types to configure bloom filters on label columns. Bloom filters enable efficient query-time filtering by allowing readers to skip row groups that definitely don't contain a target value. New APIs: - Format::with_bloom_filter() - create writer with bloom filter config - format.bloom_filter_config() - getter for rotation - CaptureManager::new_parquet_with_bloom_filter() - CaptureManager::new_multi_with_bloom_filter() Backwards compatible - existing Format::new() and new_parquet() still work unchanged using BloomFilterConfig::default(). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.