Add an efficient reservoir sampling aggregator#1214
Open
marcusb wants to merge 1 commit intotwitter:developfrom
Open
Add an efficient reservoir sampling aggregator#1214marcusb wants to merge 1 commit intotwitter:developfrom
marcusb wants to merge 1 commit intotwitter:developfrom
Conversation
|
|
Author
|
JMH benchmark results (Intel Core i9-10885H): |
Author
marcusb
force-pushed
the
sampling
branch
4 times, most recently
from
3d8bd4b to
a0b526b
Compare
Author
|
|
marcusb
force-pushed
the
sampling
branch
2 times, most recently
from
036d5c2 to
431d5b7
Compare
Author
|
Another large performance boost by optimizing the aggregation for IndexedSeq.
|
This aggregator uses Li's "Algorithm L", a simple yet efficient sampling method, with modifications to support a monoidal setting. A JMH benchmark was added for both this and the old priority-queue algoritm. In a single-threaded benchmark on an Intel Core i9-10885H, this algorithm can outperform the old one by an order of magnitude or more, depending on the parameters. Because of this, the new algorithm was made the default for Aggregtor.reservoirSample(). Unit tests were added for both algorithms. These are probabilistic and are expected to fail on some 0.1% of times, per test case (p-value is set to 0.001). Optimized overloads of aggregation methods append/appendAll were added that operate on IndexedSeqs. These have efficient random access and allow us to skip over items without examining each one, so sublinear runtime can be achieved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This aggregator uses Li's "Algorithm L", a simple yet efficient
sampling method, with modifications to support a monoidal setting.
A JMH benchmark was added for both this and the old priority-queue
algoritm. In a single-threaded benchmark on an Intel Core i9-10885H,
this algorithm can outperform the old one by an order of magnitude or
more, depending on the parameters.
Because of this, the new algorithm was made the default for
Aggregtor.reservoirSample().
Unit tests were added for both algorithms. These are probabilistic and
are expected to fail on some 0.1% of times, per test case (p-value is
set to 0.001).