Skip to content

Multi-threaded reductions #549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
FrancescAlted wants to merge 91 commits into
base: main
Choose a base branch
from
Open

Multi-threaded reductions #549

FrancescAlted wants to merge 91 commits into from

Conversation

@FrancescAlted
Copy link
Member

@FrancescAlted FrancescAlted commented Jan 4, 2026

This adds support for multi-threaded reductions. Only some scenarios are supported:

  • non-strided partitions
  • in-memory arrays
  • argmax() and argmin() are not supported yet

When these conditions are not met, a fallback to existing, non-threaded machinery, is made.

In addition, support for multi-threaded evaluation of expressions (e.g. "(a + b) * c +1") is here too.

All tests are passing now, but more review needs to be done, as this is quite intrusive for the computing engine.

@FrancescAlted
Very preliminary attempt to use the miniexpr library
aeee83b
@FrancescAlted
Upgrade version of miniexpr
8358c59
@FrancescAlted
Using me_compile_chunk in register prefilter
0483cd1
@FrancescAlted
Use me_dtype_from_numpy and others
f1986cc
@FrancescAlted
First preliminary version that works with miniexpr (for simple cases)
8c6249d
@FrancescAlted
miniexpr prefilter doesn't hold the gil anymore
cd0c4da
@FrancescAlted
miniexpr prefilter finally works in multithread mode!
42189d9
@FrancescAlted
Fix a compilation error on linux
a2a7259
@FrancescAlted
Be more conservative for blocksize in apple silicon
f599cdb
@FrancescAlted
Add a comparsion against numexpr
3bf54c9
@FrancescAlted
Update to latest miniexpr sources
ef9dc80
@FrancescAlted
Fix a leak
e345d9c
@FrancescAlted
Add a comparison against NumPy
655efec
@FrancescAlted
Selectively try miniexpr, if possible
1d56614
@FrancescAlted
Create a new context for each input
cd26133
@FrancescAlted
Updated to latest miniexpr (bools as result of comparsions)
b3742f8
@FrancescAlted
Fixed many things with miniexpr backend
36065c3
@FrancescAlted
More defensive protection when calling C funcs
232e485
@FrancescAlted
Disable parallel mode for numba
6da99ae
@FrancescAlted
Better examplanation on why we disable parallel
ac430ac
@FrancescAlted
Fix many issues with type casting in miniexpr
c51f528
@FrancescAlted
Add inverse hyperbolic funcs
4be0758
@FrancescAlted
Add conj and imag complex funcs
0abfb65
@FrancescAlted
Add more functions to miniexpr
d1d922e
@FrancescAlted
New where() func in miniexpr
620c980
@FrancescAlted
Support for windows
5545a81
@FrancescAlted
Changing to clang-cl on win
d88dcbb
@FrancescAlted
Upgrade version of miniexpr
d65f2dc
@FrancescAlted
Changing MSVC to clang-cl on win (II)
147cb80
@FrancescAlted
Changing MSVC to clang-cl on win (III)
9e8f677
@FrancescAlted
Clear output buffer
fb9f296
@FrancescAlted
Add a benchmark for reductions (preliminary)
47cdf2a
@FrancescAlted
Upgrade to latest openzl commit in c-blosc2
999a1b6
@FrancescAlted
Optimization for dispose outputs in reductions
5ceba21
@FrancescAlted
Allow miniexpr to take care of reductions with multi-operand expressions
d7b0f21
@FrancescAlted
Improvements in reduction benchs
8bce9f4
@FrancescAlted
More complex expression in bench
4834614
@FrancescAlted
Update to latest miniexpr, and some nicer stats in benchs
ec65a7d
@FrancescAlted
Minor fix
54748a3
@FrancescAlted
Improved conditions determining when miniexpr can enter into action
e70a31a
@FrancescAlted
Benefit Intel architectures for now
58baecc
@FrancescAlted
New (faster) way of evaluating with miniexpr
eb43bf2
@FrancescAlted
Use the new version of miniexpr as a library
07ccba9
@FrancescAlted
Relax precision checking for tan in miniexpr
a179ea1
@FrancescAlted
Only attempt to compile the static version of miniexpr
4e65f59
@FrancescAlted
Set Position Independent Code when compiling miniexpr
cda72e2
@FrancescAlted
Remove torch as a test requirement (too heavy dependency)
a604af2
@FrancescAlted
Deactivate openzl on wasm platforms (requires too advanced C11)
ac019cb
@FrancescAlted
More fixing issues on windows
d521d4b
@FrancescAlted
Protect tests that depend on torch
677cfe6
@FrancescAlted
More granularity in protecting tests requiring torch
e55bf99
@FrancescAlted
Fix result checks
65d2738
@FrancescAlted
Fix result checks
891c9ae
@FrancescAlted
Fetch the latest commit only
a05ed3c
@FrancescAlted
Fix PATH parsing issues on win
55d5e8f
@FrancescAlted
miniexpr is not designed for WASM (it does not support multithreading)
e099898
@FrancescAlted
Disabling miniexpr for windows, as it has some issues yet
c4367ef
@FrancescAlted
Hook for ignoring transient request failures for tests requiring network
57de907
@FrancescAlted
Switch to openzl branch in c-blosc2
651e6aa
@FrancescAlted
Switch to *add_openzl* branch in c-blosc2
5b7afed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@FrancescAlted