Adjust to run with newer dependencies#224
Open
ximion wants to merge 1 commit intodenisecailab:masterfrom
Open
Conversation
545dc4a to
79ad439
Compare
Contributor
Author
|
So, the Windows and macOS Conda environments don't seem to resolve on your CI, and ideally demand Python 3.10 - since that's the latest release anyway, I updated the dependencies again to work with the most recent Python. Of course, this combination isn't tested as much as the previous one. On my initial tests I didn't find any issues, but I do want to run this on larger datasets and maybe make some tweaks before claiming it's ready (so please don't merge this PR just yet). |
9be024c to
a8835b0
Compare
a8835b0 to
e233ac2
Compare
e233ac2 to
c0b31c0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi!
I needed a newer Dask version and a newer numpy and xarray (as well as a more recent ffmpeg for better AV1 support) for some custom preprocessing on our data before running Minian, which sent me down an insane rabbit hole to update the Minian code to work with newer dependencies.
So, this is what I ended up with. The generated output for the sample video looks the same as before (but this patch only contains minimal modifications to the Jupyter notebook), and this code works fine with my data.
I also apparently solved #214 somehow (I had this one once before, and after updating Holoviews it never showed up again) and this change also definitely fixes #152, the bane of my existence (dealing with sparse/dense xarrays was a pain as not every OPs are allowed with the same parameters on either, so I do densify arrays explicitly in at some points now for some operations, and sparsify arrays that were previously dense - all of this can result in higher peak memory usage in my testing, so to run Minian on larger datasets (like my 15min movies) you really need to have 16GB of RAM or more memory/swap space.
Some deprecation warnings are still left, but some of those also existed before and could be resolved at a later time (and I wanted to keep this patch small).
This patch also switches Numba's threading layer to OpenMP, because often it switches to TBB which is not threadsafe and causes all kind of errors when combined with dask (and also thousands of warning messages), and in addition to running without random crashes, the OMP backend was also running a lot faster during my testing.
Minian also runs on Python 3.9+ now.
Thanks for considering this patch!
Cheers,
Matthias