Load .jpk-qi-data metadata using jpk-reader-rs package#45
Load .jpk-qi-data metadata using jpk-reader-rs package#45bicarlsen wants to merge 2 commits intoAFM-analysis:masterfrom
.jpk-qi-data metadata using jpk-reader-rs package#45Conversation
|
This also incorporates #44. |
remove backslash from hour and minute while parsing time. refactor optimize get_metadata
|
This new version optimizes Once this is more thouroughly tested, it should be good to go. There may also be some optimizations that can be applied elsewhere:
|
|
Thanks @bicarlsen for the effort, looks great.
|
| @@ -1,19 +1,18 @@ | |||
| import numpy as np | |||
| import jpk_reader_rs as jpk_rs | |||
There was a problem hiding this comment.
jpk_rs isn't used here.
If you lint the package using flake8 (pip install flake8) (flake8 afmformats tests)
You'll see some small style issues
| "igor2>=0.5.0", # Asylum Research .ibw file format | ||
| "jprops", # JPK file format | ||
| "jprops", # JPK file format | ||
| "jpk-reader-rs", # JPK QI data (`.jpk-qi-data`) |
There was a problem hiding this comment.
This doesn't install as is. IT requires Rust and Cargo to install, apparently. I get the issue locally and also the CI/CD has the same issue (see https://github.com/AFM-analysis/afmformats/actions/runs/21680235950/job/62512249369?pr=45)
| self._hierarchy = "single" | ||
| elif self.files_contains("index/"): | ||
| self._hierarchy = "indexed" | ||
|
|
There was a problem hiding this comment.
As above, if you run flake8, you'll see some lint issues e.g. W293 blank line contains whitespace
2 participants
Speeds up loading
.jpk-qi-datafiles by using thejpk-reader-rspackage to load metadata.Addresses #41, reducing load times of standard files by ~10x. For the test file it reduced the load time from more than 120 minutes to about 15.
I kept as much of the original
JPKReaderas possible for this first pass. There is still quite some performance to be gained though, as the same test file loads in 20 seconds from the Rust crate.While probably not yet ready to be merged, I'm hoping can use it so we can address any issues that arise and continue to imporve upon performance.