Consolidate load functions, add schema for casting, and add CSV options for reading #2423
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… load_edges_from_polars that internally calls to_pandas() on the polars dataframe. Fireducks works.
… in rust instead of obtaining each column of each batch from Python individually.
…g up. Committing benchmarks and tests that check graph equality when using different ingestion pathways.
…id of them and always stream. Added benchmark for loading from fireducks.
…e_props, that all use the __arrow_c_stream__() interface. If a data source is passed with no __len__ function, we calculate the len ourselves. Updated ingestion benchmarks to also test pandas_streaming, fireducks_streaming, polars_streaming
# Conflicts: # python/python/raphtory/__init__.pyi
…renamed to load_edge_metadata/load_node_metadata.
…if the data source provides __len__(), and if not, the loading/progress bar for loading nodes and edges doesn't show progression, only iterations per second.
…ess bar for loading updates properly when using the __arrow_c_stream__ interface.
…ingestion pathways
…dge_deletions_from_df
# Conflicts: # python/python/raphtory/vectors/__init__.pyi
…vent graphs/persistent graphs.
# Conflicts: # python/python/raphtory/graphql/__init__.pyi # python/python/raphtory/vectors/__init__.pyi
…ta from python. Replaced it with PyRecordBatchReader::from_arrow_pycapsule for safety and future changes.
…rrow format at once. Now stream 1 mil rows at a time.
… function on PyProperties. Added test for schema casting.
# Conflicts: # raphtory/src/python/graph/io/arrow_loaders.rs
…ify what type to cast columns to.
…nested type using pyarrow Table. Cast whole RecordBatch at once now using StructArray.
…taTypes can be extracted from Python without feature gating behind arrow (larger dependency). Refactored data_type_as_prop_type to be in raphtory-api as long as any of "arrow", "storage", or "python" features is enabled, since they all have dep:arrow-schema.
…rison for PropType. Fixed previous tests and added tests for dict schema input, pyarrow types, nested (StructArray) properties, nested schemas, mixed and matched PropType and pyarrow types, both in property and in schema,...
…ssed but no CSV files were detected.
# Conflicts: # python/python/raphtory/__init__.pyi # python/python/raphtory/iterables/__init__.pyi # python/python/raphtory/node_state/__init__.pyi # python/tests/test_ingestion_equivalence_df.py # python/tests/test_load_from_df.py # raphtory-api/src/python/mod.rs # raphtory/src/python/graph/graph.rs # raphtory/src/python/graph/graph_with_deletions.rs # raphtory/src/python/graph/io/arrow_loaders.rs # raphtory/src/python/packages/base_modules.rs
…d parquet/csv). Make sure each ingestion path returns the same node ids.
… well as failures from malformed inputs
…lformed (or any column). Added tests for malformed inputs in csv.
…umn is not found. removed extra_field parquet test bc it didn't work. cleaned up test file.
…e_deletions for csv files/directories
…v optional-dependencies in pyproject.toml. General clean-up before adding other functions (load_edge, load_node_metadata, ...) in python graph.
…unctions. Added load_edges, load_node_metadata, load_edge_metadata functions to PyGraph and PyPersistentGraph. Removed Pandas loaders.
…folder which is not available in the crate root. Fixed parquet_loaders.rs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Takes care of #2394
Not included in the issue above: