Consolidate load functions, add schema for casting, and add CSV options for reading#2423

Draft

arienandalibi wants to merge 61 commits intomasterfrom

consolidate_load_functions

Commits on Nov 11, 2025

Added tests for loading edges from polars and from fireducks. Added a load_edges_from_polars that internally calls to_pandas() on the polars dataframe. Fireducks works.
arienandalibi
committed

Commits on Nov 16, 2025

Adding loading of data (only edges for now) from arrow directly
arienandalibi
committed

Commits on Nov 17, 2025

Adding loading of data (only edges for now) from arrow with streaming in rust instead of obtaining each column of each batch from Python individually.
arienandalibi
committed

Commits on Nov 18, 2025

Commits on Nov 21, 2025

Commits on Nov 22, 2025

Commits on Nov 24, 2025

Commits on Nov 27, 2025

Commits on Nov 28, 2025

Commits on Dec 2, 2025

Commits on Dec 4, 2025

Adding load_nodes function in python that can take any input from the following: object with __arrow_c_stream__(), parquet file/directory, csv file/directory. Added arrow-csv as a dependency to loa…
arienandalibi
committed

Commits on Dec 8, 2025

Added schema argument to load_nodes function
arienandalibi
committed

Commits on Dec 10, 2025

Added casting of nested datatypes in the data source. Added test for nested type using pyarrow Table. Cast whole RecordBatch at once now using StructArray.
arienandalibi
committed

Commits on Dec 11, 2025

Commits on Dec 12, 2025

Added CSV options for when loading CSV. Errors if CSV options were passed but no CSV files were detected.
arienandalibi
committed

Commits on Dec 17, 2025

Added btc_dataset tests for loading/casting from different sources as well as failures from malformed inputs
arienandalibi
committed

Commits on Dec 18, 2025

Commits on Dec 20, 2025