Add batching support to fetch_df_by_partition for improved performance with large file counts

[Ronxvier]

Adds batching support to fetch_df_by_partition with use_batching and batch_size parameters. When enabled, files are processed in smaller batches instead of loading all files into memory simultaneously, which should reduce peak memory usage if users have extremely large file counts. Uses the existing fetch_dfs_by_paths_batching function. Maintains backward compatibility - existing code works unchanged.

[Copilot]

Pull Request Overview

This PR adds optional batching functionality to the fetch_df_by_partition function to improve memory efficiency when processing large numbers of files. The change introduces two new optional parameters to control batching behavior while maintaining full backward compatibility.

• Added use_batching and batch_size parameters to fetch_df_by_partition
• Implemented fetch_dfs_by_paths_batching function to handle batched file processing
• Enhanced function documentation with proper parameter descriptions

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}