Refactor mdf_reader to use polars

[jtsiddons]

An inprogress re-write of the pandas components of mdf_reader into polars. This could allow for improved performance in terms of memory usage and speed.

Todo:

• Match Unit Tests
• Current implementation uses pandas.MultiIndex for the columns, allowing core.YR access. Polars does not support this behaviour.
• Handling of missing_values and fields to ignore
• Remove re-conversion to pandas.DataFrame
• Allow chunking
• Decoding Step
• Conversion Step
• Validation Step

[jtsiddons]

@ludwiglierhammer: Do I need to add polars to the ci requirements/environment files?

[ludwiglierhammer]

@ludwiglierhammer: Do I need to add polars to the ci requirements/environment files?

Yes please.

[ludwiglierhammer]

@jtsiddons: In general, we have to read the input data line by line, as the sentinals etc can vary from line to line. I just wanted to let you konw, not that you are putting time and effort into it.

[jtsiddons]

@jtsiddons: In general, we have to read the input data line by line, as the sentinals etc can vary from line to line. I just wanted to let you konw, not that you are putting time and effort into it.

Using polars I scan the column for the sentinal. I then separate the section (as a single column) if the column is present (values are None if the sentinal is not present). I then sequentially slice into the section column splitting out all of the fields.

If the sentinal is None or "", then it is assumed that the section is guaranteed to be present. And the slicing/splitting is performed.

No splitting on values is performed if the sentinal is otherwise missing. Slicing on None results in two values of None.

The operations are performed on the correct lines.

[ludwiglierhammer]

@jtsiddons: I did some code restructuring to unify code snippets in mdf_reader and cdm_mapper. Unfortunately, I created some merge conflicts in this PR. Could you solve them or should we try to fix them together? This conflicts are only resolveable using the command line.

[jtsiddons]

@jtsiddons: I did some code restructuring to unify code snippets in mdf_reader and cdm_mapper. Unfortunately, I created some merge conflicts in this PR. Could you solve them or should we try to fix them together? This conflicts are only resolveable using the command line.

No worries - I can fix the conflicts. I am working on other things this week so I'll resolve them on Monday morning.

[jtsiddons]

@ludwiglierhammer: Have moved decode/convert and validate steps into a _read_loop method, meaning that only a single TextParser loop is required. This avoids significant refactor of the Configurator class to allow for conversion/validation in the open_* methods.

[jtsiddons]

Current status - re-factored validators for polars.DataFrame, mask is now generated in Configurator.open_* method, and passed as input to validator functions.

Next steps:

• Ensure that the mask values are correctly assigned for missing sections/disabled sections/ignored sections/missing values
• Ensure polars -> pandas type conversion is correct (note: pandas has two int64 types: int64 is not nullable, Int64 is nullable)
• Ensure column names are not converted to tuple if only one component to the name.

edit: add extra todo step

[jtsiddons]

May be of some use for this: https://narwhals-dev.github.io/narwhals/

Allows for interoperability between python DataFrame libraries (e.g. polars and pandas).