TextFileReader needed?

[ludwiglierhammer]

Current state

Optionally, you can select a chunksize when reading the data. If this is the case, the data is read as a TextFileReader object for iterations; see pandas.read_csv. This complicates the code because manipulating TextFileReader objects causes iterations and high I/O loads. In earlier pandas versions, reading chunked data freed up memory and made the procedure faster.

Question

Is it necessary to make use of chunked TextFileReader objects are can newer pandas versions can handle this issue?

Possible Solution

Get rid of TextFileReader objects.

Advantage

Simplify the code.

[ludwiglierhammer]

Hi @jtsiddons, do you set chunksizes when using the mdf_reader? Because newer pandas versions can process large inputs, I think we do not need this anymore.

[jtsiddons]

Hi @ludwiglierhammer, we have on ocassion used the chunk-size component - mainly for the larger ICOADS NRT files, or ICOADS total files after 2000 or so. The reader for these files seem to use a lot of memory, I'm not sure the chunksize parameter makes much difference.

I think the memory usage problems occur after the initial read - do you know if the chunks are combined after the initial read?

Can we remove the python 3.11.9 issue from the README? I've not experienced any issues on 3.11.9. Is 3.11.9 the version in the ci-tests?

[ludwiglierhammer]

@jtsiddons: The chunks are not combined after the initial read. When manipulating the TextFileReader object it is opened chunk by chunk. For each chunk the manipulation will be done. Then the manipulated chunk is saved in a StringIO buffer and written to disk.
Afterwards the file on disk is read in again as a TextFileReader object. Is yields high I/O loads.

E.g. see the difference between pd.DataFrame and TextFileReader objects.

Can you please test your larger ICOADS files without the chunk-size component? Do you see any differences?

[ludwiglierhammer]

@jtsiddons: The python 3.11. test succesfully runs with python 3.11.9. I'll remove the python 3.11.9 issue from the README. Thanks for the hint.

[jtsiddons]

@ludwiglierhammer: Thanks for the info. I'll run a larger file on one of our servers on Monday morning.

[jtsiddons]

@ludwiglierhammer Do you want me to test on main or v0.3.0? I can look at running mdf_reader.read through the memory-profiler package too

e: branch name

[ludwiglierhammer]

@jtsiddons:You can run the tests on your fork. It does not matter which branch you actually use. No changes have been made to the corresponding code snippets since v3.0.0.

Do you mean this package: https://pypi.org/project/memory-profiler/
Unfortunately, this is no longer maintained. I do not know this package. Is it trustworthy?

[jtsiddons]

@ludwiglierhammer - good spot - I hadn't noticed it was no-longer maintained. I've not had any issues with it in the past. I'll leave the profiler for now.

[jtsiddons]

@ludwiglierhammer: Update from run with chunksize=None. I have cancelled the job it was close to memory limits (~200G). It did not get past "Getting data string from source stage". I did select one of the largest ICOADS NRT Total files.

Manual reporting, data from top.
Python : 3.11.3
OS: Rocky 9

in_file = "./IMMA1_R3.0.2T_2023-12"
out_dir = "./mem_test_out"

chunksize = None
mdf_reader.read(in_file, data_model="imma1", out_path=out_dir, chunksize=chunksize)

File: IMMA1_R3.0.2T_2023-12
Size on disk : 3.0G (3_117_986_480)
Lines in file: 6_195_778

chunksize=None :

Stage	VIRT	RES	TIME+
"Getting data string from source"	65.2g	64.0g	30:00
^^^	95.9g	94.4g	45:04
^^^	127.0g	125.1g	60:01
^^^	157.7g	155.4g	75:00
^^^	180.1g	177.7g	90:02

File "CDM_READER_MAPPER/./test_mem_use.py", line 17, in <module>
    main()
  File "CDM_READER_MAPPER/./test_mem_use.py", line 12, in main
    mdf_reader.read(in_file, data_model="imma1", out_path=out_dir, chunksize=chunksize)
  File "CDM_READER_MAPPER/cdm_reader_mapper/mdf_reader/read.py", line 321, in read
    return MDFFileReader(
           ^^^^^^^^^^^^^^
  File "CDM_READER_MAPPER/cdm_reader_mapper/mdf_reader/read.py", line 241, in read
    self.data, self.isna = self._open_data(
                           ^^^^^^^^^^^^^^^^
  File "CDM_READER_MAPPER/cdm_reader_mapper/mdf_reader/utils/auxiliary.py", line 512, in _open_data
    df, self.missings = self._read_sections(
                        ^^^^^^^^^^^^^^^^^^^^
  File "CDM_READER_MAPPER/cdm_reader_mapper/mdf_reader/utils/auxiliary.py", line 476, in _read_sections
    df = TextParser.apply(
         ^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/pandas/core/frame.py", line 9423, in apply
    return op.apply().__finalize__(self, method="apply")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/pandas/core/generic.py", line 5931, in __finalize__
    @final

KeyboardInterrupt

I will run with chunksize set this afternoon.

e: complete incomplete sentence

[jtsiddons]

Results with chunksize. The reading step uses much less memory, so obviously this does have an effect. However, even with chunksize I would not be able to run this on a compute/data centre server such as JASMIN for example, where memory limits tend to apply.

chunksize=100_000 :
NOTE - Got to > 100g for a few seconds at 117:00 (end of read?)
NOTE - Got to > 150g for a few seconds at 162:00 (end of validation?)

Stage (approximate)	VIRT	RES	TIME+
"Getting data string from source"	9.9g	9.3g	29:43
^^^	12.9g	12.3g	45:00
^^^	16.0g	15.4g	61:30
^^^	18.7g	18.1g	75:00
^^^	21.6g	21.0g	90:00
^^^	25.5g	24.8g	108:38
^^^ - "Convert/Decode"?	57.0g	56.4g	116:54
^^^	71.1g	70.5g	117:57
"Validate Entries"	108.6g	108.0g	141:57
"Validate Entries"	113.0g	112.4g	159:43
"Writing Data Files"	144.3g	143.7g	162:54

Total elapsed ~ 175 mins

I may re-run the test at some-point for a (large) subset of the data (say 2 million rows), to see how the post-read stages look memory-wise.

[jtsiddons]

Final output size on disk:

 54K Jun 24 15:41 atts.json
4.0G Jun 24 15:41 data.csv
8.9G Jun 24 15:41 mask.csv

[ludwiglierhammer]

@jtsiddons: Thanks for your tests. We have to think how we deal with such large files. Maybe we should use dask.DataFrames instead of pandas DataFrames. One can simply read csv tables in a dask DataFrame using dask.dataframe.read_csv. I think this could speed up processing large files and reduce the memory load.