Redesign parsing of lumisection histograms

[vicha-w]

Currently, histogram parsing in Histogram File Manager on GUI forces users to send the parsing request twice when the file contains 1D and 2D histograms, such as in DQMIO files that contain both.

We need to redesign the parsing steps so that users can send the request to parse one file once, and the machine should figure out automatically which histograms go to which category (1D, 2D, run, lumi) and which file format it is (CSV or DQMIO).

[GuillermoFidalgo]

Posting here some old notes relevant to this task

• Implement some robustness against pods being killed and extend the resources limits
• Try to implement some robustness to the parsing script with things like :
  • a queue
  • Dict or list with info of files that were not parsed or skipped
  • status flag {status: [”submitted”| “parsed”]}
  • Use these to add some checks after the parsing script has finished executing and verify the status of each submitted file.

[GuillermoFidalgo]

Above comments arise from the fact that when submitting large quantities of files, some of them are "submitted" but never parsed. The API returns an OK response because it receives the request for a file but inspection of the Playground shows few of them to never be process (basically skipped)

relevant slides here