Find alternative ways to offload resource-heavy and time-consuming tasks

[nothingface0]

Description

Currently, on each parsing command issued for a HistogramDataFile, a new thread is spawned (from within Django) to parse the file.
Several issues with this approach:

• Django carries the burden of parsing files, which it should not
• The pod is pretty much overloaded, leading even to pod restarts.
• There is no way to prevent multiple Threads being spawned on the same file, wasting resources.

Next steps:

• Think about implementing a separate process which parses files, maybe even deployed on a separate pod, with its own API.
• Create a new DB model where a queue can be implemented for the separate process to check for files to parse. This has the added advantage of being able to resume file parsing after a crash, as the entry in the queue has not been cleared.

[nothingface0]

Asynchronous tasks that need to be offloaded to a separate process:

• Histogram File parsing (Very high cpu usage, takes a long time)
• RunRegistry/OMS API queries (takes a long time, may even timeout)

Celery seems to be an overkill. Perhaps simple Threads will do? (Histogram file parsing is already done this way)

---

Note: It must be taken into account that tasks may fail. Should there be a periodic task which cleans up/resumes those operations? Maybe the tasks are just put in a queue/table, and this periodic task checks what should be started/resumed/cleaned up.

---

Sources

1. https://stackoverflow.com/questions/21945052/simple-approach-to-launching-background-task-in-django
2. https://stackoverflow.com/questions/17601698/can-you-perform-multi-threaded-tasks-within-django