question: is there a way to limit ephemeral storage usage of spark pods?

[maxgruber19]

When running very big jobs via spark we see that sometimes the ephemeral volumes of executors are filled endlessly. That leads to fully used ephemeral storage of the node which then leads to consequences for other pods on that node because they are running out of ephemeral storage as well. We had some trino downtime because of that.

The Volume causing it in the executor looks like the following, seems like a kind of spilling.

A second thing which is arising out of it, is a second driver spawning in a magical way. I can't explain it yet but in rare cases, the spark application suddenly has 2 running drivers. I'll do my best to provide further details on that.

question

is there a way to configure ephemeral storage limits (and maybe more properties) globally for all spark applications the operator applies? it's okay for us to fail one application, but it's a big mess if any application can kill nodes by starting to spill on the nodes filesystem

[razvan]

Hi,

yeah that looks like shuffle data. You can try creating spark-local-dir-* volumes as described here which spark will use instead of the ephemeral volume.

To create the volumes/mounts you can use the SparkApplication CRD fields or podOverrides in addition to the Spark configuration properties.

[maxgruber19]

Thanks as usual @razvan, we will have a look on that

Do you plan to add a configuration to be set on operator level which sets a couple of properties for all spark applications deployed in the cluster? We have ~200 spark applications and some are written / deployed by customers and it would be a mess to add those "global configs" for each of them

[razvan]

Do you plan to add a configuration to be set on operator level which sets a couple of properties for all spark applications deployed in the cluster?

It's not planned yet, but it does sound like a good idea. I can only promise that we'll discuss it in the next planning meeting and I'll update this issue here.

[maxgruber19]

Edit: The problem with two drivers of the same application is added in a separate issue #646