BUG: spark application can have multiple drivers at once

[maxgruber19]

Affected Stackable version

No response

Affected Apache Spark-on-Kubernetes version

No response

Current and expected behavior

Disclaimer, we're still in 25.3.0 with spark but currently upgrading to 25.11.0 - if you fixed that in between, just let me know but i didn't find a related issue fitting my problem

We found the cause for the situation of two or more same spark applications running at the same time. You can reproduce that by just deleting the spark-submit pod ungracefully (which is done in case of evictions or node outages) and even when gracefully shutting down the pod. After a couple of seconds, a new spark-submit pod is booting up, starting a new driver and the driver starts its executors. The problem: old driver and executors are still running. That leads to corrupted data, when to spark apps update the same dataset at the same time.

I expect a YARN like behaviour, when the submit is killed then a) the whole application should fail or b) the application should not be restarted and running application can continue its doing

Issue should be easily reproducible

Possible solution

No response

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

[razvan]

Thanks for the detailed explanation and sorry for the trouble.

a) is the correct fix but it's impossible to implement right now without a bigger change first. We'll have to drop the use of Job workloads and move the management of the spark-submit pod to the operator.

We'll implement b) as it's easy and hopefully helps a bit.

[maxgruber19]

Thank you for the quick reply as usual.

Both a) and b) are fine for us in the first place so we're looking forward for it anyways. Maybe a suggestion from my side, as it may not require so many changes: Isn't it possible to set the retry / backoff of the spark job to zero so that the job is not restarted that way? Maybe that's even less complicated

I would like to propose that the spark application is listed as UNKNOWN or something so that the submitter (custom airflow operator in our case) has the chance to decide what to do in that case. It would be a mess if the spark application continues but its status field is never updated in the end

[maxgruber19]

I see disabling the backoff is exactly what you did in your fix commit, thank you very much