feat(docs): KEP- Spark Client for Kubeflow SDK

[Shekharrajak]

Ref https://docs.google.com/document/d/1l57bBlpxrW4gLgAGnoq9Bg7Shre7Cglv4OLCox7ER_s/edit?tab=t.0

PR: #158

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 19438736784

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 66.596%

---

Totals
Change from base Build 19231750341:	0.0%
Covered Lines:	2506
Relevant Lines:	3763

---

💛 - Coveralls

[Shekharrajak]

Users can extend the backend, if they want to have any specific changes or different way to connect or submit spark job

[andreyvelich]

Thanks for this great effort @Shekharrajak!
I left my initial thoughts.

[andreyvelich]

For Trainer, backends represent various job submission (local subprocess, container, and Kubernetes). I am not sure if we can replicate it for Spark.

[Shekharrajak]

Here we have job submission using K8S Operator backend, Spark Connector backend , Gateway backend (not implemented - we just have abstract class).

[andreyvelich]

I am wondering what is the main motivation to separate SessionClient() and BatchClient()?
Alternatively, we can just have unified SparkClient() which has sessions and batch APIs:

submit_job() <-- to create Spark Application and submit batch job
connect() <-- to create session and connect to existing Spark cluster

[Shekharrajak]

• BatchSparkClient users never see connect(), create_session() methods
• SparkSessionClient users never see submit_application(), wait_for_job_status() methods
• This prevents runtime errors: Can't call wait_for_job_status() on a session object.

[Shekharrajak]

This helps user to know which methods are available clearly :

 client = SparkClient()
  # User sees BOTH batch AND session methods
  client.submit_job(...)        # For batch
  client.connect(...)           # For session
  client.get_job(...)           # Works with connect() or batch()  ? 

 # event if we take arg in config: 
  client = SparkClient(mode="batch")
  client.create_session(...)  # IDE will not show error, but runtime error 

[andreyvelich]

Can you explain reason of creating various backends? Can we just have an API: SparkClient().connect() which creates SparkConnect CR and connects to the existing cluster as we discussed?

[Shekharrajak]

The batch backend will have APIs like submit_application, wait_for_completion, get_logs, .. where user will just submit the job and can check the logs/results.
example: https://github.com/kubeflow/sdk/pull/158/files#diff-e692a5819ee6b1dc00cba3b58e91f058c0022d3ca9aa6f3ee468126f245eef89R89

But with interactive session spark client user will be able to run interactive SQL queries and DataFrame operations.
example: https://github.com/kubeflow/sdk/pull/158/files#diff-a5011f48c9d6d16ff6ddd65588f65a7c78abf5fbeb121cccb693c0892ce3a5aeR275

[andreyvelich]

I am wondering if there is a way to allow users bypass function to SparkApplication similar to Trainer API: https://github.com/kubeflow/sdk?tab=readme-ov-file#run-your-first-pytorch-distributed-job

That might be interesting to explore how we can allow to submit SparkApplication without building an image.

[andreyvelich]

This is pretty interesting experience for dev -> prod Spark lifecycle.
cc @shravan-achar @akshaychitneni @bigsur0 to explore.

[andreyvelich]

Suggested change

	trainer_client.train(
	name="train-model",
	func=train_func,
	num_nodes=4,
	)
	trainer_client.train(
	trainer=CustomTrainer(
	func=train_func,
	num_nodes=4
	)
	)

[andreyvelich]

cc @kubeflow/wg-pipeline-leads to explore