feat: add SlurmQos CRD for declarative QOS management#158
feat: add SlurmQos CRD for declarative QOS management#158faganihajizada wants to merge 1 commit intoSlinkyProject:mainfrom
Conversation
faganihajizada
force-pushed
the
feat/qos
branch
from
58a389e to
09c0afa
Compare
faganihajizada
force-pushed
the
feat/qos
branch
from
09c0afa to
fe8af86
Compare
|
SlinkyProject/slurm-client#2 merge and release is needed before merging this PR |
|
Accounting to Controller is 1:N. QoS is global to all Controller CR using the same Accounting CR. Therefore, architecturally the Another issue that is not handled in the code is ownership of the QoS in Slurm. If I were to define 2 SlurmQoS both targeting the same QoS by name (e.g. normal), what stops the endless update/overwrite. Who wins and how? How is rejection handled? I see why a raw v44 QoS is transparently passed from CR to client, but that is certainly a precarious thing to do. If the operator changes Slurm API version, the raw QoS may silently break if the Slurm API changed subtly. And the operator does not let the admin specify Slurm API version. |
| slurmQos, err := buildV0044Qos(qosCopy) | ||
| if err != nil { | ||
| statusErr := r.syncStatus(ctx, qosCopy, false, nil, fmt.Sprintf("failed to build QOS payload: %v", err)) | ||
| if statusErr != nil { | ||
| logger.Error(statusErr, "failed to update status") | ||
| } | ||
| return err | ||
| } | ||
|
|
||
| body := slurmapi.V0044OpenapiSlurmdbdQosResp{ | ||
| Qos: []slurmapi.V0044Qos{slurmQos}, | ||
| } | ||
| qosObj := &slurmtypes.V0044Qos{} | ||
| qosObj.Name = slurmQos.Name | ||
| if err := slurmClient.Create(ctx, qosObj, body); err != nil { | ||
| statusErr := r.syncStatus(ctx, qosCopy, false, nil, fmt.Sprintf("failed to create/update QOS: %v", err)) | ||
| if statusErr != nil { | ||
| logger.Error(statusErr, "failed to update status") | ||
| } | ||
| return err | ||
| } |
There was a problem hiding this comment.
All Slurm API handling should be abstracted through the slurmcontrol concept.
2 participants
Summary
Add declarative Slurm QOS management via a new
SlurmQosCRD with controller and webhook. QOS policies are reconciled against the Slurm REST API (slurmdb endpoints), writing directly to the accounting database: noscontrol reconfigurerequired.This supports the strategic shift from partition-centric to QOS-centric scheduling: fewer partitions, fine-grained preemption hierarchies, live policy updates, and lower scheduler load.
New CRD:
SlurmQoscontrollerRefscopes to a Controller CR (requires accounting enabled)descriptionfor human-readable QOS descriptionconfigasruntime.RawExtensionmetadata.namemaps directly to the Slurm QOS nameController (
internal/controller/slurmqos/):ready,id(Slurm-assigned), andSyncedconditionWebhook (
internal/webhook/slurmqos_webhook.go):Helm chart:
qosmap in values.yaml (gated byaccounting.enabled)Depends on: SlinkyProject/slurm-client#2
Breaking Changes
N/A
Testing Notes
helm unittest helm/slurm/ --file tests/qos_test.yamlkubectl patchreflected in Slurm within secondsSynced=Falsewith clear message when accountingRef is nilmake generate && make manifestsAdditional Context
configfield usesruntime.RawExtensionwhich is new to this operator's CRDs.Justified because QOS config maps to arbitrary JSON for the REST API (unlike partition config which is a flat key=value string). This is the standard K8s approach for embedding opaque JSON in CRDs.
metadata.namemust match the exact Slurm QOS name. ThecontrollerRefprovides cluster scoping.accounts={"nvidia": "denied"}) are out of scope: they require a separateSlurmAssociationCRD as a follow-up.DENY_LIMIT, notDenyOnLimit).Known Limitations
accounts={"account_name": "denied"}) cannot be managed via theV0044QosREST API. These requiresacctmgrassociations. Follow-up: aSlurmAssociationCRD using the same architecture pattern. Theslurm-clientalready has generated OpenAPI types for associations. I will create separate PR.normalQOS needsmax_tres_pu.nodes = 0.5 * cluster_node_count. Users must compute and set the absolute value at deploy time. The Helm values comments document the formula.