Conformance: Adds Data Parallelism Test #1769
Conversation
k8s-ci-robot
added
the
area/conformance-test
|
@danehans: The label(s) In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: danehans The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
approved
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>
| targetPorts: | ||
| - number: 3000 | ||
| - number: 3002 | ||
| - number: 3004 |
There was a problem hiding this comment.
While this is an interesting configuration, I don't think you could do this with a real vLLM server
| spec: | ||
| containers: | ||
| - name: echoserver-3000 | ||
| image: gcr.io/k8s-staging-gateway-api/echo-basic:v20240412-v1.0.0-394-g40c666fd |
There was a problem hiding this comment.
Other than your use of non-contiguous ports here, why not not use llm-d-inference-sim which supports --data-parallel-size=N ?
| { | ||
| name: "DP routes only to all pods (EPP returns all; ranks balanced internally)", | ||
| podIPsToBeReturnedByEPP: []string{podIPs[0], podIPs[1], podIPs[2]}, | ||
| expectAllRequestsRoutedWithinPodNames: []string{podNames[0], podNames[1], podNames[2]}, |
There was a problem hiding this comment.
You are only checking Pod IPs. Shouldn't youalso be checking Pod Ports?
|
|
||
| // Filter selects pods that match the IP addresses specified in the request header. | ||
| // Filter selects pods whose IPs match any value in the "test-epp-endpoint-selection" header. | ||
| // Values may be "IP" or "IP:port"; ports (ranks) are ignored here because DP fan-out happens later. |
There was a problem hiding this comment.
Why do you want to ignore the port? I would have thought you would add code that if a port was specified, it filters by both IP and port.
Without that how do you know that DP really worked and sent requests to a "non-base" of the model server?
3 participants
What type of PR is this?
/kind test
/area conformance-test
What this PR does / why we need it:
v20251023-d788a2c.Which issue(s) this PR fixes:
Fixes #1680
Does this PR introduce a user-facing change?: