feat(completions): add native gRPC pipeline typing for /v1/completions

[vschandramourya]

Description

Problem

This PR continues the /v1/completions rollout by making completions a
first-class native request type in the gRPC pipeline.

Even with a validated request contract, the gRPC pipeline did not yet treat
completions as a native request type, which made /v1/completions
impossible to support cleanly through the same pipeline contract used by
chat, generate, embeddings, and other endpoints.

Solution

Introduce CompletionRequest as a first-class gRPC pipeline request type and
extend shared pipeline orchestration to carry completion requests and
completion responses natively.

This PR adds the native completion pipeline type, metadata flow, and
completion entrypoint, while leaving the actual completion stage wiring to the
later stacked PRs.

Changes

• Add RequestType::Completion
• Add FinalResponse::Completion
• Add completion request context construction and typed accessors
• Update dispatch metadata to read completion model IDs directly
• Make Harmony/shared stages exhaustive for completion requests
• Add execute_completion() to the gRPC request pipeline
• Temporarily keep regular request-building and response-processing
  delegators buildable until native completion stages land in the stacked follow-up PRs

Test Plan

• cargo test -p smg completion --quiet
• cargo clippy -p smg --all-targets --all-features -- -D warnings

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings passes
• (Optional) Documentation updated
• (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: e209ad96-3865-45fd-89aa-6c7a16ec16f5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/completions-pipeline-plumbing

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the /v1/completions functionality more deeply into the gRPC pipeline by establishing it as a native request type. This foundational change involves defining new data structures for completion requests and responses, updating the request context to manage these types, and introducing a dedicated execution path within the gRPC pipeline. The changes also include robust validation for completion parameters and ensure that existing pipeline stages correctly acknowledge the new request type, laying the groundwork for subsequent feature development.

Highlights

• Native gRPC Pipeline Support for Completions: The /v1/completions endpoint is now treated as a first-class native request type within the gRPC pipeline, enabling cleaner integration and future development.
• New Request and Response Types: Introduced RequestType::Completion and FinalResponse::Completion to represent completion requests and responses natively within the gRPC context.
• Completion Request Context and Accessors: Added dedicated constructors (for_completion) and typed accessors (completion_request, completion_request_arc) for CompletionRequest within the RequestContext.
• Enhanced Request Validation: Implemented comprehensive validation rules for the CompletionRequest structure, including checks for prompt emptiness, stream_options usage, min_tokens vs max_tokens, and best_of vs n parameters.
• Pipeline Entrypoint Added: A new execute_completion() function was added to the RequestPipeline, serving as the dedicated entrypoint for processing completion requests through the gRPC pipeline.
• Harmony Model Compatibility Updates: Updated Harmony-specific request building and response processing stages to explicitly handle Completion requests as unsupported, providing clearer error messages.

Changelog

• crates/protocols/src/completion.rs
  • Added Validate derive to CompletionRequest for declarative validation.
  • Imported validator::Validate and crate::validated::Normalizable.
  • Added #[validate] attributes to various fields within CompletionRequest (e.g., prompt, max_tokens, temperature, top_p, n, logprobs, stop, presence_penalty, frequency_penalty, best_of, top_k, min_p, min_tokens, repetition_penalty).
  • Implemented Normalizable trait for CompletionRequest.
  • Added validate_completion_prompt function to ensure prompts are not empty.
  • Added validate_completion_cross_parameters function for interdependent parameter validation (e.g., stream_options with stream, min_tokens vs max_tokens, single structured output constraint, best_of vs n).
• model_gateway/src/routers/grpc/common/stages/dispatch_metadata.rs
  • Updated the match statement in DispatchMetadataStage::execute to include RequestType::Completion for extracting the model ID.
• model_gateway/src/routers/grpc/context.rs
  • Imported CompletionRequest and CompletionResponse from openai_protocol::completion.
  • Added Completion(Arc<CompletionRequest>) variant to the RequestType enum.
  • Added Completion variant to the FinalResponse enum.
  • Implemented for_completion constructor in RequestContext to create a context for completion requests.
  • Added completion_request and completion_request_arc methods to RequestContext for accessing completion request data.
  • Updated the is_stream method in RequestContext to handle RequestType::Completion.
• model_gateway/src/routers/grpc/harmony/stages/request_building.rs
  • Refactored the match statement in HarmonyRequestBuildingStage::execute to explicitly handle RequestType::Completion, RequestType::Embedding, RequestType::Classify, and RequestType::Messages as unsupported, providing specific error messages for each.
• model_gateway/src/routers/grpc/harmony/stages/response_processing.rs
  • Refactored the match statement in HarmonyResponseProcessingStage::execute to explicitly handle RequestType::Completion, RequestType::Embedding, RequestType::Classify, and RequestType::Messages as unsupported, providing specific error messages for each.
• model_gateway/src/routers/grpc/pipeline.rs
  • Imported CompletionRequest from openai_protocol::completion.
  • Updated error handling in execute_chat and execute_generate to include FinalResponse::Completion as an unexpected response type.
  • Added a new asynchronous function execute_completion to handle the complete pipeline for completion requests, including metrics recording and error handling.
  • Updated error handling in get_chat_response to include FinalResponse::Completion as an unexpected response type.
• model_gateway/src/routers/grpc/regular/stages/request_building.rs
  • Added RequestType::Completion to the list of request types that are not supported by the regular request building stage, resulting in a not_supported_in_regular error.
• model_gateway/src/routers/grpc/regular/stages/response_processing.rs
  • Added RequestType::Completion to the list of request types that are not supported by the regular response processing stage, resulting in a not_supported_in_regular error.
• model_gateway/src/server.rs
  • Changed the v1_completions endpoint handler to use ValidatedJson(body) instead of Json(body) for CompletionRequest, enabling automatic validation.
• model_gateway/tests/spec/completion.rs
  • Added a new test file for CompletionRequest validation.
  • Implemented base_request helper function to create a default CompletionRequest.
  • Added unit tests to verify that an empty prompt string or array items result in validation errors.
  • Added a unit test to confirm that a valid prompt array passes validation.
  • Added a unit test to ensure stream_options requires stream to be enabled.
  • Added a unit test to check that best_of must be greater than or equal to n.

Activity

• The pull request author, vschandramourya, has implemented the initial plumbing for native gRPC pipeline typing for /v1/completions.
• The author has provided a detailed description outlining the problem, solution, and specific changes.
• The author has confirmed that cargo test -p smg completion --quiet and cargo clippy -p smg --all-targets --all-features -- -D warnings pass, indicating initial code quality and test coverage for the new completion features.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively introduces CompletionRequest as a native type within the gRPC pipeline, which is a great step towards cleanly supporting the /v1/completions endpoint. The addition of comprehensive validation for CompletionRequest at the protocol level is a significant improvement for robustness. The new pipeline entrypoint execute_completion and the necessary updates across the context and stages are well-implemented and consistent with the existing architecture. I have a couple of minor suggestions to improve code clarity and style, as detailed in the comments.

[gemini-code-assist]

This is a bit verbose for counting boolean flags. You can make this more concise and arguably more idiomatic by casting the booleans to integers and summing them up. This aligns with prioritizing code simplicity and clarity.

Suggested change

	let constraint_count = [
	req.regex.is_some(),
	req.ebnf.is_some(),
	req.json_schema.is_some(),
	]
	.iter()
	.filter(|&&enabled| enabled)
	.count();
	let constraint_count = req.regex.is_some() as u8
	+ req.ebnf.is_some() as u8
	+ req.json_schema.is_some() as u8;

References

1. Prioritize code simplicity and clarity over micro-optimizations, especially when the performance gain is negligible for typical use cases (e.g., iterating over a small number of items).

[CatherineSue]

Overall LGTM. Just a few issues regarding the request_type and final_response format.

[CatherineSue]

The original code already supports request_type. We don't need to boilerplate here anymore.

You can simply do | RequestType::Completions

[CatherineSue]

Same here.

[CatherineSue]

We might need to do a Fmt impl for FinalResponse like what we did for RequestType. Then we would simply do response_type @ here.

[CatherineSue]

Please don't open a stack PR. Merge commit to main instead.

[CatherineSue]

hi @vschandramourya , the PR looks great. Only couple of issues need to be addressed. Posted above.

I do recommend to close this PR and open another one to merge into main.