feat(inference): multi-route proxy with alias-based model routing

[cosmicnet]

Summary

Adds multi-route inference proxy support, allowing sandboxed agents to reach multiple LLM providers (OpenAI, Anthropic, NVIDIA, Ollama) through a single inference.local endpoint. Agents select a backend by setting the model field to an alias name. Also adds Ollama native API support and Codex URL pattern matching.

Related Issue

Closes #203

Changes

• Proto: Add InferenceModelEntry message (alias, provider_name, model_id); add models repeated field to set/get request/response messages
• Server: upsert_multi_model_route() validates and stores multiple alias→provider mappings; resolves each entry into a separate ResolvedRoute at bundle time
• Router: select_route() implements alias-first, protocol-fallback selection; proxy_with_candidates/proxy_with_candidates_streaming accept optional model_hint
• Sandbox proxy: Extracts model field from request body as model_hint for route selection
• Sandbox L7: Add /v1/codex/*, /api/chat, /api/tags, /api/show inference patterns
• Backend: build_backend_url() always strips /v1 prefix to support both versioned and non-versioned endpoints (e.g. Codex)
• Core: Add OLLAMA_PROFILE provider profile with native + OpenAI-compat protocols
• CLI: --model-alias ALIAS=PROVIDER/MODEL flag (repeatable, conflicts with --provider/--model)
• Architecture docs: Updated inference-routing.md with all new sections

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[cosmicnet]

I have read the DCO document and I hereby sign the DCO.

[Copilot]

Pull request overview

Adds multi-route inference proxying so sandboxes can route inference.local requests to multiple LLM backends by using a model alias in the request body.

Changes:

• Extends the inference proto + gateway storage to support multiple (alias, provider_name, model_id) entries per route.
• Adds alias-first route selection in the router and passes a model_hint extracted from sandbox request bodies.
• Expands sandbox L7 inference patterns and adds an Ollama provider profile + endpoint validation probe.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
proto/inference.proto	Adds InferenceModelEntry and models fields for multi-model inference config.
crates/openshell-server/src/inference.rs	Implements multi-model upsert + resolves each alias into separate ResolvedRoute entries.
crates/openshell-sandbox/src/proxy.rs	Extracts model from JSON body and forwards it as model_hint to the router.
crates/openshell-sandbox/src/l7/inference.rs	Adds Codex + Ollama native API patterns and tests.
crates/openshell-router/src/lib.rs	Adds select_route() and extends proxy APIs to accept model_hint.
crates/openshell-router/src/backend.rs	Adds Ollama validation probe and changes backend URL construction behavior.
crates/openshell-router/tests/backend_integration.rs	Updates tests for new proxy function signatures and /v1 endpoint expectations.
crates/openshell-core/src/inference.rs	Adds OLLAMA_PROFILE (protocols/base URL/config keys).
crates/openshell-cli/src/run.rs	Adds gateway_inference_set_multi() to send multi-model configs.
crates/openshell-cli/src/main.rs	Adds --model-alias ALIAS=PROVIDER/MODEL CLI flag and dispatch.
architecture/inference-routing.md	Documents alias-based route selection, new patterns, and multi-model route behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cosmicnet]

@pimlock Happy to address any feedback or questions. Let me know if you'd like anything restructured or split differently.

[johntmyers]

The use of inference.local was to provide a default model at a minimum for all sandboxes to have access to (if configured). We're cautious not to bloat the embedded sandbox inference router to support arbitrary upstream providers and models and also turn this into a larger scale model router that will have consistent maintenance. We're still determining what level of routing support we should have on our roadmap.

I am curious, if you need such level of routing support, have you considered setting up a dedicated proxy/router that is accessible outside of the sandbox and just configuring access to it with network policies? This is a typical pattern we have several users follow.

[pimlock]

Is there a reason for using ollama inference protocol, rather than OpenAI one? Is there something extra that ollama supports that cannot be accessed through OpenAI one?

[cosmicnet]

Ollama exposes native endpoints (/api/chat, /api/tags, /api/show) that provide capabilities not available through its OpenAI-compatible layer:

/api/tags lists all locally available models (no OpenAI equivalent)
/api/show returns model metadata: parameters, template, license, quantization info
/api/chat supports Ollama-specific options like num_ctx, num_predict, temperature variants, and raw mode
The OLLAMA_PROTOCOLS list includes both native and OpenAI-compatible protocols (openai_chat_completions, openai_completions, model_discovery), so agents can use either interface. The native protocols are there so tools that use the Ollama client library directly (which targets /api/*) work through inference.local without needing to switch to the OpenAI-compat paths.

If you'd prefer to keep it simpler and only support Ollama through its OpenAI-compat layer, I can drop the native patterns and the ollama_chat/ollama_model_discovery protocols. The tradeoff is that model discovery (/api/tags) and agent tooling that uses the Ollama SDK directly wouldn't work.

[cosmicnet]

The use of inference.local was to provide a default model at a minimum for all sandboxes to have access to (if configured). We're cautious not to bloat the embedded sandbox inference router to support arbitrary upstream providers and models and also turn this into a larger scale model router that will have consistent maintenance. We're still determining what level of routing support we should have on our roadmap.

I am curious, if you need such level of routing support, have you considered setting up a dedicated proxy/router that is accessible outside of the sandbox and just configuring access to it with network policies? This is a typical pattern we have several users follow.

Thanks for the feedback. This PR follows the approach outlined in #203 (option B: single record with repeated entries, alias-first selection with protocol fallback, model hint from the request body). I appreciate that was closed off citing the replacement issue #207, but that covers a different concern. System vs user inference is about who the route serves, not how many backends it can reach. This PR already accommodates that split through the system route guard and separate sandbox.inference.local endpoint.

On the external proxy: it's a valid pattern, but the overhead feels disproportionate here. This is a static alias lookup table. There's no load balancing, retries, rate limiting, or discovery. The maintenance surface is one function, one proto field, and one server method. For users with 2-3 providers, standing up a separate proxy service is a lot of ceremony for a lookup table.

More broadly, my understanding is that NemoClaw/OpenShell is positioned as an enterprise-ready platform for running AI agents securely out of the box. In that context, multi-model access feels like a baseline expectation rather than an edge case. Agents routinely need a fast cheap model for simple tasks and a more capable one for complex reasoning, or a specialised model for specific domains. If each of those requires its own external proxy and network policy, that's a significant barrier to the "out of the box" experience. Maybe I'm misunderstanding the intended scope, but it's hard to see how single-model inference serves that use case long term.

If the team has decided this doesn't belong in the embedded proxy, I can scope this down to just the Ollama native API support and Codex pattern matching (commits 1-2) and drop the multi-model routing. Happy to go either way.