Docs: Create EPP Operations Guide

[danehans]

Questions such as "How many InferencePools does an EPP support?" have been asked several times recently. A user guide should be added that provides steps for performing common EPP operations tasks.

[kfswain]

This issue's context is currently scoped to those familiar with GIE, and I think it will need to stay that way given the subject matter.

However, if someone feels confident in picking this issue up, but requires more context, feel free to ping anyone here: https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/OWNERS_ALIASES

[danehans]

cc: @smarterclayton @kfswain @ahg-g @nirrozenbaum, please add comments for EPP operations tasks that should be documented. Here are a few that come to mind:

1. Upgrades/Rollbacks
2. Troubleshooting
3. Scaling EPP or provide auto scaling details (when supported).
4. Monitoring and observability- We already have an EPP metrics doc and Grafana dashboards but users still need to manually stitch things together and fill-in some gaps.
5. Securing EPP- Currently EPP uses a self-signed cert (xref). Should cert manager be used for this doc?
6. EPP Perf Optimization- How can a user optimize EPP for perf based on common use cases.

Should the guide cover inference platform and inference workload tasks?

[nirrozenbaum]

• Building EPP with pluggable scheduler logic, e.g., configure scheduler plugins during build time (support added in Move filter and scorer plugins registration to a separate file #729, usage added in Weighted scorers #737).
• How to configure EPP collected metrics (when supported) as documented in extensible data layer: EPP should allow configurable metrics collection #703.
• not exactly EPP, but also good candidate for something to document - how to use BBR together with EPP

other than operations tasks, I think this document should explain the mental model of EPP and the why.

[danehans]

I think this document should explain the mental model of EPP and the why.

IMHO, this should be in the concepts section of the docs.

[danehans]

Additional background for the Securing EPP section:

By default, the --secure-serving=true and --cert-path="" CLI args are set for the EPP, causing the gRPC ext-proc server to create and use a self-signed certificate that is stored in memory. Since the Gateway acts as the client, it must disable TLS verification for the EPP's certificate, so it will not check the cert for validity. This makes the Gateway<>EPP ext-proc connection insecure. The decision on which cert to use is a property of the EPP deployment (potentially InferencePool?). The decision of how to connect to the EPP and verify that the cert is a Gateway responsibility. The operations guide should clarify this since there has been confusion about this in the past.

[danehans]

From discussions with @robscott, the plan is to use BackendTLSPolicy, which targets the EPP Service, to convey the cert information required for secure Gateway<->EPP comms.