add setup_namespace_services #4167

Rob-Johnson · 2025-11-21T08:59:43Z

building on @wjhoward's work in #4164 manage per-namespace k8s services in a cluster. largely based on what had previously been build in setup_istio_mesh.py, which is now unused.
tidy up what's left of the old setup_istio_mesh code, too.

Notable changes:

I'm using k8s server side apply to handle applying patches to the service. I think this is now the recommended way?
adding the name param to the ContainerPort we add for the service. This means I don't have to lookup the correct container_port for the service (and hope they're not different between backends in a service), but k8s can figure out which one to use by name. NB: will trigger a big bounce

Limitations:

this won't support cross-region advertising. we'll only see matching backends from the cluster the service belongs to. it also won't support the advertise/discover keys - everything in the cluster will be reachable.
I've chosen to put the service in the matching k8s namespace that the smartstack namespace belongs to (whichever service smartstack.yaml is in). that means:
- any backends for a service/instance that register in a different k8s namespace to the default paastasvc-$service won't show up.
- any backends for a service/instance that has a value in registrations for a namespace belonging to a different service won't show up.

docs/source/generated/paasta_tools.rst

this manages per-namespace k8s services in a cluster. largely based on what had previously been build in setup_istio_mesh.py, which is now unused. tidy up what's left of the old setup_istio_mesh code, too.

nemacysts

mostly just nits outside of get_service_kube_namespace() probably needing to actually read the service config to get the correct k8s namespace

nemacysts · 2025-11-25T12:28:46Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+def get_service_kube_namespace(service: str) -> str:
+    return f"paastasvc-{sanitise_kubernetes_name(service)}"


i think we'll need to load the actual config so that we can use the namespace getter - we allow folks to set custom namespaces and there's a small number of services still in the paasta namespace :(

nemacysts · 2025-11-25T12:30:49Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+def is_external_routing_enabled(config: Mapping) -> bool:
+    """Check if external routing is enabled for a namespace config.
+
+    Args:
+        config: Namespace configuration dict
+
+    Returns:
+        True if routing.external is set to True
+    """
+    routing_cfg = config.get("routing", {})
+    if not isinstance(routing_cfg, Mapping):
+        return False
+    return bool(routing_cfg.get("external"))


imo, this might be better placed in the instance class (and we can probably assume that these are in the right format/type due to the schema, right?)

nemacysts · 2025-12-02T15:03:36Z

paasta_tools/kubernetes_tools.py

+        if prometheus_port and all(
+            port.container_port != prometheus_port for port in ports
+        ):
+            ports.append(V1ContainerPort(container_port=prometheus_port))


Suggested change

if prometheus_port and all(

port.container_port != prometheus_port for port in ports

):

ports.append(V1ContainerPort(container_port=prometheus_port))

if prometheus_port and prometheus_port != self.get_container_port():

ports.append(V1ContainerPort(container_port=prometheus_port))

should be the same since the only other port we're adding a V1ContainerPort for is the app one, right?

nemacysts · 2025-12-02T15:06:07Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+    try:
+        with open(file_path) as f:
+            svc_namespaces = yaml.safe_load(f)
+            if not isinstance(svc_namespaces, Mapping):


imo, we can probably skip the isinstance checks - we should be guaranteed to see valid smartstack.yaml files since there's a schema

nemacysts · 2025-12-02T15:07:47Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+    return f"paastasvc-{sanitise_kubernetes_name(service)}"
+
+
+def is_external_routing_enabled(config: Mapping) -> bool:


heh, i guess we don't have a typed dict for smartstack configs - maybe next year (:p)!

nemacysts · 2025-12-02T15:31:53Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+    patch_kwargs = dict(
+        name=service.metadata.name,
+        namespace=kube_namespace,
+        body=serialized,
+        field_manager=FIELD_MANAGER,
+        force=True,
+        content_type="application/apply-patch+yaml",
+    )
+    if dry_run:
+        patch_kwargs["dry_run"] = "All"


imo, it'd be nice to keep using client-side apply just to avoid having to manage these kwargs dicts :p

(paasta is the only thing managing these, so SSA/client-side shouldn't be particularly different)

it is somewhat nice to have the server do validation for dry-runs, but not 100% sure how much more safety that buys us since these are pretty simple objects

nemacysts · 2025-12-02T15:32:36Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+        try:
+            kube_client.core.create_namespaced_service(**create_kwargs)
+            created = True
+            if not dry_run:


technically we don't need this since we're setting dry_run to true, right?

nemacysts · 2025-12-02T15:36:48Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+
+        create_kwargs = dict(namespace=kube_namespace, body=service)
+        if dry_run:
+            create_kwargs["dry_run"] = "All"


if we wanna SSA this, we're missing the other kwargs here, right?

nemacysts · 2025-12-02T15:39:06Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+            if created and not dry_run:
+                log.warning(
+                    f"Rolling back Service {kube_namespace}/{service.metadata.name} after apply error"
+                )
+                try:
+                    kube_client.core.delete_namespaced_service(
+                        name=service.metadata.name,
+                        namespace=kube_namespace,
+                    )
+                except Exception:
+                    log.exception(
+                        f"Failed to rollback Service {kube_namespace}/{service.metadata.name} after apply error"
+                    )
+            raise


hmm, I wonder if we should just create the Service as we'd want it to look over Create+Patch?

(i think that's the pattern we take in skj and whatnot - 3 different paths for create vs modify vs delete)

nemacysts · 2025-12-02T15:44:16Z

paasta_tools/setup_smartstack_namespace_k8s_svcs.py

+        svc = build_namespace_service(kube_namespace, namespace)
+        try:
+            server_side_apply_service(kube_client, kube_namespace, svc, dry_run)
+            yield (True, f"Reconciled Service {kube_namespace}/{svc.metadata.name}")


I assume we're using an iterator here to avoid having to maintain a list of results to return at the end?

Rob-Johnson force-pushed the namespace-service-manager branch 6 times, most recently from af4acb5 to 529251a Compare November 24, 2025 14:10

Rob-Johnson requested review from cuza, diwakar-dubey, ilkinmammadzada, nemacysts and wjhoward November 24, 2025 14:19

Rob-Johnson marked this pull request as ready for review November 24, 2025 14:19

Rob-Johnson requested a review from a team as a code owner November 24, 2025 14:19

Rob-Johnson requested a review from itsgc November 25, 2025 11:55

ilkinmammadzada reviewed Nov 25, 2025

View reviewed changes

docs/source/generated/paasta_tools.rst Outdated Show resolved Hide resolved

ilkinmammadzada previously approved these changes Nov 28, 2025

View reviewed changes

add setup_namespace_services

3953cb2

this manages per-namespace k8s services in a cluster. largely based on what had previously been build in setup_istio_mesh.py, which is now unused. tidy up what's left of the old setup_istio_mesh code, too.

Rob-Johnson dismissed ilkinmammadzada’s stale review via 3953cb2 November 28, 2025 15:11

Rob-Johnson force-pushed the namespace-service-manager branch from 529251a to 3953cb2 Compare November 28, 2025 15:11

nemacysts reviewed Dec 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add setup_namespace_services #4167

add setup_namespace_services #4167

Rob-Johnson commented Nov 21, 2025 •

edited

Loading

Uh oh!

Uh oh!

nemacysts left a comment

Uh oh!

nemacysts Nov 25, 2025

Uh oh!

nemacysts Nov 25, 2025

Uh oh!

nemacysts Dec 2, 2025

Uh oh!

nemacysts Dec 2, 2025

Uh oh!

nemacysts Dec 2, 2025

Uh oh!

nemacysts Dec 2, 2025

Uh oh!

nemacysts Dec 2, 2025

Uh oh!

nemacysts Dec 2, 2025

Uh oh!

nemacysts Dec 2, 2025

Uh oh!

nemacysts Dec 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		def get_service_kube_namespace(service: str) -> str:
		return f"paastasvc-{sanitise_kubernetes_name(service)}"

		return f"paastasvc-{sanitise_kubernetes_name(service)}"


		def is_external_routing_enabled(config: Mapping) -> bool:

add setup_namespace_services #4167

Are you sure you want to change the base?

add setup_namespace_services #4167

Conversation

Rob-Johnson commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nemacysts left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Rob-Johnson commented Nov 21, 2025 •

edited

Loading