Provide a way to debug IOCs in cluster

[gilesknap]

I would like to add a function to ec like this:-

ec debug services/bl01t-ea-test-01

which will

• stop the existing ioc if it is running (whatever version of it is running already - needs to be agnostic about version)
• create a raw K8S manifest for debugging the developer version of the IOC mentioned in services/bl01t-ea-test-01/values.yaml
• apply that manifest to the cluster
• kubectl copy services/bl01t-ea-test-01/values.yaml to pod:/epics/ioc/config
• show instructions for how to connect vscode to the pod and start the IOC
• also show instructions on how to connect phoebus to the PVs in that pod (I suggest that it runs hostNetwork=false but we can make a service and point phoebus at that -- this involves extracting which IP the service was assigned - or maybe we can use my in cluster gateway approach instead)

This is not an urgent requirement but certainly nice to have. But it raises another important question that I'll put in another issue.

[gilesknap]

Debugging IOCs in-cluster

These are the steps I performed to debug an IOC in cluster. These should be automated by the ec tool.

1. Change the IOCs ioc-template chart values to make it debug compatible:

   •   ioc-instance:
         image: USE_DEVELOPER_TARGET:VERSION
       
         # don't make the IOC the entrypoint
         startCommand: sleep
         startArgs: infinity
       
         # disable host networking to allow root user
         hostNetwork: false
       
         # choose root user for write access to the container filesystem
         securityContext:
           runAsUser: 0
           runAsGroup: 0
           readOnlyRootFilesystem: false
           runAsNonRoot: false
     
     
         # mount the config map elsewhere so we can copy and edit it
         # i.e. `cp -r /tmp/config /epics/ioc/` before starting the IOC
         iocConfig: /tmp/config
       
         # generous resources to handle the extra overheads of debugging
         resources:
           limits:
             cpu: 4
             memory: 4Gi
           requests:
             cpu: 1
             memory: 1Gi
     

2. If you do need to access the PVs from outside the cluster then we should deploy a gateway to make that possible. Example here https://github.com/gilesknap/t03-services/tree/main/services/gateways

3. Stop the existing runtime IOC - UNFORTUNATELY for ArgoCD based beamlines you will need to remove the IOC (you can set the removed flag in apps/values.yaml to do so). This is because helm cannot update a non-helm deployment.

4. Start the new debugging IOC

   • ec deploy-local services/my-debugging-ioc

5. Setup environment for the cluster: e.g.

   • module load pollux

6. Launch vscode - must be the first instance to pick up the cluster config. Make sure you have the kubernetes and remote development extensions.

7. In vscode go to the kubernetes explorer pane:

   • Go to namespaces, right click → Use Namespace on the beamline namespace
   • Go to Workloads → Pods → IOC and right click → Attach Visual Studio Code

8. ⁠ Once attached Open Folder → /epics

9. ⁠Prepare the config folder. Open a terminal and:

   • cp -r /tmp/config /epics/ioc/config

10. /epics has a .vscode folder which configures some debug launchers:

    • (gdb) Attach to IOC
    • Open a core dump (C++)
    • Launch IOC in debugger