Controller Runtime Exercise

This repo contains a learn-by-doing style exercise to familiarize users with the inner workings of the controller runtime. Many projects, like Argo CD, Istio, and everything that leverages the Operator Framework. The most common use case is to think of Kubernetes resources (those little yaml definitions) as defining the intended state of the world, and services that leverage the Controller Runtime are actors that will "reconcile" the state of the world to the intended state.

Level vs Edge Based Reconciliation

The Controller Runtime can leverage both level and edge based reconiliation. Edge based watches for events, like resource creation, deletion, or updates and reconciles the state of the world to match the intent. Level based periodically reads in the current intent, and reconciles so the state of the world matches the intent.

Getting Started

Initial Setup

1. Create a kubernetes cluster via the Azure CLI or Azure Portal. There is optionally a command in the Makefile to deploy a KiND cluster, but it's recommended to run against a real cluster.

2. Use the commands in the Makefile to build and deploy the CRD's, Controller, and example manifests. Then continue with the steps below.

Your Task

You are the head of your dev ops department, and you'd like to provide your engineers with a simpler interface for defining their services with a common set of K8s best practices. To do so, you plan to defin a new K8s resource MyApp that provides a smaller API surface, and a lot of defaults around resources, pod disruption budgets, and more.

1. Modify the Reconcile method to create a Deployment with the provided image, a default set of resources (CPU and mem), and a pod disruption budget.

2. Add some custom metrics to the metrics handler, and enable the metrics handler. What's an example metric you might want to keep track of?

3. Enable leader election

4. Use a custom queue that orders Reconciliation by name, lexicographically.

5. Modify the reconciler to ignore MyApp's with labels reconciler: ignore

6. When a MyApp is deleted, remove all children.

7. Make sure the status of MyApp is kept up to date.

8. Migreate the controller runtime usage to leverage the KubeBuilder

Testing Your Understanding

Answer the following questions to test your understanding. You may want to tweak your code or add print/debug statements to test how it works. For each of the below questions also try to come up with an example scenario where this feature would be useful.

1. Which controller option controls how often Level-based reconciliation occurs?

2. Which event types are passed in on level based reconciliation?

3. What is the difference between the builder.Builder's For, Owns and Watches methods?

4. What happens if 10 updates to the same object occur in rapid succession (ie: before a single Reconcile occurs)? How many times is Reconcile called, and with which version of the object?

5. How can you control the speed of reconciliation?

6. How can you retry a failed reconiliation at a later time?

7. What happens to child objects if you delete a watched object?

8. Does the reconciler trigger on updates to the watched object, updates to the child object, or both?

9. Answer the above question, but for children of children? ie: 1 controller that creates a child object, that in turn creates a child object (ie: creating a deployment, will in turn create Pods)

10. What happens if a Reconciliation fails, and new updates come in?

11. What happens if a reconciliation fails to a child's child objects (ie: a deployments pods)?

12. If a single resource is in a failed state, does it block reconciliation of other objects?

13. A ReconcileRequest only has the NamespacedName. How do you get the full object? Is this object cached, or result in an API call to the k8s master?

14. What metrics does the controller runtime emit? Describe what some of those metrics represent. You can use the kubectl port-forward command to discover this.

15. What is leader election, and when would you use it?

16. What is the difference between Kubebuilder and controller runtime?

17. How does the Pod Disruption Budget work during a rollout? What about scaling down nodes? And when evicting a pod?