graceful shutdown of ingesters (grafana#195)

* graceful shutdown of ingesters

Signed-off-by: Tom Hayward <thayward@infoblox.com>

* tolerate auto-scale-down now that it is safe

Signed-off-by: Tom Hayward <thayward@infoblox.com>

* allow overriding preStop hook

Signed-off-by: Tom Hayward <thayward@infoblox.com>

Author: kd7lxl

CHANGELOG.md

@@ -5,6 +5,7 @@
 * [FEATURE] Add autoscaler for queriers #190
 * [FEATURE] Add autoscaler for distributors #189
 * [FEATURE] Add autoscaler for ingesters #182
+* [ENHANCEMENT] Graceful shutdown of ingesters #195
 * [ENHANCEMENT] Define namespace in templates #184
 * [ENHANCEMENT] Use FQDN for memcached addresses #175
 * [ENHANCEMENT] Optionally generate endpoints for `X-Scope-OrgID` injection (multi-tenancy) #180

README.md

@@ -466,7 +466,8 @@ Kubernetes: `^1.19.0-0`
 | ingester.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey | string | `"kubernetes.io/hostname"` |  |
 | ingester.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].weight | int | `100` |  |
 | ingester.annotations | object | `{}` |  |
-| ingester.autoscaling.behavior.scaleDown.selectPolicy | string | `"Disabled"` | Scaledown procedure varies, so automatic scaledown is disabled Ref: https://cortexmetrics.io/docs/guides/ingesters-scaling-up-and-down/#scaling-down |
+| ingester.autoscaling.behavior.scaleDown.policies | list | `[{"periodSeconds":1800,"type":"Pods","value":1}]` | see https://cortexmetrics.io/docs/guides/ingesters-scaling-up-and-down/#scaling-down for scaledown details |
+| ingester.autoscaling.behavior.scaleDown.stabilizationWindowSeconds | int | `3600` | uses metrics from the past 1h to make scaleDown decisions |
 | ingester.autoscaling.behavior.scaleUp.policies | list | `[{"periodSeconds":1800,"type":"Pods","value":1}]` | This default scaleup policy allows adding 1 pod every 30 minutes. Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior |
 | ingester.autoscaling.enabled | bool | `false` |  |
 | ingester.autoscaling.maxReplicas | int | `30` |  |
@@ -481,6 +482,7 @@ Kubernetes: `^1.19.0-0`
 | ingester.extraVolumeMounts | list | `[]` |  |
 | ingester.extraVolumes | list | `[]` |  |
 | ingester.initContainers | list | `[]` |  |
+| ingester.lifecycle.preStop | object | `{"httpGet":{"path":"/ingester/shutdown","port":"http-metrics"}}` | The /shutdown preStop hook is recommended as part of the ingester scaledown process, but can be removed to optimize rolling restarts in instances that will never be scaled down or when using chunks storage with WAL disabled. https://cortexmetrics.io/docs/guides/ingesters-scaling-up-and-down/#scaling-down |
 | ingester.livenessProbe.httpGet.path | string | `"/ready"` |  |
 | ingester.livenessProbe.httpGet.port | string | `"http-metrics"` |  |
 | ingester.livenessProbe.httpGet.scheme | string | `"HTTP"` |  |

templates/ingester/ingester-dep.yaml

@@ -103,6 +103,10 @@ spec:
             {{- if .Values.ingester.env }}
             {{ toYaml .Values.ingester.env | nindent 12 }}
             {{- end }}
+          {{- with .Values.ingester.lifecycle }}
+          lifecycle:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
         {{- with .Values.ingester.extraContainers }}
         {{ toYaml . | nindent 8 }}
         {{- end }}

templates/ingester/ingester-statefulset.yaml

@@ -151,4 +151,8 @@ spec:
             {{- if .Values.ingester.env }}
               {{- toYaml .Values.ingester.env | nindent 12 }}
             {{- end }}
+          {{- with .Values.ingester.lifecycle }}
+          lifecycle:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
 {{- end -}}

values.yaml

@@ -537,9 +537,14 @@ ingester:
     targetMemoryUtilizationPercentage: 80
     behavior:
       scaleDown:
-        # -- Scaledown procedure varies, so automatic scaledown is disabled
-        # Ref: https://cortexmetrics.io/docs/guides/ingesters-scaling-up-and-down/#scaling-down
-        selectPolicy: Disabled
+        # -- see https://cortexmetrics.io/docs/guides/ingesters-scaling-up-and-down/#scaling-down for scaledown details
+        policies:
+          - type: Pods
+            value: 1
+            # set to no less than 2x the maximum between -blocks-storage.bucket-store.sync-interval and -compactor.cleanup-interval
+            periodSeconds: 1800
+        # -- uses metrics from the past 1h to make scaleDown decisions
+        stabilizationWindowSeconds: 3600
       scaleUp:
         # -- This default scaleup policy allows adding 1 pod every 30 minutes.
         # Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior
@@ -548,6 +553,17 @@ ingester:
             value: 1
             periodSeconds: 1800
 
+  lifecycle:
+    # -- The /shutdown preStop hook is recommended as part of the ingester
+    # scaledown process, but can be removed to optimize rolling restarts in
+    # instances that will never be scaled down or when using chunks storage
+    # with WAL disabled.
+    # https://cortexmetrics.io/docs/guides/ingesters-scaling-up-and-down/#scaling-down
+    preStop:
+      httpGet:
+        path: "/ingester/shutdown"
+        port: http-metrics
+
   ## DEPRECATED: use persistentVolume.subPath instead
   persistence:
     subPath: