Merge pull request #255 from grafana/alert-on-unshipper-blocks

Added alert CortexIngesterHasUnshippedBlocks

Author: pracucci

CHANGELOG.md

@@ -9,6 +9,7 @@
   - Alerts: added "CortexBucketIndexNotUpdated" (bucket index only) and "CortexTenantHasPartialBlocks"
 * [ENHANCEMENT] The name of the overrides configmap is now customisable via `$._config.overrides_configmap`. #244
 * [ENHANCEMENT] Added flag to control usage of bucket-index, and enable it by default when using blocks. #254
+* [ENHANCEMENT] Added the alert `CortexIngesterHasUnshippedBlocks`. #255
 * [BUGFIX] Honor configured `per_instance_label` in all panels. #239
 * [BUGFIX] `CortexRequestLatency` alert now ignores long-running requests on query-scheduler. #242
 * [BUGFIX] Honor configured `job_names` in the "Memory (go heap inuse)" panel. #247

cortex-mixin/alerts/blocks.libsonnet

@@ -39,6 +39,24 @@
             message: 'Cortex Ingester {{ $labels.namespace }}/{{ $labels.instance }} has not shipped any block in the last 4 hours.',
           },
         },
+        {
+          // Alert if the ingester has compacted some blocks that haven't been successfully uploaded to the storage yet since
+          // more than 1 hour. The metric tracks the time of the oldest unshipped block, measured as the time when the
+          // TSDB head has been compacted to a block. The metric is 0 if all blocks have been shipped.
+          alert: 'CortexIngesterHasUnshippedBlocks',
+          'for': '15m',
+          expr: |||
+            (time() - cortex_ingester_oldest_unshipped_block_timestamp_seconds > 3600)
+            and
+            (cortex_ingester_oldest_unshipped_block_timestamp_seconds > 0)
+          |||,
+          labels: {
+            severity: 'critical',
+          },
+          annotations: {
+            message: "Cortex Ingester {{ $labels.namespace }}/{{ $labels.instance }} has compacted a block {{ $value | humanizeDuration }} ago but it hasn't been successfully uploaded to the storage yet.",
+          },
+        },
         {
           // Alert if the ingester is failing to compact TSDB head into a block, for any opened TSDB. Once the TSDB head is
           // compactable, the ingester will try to compact it every 1 minute. Repeatedly failing it is a critical condition

cortex-mixin/docs/playbooks.md

@@ -92,6 +92,13 @@ If the ingester hit the disk capacity, any attempt to append samples will fail.
 
 Same as [`CortexIngesterHasNotShippedBlocks`](#CortexIngesterHasNotShippedBlocks).
 
+### CortexIngesterHasUnshippedBlocks
+
+This alert fires when a Cortex ingester has compacted some blocks but such blocks haven't been successfully uploaded to the storage yet.
+
+How to **investigate**:
+- Look for details in the ingester logs
+
 ### CortexIngesterTSDBHeadCompactionFailed
 
 This alert fires when a Cortex ingester is failing to compact the TSDB head into a block.