Idea: Runbook Execution Engine — pup runbooks

[platinummonkey]

Overview

This proposes adding a runbook execution engine to pup, enabling teams to define, discover, and execute operational runbooks directly from the CLI. Runbooks could cover deployments, incident remediation, routing fixes, maintenance windows, and any other repeatable operational procedure.

---

Motivation

Operational runbooks today are scattered — Confluence docs, Notion pages, shell scripts, internal wikis. They go stale, they aren't discoverable, and they're not executable. pup already has authenticated access to the Datadog API, making it the ideal host for runbooks that combine:

• Datadog data (monitors, incidents, metrics, logs, SLOs, services)
• Datadog Workflows (trigger automated workflow runs)
• Custom internal tools (shell commands, internal APIs, scripts)
• Human checkpoints (prompts for confirmation before destructive steps)

---

Proposed Interface

# Discover runbooks
pup runbooks list
pup runbooks list --tag=service:payments
pup runbooks list --tag=env:production --tag=type:deployment

# Inspect a runbook
pup runbooks describe deploy-service
pup runbooks describe rollback-feature-flag

# Execute a runbook
pup runbooks run deploy-service --set SERVICE=payments --set VERSION=2.4.1
pup runbooks run incident-triage --set INCIDENT_ID=12345
pup runbooks run maintenance-window --set SERVICE=redis --set DURATION=30m

# Validate a runbook without executing
pup runbooks validate deploy-service

# Import a shared runbook from a URL or path
pup runbooks import https://internal.corp/runbooks/deploy-template.yaml
pup runbooks import ./shared-runbooks/

---

Storage

Runbooks live in pup's config directory:

~/.config/pup/
  runbooks/
    deploy-service.yaml
    rollback-feature-flag.yaml
    incident-triage.yaml
    lib/                    # imported/shared templates
      deploy-template.yaml

---

Runbook Format

YAML-based with templating (Tera or Handlebars-style {{ var }} syntax):

name: deploy-service
description: Deploy a service to a target environment with health validation
tags:
  service: "{{ SERVICE }}"
  env: "{{ ENV | default: \"staging\" }}"
  type: deployment
vars:
  SERVICE:
    description: Name of the service to deploy
    required: true
  VERSION:
    description: Version/tag to deploy
    required: true
  ENV:
    description: Target environment
    default: staging

steps:
  - name: Check current SLO status
    kind: pup
    run: slos list --service={{ SERVICE }} --env={{ ENV }}
    on_failure: warn  # don't block, just surface

  - name: Verify no active incidents
    kind: pup
    run: incidents list --state=active --service={{ SERVICE }}
    assert:
      empty: true
      message: "Active incidents detected for {{ SERVICE }} — confirm before continuing"
    on_failure: confirm  # require human sign-off

  - name: Trigger deployment workflow
    kind: datadog-workflow
    workflow_id: abc123-workflow-id
    inputs:
      service: "{{ SERVICE }}"
      version: "{{ VERSION }}"
      environment: "{{ ENV }}"

  - name: Watch deployment monitor
    kind: pup
    run: monitors get 98765
    poll:
      interval: 30s
      timeout: 10m
      until: "status == OK"

  - name: Notify on-call
    kind: shell
    run: "notify-oncall --service={{ SERVICE }} --version={{ VERSION }} --env={{ ENV }}"
    optional: true

Sequence: deploy-service

sequenceDiagram
    actor Operator
    participant pup
    participant DatadogAPI as Datadog API
    participant DDWorkflow as Datadog Workflows
    participant Shell as Shell Tools

    Operator->>pup: pup runbooks run deploy-service --set SERVICE=payments --set VERSION=2.4.1

    pup->>DatadogAPI: GET /slos?service=payments&env=staging
    DatadogAPI-->>pup: SLO status

    pup->>DatadogAPI: GET /incidents?state=active&service=payments
    DatadogAPI-->>pup: incident list

    alt Active incidents found
        pup->>Operator: ⚠️ Active incidents detected. Proceed?
        Operator->>pup: confirm
    end

    pup->>DDWorkflow: POST /workflows/abc123/run {service, version, environment}
    DDWorkflow-->>pup: execution started

    loop Poll every 30s (timeout 10m)
        pup->>DatadogAPI: GET /monitors/98765
        DatadogAPI-->>pup: monitor status
    end

    opt notify-oncall installed
        pup->>Shell: notify-oncall --service=payments --version=2.4.1
        Shell-->>pup: notified
    end

    pup-->>Operator: ✅ Deployment complete

Loading

---

Example: Incident Triage Runbook

name: incident-triage
description: First-responder checklist for a service incident
tags:
  type: incident
  phase: triage
vars:
  INCIDENT_ID:
    description: Datadog incident ID
    required: true
  SERVICE:
    description: Affected service
    required: true

steps:
  - name: Pull incident details
    kind: pup
    run: incidents get {{ INCIDENT_ID }}

  - name: Check recent deployments
    kind: pup
    run: deployments list --service={{ SERVICE }} --from=2h

  - name: Fetch recent error logs
    kind: pup
    run: logs search --query="service:{{ SERVICE }} status:error" --from=30m --limit=25

  - name: Check related monitors
    kind: pup
    run: monitors list --tag="service:{{ SERVICE }}" --status=alert

  - name: Open Datadog workflow — auto-mitigation
    kind: datadog-workflow
    workflow_id: xyz-auto-mitigation-workflow
    inputs:
      incident_id: "{{ INCIDENT_ID }}"
      service: "{{ SERVICE }}"

  - name: Run internal diagnostics
    kind: shell
    run: "diag-tool --service={{ SERVICE }} --format=summary"
    optional: true

Sequence: incident-triage

sequenceDiagram
    actor Operator
    participant pup
    participant DatadogAPI as Datadog API
    participant DDWorkflow as Datadog Workflows
    participant Shell as Shell Tools

    Operator->>pup: pup runbooks run incident-triage --set INCIDENT_ID=12345 --set SERVICE=payments

    pup->>DatadogAPI: GET /incidents/12345
    DatadogAPI-->>pup: incident details

    pup->>DatadogAPI: GET /deployments?service=payments&from=2h
    DatadogAPI-->>pup: recent deployments

    pup->>DatadogAPI: GET /logs?query=service:payments status:error&from=30m
    DatadogAPI-->>pup: recent error logs

    pup->>DatadogAPI: GET /monitors?tag=service:payments&status=alert
    DatadogAPI-->>pup: alerting monitors

    pup->>DDWorkflow: POST /workflows/xyz-auto-mitigation/run {incident_id, service}
    DDWorkflow-->>pup: mitigation triggered

    opt diag-tool installed
        pup->>Shell: diag-tool --service=payments --format=summary
        Shell-->>pup: diagnostics output
    end

    pup-->>Operator: ✅ Triage complete — review output above

Loading

---

Example: Feature Flag Rollback

name: rollback-feature-flag
description: Safely roll back a feature flag and validate impact
tags:
  type: rollback
  service: "{{ SERVICE }}"
vars:
  FLAG_KEY:
    required: true
  SERVICE:
    required: true

steps:
  - name: Get current flag state
    kind: pup
    run: feature-flags get {{ FLAG_KEY }}

  - name: Check error rate before rollback
    kind: pup
    run: metrics query --query="avg:trace.servlet.request.errors{service:{{ SERVICE }}}" --from=15m

  - name: Confirm rollback
    kind: confirm
    message: "Rolling back {{ FLAG_KEY }} for {{ SERVICE }}. Proceed?"

  - name: Disable flag
    kind: pup
    run: feature-flags update {{ FLAG_KEY }} --enabled=false

  - name: Monitor error rate after rollback
    kind: pup
    run: metrics query --query="avg:trace.servlet.request.errors{service:{{ SERVICE }}}" --from=5m
    poll:
      interval: 30s
      timeout: 5m
      until: "decreasing"

Sequence: rollback-feature-flag

sequenceDiagram
    actor Operator
    participant pup
    participant DatadogAPI as Datadog API

    Operator->>pup: pup runbooks run rollback-feature-flag --set FLAG_KEY=new-checkout --set SERVICE=payments

    pup->>DatadogAPI: GET /feature-flags/new-checkout
    DatadogAPI-->>pup: flag state (enabled: true)

    pup->>DatadogAPI: GET /metrics?query=avg:trace.servlet.request.errors{service:payments}&from=15m
    DatadogAPI-->>pup: baseline error rate

    pup->>Operator: Rolling back new-checkout for payments. Proceed?
    Operator->>pup: confirm

    pup->>DatadogAPI: PATCH /feature-flags/new-checkout {enabled: false}
    DatadogAPI-->>pup: flag disabled

    loop Poll every 30s (timeout 5m)
        pup->>DatadogAPI: GET /metrics?query=avg:trace.servlet.request.errors{service:payments}&from=5m
        DatadogAPI-->>pup: current error rate
    end

    pup-->>Operator: ✅ Rollback complete — error rate decreasing

Loading

---

Example: Redis Maintenance Window

name: maintenance-window
description: Schedule a Datadog maintenance window and drain traffic before maintenance
tags:
  type: maintenance
vars:
  SERVICE:
    required: true
  DURATION:
    default: 30m
    description: Maintenance window duration (e.g. 30m, 1h)

steps:
  - name: Create Datadog downtime
    kind: pup
    run: downtimes create --scope="service:{{ SERVICE }}" --duration={{ DURATION }} --message="Scheduled maintenance"
    capture: DOWNTIME_ID

  - name: Drain traffic
    kind: shell
    run: "lb-cli drain --service={{ SERVICE }}"

  - name: Wait for drain
    kind: pup
    run: metrics query --query="sum:requests{service:{{ SERVICE }}}" --from=1m
    poll:
      interval: 10s
      timeout: 5m
      until: "value < 5"

  - name: Notify — ready for maintenance
    kind: confirm
    message: "Traffic drained. Proceed with maintenance on {{ SERVICE }}?"

  - name: Cancel downtime when done
    kind: pup
    run: downtimes delete {{ DOWNTIME_ID }}
    when: always  # run even if earlier steps fail

Sequence: maintenance-window

sequenceDiagram
    actor Operator
    participant pup
    participant DatadogAPI as Datadog API
    participant Shell as Shell Tools

    Operator->>pup: pup runbooks run maintenance-window --set SERVICE=redis --set DURATION=30m

    pup->>DatadogAPI: POST /downtimes {scope: service:redis, duration: 30m}
    DatadogAPI-->>pup: downtime created (ID: 55123)

    pup->>Shell: lb-cli drain --service=redis
    Shell-->>pup: drain initiated

    loop Poll every 10s (timeout 5m)
        pup->>DatadogAPI: GET /metrics?query=sum:requests{service:redis}&from=1m
        DatadogAPI-->>pup: request count
    end

    pup->>Operator: Traffic drained. Proceed with maintenance on redis?
    Operator->>pup: confirm

    Note over Operator,Shell: Operator performs maintenance work

    pup->>DatadogAPI: DELETE /downtimes/55123
    DatadogAPI-->>pup: downtime cancelled

    pup-->>Operator: ✅ Maintenance window complete

Loading

---

Step Kinds

Kind	Description
pup	Run any pup subcommand; output available for assertions/polling
shell	Run an arbitrary shell command (uses $PATH, optional: true skips if not found)
datadog-workflow	Trigger a Datadog Workflow by ID with structured inputs
confirm	Pause for human confirmation before continuing
http	Make an HTTP request to an internal API

---

Templating & Imports

Runbooks can import shared templates to avoid duplication:

# deploy-service.yaml
import:
  - lib/health-check-template.yaml
  - lib/notify-template.yaml

name: deploy-service
# ... vars and steps, can reference imported step names

Shared templates define reusable step groups, making it easy for platform teams to distribute approved runbook fragments across services.

---

Open Questions

1. Templating engine — Tera (Rust-native) vs Handlebars vs Liquid? Tera seems like the natural fit.
2. Remote runbook sources — Should pup runbooks import support Git refs (e.g. a shared internal repo)?
3. Execution history — Should pup record runbook execution history locally (what ran, when, outcome)?
4. Secret injection — Should vars support reading from the OS keychain (kind: secret) to avoid passing tokens on the CLI?
5. Dry-run mode — pup runbooks run --dry-run to print the resolved plan without executing.
6. Datadog Workflow discovery — Could pup runbooks list surface available Datadog Workflows alongside local runbooks?

---

Why pup Is the Right Host

• Already authenticated with Datadog (OAuth2 + API keys)
• Config directory (~/.config/pup/) is a natural home for user-defined runbooks
• Cross-platform, single binary — easy to distribute in CI/CD or developer machines
• pup commands become first-class runbook primitives without any extra setup
• Composable with any shell tool the team already uses

Would love feedback on the format design, step kinds, templating approach, and whether there's appetite for this as a first-class feature.

[Chriscbr]

The idea reminds me a bit of http://docs.aws.amazon.com/systems-manager/latest/userguide/automation-documents.html from AWS (did not use the product, but I heard of it). May be worth perusing the docs to see if there's anything worth taking inspiration from.

suggestion: Maybe instead of:

pup runbooks run deploy-service --set SERVICE=payments --set VERSION=2.4.1

this may feel more obvious?:

pup runbooks run deploy-service --arg SERVICE=payments --arg VERSION=2.4.1

[platinummonkey]

The prototype was merged! I'm curious how people will use this

[platinummonkey]

In addition a new "extensions" feature was added to extend pup with custom scripts and it will auto inject the credentials into those scripts