How to setup Pulumi to use Azure Blob Storage for state and Azure Key Vault for secret provider

Overview

Steps to create a Pulumi Project with Azure Blob Storage as the state storage and Azure Key Vault to encrypt secrets.

Two things are needed to accomplish this:

1. Azure Blob Storage Account
2. Azure Key Vault

These resources will be in its own resource group and the name by convention will be <project-name>-state-rg. Also, the following naming conventions are highly recommended:

• For Storage account, <projectname>infrasa
• the storage container, pulumistate and
• Key vault, <project-name>-infra-kv

The next steps uses the Azure CLI to create the resources, remember to confirm that you have the latest version installed by reviewing the result of az --version. Also, it is a good time to proceed to install Pulumi.

Note: It is assumed that the following commands will be running from a Bash terminal environment.

Setup

As a good practice please confirm in which Azure account and subscription you are login.

az account show --output table

To create the main resource group for this project execute below script, you should have privilege to create resources in the subscription used.

az group create --name <project-name>-state-rg --location <location>

Tip: to generate a list of Azure locations, az account list-locations -o table.

Now we are going to create the storage account.

az storage account create --name <projectname>infrasa --resource-group <project-name>-state-rg --location <location> --sku Standard_LRS --access-tier Hot --https-only true --kind StorageV2

Next we need to create two environment variables. But first we are going to obtain the key to the storage account that we have created.

az storage account keys list -g <project-name>-state-rg -n <projectname>infrasa

You can use any of the keys available in the items named value in the result object of the above query.

Now the environment variables:

export AZURE_STORAGE_ACCOUNT="<projectname>infrasa"
export AZURE_STORAGE_KEY=<account-key-from-storage-account>

These variables are important to complete the creation of the container below:

az storage container create --account-name <projectname>infrasa --name pulumistate

Checkpoint

So far we have created the blob storage to hold the state files that Pulumi will use to track changes in our infrastructure as code (IaC) solution. Now will proceed to create the Azure Key Vault object.

az keyvault create --name <project-name>-infra-kv --resource-group <project-name>-state-rg --location <location>

Now let’s create the encryption key needed by Pulumi to protect our secrets.

az keyvault key create --vault-name <project-name>-infra-kv --name PulumiStateDev

When the script execution is completed, save the uri generated and located in the item named "kid" of the result object. This is the key-identifier-uri-address that will be used to configure the secret provider later.

The previous steps create an encryption key but we need to setup proper permissions for the user to be able to encrypt and decrypt the Pulumi secrets. This will be executed in the Azure Portal. Go to the newly created Azure Key Vault. In the left sidebar search for Settings > Access policies. The selected user for management should have at least the following keys permissions: Get, Decrypt and Encrypt.

Important: To allows Pulumi use the credentials from the CLI, we need to create the following environment variable:

export AZURE_KEYVAULT_AUTH_VIA_CLI=true

At this moment you must have the required infrastructure configured to be able to use Azure as the backend for the state and the secret provider.

Startup Pulumi

Define the folder structure where the project will reside. Pulumi will be configure in the infra folder.

• Project Name
  • app
  • functions
  • infra
  • pipelines
  • web

Tip: To create the above structure use: mkdir -p <project-name>/{app,functions,infra,pipelines,web}

Go to the infra directory. Proceed to login to Pulumi using the Azure Blob storage for state endpoint.

pulumi login azblob://pulumistate

Now proceed to initiate the project using the Azure Key Vault as the secret provider. For this we’ll need the Key Identifier uri address. This uri address starts with https but for Pulumi we need to replace this with azurekeyvault.

You will need to manually create a new key in Azure KeyVault, this will give you the referenced in the command below.

pulumi new --secrets-provider="azurekeyvault://<azure-keyvault-uri>/keys/<your-keyvault-key>"

TODO - explain some error that could happen and the non-interactive issue if using bash directly in Windows, there is no problem from Windows Terminal or Visual Studio Code terminal (Pulumi Issue 5407). Try to add the same example for PowerShell.