Switching to new SDK provider tool for makefiles and adding to mise #1822

Graham-Pedersen · 2025-11-11T21:02:47Z

This change is part of our refactoring of how SDK generation works for providers. It alters the makefile targets to use the tool, and adds the new tool to mise.

This change is ready to be merged and rolled out to the bridged providers

As part of this rollout we are also going to migrate overlays into a new package structure. This has been done for AWS, and I have identified the other 3 packages that need that change (pulumi-gcp, pulumi-openstack, and pulumi-azure). In addition each bridged provider will need another commit to remove the 4 SDKs that no longer need to be included in source

corymhall · 2025-11-11T21:29:08Z

provider-ci/internal/pkg/migrations/migrate_cimgmt_to_mise.go

 	entries := pluginsToToolEntries(pluginList)
+
+	//Manually adding the new SDK tool to this list until we support dynamically adding new dependencies
+	sdkToolPlugin := sectionEntry{"go:github.com/pulumi/provider-sdk-builder", "latest"}


long term we'll probably want to switch to using a binary which is published on the GitHub release.

"github:pulumi/provider-sdk-builder" = "latest"

We can do that. I would just need to get the binaries getting built and included on the release. Any reason we prefer the binaries to go installation though?

I think we will run into less issues long term by using a built binary. This method uses go install which can lead to issues. https://golangci-lint.run/docs/welcome/install/#install-from-sources

golangci-lint is a bit of an odd duck and that guidance isn't true in general. Sticking with go install should be fine for something this simple.

corymhall · 2025-11-14T13:39:48Z

provider-ci/internal/pkg/templates/base/Makefile

 	@touch $@

 # Build the provider and all SDKs and install ready for testing
-build: .make/mise_install provider build_sdks install_sdks#{{- if .Config.RegistryDocs }}# build_registry_docs#{{- end }}#


I'm assuming that install is now run as part of build in the tool?

In the latest version I actually have changed this. You can pass in an SDK to build. It is run as part of the build in root though also

Can you speak to this? Does build include install or not?

corymhall · 2025-11-14T13:43:59Z

provider-ci/internal/pkg/templates/base/Makefile

-install_sdks:#{{ range .Config.Languages }}# install_#{{ . }}#_sdk#{{ end }}#
-.PHONY: development only_build build generate generate_sdks build_sdks install_sdks mise_install mise_env
+# Creates all generated files of which the schema is committed
+generate: build_sdks schema#{{- if .Config.RegistryDocs }}# build_registry_docs#{{- end }}#


Are we still generating the SDKs? If so I think we should still separate out the generate from build. Building the sdks can take a while and it's nice to not have to run it if you don't have to.

I have tried to solve that problem by only triggering a build when the schema has changed. I am anticipating we will still always generate them once (at least in this current world)

I think I'm more suggesting that when iterating I might just want to regenerate the SDKs and not build the SDKs. But that's probably pretty uncommon.

Yeah you can do that by invoking the build tool directly, but I also thought that was uncommon and didn't feel it needed to be a target in the makefile. Happy to change if you feel strongly though!

corymhall · 2025-11-14T13:47:24Z

provider-ci/internal/pkg/templates/base/Makefile

-	@echo "  generate_sdks Generate all SDKs"
 	@echo "  build_sdks    Build all SDKs"
-	@echo "  install_sdks  Install all SDKs"
 	@echo "  provider_dist Build and package the provider for all platforms"
 	@echo ""
 	@echo "Tool Targets"
 	@echo "  ci-mgmt     Re-generate CI configuration from .ci-mgmt.yaml"
 	@echo "  debug_tfgen Start a debug server for tfgen"
 	@echo ""
 	@echo "Internal Targets (automatically run as dependencies of other targets)"
 	@echo "  prepare_local_workspace  Prepare for building"
 	@echo "  mise_install             Install tools with mise"
 	@echo "  upstream                 Initialize the upstream submodule, if present"
 	@echo ""
-	@echo "Language-Specific Targets"
-	@echo "  generate_[language]    Generate the SDK files ready for committing"
-	@echo "  build_[language]       Build the SDK to check correctness"
-	@echo "  install_[language]_sdk Install the SDK ready for testing"
-	@echo ""


Don't we still need all these targets? If I'm running locally and just want the nodejs SDKs installed I can just run the language specific targets.

If the new build target also runs install then we probably don't need this anymore now that we have the SDK_LANG

provider-ci/internal/pkg/templates/base/Makefile

.github/workflows/update-workflows.yml

Removing step to install go and allowing mise to handle installation Updating to support new SDK generation tool updating download SDK action to create directory if missing Updating the pulumi package publish workflows adding provider version to new SDKs and updating the non java publish workflow to use the latest provider publisher commit Going to download the SDKs before the verify step since they are no longer mastered in source control

…doption

…e in the sdk directory Switching the build SDK workflow to use the pulumi bot token switching install to use the sdk tool directly instead of a make target

.github/workflows/update-workflows.yml

blampe · 2025-12-02T20:00:16Z

.github/workflows/update-workflows.yml

      - name: Setup Python
        uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6
        with:
          python-version: 3.9
      - name: Install pulumictl


This and the other steps can be removed if we're installing with mise now.

I will rip all these out and retest.

blampe · 2025-12-02T20:01:33Z

provider-ci/internal/pkg/migrations/mise.toml

 "aqua:gradle/gradle-distributions" = '7.6.6'
 golangci-lint = "1.64.8" # See note about about overrides if you need to customize this.
 "npm:yarn" = "1.22.22"
+"go:github.com/pulumi/provider-sdk-builder" = "latest"


We should pin this if it's going to be going out everywhere.

blampe · 2025-12-02T20:02:20Z

provider-ci/internal/pkg/migrations/migrate_cimgmt_to_mise.go

+	//Manually adding the new SDK tool to this list until we support dynamically adding new dependencies
+	sdkToolPlugin := sectionEntry{"go:github.com/pulumi/provider-sdk-builder", "latest"}
+	entries = append(entries, sdkToolPlugin)


I don't think you need this, the mise.toml in this directory should already capture the default.

I was testing off an old branch of this package, but I know we did need it for a time. Repos that already had a mise.toml file to add it to the existing ones.

Let me rip it out and check when I retest.

I know this file has changed a lot since I wrote that code, because I had like 10 sets of merge conflicts in here when I was rebasing my changes onto mainline haha

blampe · 2025-12-02T20:03:58Z

provider-ci/internal/pkg/templates/base/.github/workflows/build_sdk.yml

        with:
          version: 2025.11.6
          github_token: ${{ steps.esc-secrets.outputs.PULUMI_BOT_TOKEN }}
+          cache_key: "mise-{{platform}}-{{file_hash}}"


We should be using the default cache key everywhere now.

Suggested change

cache_key: "mise-{{platform}}-{{file_hash}}"

provider-ci/internal/pkg/templates/base/.github/workflows/build_sdk.yml

blampe · 2025-12-02T20:06:10Z

provider-ci/internal/pkg/templates/base/.github/workflows/test.yml

 #{{- end }}#
    #{{- if .Config.IntegrationTestProvider }}#
+    - name: Install dependencies
+      run: provider-sdk-builder install  --sdkLocation ./sdk/${{ matrix.language}} -l ${{ matrix.language}} --installLocation ${{ github.workspace }}


This is more of a nit but is this step less of an "install" and more of a "link local SDK"?

Also formatting

Suggested change

run: provider-sdk-builder install --sdkLocation ./sdk/${{ matrix.language}} -l ${{ matrix.language}} --installLocation ${{ github.workspace }}

run: provider-sdk-builder install --sdkLocation ./sdk/${{ matrix.language }} -l ${{ matrix.language }} --installLocation ${{ github.workspace }}

Happy to make those changes, and yes that was probably not the correct name

blampe · 2025-12-02T20:07:47Z

provider-ci/internal/pkg/templates/base/.github/workflows/test.yml

        language: ${{ matrix.language }}
    - name: Restore makefile progress
-      run: make --touch provider schema build_${{ matrix.language }}
+      run: make --touch provider schema build_sdks SDK_LANG=${{ matrix.language }} 


Why not invoke provider-sdk-builder directly?

Yeah thats not a bad idea. I was looking at this and scratching my head right now too, because I don't believe it should even be building the SDK in this step....

I am just going to remove this build_sdk call. I don't think its right to trigger that behavior here, and I think as I side effect of how we use our sentinel files I am not sure it actually was building.... 🤷

Yeah this is using --touch so it's not actually building. We do this so that later when we run make test it thinks (correctly) that the SDKs have already been built. Without this make test will re-build the SDKs.

Oh that makes sense. For some reason I was reading the touch flag as only touching the sentinel file for the first target, but obviously that is not how it works

Thanks for clarifying!

blampe · 2025-12-02T20:11:07Z

provider-ci/internal/pkg/templates/base/.github/workflows/verify-release.yml

+      - name: Download python SDK
+        uses: ./.github/actions/download-sdk
+        with:
+          language: python


This seems counter to what the workflow is trying to do, and if we needed to add this step this might point towards a bug somewhere.

In particular I think this workflow is trying to test the end-to-end experience from the user's perspective, which means that it expects to find all the relevant SDKs and binaries published and available from the usual places.

What we're doing here is grabbing an SDK that was generated earlier. Why weren't we able to grab the SDK from npm, pypi, etc.?

Let me take a deeper look

I cannot lie, I have no memory of writing this step anymore.

Is this what was causing your issue with windows in verify-release?

I believe it was one of them yes

Re: Windows that would make sense. If this isn't needed then we can continue to test Windows in this workflow, right?

Ok I reviewed and re-tested and these are required for the verify release workflows. What was happening is that the old verify release workflow was relying on the SDKs that use to be checked into source. You can see the failure when it does not have this code in the following workflow when it tries to set up the java SDK to run: https://github.com/pulumi/pulumi-random/actions/runs/19482295415/job/55757193191

I think that is just a side-effect of how the setup-java action works. It must expect a local java project in certain cases and it was a coincidence that we had the SDKs checked out. I think it might be because the setup-java action tries to restore the gradle cache and looks for local projects. Maybe we can just turn that off or trick it somehow?

Yes I think you might be correct. I think Bryce just linked me something to that effect as well. I am going to try to build this with the plugin changes that Bryce made yesterday to see if that unblocks me.

Failing that I can revert this workflow to how it was, and disable that caching.

Graham-Pedersen · 2025-12-02T21:29:16Z

provider-ci/internal/pkg/templates/base/.github/workflows/test.yml

-    - name: Install dependencies
-      run: make install_${{ matrix.language}}_sdk


I need to add this back in but update it to the latest way. Next version will have a change. This is why the XYZ provider fails to build

Is this still outstanding?

corymhall · 2025-12-03T10:53:39Z

provider-ci/internal/pkg/migrations/migrate_cimgmt_to_mise.go

 	entries := pluginsToToolEntries(pluginList)
+
+	//Manually adding the new SDK tool to this list until we support dynamically adding new dependencies
+	sdkToolPlugin := sectionEntry{"go:github.com/pulumi/provider-sdk-builder", "latest"}


I think we will run into less issues long term by using a built binary. This method uses go install which can lead to issues. https://golangci-lint.run/docs/welcome/install/#install-from-sources

corymhall · 2025-12-03T10:56:35Z

provider-ci/internal/pkg/templates/base/.github/workflows/test.yml

        language: ${{ matrix.language }}
    - name: Restore makefile progress
-      run: make --touch provider schema build_${{ matrix.language }}
+      run: make --touch provider schema build_sdks SDK_LANG=${{ matrix.language }} 


Yeah this is using --touch so it's not actually building. We do this so that later when we run make test it thinks (correctly) that the SDKs have already been built. Without this make test will re-build the SDKs.

corymhall · 2025-12-03T10:58:05Z

provider-ci/internal/pkg/templates/base/.github/workflows/verify-release.yml

+      - name: Download python SDK
+        uses: ./.github/actions/download-sdk
+        with:
+          language: python


Is this what was causing your issue with windows in verify-release?

corymhall · 2025-12-03T11:10:08Z

provider-ci/internal/pkg/templates/base/Makefile

-install_sdks:#{{ range .Config.Languages }}# install_#{{ . }}#_sdk#{{ end }}#
-.PHONY: development only_build build generate generate_sdks build_sdks install_sdks mise_install mise_env
+# Creates all generated files of which the schema is committed
+generate: build_sdks schema#{{- if .Config.RegistryDocs }}# build_registry_docs#{{- end }}#


I think I'm more suggesting that when iterating I might just want to regenerate the SDKs and not build the SDKs. But that's probably pretty uncommon.

corymhall · 2025-12-03T11:11:59Z

provider-ci/internal/pkg/templates/base/Makefile

-	@echo "  generate_sdks Generate all SDKs"
 	@echo "  build_sdks    Build all SDKs"
-	@echo "  install_sdks  Install all SDKs"
 	@echo "  provider_dist Build and package the provider for all platforms"
 	@echo ""
 	@echo "Tool Targets"
 	@echo "  ci-mgmt     Re-generate CI configuration from .ci-mgmt.yaml"
 	@echo "  debug_tfgen Start a debug server for tfgen"
 	@echo ""
 	@echo "Internal Targets (automatically run as dependencies of other targets)"
 	@echo "  prepare_local_workspace  Prepare for building"
 	@echo "  mise_install             Install tools with mise"
 	@echo "  upstream                 Initialize the upstream submodule, if present"
 	@echo ""
-	@echo "Language-Specific Targets"
-	@echo "  generate_[language]    Generate the SDK files ready for committing"
-	@echo "  build_[language]       Build the SDK to check correctness"
-	@echo "  install_[language]_sdk Install the SDK ready for testing"
-	@echo ""


If the new build target also runs install then we probably don't need this anymore now that we have the SDK_LANG

corymhall · 2025-12-03T11:12:30Z

provider-ci/test-providers/acme/.github/actions/download-sdk/action.yml

      shell: bash
-      run: tar -zxf ${{ github.workspace }}/sdk/${{ inputs.language }}.tar.gz -C ${{ github.workspace }}/sdk/${{ inputs.language }}
+      run: mkdir -p ${{ github.workspace }}/sdk/${{ inputs.language }} && tar -zxf ${{ github.workspace }}/sdk/${{ inputs.language }}.tar.gz -C ${{ github.workspace }}/sdk/${{ inputs.language }}
+    - name: Uncompress SDK folder (Windows)


Is this only needed because of verify-release?

It is used for more steps than just that. Several steps download the SDK. We needed to make this change because tar will fail if the destination folder does not exist without the flag I added. Since the SDK folders are not checked in this is needed for the verify and test workflows.

I'm referring specifically to the (Windows) piece

Oh, yeah that is needed for the verify release windows. I left it (even though I disabled the job) as I do anticipate adding it back in at a later date

Per the other comment we don't want to download SDKs in the verify-release job since we want to use the published SDKs.

Would you mind if I tackled that at a different time when I go after the verify-release job?

As I understand it we don't (or shouldn't) need to touch verify-release, so let's roll those changes back before shipping this.

Sounds good

…ree different versions of it that existed

EronWright · 2025-12-03T23:07:58Z

provider-ci/internal/pkg/templates/base/Makefile

+
+install_sdks: .make/install_sdks
+.make/install_sdks: .make/build_sdks
+	provider-sdk-builder build-sdks --language $(SDK_LANG)


Does provider-sdk-builder build-sdks install too? This seems too similar to .make/build_sdks.

You just caught a huge typo! Thank you so much. The 'build-sdks' argument passed needs to be instead be 'install'.

provider-ci/internal/pkg/templates/base/.github/workflows/publish.yml

provider-ci/internal/pkg/templates/base/.github/workflows/verify-release.yml

EronWright · 2025-12-03T23:18:12Z

provider-ci/test-providers/acme/.github/workflows/build_sdk.yml

        # dependencies, commit the updated SDK and push it back to the PR. The
        # job will still be marked as a failure.
-        if: failure() && steps.worktreeClean.outcome == 'failure' && contains(github.actor, 'renovate') && github.event_name == 'pull_request'
+        if: failure() && contains(github.actor, 'renovate') && github.event_name == 'pull_request'


Is this change intentional that all failures (not just dirty worktrees caused by SDK generation) will trigger a commit?

Uhh no it was not. My intent was to remove the worktree clean step. Very good catch let me rework it.

I have just ripped this whole step of the workflow out

Graham-Pedersen · 2025-12-03T23:33:21Z

Also enough people have asked for a compile target I will add that back in

guineveresaenger · 2025-12-03T23:58:52Z

provider-ci/internal/pkg/migrations/migrate_cimgmt_to_mise.go

 	// If we have any plugins overrides, move them to the .config/mise.toml
 	pluginList := nodeToPluginEntries(plugins)
 	entries := pluginsToToolEntries(pluginList)
+


nit: surprise blank line

EronWright

Some feedback to be addressed.

…ed a missing minor version comment

EronWright · 2025-12-04T19:36:12Z

provider-ci/internal/pkg/templates/base/.github/workflows/test.yml

-    - name: Install dependencies
-      run: make install_${{ matrix.language}}_sdk


Is this still outstanding?

EronWright · 2025-12-04T19:37:16Z

provider-ci/internal/pkg/templates/base/Makefile

 	@touch $@

 # Build the provider and all SDKs and install ready for testing
-build: .make/mise_install provider build_sdks install_sdks#{{- if .Config.RegistryDocs }}# build_registry_docs#{{- end }}#


Can you speak to this? Does build include install or not?

EronWright · 2025-12-04T19:38:06Z

provider-ci/internal/pkg/templates/base/Makefile

+	@echo "  build_sdks    Generate and compile all SDKs"
+	@echo "  generate_sdks Generates all SDKs"
+	@echo "  compile_sdks  Compile all previously generated SDKs"
+	@echo "  install_sdks  Install SDKs. Installs nodejs and dotnet SDKs. Noop for go, java, and python, "


nit: little typo here? (, )

This reverts commit 9f83ae4.

Graham-Pedersen requested review from blampe, corymhall and guineveresaenger November 11, 2025 21:02

corymhall reviewed Nov 14, 2025

View reviewed changes

Graham-Pedersen added 4 commits December 2, 2025 11:50

Adding windows support to the download-sdk action

54bc2a6

Removing windows latest from the verify-release step to unblock SDK a…

2079f9e

…doption

Fixing the clean target to not destroy the hand checked in go.mod fil…

079e7b0

…e in the sdk directory Switching the build SDK workflow to use the pulumi bot token switching install to use the sdk tool directly instead of a make target

Graham-Pedersen force-pushed the new-sdk-tool branch from 8bcd4fa to 079e7b0 Compare December 2, 2025 19:51

blampe reviewed Dec 2, 2025

View reviewed changes

Graham-Pedersen commented Dec 2, 2025

View reviewed changes

corymhall reviewed Dec 3, 2025

View reviewed changes

Graham-Pedersen requested a review from a team December 3, 2025 17:10

Graham-Pedersen added 4 commits December 3, 2025 13:10

Addressing PR feedback and fixing a small issue

54ae590

Making link SDKs universal for the test workflows and removing the th…

cd9c3f8

…ree different versions of it that existed

Addressing PR feedback and fixing a small issue

4a98803

forgot about a change I made for language support in the install target

afdb672

EronWright reviewed Dec 3, 2025

View reviewed changes

Responding to PR feedback

0b9c625

guineveresaenger approved these changes Dec 4, 2025

View reviewed changes

EronWright suggested changes Dec 4, 2025

View reviewed changes

Graham-Pedersen added 4 commits December 4, 2025 09:00

removing a blank whitespace line

762dae9

Adding a generate target, fixed a typo in the compile target, and add…

d471f06

…ed a missing minor version comment

Adding missing provider name flag

5c537f8

testing verify release changes

f18cb6c

EronWright reviewed Dec 4, 2025

View reviewed changes

Graham-Pedersen added 3 commits December 5, 2025 09:29

Reverting verify-release workflow but without gradle cache

9f83ae4

Revert "Reverting verify-release workflow but without gradle cache"

c1d1d9c

This reverts commit 9f83ae4.

Further testing changes to the verify release workflow

48ed6c8

updating how we use mise in the update provider workflow

9448670

pose self-assigned this Dec 10, 2025

pose added this to the 0.130 milestone Dec 22, 2025

pose removed their assignment Dec 22, 2025

	run: provider-sdk-builder install --sdkLocation ./sdk/${{ matrix.language}} -l ${{ matrix.language}} --installLocation ${{ github.workspace }}
	run: provider-sdk-builder install --sdkLocation ./sdk/${{ matrix.language }} -l ${{ matrix.language }} --installLocation ${{ github.workspace }}

		- name: Install dependencies
		run: make install_${{ matrix.language}}_sdk

Switching to new SDK provider tool for makefiles and adding to mise #1822

Are you sure you want to change the base?

Switching to new SDK provider tool for makefiles and adding to mise #1822

Uh oh!

Conversation

Graham-Pedersen commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Graham-Pedersen commented Nov 11, 2025 •

edited

Loading