Integrate deployment metadata service for locking and state#4856
Draft
shreyas-goenka wants to merge 11 commits intomainfrom
Draft
Integrate deployment metadata service for locking and state#4856shreyas-goenka wants to merge 11 commits intomainfrom
shreyas-goenka wants to merge 11 commits intomainfrom
Conversation
Add client integration with the deployment metadata service API for server-side deployment locking and resource state tracking. Gated behind DATABRICKS_BUNDLE_DEPLOYMENT_SERVICE=true environment variable. Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
|
Commit: 342fef8
|
- Use background context with timeout for CompleteVersion in defer blocks, so the lock is released even if the parent context is cancelled (e.g. Ctrl+C) - Add nil state.ID guard in destroy to avoid querying with zero UUID - Fix misleading --force-lock error message to explain lock expiry behavior - Fix import ordering Co-authored-by: Isaac
Move the deployment metadata service client from bundle/deploy/metadata/service to libs/tempdms with SDK-style method signatures (single request struct param). When the protos land in the Go SDK, migration is just an import path change. Unify deploy and destroy flows: instead of separate *WithMetadataService functions that duplicated all mutator calls, the core logic stays in Deploy() and Destroy() with conditional lock management based on the env var. Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
The proto HTTP bindings use `body: "deployment"`, `body: "version"`, and `body: "operation"` for Create endpoints, which means only the sub-message goes in the request body. The identifier fields (deployment_id, version_id, resource_key) must be passed as query parameters. Previously these fields were incorrectly included in the request body, which would cause "required field missing" errors against the real service. Also updates the test server to read these fields from query parameters instead of the body, so acceptance tests validate the real API contract. Co-authored-by: Isaac
- Rename VersionCompleteLeaseExpire to VersionCompleteLeaseExpired to match proto enum VERSION_COMPLETE_LEASE_EXPIRED. - Remove redundant "parent" query parameter from ListResources (the deployment ID is already in the URL path). - Add acceptance test for the deployment metadata service integration that validates the correct API call sequence during deploy and destroy. Co-authored-by: Isaac
shreyas-goenka
had a problem deploying
to
test-trigger-is
GitHub Actions
Failure
Use print_requests.py to print all requests to /bundle endpoints at each stage (deploy and destroy) for clear visibility into the API call sequence. Co-authored-by: Isaac
shreyas-goenka
had a problem deploying
to
test-trigger-is
GitHub Actions
Failure
- Rename libs/tempdms package to libs/tmpdms - Rename env var to DATABRICKS_BUNDLE_MANAGED_STATE - Use lineage from resources.json as deployment ID - Write _deployment_id file to state directory - Remove postApplyHook, add inline OperationReporter - Set heartbeat interval to 30 seconds Co-authored-by: Isaac
shreyas-goenka
had a problem deploying
to
test-trigger-is
GitHub Actions
Failure
Fix map[string]string -> map[string]any in tmpdms API client for SDK v0.126.0 compatibility. Generate golden files for metadata-service acceptance test showing the full deploy/destroy request flow. Co-authored-by: Isaac
shreyas-goenka
had a problem deploying
to
test-trigger-is
GitHub Actions
Failure
…tion - Change all enum types from int to string using proto enum name strings (e.g. "OPERATION_ACTION_TYPE_CREATE" instead of 4), matching proto-over-HTTP serialization format. - Report failed operations to the metadata service with error messages, not just successful ones. - Enforce direct deployment engine for managed state (early return). - Extract acquireMetadataLock helper to deduplicate deploy/destroy lock blocks. - Add deploy-error acceptance test verifying failed operation reporting. Co-authored-by: Isaac
shreyas-goenka
had a problem deploying
to
test-trigger-is
GitHub Actions
Failure
When the DATABRICKS_LITESWAP_ID environment variable is set, wrap the SDK HTTP transport to inject the x-databricks-traffic-id header on all API requests. This routes traffic to the liteswap service instance for E2E testing against dev deployments. Usage: DATABRICKS_LITESWAP_ID=my-env databricks bundle deploy Co-authored-by: Isaac
shreyas-goenka
had a problem deploying
to
test-trigger-is
GitHub Actions
Failure
The LiteswapID() function was never called; workspace.go reads DATABRICKS_LITESWAP_ID via os.Getenv directly. Co-authored-by: Isaac
shreyas-goenka
had a problem deploying
to
test-trigger-is
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/api/2.0/bundle/) for server-side deployment locking and resource state trackingINITIAL_REGISTERfor first-time tracking)Gated behind
DATABRICKS_BUNDLE_DEPLOYMENT_SERVICE=trueenvironment variable. Zero behavior change when the flag is off.New files
bundle/env/deployment_metadata.go— env var definitionbundle/deploy/metadata/service/types.go— Go structs matching the proto APIbundle/deploy/metadata/service/client.go— HTTP client for all deployment metadata endpointsbundle/deploy/metadata/service/heartbeat.go— background lock renewalbundle/phases/deploy_metadata.go— new deploy flow with metadata servicebundle/phases/destroy_metadata.go— new destroy flow with metadata serviceModified files
bundle/deploy/state_update.go— exportLoadState()functionbundle/phases/deploy.go— feature flag checkbundle/phases/destroy.go— feature flag checkTest plan
[[Server]]stubs for deploy/destroy flowsThis pull request was AI-assisted by Isaac.