Bedrock fix (#5642)

* fix(bedrock): Poll for model Active status before creating deployment

Add _wait_for_model_active() to poll get_custom_model until the model
reaches Active status before calling create_custom_model_deployment.
This fixes ValidationException when the custom model is not yet ready
for deployment after create_custom_model returns.

* feat(bedrock): Harden BedrockModelBuilder for production readiness

Extract _is_nova_model() helper to eliminate duplicated Nova
detection logic across deploy() and _get_s3_artifacts(). Uses
getattr with safe defaults instead of fragile hasattr chains.

Add input validation to deploy() and create_deployment():
- Raise ValueError when model_package is not set
- Raise ValueError when custom_model_name or role_arn missing
  for Nova deployments
- Raise ValueError when model_arn is empty in create_deployment

Move json and urlparse imports to module level (were previously
imported inside _get_checkpoint_uri_from_manifest). Replace
f-string logging with lazy %s formatting throughout.

Initialize status=None before the polling loop in
_wait_for_model_active to avoid UnboundLocalError if the loop
body never executes.

Rewrite unit tests (43 tests) with full coverage:
- _is_nova_model: recipe_name, hub_content_name, case
  insensitivity, missing base_model, None fields
- __init__: None model, TrainingJob, ModelPackage
- Client singletons: caching, injection
- _fetch_model_package: ModelPackage, TrainingJob, ModelTrainer,
  unknown type
- _get_s3_artifacts: None package, non-Nova, Nova delegation,
  Nova fallback
- _get_checkpoint_uri_from_manifest: success, missing key,
  NoSuchKey, not TrainingJob, no artifacts, invalid JSON
- _wait_for_model_active: immediate, polling, Failed, timeout
- create_deployment: polling chain, extra kwargs, empty/None ARN
- deploy: non-Nova, Nova full chain, hub_content_name detection,
  default deployment name, tags, missing params, None stripping

Add integration tests for Nova E2E deployment:
- Training job existence and status verification
- Builder creation and Nova detection via _is_nova_model
- S3 artifacts checkpoint validation
- Full deploy-with-polling flow (marked @pytest.mark.slow)
- Timeout behavior on bogus ARN
- Validation error paths (no model_package, empty model_arn)
- Resource cleanup fixture for deployments and custom models

* feat(bedrock): Add deployment status polling after CreateCustomModelDeployment

Previously create_deployment() only polled for the custom model
to reach Active status before calling CreateCustomModelDeployment,
but did not wait for the deployment itself to become Active. This
caused callers to receive a deployment ARN that was still in
Creating state, requiring manual polling in user code.

Add _wait_for_deployment_active() that polls
get_custom_model_deployment until status reaches Active, raises
RuntimeError on Failed, and times out after max_wait seconds
(default 3600s, poll interval 30s).

Wire it into create_deployment() so the full flow is now:
1. _wait_for_model_active (poll model creation)
2. create_custom_model_deployment (API call)
3. _wait_for_deployment_active (poll deployment creation)

Gracefully skips deployment polling if the API response does not
contain a customModelDeploymentArn.

Unit tests (48 passing):
- _wait_for_deployment_active: immediate Active, polling,
  Failed status, timeout
- create_deployment: full model+deployment polling chain,
  skip polling when no ARN in response
- deploy Nova chain: updated to verify deployment polling

* fix(integ): Fix region handling and add get-or-create Nova training job

The TestModelCustomizationDeployment integ tests were failing with
DescribeTrainingJob 'Requested resource not found' because the
SageMaker SDK caches the first session's region internally. The
session-scoped cleanup_e2e_endpoints fixture (autouse) was creating
a session in us-east-1 (default) before the class fixtures could
set us-west-2, causing all subsequent TrainingJob.get calls to hit
the wrong region.

Fix by setting AWS_DEFAULT_REGION=us-west-2 in the
cleanup_e2e_endpoints fixture before any SageMaker session is
created.

Add tests/integ/conftest.py with a session-scoped
nova_training_job_name fixture that implements get-or-create:
- Checks if sdk-integ-nova-micro-sft exists and is Completed
- If InProgress, waits for completion
- If not found, uploads minimal training data to S3 and launches
  a Nova Micro SFT training job via SFTTrainer
- Reused across test_bedrock_nova_e2e.py and
  TestBedrockNovaDeployment in test_model_customization_deployment

Update both Nova test files to use the shared fixture instead of
hardcoded training job names.

* fix(integ): Use SAGEMAKER_REGION for cross-region training job lookup

The SageMaker SDK's SageMakerClient reads SAGEMAKER_REGION env var
at init time and caches the region for all subsequent API calls.
The cleanup_e2e_endpoints session fixture was the first to create
a SageMakerClient (in the default region), which then poisoned all
subsequent TrainingJob.get calls regardless of the region parameter.

Fix by setting SAGEMAKER_REGION=us-west-2 in cleanup_e2e_endpoints
before any SDK session is created, since all resources in this test
file live in us-west-2. The env var is restored after cleanup.

In CodeBuild (us-west-2) this is a no-op since the default region
already matches. The other test files (triton, tei, tgi) are not
affected since they have their own fixtures and don't import from
this file.

* refactor(integ): Replace Nova integ tests with example notebook

Remove us-east-1 Nova integration tests that cannot run in the
us-west-2 CodeBuild environment:
- Delete tests/integ/test_bedrock_nova_e2e.py
- Delete tests/integ/conftest.py (Nova get-or-create fixture)
- Remove TestBedrockNovaDeployment class from
  test_model_customization_deployment.py

Add example_notebooks/bedrock_nova_deployment.ipynb covering the
full Nova workflow: SFTTrainer fine-tuning, BedrockModelBuilder
deploy with model+deployment polling, inference, and cleanup.

The BedrockModelBuilder source code and unit tests (48 passing)
are unchanged. The us-west-2 integ tests for non-Nova Bedrock
deployment (TestModelCustomizationDeployment) remain.

* docs: Add Bedrock model builder example notebooks

Add notebooks demonstrating Bedrock deployment workflows:
- bedrock-modelbuilder-deployment-nova.ipynb: Nova model deployment
  via BedrockModelBuilder with SFTTrainer fine-tuning
- boto3_deployment_notebook.ipynb: Direct boto3 Bedrock deployment
- model_builder_deployment_notebook(1).ipynb: ModelBuilder deployment
- 07-ml-model-development(1).ipynb: ML model development workflow
- sagemaker-serve/example_notebooks/bedrock_nova_deployment.ipynb:
  Clean Nova deployment example with polling, inference, and cleanup

* docs: Add Bedrock model builder example notebooks

Add notebooks demonstrating Bedrock deployment workflows:
- bedrock-modelbuilder-deployment-nova.ipynb: Nova model deployment
  via BedrockModelBuilder with SFTTrainer fine-tuning
- boto3_deployment_notebook.ipynb: Direct boto3 Bedrock deployment
- model_builder_deployment_notebook(1).ipynb: ModelBuilder deployment
- 07-ml-model-development(1).ipynb: ML model development workflow
- sagemaker-serve/example_notebooks/bedrock_nova_deployment.ipynb:
  Clean Nova deployment example with polling, inference, and cleanup

* fix(serve): Update Nova Bedrock deployment notebook with working e2e flow

Simplify notebook to use existing completed training job with
BedrockModelBuilder deploy flow. Fix Nova inference content format
to use array of {text: ...} objects. Remove broken SFTTrainer cells
that fail due to botocore service model mismatch.

Author: nargokul

07-ml-model-development(1).ipynb

@@ -0,0 +1,210 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Deploy a Fine-Tuned Nova Model to Amazon Bedrock\n",
+    "\n",
+    "This notebook demonstrates how to deploy a fine-tuned Amazon Nova model to\n",
+    "Amazon Bedrock using `BedrockModelBuilder`.\n",
+    "\n",
+    "The workflow:\n",
+    "1. Retrieve a completed Nova SFT training job\n",
+    "2. Create a `BedrockModelBuilder` from the training job\n",
+    "3. Deploy to Bedrock — the builder automatically:\n",
+    "   - Detects the model as Nova\n",
+    "   - Reads the checkpoint URI from the training job manifest\n",
+    "   - Calls `CreateCustomModel` and polls until Active\n",
+    "   - Calls `CreateCustomModelDeployment` and polls until Active\n",
+    "4. Test inference\n",
+    "5. Clean up resources\n",
+    "\n",
+    "**Prerequisites:**\n",
+    "- AWS credentials with SageMaker and Bedrock access\n",
+    "- `sagemaker-serve` package installed\n",
+    "- A completed Nova SFT training job\n",
+    "- An IAM role with Bedrock and SageMaker permissions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os, json, time, random, boto3\n",
+    "\n",
+    "REGION = \"us-east-1\"\n",
+    "os.environ[\"AWS_DEFAULT_REGION\"] = REGION\n",
+    "os.environ[\"SAGEMAKER_REGION\"] = REGION\n",
+    "\n",
+    "from sagemaker.core.helper.session_helper import get_execution_role\n",
+    "role_arn = get_execution_role()\n",
+    "print(f\"Role: {role_arn}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 1: Retrieve the completed training job\n",
+    "\n",
+    "Use an existing completed Nova SFT training job. Replace the job name with your own."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sagemaker.core.resources import TrainingJob\n",
+    "\n",
+    "training_job = TrainingJob.get(training_job_name=\"nova-textgeneration-micro-sft-20251208154822\")\n",
+    "print(f\"Training job: {training_job.training_job_name}\")\n",
+    "print(f\"Status:       {training_job.training_job_status}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 2: Deploy to Bedrock with BedrockModelBuilder\n",
+    "\n",
+    "The builder handles the full deployment flow:\n",
+    "- Fetches the model package from the training job\n",
+    "- Detects it as a Nova model\n",
+    "- Reads the checkpoint URI from the training output manifest\n",
+    "- Creates a Bedrock custom model and polls until Active\n",
+    "- Creates a deployment and polls until Active"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sagemaker.serve.bedrock_model_builder import BedrockModelBuilder\n",
+    "\n",
+    "builder = BedrockModelBuilder(model=training_job)\n",
+    "print(f\"Model package: {builder.model_package}\")\n",
+    "print(f\"S3 artifacts:  {builder.s3_model_artifacts}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "rand = random.randint(1000, 9999)\n",
+    "custom_model_name = f\"nova-e2e-{rand}-{int(time.time())}\"\n",
+    "deployment_name = f\"{custom_model_name}-dep\"\n",
+    "\n",
+    "print(f\"Deploying as: {custom_model_name}\")\n",
+    "print(\"This will poll for model creation and deployment — may take several minutes...\")\n",
+    "\n",
+    "response = builder.deploy(\n",
+    "    custom_model_name=custom_model_name,\n",
+    "    role_arn=role_arn,\n",
+    "    deployment_name=deployment_name,\n",
+    ")\n",
+    "\n",
+    "deployment_arn = response.get(\"customModelDeploymentArn\")\n",
+    "print(f\"\\nDeployment ARN: {deployment_arn}\")\n",
+    "print(\"Deployment is Active and ready for inference.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 3: Test inference\n",
+    "\n",
+    "Once the deployment is Active, invoke it via the Bedrock Runtime API.\n",
+    "Nova expects `content` as an array of objects with a `text` key."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "bedrock_runtime = boto3.client(\"bedrock-runtime\", region_name=REGION)\n",
+    "\n",
+    "resp = bedrock_runtime.invoke_model(\n",
+    "    modelId=deployment_arn,\n",
+    "    contentType=\"application/json\",\n",
+    "    body=json.dumps({\n",
+    "        \"messages\": [{\"role\": \"user\", \"content\": [{\"text\": \"What is 7 + 7?\"}]}]\n",
+    "    }),\n",
+    ")\n",
+    "\n",
+    "result = json.loads(resp[\"body\"].read())\n",
+    "print(f\"Response: {json.dumps(result, indent=2)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 4: Cleanup\n",
+    "\n",
+    "Delete the deployment and custom model to avoid ongoing charges."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "bedrock = boto3.client(\"bedrock\", region_name=REGION)\n",
+    "\n",
+    "dep_info = bedrock.get_custom_model_deployment(\n",
+    "    customModelDeploymentIdentifier=deployment_arn\n",
+    ")\n",
+    "model_arn = dep_info.get(\"modelArn\")\n",
+    "\n",
+    "# Delete deployment first\n",
+    "try:\n",
+    "    bedrock.delete_custom_model_deployment(\n",
+    "        customModelDeploymentIdentifier=deployment_arn\n",
+    "    )\n",
+    "    print(f\"Deleted deployment: {deployment_arn}\")\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed to delete deployment: {e}\")\n",
+    "\n",
+    "# Then delete the custom model\n",
+    "try:\n",
+    "    bedrock.delete_custom_model(modelIdentifier=model_arn)\n",
+    "    print(f\"Deleted custom model: {model_arn}\")\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed to delete custom model: {e}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}