Support unified pre/post processing for Nova MME model

[nathaliellenaa]

Description

Create a unified pre/post processing function for Nova MME model

Sample remote model payload

POST /_plugins/_ml/models/_register
{
    "name": "Bedrock Nova Multi-Modal Embedding model",
    "function_name": "remote",
    "description": "test model",
    "connector": {
        "name": "Amazon Bedrock Connector: embedding",
        "description": "The connector to bedrock nova multi-modal embedding model",
        "version": 1,
        "protocol": "aws_sigv4",
        "parameters": {
            "region": "{{aws_region}}",
            "service_name": "bedrock",
            "model": "amazon.nova-2-multimodal-embeddings-v1:0",
            "dimensions": 1024,
            "normalize": true,
            "embeddingTypes": [
                "float"
            ],
            "imageFormat": "jpeg",
            "detailLevel": "STANDARD_IMAGE",
            "videoFormat": "mp4",
            "audioFormat": "mp3",
            "input_docs_processed_step_size": 1
        },
        "credential": {
            "access_key": "{{nova_access_key}}",
            "secret_key": "{{nova_secret_key}}",
            "session_token": "{{nova_session_token}}"
        },
        "actions": [
            {
                "action_type": "predict",
                "method": "POST",
                "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
                "headers": {
                    "content-type": "application/json",
                    "x-amz-content-sha256": "required"
                },
                "request_body": "{\n  \"taskType\": \"SINGLE_EMBEDDING\",\n  \"singleEmbeddingParams\": {\n    \"embeddingPurpose\": \"GENERIC_INDEX\",\n    \"embeddingDimension\": ${parameters.dimensions},\n    \"text\": {\n      \"truncationMode\": \"END\",\n      \"value\": \"${parameters.inputText:-null}\"\n    },\n    \"image\": {\n      \"detailLevel\": \"${parameters.detailLevel:-null}\",\n      \"format\": \"${parameters.imageFormat:-null}\",\n      \"source\": {\n        \"bytes\": \"${parameters.inputImage:-null}\"\n      }\n    },\n    \"video\": {\n      \"format\": \"${parameters.videoFormat:-null}\",\n      \"source\": {\n        \"bytes\": \"${parameters.inputVideo:-null}\"\n      },\n      \"embeddingMode\": \"AUDIO_VIDEO_COMBINED\"\n    },\n    \"audio\": {\n      \"format\": \"${parameters.audioFormat:-null}\",\n      \"source\": {\n        \"bytes\": \"${parameters.inputAudio:-null}\"\n      }\n    }\n  }\n}",
                "pre_process_function": "connector.pre_process.bedrock.nova.embedding",
                "post_process_function": "connector.post_process.bedrock.nova.embedding"
            }
        ]
    }
}

Predict with different modalities

// Text embedding
POST /_plugins/_ml/models/HWwkhJoB---ynWQ14tgy/_predict
{
  "parameters": {
    "inputText": "hello world"
  }
}

// Image embedding
POST /_plugins/_ml/models/HWwkhJoB---ynWQ14tgy/_predict
{
  "parameters": {
    "inputImage": "{{image_base64}}"
  }
}

// Video embedding
POST /_plugins/_ml/models/HWwkhJoB---ynWQ14tgy/_predict
{
  "parameters": {
    "inputVideo": "{{video_bytes}}"
  }
}

// Audio embedding
POST /_plugins/_ml/models/HWwkhJoB---ynWQ14tgy/_predict
{
  "parameters": {
    "inputAudio": "{{audio_bytes}}"
  }
}

Note: Run predict with multiple modalities will lead to this error

POST /_plugins/_ml/models/HWwkhJoB---ynWQ14tgy/_predict
{
  "parameters": {
    "inputText": "hello world",
    "inputImage": "{{image_base64}}",
    "inputVideo": "{{video_bytes}}",
    "inputAudio": "{{audio_bytes}}"
  }
}

{
    "error": {
        "root_cause": [
            {
                "type": "status_exception",
                "reason": "Error from remote service: {\"message\":\"Malformed input request: #/singleEmbeddingParams: #: 4 subschemas matched instead of one, reformat your input and try again.\"}"
            }
        ],
        "type": "status_exception",
        "reason": "Error from remote service: {\"message\":\"Malformed input request: #/singleEmbeddingParams: #: 4 subschemas matched instead of one, reformat your input and try again.\"}"
    },
    "status": 400
}

Related Issues

Resolves #4365

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ylwu-amzn]

Can you also test this way

POST /_plugins/_ml/models/HWwkhJoB---ynWQ14tgy/_predict?algorithm=text_embedding
{
  "text_docs": [
    "hello world", "how are you"
  ]
}

The result should return two embeddings.
Use the same way to test other modality

[nathaliellenaa]

Can you also test this way

POST /_plugins/_ml/models/HWwkhJoB---ynWQ14tgy/_predict?algorithm=text_embedding
{
  "text_docs": [
    "hello world", "how are you"
  ]
}

The result should return two embeddings. Use the same way to test other modality

I tested with both text_embedding preprocess (connector.pre_process.bedrock.nova.text_embedding)and unified preprocess function. Both return single embedding

{
    "inference_results": [
        {
            "output": [
                {
                    "name": "sentence_embedding",
                    "data_type": "FLOAT32",
                    "shape": [
                        1024
                    ],
                    "data": [...
                    ]
                }
            ],
            "status_code": 200
        }
    ]
}

[nathaliellenaa]

Discussed offline, adding "input_docs_processed_step_size": 1 to the connector parameters solve the issue. Both text_embedding preprocess and unified preprocess function returns two embeddings. Tested image embedding and it returns two embeddings

POST /_plugins/_ml/models/tTwnhZoBuRZH0YwHn4I_/_predict?algorithm=text_embedding
{
  "text_docs": [
    "{{image_base64}}", "{{image_base64}}"
  ]
}

Encountered this error when testing video and audio multiple embeddings with the unified preprocess function.

{
    "error": {
        "root_cause": [
            {
                "type": "status_exception",
                "reason": "Error from remote service: {\"message\":\"Malformed input request: #/singleEmbeddingParams/text/value: expected maxLength: 50000, actual: 294244, reformat your input and try again.\"}",
                "suppressed": [
                    {
                        "type": "status_exception",
                        "reason": "Error from remote service: {\"message\":\"Malformed input request: #/singleEmbeddingParams/text/value: expected maxLength: 50000, actual: 294244, reformat your input and try again.\"}"
                    }
                ]
            }
        ],
       ...
    "status": 400
}

This is because the detectModalityParameter method cannot detect the modality type properly and it will go with the default value, which is text

ml-commons/common/src/main/java/org/opensearch/ml/common/connector/functions/preprocess/NovaMultiModalEmbeddingPreProcessFunction.java

Lines 51 to 70 in ae40e21

	private String detectModalityParameter(String input) {
	try {
	if (input.contains("\"text\"")) {
	return "inputText";
	}
	if (input.contains("\"image\"")) {
	return "inputImage";
	}
	if (input.contains("\"video\"")) {
	return "inputVideo";
	}
	if (input.contains("\"audio\"")) {
	return "inputAudio";
	}
	return "inputText";
	} catch (Exception e) {
	log.warn("Failed to detect modality from input, defaulting to text: {}", e.getMessage());
	return "inputText";
	}
	}