What did you find confusing? Please describe.
Huggingface have documented how to use the sagemaker pytorch inference API in order to host their models. They make it quite clear that you must supply model_fn and then either transform_fn or (input_fn, predict_fn and output_fn). By using transform_fn you can have fine control of batch size for example, allowing you to handle large requests (in particular I have an issue where my batch transform jobs continuously die because the minimum payload of 1MB is way to large for my model - due to the large intermediate matrices I..e probabilities = batch_szie x num_labels)
I cannot find any mention of transform_fn in the documentation - https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html
It is mentioned in passing in one of the examples - https://sagemaker-examples.readthedocs.io/en/latest/frameworks/pytorch/get_started_mnist_deploy.html
Describe how documentation can be improved
Document the use of transform_fn as an alternative to input_fn, predict_fn and output_fn
Additional context
[Add any other context or screenshots about the documentation request here.]
This is how I was aware of transform_fn:
https://aws.amazon.com/blogs/machine-learning/run-computer-vision-inference-on-large-videos-with-amazon-sagemaker-asynchronous-endpoints/
The I found this:
(https://huggingface.co/docs/sagemaker/inference)
What did you find confusing? Please describe.
Huggingface have documented how to use the sagemaker pytorch inference API in order to host their models. They make it quite clear that you must supply
model_fnand then eithertransform_fnor (input_fn,predict_fnandoutput_fn). By usingtransform_fnyou can have fine control of batch size for example, allowing you to handle large requests (in particular I have an issue where my batch transform jobs continuously die because the minimum payload of 1MB is way to large for my model - due to the large intermediate matrices I..e probabilities = batch_szie x num_labels)I cannot find any mention of
transform_fnin the documentation - https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.htmlIt is mentioned in passing in one of the examples - https://sagemaker-examples.readthedocs.io/en/latest/frameworks/pytorch/get_started_mnist_deploy.html
Describe how documentation can be improved
Document the use of
transform_fnas an alternative toinput_fn,predict_fnandoutput_fnAdditional context
[Add any other context or screenshots about the documentation request here.]
This is how I was aware of
transform_fn:https://aws.amazon.com/blogs/machine-learning/run-computer-vision-inference-on-large-videos-with-amazon-sagemaker-asynchronous-endpoints/
The I found this:
(https://huggingface.co/docs/sagemaker/inference)