Installing asyncpg on OS Amazon Linux

[StayFoolisj]

Hey I'm planning to use asyncpg as a part of my data pipeline infrastructure. All my pipelines run as AWS Lambda functions which themselves run the OS Amazon Linux. https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

I tried installing asyncpg in this environment, but at runtime I'm getting:

[ERROR] Runtime.ImportModuleError:  No module named 'asyncpg.protocol.protocol'

I'm adding asyncpg as a dependency by creating a Lambda layer as described here:
https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

I searched around in issues and elsewhere but failed to find this problem described. Anyone aware of the cause of this and perhaps have a suggestion for workaround?

[elprans]

Lambda requires extra hoops to be jumped through for binary packages. See, e.g. this post

[StayFoolisj]

I tried another approach than your example. I spun up a server with Amazon Linux, installed asyncpg there and imported those files to my Lambda function but same error as I posted above. Will debug a bit more, if no success I'll attempt the docker solution in your example link.

[StayFoolisj]

Followed the tutorial in your post, this works! Thank you so much :)

[angiodianxin]

I faced the same problem. And I solved it in a way that did not rely on external tools.

As described in official Documentation by AWS, we had to specify the pip install command with --platform and other options.
Also note that the architecture must be correctly specified in the runtime configuration.
Remember to specify asyncpg as Extras if you are using asyncpg with dependencies from other libraries such as the "databases" library.

[angiodianxin]

x86_64

pip install \
--platform manylinux2014_x86_64 \
--target=package \
--implementation cp \
--python-version <version> \
--only-binary=:all: --upgrade \
<package_name>

arm64

pip install \
--platform manylinux2014_aarch64 \
--target=package \
--implementation cp \
--python-version <version> \
--only-binary=:all: --upgrade \
<package_name>