Connecting to an AWS RDS database inside a VPC

[fideliocc]

Hi,
I plan to use faast.js to store an array of rows in a MySQL database created with AWS RDS. So, this DB is inside a VPC, so the Lambda function needs permissions to establish the connection to the host VPC. Could fellas please share some guidance or experience for the config in this scenario? Can't find any options for this.

[fideliocc]

UPDATE:
Reading a little more, I see it's possible to pass "awsLambdaOptions" as an attribute inside config for faast.faastAws() method. So, inside it I declared the security group ID and subnet ID as part of VpcConfig. See an example below:

 const faastFunctions = await faast.faastAws(methods, {
        // credentials,
        mode: 'queue',
        memorySize: 512,
        timeout: 40,
        region: 'sa-east-1',
        awsLambdaOptions: {
            VpcConfig: { 
                SecurityGroupIds: [ "string" ],
                SubnetIds: [ "string" ]
             }
        }
    });

Now, the execution is stopped from the first Faast function, giving no logs. Some ideas?

[acchou]

Right now faast.js might not be the best fit for use with VPC. According to the AWS docs (https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html), it can take up to a minute to connect the elastic network interface and during this time you'll just have to wait. Since faast.js recreates lambda functions each time you initiate execution, this means you'll pay this latency each time you start up faast. Not ideal.

Also, I haven't done testing with a VPC so I'm not sure what would be involved in configuring faast.js to use one. From what I can gather from the docs, it appears the issue might be the execution role permissions. You may need to create your own role and use the RoleName option to use this instead of the default one faast.js creates for you. See https://faastjs.org/docs/api/faastjs.awsoptions.rolename#awsoptionsrolename-property

I would be surprised if this worked, however, since the default role has admin privileges and any role you create would be more restrictive...

[fideliocc]

This article I read a time ago to better understand some RDS limitations with Lambda explain the ENIs problem and how it can been solved with AWS Hyperlane (https://blog.thundra.io/can-lambda-and-rds-play-nicely-together, section "AWS Improvements to Lambda's VPC Networking"). This indicates VPC connection time and cold starts are not a problem anymore. What do you think about it? I'll be testing this approach and updating you soon...

[fideliocc]

Thank you so much for your support @acchou. I've tested my process with the RDS MySQL DB outside the VPC and everything ran well sending 10-20 rows to store. But a new warning appears when I send a large process (500 - 2000 rows):

faast:warning InvalidParameterValue: One or more parameters are invalid. Reason: Message must be shorter than 262144 bytes.

Do you identify what this boundary is related to?

[acchou]

Any tips for getting things to work with VPC? I could add some information to the documentation to help future users.

The warning is caused by sending argument values to your functions that exceed 256kb, which is the limit for an SQS message in queue mode. See: https://faastjs.org/docs/api/faastjs.commonoptions.mode#remarks

You have a choice as to what to do:
(1) Change to https mode, which allows for a higher 6MB limit
(2) Create objects in S3 and pass references to them instead of passing them directly as arguments.
(3) Reduce the size of the arguments; you might be sending much more data than you realize through references to data you didn't intend to send.

[acchou]

Also, if you don't mind, could you tell me a little more about your use case? We can also pick this up in the discord channel: https://discord.gg/VxuCa9FN

[fideliocc]

@acchou I really appreciate your help. I'd been a little busy with other projects, but I finally tested faast.js to store 5000+ rows to a MySQL database and the process was successfully done! I'll be building a better project, I expect to store 20k+ rows in a single batch operation. Hope to share my conclusions very soon. One of the limitations, of course, is that currently, it is not able to connect to a VPC.

[fideliocc]

If it is possible I can join the Discord channel

[acchou]

Try here: https://discord.gg/F3aqjb3