Using Lambda with SQS
note
This process has a flaw: The queue continues as soon as the render is triggered, not as soon as the render has finished. This means the renders do not all execute serially. We are working on an improved version of a Lambda queueing strategy, in the meanwhile be aware of this behavior.
This guide will show you how to use Amazon SQS (Simple Queue Service) and Remotion's renderMediaOnLambda() API.
Queues are used to park requests to make way for the underlying resources to cope up demand. Since AWS Lambda is subject to a concurrency limit, you can use SQS to queue renders in the background and you notify the user when the render process is completed by sending them an email or using other means of notification.
To supplement this guide, two projects have been created:
- The remotion-app contains a Remotion composition and utility scripts for deploying and deleting Remotion Lambda infrastructure in AWS. Note that this is the same application as from the Serverless Framework guide.
- The apigw-sqs-app (API Gateway SQS) contains a CDK project that deploys two Lambda functions. The enqueue-function function, when invoked, queues JSON data sent by the user; this can be an input payload for the Remotion Lambda function. The render-lambda-function function listens to enqueued message from SQS, processes it and renders videos.
The enqueue-function is configured to be invoked through API Gateway and is secured by Cognito. The API Gateway and Cognito setup is automatically created by the CDK Stack deployment upon execution of cdk deploy.
This guide assumes that you have knowledge in using CDK with TypeScript and AWS SQS knowledge. The AWS Cloud Development Kit (CDK) has been chosen to provision infrastructure as it is an official library from AWS.
remotion-app
- Follow the same setup instruction from remotion-app guide as we will just re-use the application.
apigw-sqs-app
In the following step, the Lambda functions enqueue-function and render-lambda-function will be deployed to your AWS account. This guide is designed to be executed on your local machine.
The project will create all the resources defined by the CDK stack, including setting up Cognito for the project's authentication and authorization system, uploading Lambda code, generating and associating IAM roles to your AWS account.
Prerequisites
- AWS deployment profile on your local machine, to configure an AWS deployment profile on your local machine.
- A AWS policy created named
remotion-executionrole-policywhich is created from this guide. - The AWS CDK should be installed globally on your local machine. If not yet done, follow the instructions in the "Install the AWS CDK" section here.
Setup
1. Clone or download the project
The project can be found at remotion-serverless project. If not done in the previous step, clone the project using:
bash
bash
2. Go to remotion-serverless and traverse to apigw-sqs-app directory
bash
bash
3. Install dependencies
- npm
- yarn
- pnpm
bash
bash
bash
bash
bash
bash
4. Create the Remotion policy
- The
remotion-executionrole-policyshould have been created, if not, follow this guide in setting this up.
The remotion-executionrole-policy is referenced from here. The CDK stack creates a role for render-lambda-function with the remotion-executionrole-policy.
ts
ts
This creates a role in AWS with the name of remotionSQSLambdaRole with 2 policies attached:
service-role/AWSLambdaBasicExecutionRolegrants permission to the Lambda function to interact with and log information to CloudWatch. This policy is part of the AWS library of policies.remotion-executionrole-policypolicy grants permission to the Lambda function to interact to AWS services that Remotion Lambda needs access to in order to render a video. This policy is exactly the same policy from this guide.
5. Optional - Synthesize
From the apigw-sqs-app directory, execute the cdk synthesize command.
bash
bash
This command will show the CloudFormation Template that CDK will execute to your AWS account.
6. Deploy the apigw-sqs-app project
From the apigw-sqs-app directory, execute the cdk deploy command.
bash
bash
This will orchestrate the generation of resources defined in the CDK stack, which are CloudFormation templates in the background.
Deploy outputbash
Deploy outputbash
Select 'y' for the answer of the prompt and CDK will continue to deploy the Stack, with the 2 function.
final outputbash
final outputbash
The deployment will provide variables such as apigw-sqs-app-stack.region, apigw-sqs-app-stack.userPoolClientId, and apigw-sqs-app-stack.userPoolId.
8. Remove the apigw-sqs-app from your AWS account, if not needed anymore
From the apigw-sqs-app directory.
bash
bash
Combining API Gateway, Remotion Lambda and SQS together.
These are important information on how IAM roles are used by the function and instructions to enable the render-lambda-function to consume SQS messages.
From CDK stack code, here are the important parts to take note of.
- The stack creates a queue named
remotion_queue.
create queuejs
create queuejs
-
remotion_queuegrants access to 2 Lambda functions to interact with it. Each individual role are assigned to their respective Lambda function.apiIntegrationRolejs// 👇 create the apiIntegrationRole roleconst apiIntegrationRole = new IAM.Role(this, 'api-integration-role', {assumedBy: new IAM.ServicePrincipal('lambda.amazonaws.com'),managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole',),],});apiIntegrationRolejs// 👇 create the apiIntegrationRole roleconst apiIntegrationRole = new IAM.Role(this, 'api-integration-role', {assumedBy: new IAM.ServicePrincipal('lambda.amazonaws.com'),managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole',),],}); -
This role is assigned to
enqueue-function, to allow it to write CloudWatch logs.remotionSQSLambdaRolejs// 👇 create a role with custom nameconst renderFunctionLambdaRole = new Role(this, 'remotionSQSLambdaRole', {roleName: 'remotionSQSLambdaRole',assumedBy: new ServicePrincipal('lambda.amazonaws.com'),managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole',),ManagedPolicy.fromManagedPolicyName(this,'remotion-executionrole-policy','remotion-executionrole-policy',),],});remotionSQSLambdaRolejs// 👇 create a role with custom nameconst renderFunctionLambdaRole = new Role(this, 'remotionSQSLambdaRole', {roleName: 'remotionSQSLambdaRole',assumedBy: new ServicePrincipal('lambda.amazonaws.com'),managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole',),ManagedPolicy.fromManagedPolicyName(this,'remotion-executionrole-policy','remotion-executionrole-policy',),],}); -
The
render-lambda-functionhas been assigned this role, which includes access to interact with Cloudwatch and permission to access other AWS services as specified in theremotion-executionrole-policy.Grant access to the queuejs// 👇 Grant permission to publish to the queueremotionQueue.grantSendMessages(apiIntegrationRole);// 👇 grant permission to consume messages from the queueremotionQueue.grantConsumeMessages(renderFunctionLambdaRole);Grant access to the queuejs// 👇 Grant permission to publish to the queueremotionQueue.grantSendMessages(apiIntegrationRole);// 👇 grant permission to consume messages from the queueremotionQueue.grantConsumeMessages(renderFunctionLambdaRole); -
Allow the render function to listen to the queue.
Listen to queuejsremotionRenderFunction.addEventSource(new SqsEventSource(remotionQueue, {batchSize: 1,maxBatchingWindow: Duration.minutes(5),reportBatchItemFailures: true, // default to false}),);Listen to queuejsremotionRenderFunction.addEventSource(new SqsEventSource(remotionQueue, {batchSize: 1,maxBatchingWindow: Duration.minutes(5),reportBatchItemFailures: true, // default to false}),);
Interacting with the API
The API requires an authorization token to interact with it. To obtain the token:
- After successful deployment you will be given out outputs such as
apigw-sqs-app-stack.region,apigw-sqs-app-stack.userPoolClientId, andapigw-sqs-app-stack.userPoolId, which are used to authenticate with Cognito. - If you don't have a frontend which user needs to login, you can create a user and an authentication token manually for the API by following this guide.
From the guide, YOUR_USER_POOL_CLIENT_ID is apigw-sqs-app-stack.userPoolClientId and YOUR_USER_POOL_ID is the apigw-sqs-app-stack.userPoolId, the steps should be followed up to retrieving the IdToken.
The base API URL has the format of https://25w651t09g.execute-api.ap-southeast-2.amazonaws.com/dev/enqueue from the output apigw-sqs-app-stack.apiUrl.
Trigger a video generation request
render videobash
render videobash
responsebash
responsebash
This will initiate the render request of a video by taking the JSON and putting it in the queue.
On the other end, render-lambda-function will be notified by SQS that is a video render request, it will consume the data from the render request and generate the video as per instruction, the function executes renderMediaOnLambda() as part of the process.
Notes
- The deployment of Remotion Lambda is configured to be deployed only to
ap-southeast-2region to simplify the project, adjust this in the code atregion.ts. - The deployment of
apigw-sqs-appis configured to be deployed atap-southeast-2region to simplify the project, adjust this in the code atremotion-cdk-starter.ts. - Remotion packages should be bundled inside the function when deployed, you can this in
nodeModulesproperty frombundlingobject, the code for this located in here.