AWS SQS + Lambda Setup Tutorial – Step by Step

Many cloud-based applications require tasks like web applications or backends to call external services. A service that has variable loads at different times can present performance or reliability issues if it is under provisioned when traffic peaks. An attempt to counter-act this through over provisioning can lead to unnecessary costs.

One common architecture pattern is to control the load and the rate at which a service processes messages using queues. Queues act as buffers between a task and the services being called. This smooths out heavy loads that could cause the service to fail or the task to time out.

Using queues, we can control the rate at which data is processed by buffering it. What if we could automatically increase and decrease our computing capacity according to the volume of data? i.e. when queues have a higher or lower number of messages? This is where event-driven serverless architecture shines. With AWS Lambda, you can run code without provisioning or managing servers. Just upload your code and Lambda scales and runs it for you.

Now, how do you use queues to invoke the right amount of Lambda functions that match your load? Lambda comes out of the box with a feature called event source mappings. You can use it to process items from a stream or queue of AWS services that don’t invoke Lambda functions directly. We can apply this pattern with Lambda by setting queue events as event sources that trigger your Lambda function whenever a message lands in the queue.

In this article, we present a step-by-step guide on how to set up a Lambda function that responds to events of an SQS queue. We’ll start out by discussing some background of SQS and Lambda, and follow that up with the step by step tutorial in the second half.

So let’s get started.

Background on Lambda and SQS

Lambda

Lambda runs instances of a function to process events. A function is a resource that contains the code to process the events you pass into it or that other AWS services send to it.

When Lambda invokes your function it does so in an execution environment. The lifecycle of the execution environment includes the phases depicted below.

In the Init phase, Lambda creates the execution environment, downloads the code for the function and all layers, initializes any extensions, initializes the runtime, and then runs the function’s initialization code (the code outside the handler).

In the Invoke phase, Lambda invokes the function handler. The Lambda runtime passes two arguments to the function handler: event and context. When the handler exits or returns a response, it becomes available to handle another event.

The Shutdown phase is triggered if the function does not receive any invocations for a while.

SQS

SQS is an implementation of the producer-consumer design pattern (often referred to as pub-sub), where a producer is responsible for adding data, called messages, to a buffer (queue) that will be removed and processed by the consumer.

The lifecycle of a message goes as follows:

• A producer (Task) sends a message to a queue, which is distributed across the SQS servers. 
• When a consumer (Service) is ready to process messages, it consumes messages from the queue. 
• While a message is being processed, it becomes invisible to the queue for the duration of the visibility timeout.
• The consumer deletes the message from the queue after successful processing to prevent the message from being received and processed again when the visibility timeout expires. 
• SQS automatically deletes messages that have been in a queue for more than the maximum message configured retention period.

SQS supports two types of queues – standard queues and First-In-First-Out (FIFO) queues. Standard type is the default and supports at-least-once message delivery but, as it is distributed and redundant, one message can be delivered more than once if you have more than one consumer for a queue. FIFO queues give you more control at the cost of lower throughput.

Lambda and SQS

When a Lambda function subscribes to an SQS queue, meaning it uses a queue as an event source, it becomes a consumer of that queue. Lambda then polls the queue and invokes your Lambda function with an event that contains queue messages.

Lambda reads messages in batches and invokes your function once for each batch. Before invoking the function, Lambda continues to poll messages from the queue until the batch window expires or the maximum batch size is reached. You can configure the event source to buffer records for up to 5 minutes or 10,000 records of batch size, whatever comes first. There is an invocation payload size quota set by Lambda that is also a threshold for invoking a function, It tops at 6 MB each for request and response.

Each time a batch of messages is received by a function, it processes the data until it fails or succeeds. Some default behaviors upon success or failure go as follows:

• When your function successfully processes a batch, Lambda deletes its messages from the queue.
• When your function returns an error, Lambda leaves the messages in the queue.
• When some messages fail, you can configure your event source mapping to make only the failed messages visible again to avoid reprocessing all messages in a failed batch.
• If your function fails to process a message multiple times, SQS can send it to a dead-letter queue (DLQ), if you’ve configured one.

For standard queues, Lambda uses long polling to poll a queue until it becomes active. When messages are available, Lambda reads up to five batches and sends them to your function. If messages are still available, Lambda increases the number of processes that are reading batches by up to 60 more instances per minute. The maximum number of batches that an event source mapping can process simultaneously is 1,000.

SQS + Lambda Tutorial

In this tutorial, we will be using the AWS Management Console to create our Lambda function, and our queue and to connect and test everything. AWS provides many tools for you to manage services and resources, check this article for an overview and to learn when to use each tool.

• Part 1 – Creating your lambda function.
• Part 2 – Creating your queue.
• Part 3 – Subscribing the function to the queue.
• Part 4 – Editing your lambda function to receive messages.

Part 1 – Creating Your Lambda Function

First, you’ll need to head over to the Lambda section of the AWS console as seen below.

Open the functions page of the Lambda console and click Create function.

As we are not adding any dependencies we can choose ‘Author from Scratch’ and edit our code later in the console. Lambda will zip it and store it for us.

Choose a runtime and name for your function. In my case, I’ll be using Python 3.9.

Under Permissions, we need to allow our Lambda function to poll from our queue, thus select “Create a new role from AWS policy templates”, give it a name of your choice and search for SQS in “Policy Templates”. Then, select “Amazon SQS poller permissions”. Lambda will create the IAM user with the correct permissions policies.

Your configuration should look like as follows:

Go ahead and click on Create Function in the bottom right. It may take a minute or two for AWS to creation your function and the new IAM role.

Step 2 – Creating your SQS Queue

First, head over to the SQS section of the AWS console and click Create Queue. In this example, we’ll be using Standard Queue. However, you can also use FIFO if you require first-in first-out ordering. Below is a screen grab and a brief description of the parameters you have available to configure.

• Visibility timeout is the amount of time a message being processed will stay invisible before returning to the queue. If a message is not deleted by the consumer after processing it will return after this period.
• Message retention period is the maximum time a message stays in the queue awaiting processing.
• Delivery delay is the time a message will stay invisible when it is first sent to the queue before it can be read by the consumers.
• Receive message wait time is the maximum amount of time that polling will wait for messages to become available to receive. Any non-zero value sets long polling.

We’re going to leave all these settings as default for now, but you can explore them in the AWS SQS documentation.

Step 3 – Subscribing the Function to the Queue

Head back over to your Lambda Function and click on Add Trigger as seen below.

Click Select Source and search for SQS. Next, click the queue that you just created in Step 2. You can leave the Batch Size as 10 – this setting indicates to AWS Lambda the maximum number of messages you would like to receive in a single Lambda invocation. Note that you can change this to 1 to ensure 1:1 Lambda Invocation to Message processing.

Click on Add to finalize the creation of the trigger.

Step 4 – Modifying our Lambda Code to Receive SQS Messages

While staying in the Lambda section of the console, head over to the code editor to start making changes to our function.

Your code starts with a single function called lambda_handler that receives two arguments, event and context as seen in the snippet below.

def lambda_handler(event, context): 
	…
	return some_value

The first argument is the event object which is a JSON document that contains data for a function to process. The second argument is the context object, passed to your function by Lambda at runtime. This object provides methods and properties containing information about the invocation, function, and runtime environment.

Your messages will come from the event object. This JSON object is translated to a dictionary in Python and it contains the key records whose value is an array of messages in the format:

{
    "Records": [
        {
            "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
            "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
            "body": "Test message.",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1545082649183",
                "SenderId": "AIDAIENQZJOLO23YVJ4VO",
                "ApproximateFirstReceiveTimestamp": "1545082649185"
            },
            "messageAttributes": {},
            "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
            "awsRegion": "us-east-2"
        }
    ]
}

To retrieve the contents of the messages one by one, we should iterate on the records array the body field of each record. The code will become as follows:

def lambda_handler(event, context):
    records = event['Records']
    for record in records:
        content = record['body']
        print(content)
    return     {
        'statusCode': 200,
        'body': json.dumps(content)
    }

Your function should now be ready to start processing messages. You can use the SQS console and manually send messages to your queue, and confirm they are being processed using Cloudwatch Logs.

Here’s a screenshot of our Cloudwatch Log stream as it processes test events I send to the queue!

SQS and Lambda Best Practices

• Make sure that you configure the dead-letter queue on the source queue, not on the Lambda function. The dead-letter queue that you configure on a function is used for the function’s asynchronous invocation queue, not for event source queues.
• Functions should be designed to be idempotent. This means that receiving the same event multiple times does not change the result beyond the first time the event was received.
• Lambda’s by default can only execute 1000 concurrent requests at once per region. You can reduce this level of concurrency by setting the reserved concurrency setting to a lower number.