What is AWS Lambda Ephemeral Storage and How Does It Work?

Looking to understand Lambda’s 10GB ephemeral storage? This is the article for you.

Introduction to AWS’ New Feature: 10 GB Lambda Ephemeral storage

AWS recently introduced a new feature which allows users to configure up to 10 GB ephemeral storage associated with each Lambda function instance.

You see, ephemeral storage is not a new feature but being able to configure it up to 10 GB is! Previously while using AWS Lambda functions, users only had access to 512 MB of ephemeral storage for each function executed, regardless of runtime or memory configuration.

In this article, we will walk you through the benefits of Ephemeral Storage, how it works, how to estimate cost and how to implement this new feature. 

So, let’s get started…

Benefits of new feature Lambda Ephemeral storage

If you have ever used AWS Lambda for processing large files (such as PDFs, images, etc.) or running ETLs on massive dataset or for operationalizing Machine Learning models, you know the pain of putting up with runtime storage requirements and performance.

As a workaround to get more local storage, users had to mount an Elastic File System (EFS) to the Lambda function or use S3 as intermediary storage to read/write data during execution. Even though it solved issues associated with storage, users still faced performance degradations through cold start and significantly extra cost for having to use EFS, not to mention the general complexity of having to manage more moving pieces.

All of this is now solved as AWS introduced the Ephemeral storage option included in Lambda, which allows users to configure temporary storage required during each execution between 512 MB and 10 GB. 

This way, users get access to a secure, low-latency temporary file system within Lambda Functions, with the flexibility of reconfiguring the storage as required up to 10 GB.Important thing to note, Ephemeral storage is a temporary file system available within each Lambda function execution and not shared across all function invocations. For each function invocation, the function should assume availability of additional local storage provided via the Ephemeral storage option and after execution is complete, it disappears afterwards. If there is a requirement to persist the data after execution, make sure to offload it to an external storage system (such as EFS drive or Amazon S3).

Common use-cases for using Ephemeral storage in Lambda

AWS Lambda Ephemeral storage is perfect for executions that require access to a large working directory to do things like unzipping large files or downloading massive datasets during runtime.

Some of the common use-cases are listed below:

  • Data processing and ETL jobs: With up to 10 GB of ephemeral storage available, enables more complex ETLs and data-intensive workloads to run in Lambda functions.
  • Image processing: For workloads processing large image files, Lambda ephemeral storage makes it easier to use libraries while execution.
  • Decompressing files from source: Dealing with large, zipped data files has become much easier as Lambda ephemeral storage allows users to unzip files to the local file system without the need of in-memory processing.
  • Machine Learning: Lambda ephemeral storage allows you to process data intensive workloads while creating a new model and to download larger existing models for operationalization.

How Lambda Ephemeral storage works

Lambda Ephemeral storage is available at a fixed location, in the “/tmp” directory in the file system. All data stored here is encrypted at rest with AWS managed key. Storage space from this file system is scoped for each Lambda function instance. This storage is not shared between instances of Lambda functions, and it is initially empty when a new instance starts.

Estimating cost for Lambda Ephemeral Storage

With Lambda Ephemeral storage option, you can provision up to 10 GB of temporary storage. Default ephemeral storage assigned to function is 512 MB, which is free. Every megabyte utilized over 512 MB will cost extra.

Pricing model is 0.00000309 cents per GB/s.

For example, say we have a Lambda function that over the course of one month handles 10 million requests with a runtime of about 100 milliseconds and uses 1 GB of memory. It will cost you about 11.41 cents. Now if you want to add ephemeral storage of 10 GB to this function, the new price will be 11.70 cents. That’s only a difference of 29 cents per month by adding the additional 10 GB of ephemeral storage. Pretty reasonable, right?

For more details on pricing, visit https://calculator.aws

How to use Ephemeral storage up to 10GB in new or existing Lambda functions

Get started with ephemeral storage by adding it to the new Lambda function or modifying existing ones through AWS Console, AWS CLI or AWS CloudFormation.

Configure Ephemeral storage using AWS Console

In AWS Lambda Console, follow these steps to configure desired size of ephemeral storage for your Lambda function:

Step 1: Navigate to the Configuration tab. Under the General configuration section, you will find Ephemeral storage, which is set to 512 MB by default.

Step 2: Click on the Edit button. Configure ephemeral storage from 512 MB to 10 GB in 1 MB increments.

Configure Ephemeral storage using AWS CLI

Creating new Lambda function with desired size of ephemeral storage using create-function command.

See example below: 

$ aws lambda create-function \
--function-name PDFGenerator \
--runtime python3.9 \
--handler lambda_function.lambda_handler \
--code S3Bucket=myBucket, S3Key=function.zip \
--role arn:aws:iam::123456789012:role/testFunctionRole \
--ephemeral-storage '{"Size": 10240}'

Updating existing Lambda function to use desired size of ephemeral storage using update-function-configuration command.

See example below:

$ aws lambda update-function-configuration \
--function-name PDFGenerator \
--ephemeral-storage '{"Size": 10240}'

Configure Ephemeral storage using AWS CloudFormation and AWS SAM templates

Configuring Lambda function with desired size of ephemeral storage using EphemeralStorage attribute.

See example below:

ResizeFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: resizeFunction/
      Handler: app.handler
      Runtime: nodejs14.x
      Timeout: 900
      MemorySize: 10240
      EphemeralStorage:
        Size: 10240
Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts