What is AWS Lambda Ephemeral Storage and How Does It Work?

Benefits of new feature Lambda Ephemeral storage

If you have ever used AWS Lambda for processing large files (such as PDFs, images, etc.) or running ETLs on massive dataset or for operationalizing Machine Learning models, you know the pain of putting up with runtime storage requirements and performance.

As a workaround to get more local storage, users had to mount an Elastic File System (EFS) to the Lambda function or use S3 as intermediary storage to read/write data during execution. Even though it solved issues associated with storage, users still faced performance degradations through cold start and significantly extra cost for having to use EFS, not to mention the general complexity of having to manage more moving pieces.

All of this is now solved as AWS introduced the Ephemeral storage option included in Lambda, which allows users to configure temporary storage required during each execution between 512 MB and 10 GB. 

This way, users get access to a secure, low-latency temporary file system within Lambda Functions, with the flexibility of reconfiguring the storage as required up to 10 GB.Important thing to note, Ephemeral storage is a temporary file system available within each Lambda function execution and not shared across all function invocations. For each function invocation, the function should assume availability of additional local storage provided via the Ephemeral storage option and after execution is complete, it disappears afterwards. If there is a requirement to persist the data after execution, make sure to offload it to an external storage system (such as EFS drive or Amazon S3).

How Lambda Ephemeral storage works

Lambda Ephemeral storage is available at a fixed location, in the “/tmp” directory in the file system. All data stored here is encrypted at rest with AWS managed key. Storage space from this file system is scoped for each Lambda function instance. This storage is not shared between instances of Lambda functions, and it is initially empty when a new instance starts.

Estimating cost for Lambda Ephemeral Storage

With Lambda Ephemeral storage option, you can provision up to 10 GB of temporary storage. Default ephemeral storage assigned to function is 512 MB, which is free. Every megabyte utilized over 512 MB will cost extra.

Pricing model is 0.00000309 cents per GB/s.

For example, say we have a Lambda function that over the course of one month handles 10 million requests with a runtime of about 100 milliseconds and uses 1 GB of memory. It will cost you about 11.41 cents. Now if you want to add ephemeral storage of 10 GB to this function, the new price will be 11.70 cents. That’s only a difference of 29 cents per month by adding the additional 10 GB of ephemeral storage. Pretty reasonable, right?

For more details on pricing, visit https://calculator.aws

How to use Ephemeral storage up to 10GB in new or existing Lambda functions

Get started with ephemeral storage by adding it to the new Lambda function or modifying existing ones through AWS Console, AWS CLI or AWS CloudFormation.

Configure Ephemeral storage using AWS Console

In AWS Lambda Console, follow these steps to configure desired size of ephemeral storage for your Lambda function:

Step 1: Navigate to the Configuration tab. Under the General configuration section, you will find Ephemeral storage, which is set to 512 MB by default.

Step 2: Click on the Edit button. Configure ephemeral storage from 512 MB to 10 GB in 1 MB increments.

$ aws lambda create-function \
--function-name PDFGenerator \
--runtime python3.9 \
--handler lambda_function.lambda_handler \
--code S3Bucket=myBucket, S3Key=function.zip \
--role arn:aws:iam::123456789012:role/testFunctionRole \
--ephemeral-storage '{"Size": 10240}'

$ aws lambda update-function-configuration \
--function-name PDFGenerator \
--ephemeral-storage '{"Size": 10240}'

ResizeFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: resizeFunction/
      Handler: app.handler
      Runtime: nodejs14.x
      Timeout: 900
      MemorySize: 10240
      EphemeralStorage:
        Size: 10240