loading...

AWS – Creating and populating an S3 bucket with custom resources

As discussed in the introduction to CloudFormation in Chapter 1, AWS Fundamentals, it’s common for there to be cases where you need more advanced behavior than what is available by default in CloudFormation. Before custom resources, this led AWS developers down the path of doing most of their automation in CloudFormation, and then running some command-line interface (CLIs) commands to fill in the gaps. 

Fast forward to today, and the emerging pattern is to use a custom resource to delegate to an AWS Lambda function. Lambda can fill in the gaps by making API calls on your behalf. While it’s also possible to create a custom resource that communicates with your custom code via a Simple Notification Service (SNS), and a compute resource such as an Elastic Compute Cloud (EC2) instance, Lambda should be your first choice. 

In this section, you will learn how to build a custom resource that creates an Amazon Simple Storage Service (S3) bucket, and then populates that bucket with test data. It’s a simple example that might not be terribly useful, but it serves to demonstrate the basic concepts. Once you have mastered the technique, you will surely come up with some more creative uses for custom resources.

How to do it…

Follow these steps to create a lambda function, and a custom resource that makes use of that function, in order to create an S3 bucket:

  1. Log in to your account, and go to the Lambda console.
  2. Click Create function, and choose the  Author from scratch.
  3. Give your function a unique, descriptive name.
  4. Choose Node.js 8.10 as the Runtime:

The Lambda console allows you to author a function from scratch, or choose from a variety of templates
  1. Create a custom role by clicking on the IAM console link. A new screen opens, where you can create the role that will be associated with the lambda function. Give it a descriptive name.
  2. Click Edit to customize the policy. You need to give the lambda function permissions to write to CloudWatch logs, and also give it permission to administer the S3 bucket. Paste in the following code:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
  1. Continue with the statement that provides the S3 permissions:
        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::*",
                "arn:aws:s3:::*/*"
            ],
            "Action": [
                "s3:CreateBucket",
                "s3:DeleteBucket",
                "s3:PutObject",
                "s3:DeleteObject"
            ]
        }
    ]
}
  1. Click Allow, in order to associate the new role with the lambda function.
  2. Back on the Lambda console, click Create function, and paste the following code into the function code editor for index.js. This code sample continues over several blocks. Start with the required imports and constants:
/**
 * This is a custom resource handler that creates an S3 bucket 
 * and then populates it with test data.
 */

var aws = require("aws-sdk");
var s3 = new aws.S3();

const SUCCESS = 'SUCCESS';
const FAILED = 'FAILED';
const KEY = 'test_data.csv';
  1. Continue with the definition of the handler function:
exports.handler = function(event, context) {
 
    console.info('mycrlambda event', event);
    
    // When CloudFormation requests a delete, 
    // remove the object and the bucket. 
    if (event.RequestType == "Delete") {
        
        let params = {
            Bucket: event.ResourceProperties.BucketName, 
            Key: KEY
        };
  1. Continue with the code that implements a delete request:
    s3.deleteObject(params, function(err, data) {
            if (err) {
                console.log(err, err.stack);
                sendResponse(event, context, FAILED);
            } else {
                console.log('Deleted object', data);
                let params = {
                    Bucket: event.ResourceProperties.BucketName
                };
                s3.deleteBucket(params, function(err, data) {
                    if (err) {
                        console.log(err, err.stack);
                        sendResponse(event, context, FAILED);
                    } else {
                        console.log("Deleted bucket", data);
                        sendResponse(event, context, SUCCESS);
                    }
                });
            }
        });
        return;
    }
  1. We don’t take any action for an update:
    if (event.RequestType == "Update") {
        
        // Nothing to do here
        
        sendResponse(event, context, SUCCESS);
        return;
    }
 
    var params = {
        Bucket: event.ResourceProperties.BucketName
    };
     
  1. Create the bucket in response to a create request:
    s3.createBucket(params, function(err, data) {
        if (err) {
           console.log(err, err.stack); 
           sendResponse(event, context, FAILED, data);
        } else {
            console.log('Created bucket ' + 
                event.ResourceProperties.BucketName);
            // Now that we have created the bucket, populate it with test data
            params = {
               Body: '1,\"A\"\n2,\"B\"\n3,\"C\"', 
               Bucket: event.ResourceProperties.BucketName, 
               Key: KEY
            };
            s3.putObject(params, function(err, data) {
                if (err) {
                    console.log(err, err.stack); 
                    sendResponse(event, context, FAILED, data);
                } else {
                    console.log('Created object test_data.csv');
                    sendResponse(event, context, SUCCESS, data);
                }
            });
        }
    });
};
  1. Declare the following function to send responses back to the S3 signed URL, which is monitored by CloudFormation:

/**
 * Send a response to the signed URL provided by CloudFormation.
 */
function sendResponse(event, context, status, data) {
 
    var body = JSON.stringify({
        Status: status,
        Reason: "",
        PhysicalResourceId: context.logStreamName,
        StackId: event.StackId,
        RequestId: event.RequestId,
        LogicalResourceId: event.LogicalResourceId,
        Data: data
    });
  1. Continue the sendResponse function:

    console.log("body:\n", body);
 
    var https = require("https");
    var url = require("url");
 
    var parsedUrl = url.parse(event.ResponseURL);
    var options = {
        hostname: parsedUrl.hostname,
        port: 443,
        path: parsedUrl.path,
        method: "PUT",
        headers: {
            "content-type": "",
            "content-length": body.length
        }
    };
  1. End the function by making the HTTPS request:
 var request = https.request(options, function(response) {
        console.log("response.statusCode: " + 
            response.statusCode);
        console.log("response.headers: " + 
            JSON.stringify(response.headers));
        context.done();
    });
    request.on("error", function(error) {
        console.log("sendResponse Error:" + error);
        context.done();
    });
    request.write(body);
    request.end();
}
  1. Click  Save to the function. Copy the Amazon Resource Name (ARN) at the top right of the screen:

Copy the function ARN
  1. Now that you have created a lambda function that will execute your custom logic, go to the CloudFormation console.
  1. Create a file on your filesystem (the name doesn’t matter, but you should give it a .yaml extension) with the following content:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
    MyFunctionArn:
      Type: String
      Description: The ARN of the lambda function
    MyBucketName:
      Type: String
      Description: The name of the bucket to create
Resources:
  MyCustomResource:
      Type: Custom::CreateBucketWithData
      Version: "1.0"
      Properties:
        ServiceToken: !Ref MyFunctionArn
        BucketName: !Ref MyBucketName
  1. In the CloudFormation console, click Create Stack. Select Upload a template to Amazon S3, and upload the file.
  2. Give the stack a unique name, and paste the ARN from the function that you created earlier into the MyFunctionArn parameter textbox. In the MyBucketName parameter textbox, enter a globally unique name for the bucket that is to be created by your lambda function:

Stack details
  1. Click through the next two screens, and then click Create. It might take a few minutes for the stack to finish. You can watch its progress on the CloudFormation console.
  2. Once the stack is complete, go to the S3 console to confirm that the custom resource has successfully created the bucket, and the test data file within the bucket:

Bucket overview screen
  1. Delete the stack and the lambda function, so that you are left with a clean environment. This will avoid any unexpected costs in the future.

How it works…

Whenever you create, update, or delete a stack with an embedded custom resource, CloudFormation uses the ARN that you supply as a parameter to communicate with your lambda function. This ARN is referred to as the service token. The execution of the stack waits for a response from your code, but that response is asynchronous, so CloudFormation needs a way to poll for the result of the operation.

This is where S3 comes into the picture.

In your custom lambda code, when you have determined that the operation has succeeded or failed, you must report the status back to CloudFormation via a signed S3 URL, or your stack will be stuck with a CREATE_IN_PROGRESS status:

The process flow between CloudFormation, Lambda, and S3

When CloudFormation receives the response status back from Lambda, it can then continue with the stack creation if your operation succeeded, or start a rollback if your operation failed. Note that the S3 bucket in the previous diagram is the bucket that is managed by CloudFormation, not a bucket that we created ourselves. The CloudFormation management bucket serves as an inter-process communication (IPC) mechanism.

There’s more…

In this recipe, we entered our code directly into the Lambda console. While this is OK for learning exercises, for production code, you should investigate the Serverless Application Model (SAM), and AWS CodeCommit. SAM provides a simplified template syntax for creating applications that are based on lambda functions, and CodeCommit is a Git-compatible distributed version control system. It’s best practice to always store your code in version control, so that you can easily compare revisions, and share code with your peers.

SAM is an open source project that is documented here: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html.

Comments are closed.

loading...