loading...

AWS – Using a macro to create an S3 bucket for CloudTrail logs

A CloudFormation macro is a transformation that allows you to create your own shorthand, in order to inject snippets into your templates. You probably already know a little bit about macros—AWS::Include and AWS::Serverless are both macros that are hosted by CloudFormation. Now, you have the ability to write your own macros, and with some creativity, you will find that this can be a very powerful tool to add to your arsenal as an AWS administration guru.

In this recipe, you will create a simple macro that allows you to configure an AWS CloudTrail auditing trail, and a bucket to hold the audit logs, all in just a few short lines of YAML. Similar to the last recipe on custom resources, you will create a lambda function in order to implement the macro transformation.

How to do it…

Follow these steps in order to create a lambda function, and a macro that uses that function to transform your template, before launching the stack:

  1. Log in to your AWS account, and go to the Lambda console.
  2. Click Create function and Author from scratch.
  3. Give the function a descriptive name.
  4. Choose Python 3.6 as the Runtime.
  5. Choose to  Create a Custom Role. A new screen opens, where you can create the role that will be associated with the lambda function. Give it a descriptive name:

Edit the policy document for the new Lambda execution role
  1. Click  Edit to customize the policy. You need to give the lambda function permissions to write to CloudWatch logs, and to create a CloudTrail, and also give it permission to administer the S3 bucket. Paste in the following code:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        
  1. Continue with the S3 permissions:
        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::*",
                "arn:aws:s3:::*/*"
            ],
            "Action": [
                "s3:CreateBucket",
                "s3:DeleteBucket",
                "s3:PutObject",
                "s3:DeleteObject"
            ]
        },
  1. And finally, add CloudTrail permissions to the policy:
        {
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "cloudtrail:*"
            ]
        }
    ]
}
  1. Click  Allow in order to associate the new role with the lambda function.
  2. Back on the Lambda console, click  Create function:

Author from scratch
  1. Paste the following code into the function code editor for index.js. Note that this code sample continues over several blocks:
"""
Lambda Handler for the CloudTrailBucket macro.

This macro transforms a resource with type "CloudTrailBucket" into 
a bucket, a bucket policy, and a CloudTrail configuration that logs 
activity to that bucket.
"""

def lambda_handler(event, _):
    "Lambda handler function for the macro"

    print(event)
  1. Continue by inspecting the fragment for the bucket name:

    # Get the template fragment, which is the entire 
    # starting template
    fragment = event['fragment']

    bucket_name = None

    # Look through resources to find one with type CloudTrailBucket
    for k, r in fragment['Resources'].items():
        if r['Type'] == 'CloudTrailBucket':
            r['Type'] = 'AWS::S3::Bucket'
            r['DeletionPolicy'] = 'Retain'
            bucket_name = k
  1. Create the policy for the bucket (note that this code fragment is broken up over several entries):
    bucket_policy = {
        "Type" : "AWS::S3::BucketPolicy",
        "Properties" : {
            "Bucket" : {"Ref" : bucket_name},
            "PolicyDocument" : {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Sid": "AWSCloudTrailAclCheck",
                        "Effect": "Allow",
                        "Principal": { 
                            "Service":"cloudtrail.amazonaws.com"
                        },
  1. Continue with the following code:
                        "Action": "s3:GetBucketAcl",
                        "Resource": { 
                            "Fn::Join" : [
                                "", [
                                    "arn:aws:s3:::", {
                                        "Ref": bucket_name
                                    }
                                ]
                            ]
                        }
                    },
  1. Continue with the following code:
                    {
                        "Sid": "AWSCloudTrailWrite",
                        "Effect": "Allow",
                        "Principal": { 
                            "Service":"cloudtrail.amazonaws.com"
                            },
                        "Action": "s3:PutObject",
                        "Resource": { 
                            "Fn::Join" : [
                                "", [
                                    "arn:aws:s3:::", {
                                        "Ref": bucket_name
                                    }, 
                                    "/AWSLogs/", 
                                    {
                                        "Ref":"AWS::AccountId"
                                    }, "/*"
                                ]
                            ]
                        },
  1. Finish the policy with this code:

                        "Condition": {
                            "StringEquals": {
                                "s3:x-amz-acl": "bucket-owner-full-control"
                            }
                        }
                    }
                ]
            }
        }
    }
  1. Complete the transformation of the fragment and return it:
    if bucket_name:
        # Add the policy to the fragment
        fragment['Resources'][bucket_name + 'BucketPolicy'] = bucket_policy

        # Create the trail and add it to the fragment
        trail = {
            'DependsOn' : [bucket_name + 'BucketPolicy'],
            'Type' : 'AWS::CloudTrail::Trail',
            'Properties' : {
                'S3BucketName' : {'Ref': bucket_name},
                'IsLogging' : True
            }
        }
        fragment['Resources'][bucket_name + 'Trail'] = trail

    # Return the transformed fragment
    return {
        "requestId": event["requestId"],
        "status": "success",
        "fragment": fragment,
    }
  1. Go to the CloudFormation console, where you will create two new stacks. The first stack creates the macro as a named resource in your account, and this can be used in any future template.
  2. Paste the following code into a file on your filesystem:
AWSTemplateFormatVersion: "2010-09-09"
Description: "This template creates the macro"
Parameters:
  FunctionArn:
    Type: String
Resources:
  CloudTrailBucketMacro:
    Type: AWS::CloudFormation::Macro
    Properties:
      Name: CloudTrailBucket
      FunctionName: !Ref FunctionArn
  1. Select Upload a template to Amazon S3, and choose the file that you just created. Click Next, and give the stack a name. 
  2. Paste in the function ARN of the Lambda function that you created earlier in this recipe.
  3. Click Next, and then Next on the following screen.
  4. Click Create. It may take a few minutes for the stack to be created. Wait until the status is CREATE_COMPLETE, before continuing.
  5. Now you will create a new (second) stack that makes use of the macro. Go back to the CloudFormation console, and click Create stack.
  1. Paste the following code into a file on your filesystem:
AWSTemplateFormatVersion: "2010-09-09"
Transform: CloudTrailBucket
Description: "This template will be transformed by the macro"
Resources:
  MyCloudTrailBucket:
    Type: CloudTrailBucket
  1. Select  Upload a template to Amazon S3, and choose the file that you just created. Click  Next, and give the stack a name. 
  2. Click Next, and Next on the following screen.
  3. You will notice that the final confirmation screen is different than what you normally see when you create a stack. Since a macro is transforming your template, you will, instead, be creating and executing a change set:

Stack confirmation screen
  1. Check the boxes in order to acknowledge that IAM resources will be created in this stack.
  2. Click Create Change Set. You will get a summary of the resources that will be created:

Create a change set
  1. Click Execute in order to launch the stack.
  2. Once the stack is complete, go to the CloudTrail dashboard, and click View Trails. You should see your new trail, along with the newly created bucket name:

CloudTrail dashboard
  1. Click through to the S3 bucket, and inspect the contents. It may take a few minutes for the new audit logs to show up.
  2. Delete the stack to clean up the resources from this recipe.

How it works…

Macros are actually a very simple concept—take the contents of a template, pass them to a lambda function as a string, and do string manipulation on the contents in order to replace or add elements. Then, the Lambda function passes the altered contents back to CloudFormation, and the stack is executed as if it had been originally written with the new content.

You created a global resource in your account by using the AWS::CloudFormation::Macro resource, which you linked to a custom lambda function. You then created a stack that makes use of that macro in order to expand a very short and simple template into a much more complex set of resources.

In this recipe, what we have done is to replace this code:

AWSTemplateFormatVersion: "2010-09-09"
Transform: CloudTrailBucket
Description: "This template will be transformed by the macro"
Resources:
  MyCloudTrailBucket:
  Type: CloudTrailBucket

We replaced it with this:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyCloudTrailBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
  MyCloudTrailBucketBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket:
        Ref: MyCloudTrailBucket
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: AWSCloudTrailAclCheck
          Effect: Allow
          Principal:
            Service: cloudtrail.amazonaws.com
          Action: s3:GetBucketAcl
          Resource:
            Fn::Join:
            - ''
            - - 'arn:aws:s3:::'
              - Ref: MyCloudTrailBucket
        - Sid: AWSCloudTrailWrite
          Effect: Allow
          Principal:
            Service: cloudtrail.amazonaws.com
          Action: s3:PutObject
          Resource:
            Fn::Join:
            - ''
            - - 'arn:aws:s3:::'
              - Ref: MyCloudTrailBucket
              - "/AWSLogs/"
              - Ref: AWS::AccountId
              - "/*"
          Condition:
            StringEquals:
              s3:x-amz-acl: bucket-owner-full-control
  MyCloudTrailBucketTrail:
    DependsOn:
    - MyCloudTrailBucketBucketPolicy
    Type: AWS::CloudTrail::Trail
    Properties:
      S3BucketName:
        Ref: MyCloudTrailBucket
      IsLogging: true

There’s more…

Once you grasp the concept of a CloudFormation macro, you will start to see all of the many ways that macros can be used in order to simplify and standardize the templates that you create in your AWS account.

Here are a few ideas to get you started:

  • Create a macro that allows you to embed custom Python or JavaScript code directly into a template.
  • Create a shorthand library of resources that are easier to type when authoring templates.
  • Standardize the creation of resources, such as S3 buckets, so that your compliance guidelines are met with little extra effort.
  • Expand on the built-in functions, such as Fn::Join and Fn::Sub, in order to create a rich library of string manipulation for use in your templates.
  • Combine several resources, with complex dependencies, into a single easy-to-use macro resource. For example, create a private S3 bucket to host a static website, configure an Amazon CloudFront distribution to make the content public, and add a custom domain in Amazon Route 53. 

See also

  • See the AWS documentation on macros for more details: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-macros.html

Comments are closed.

loading...