loading...

AWS – Setting up a secure Amazon S3 bucket

How to run your first application on Kubernetes

Amazon S3 is one of the main services offered by AWS. It is hard to imagine implementing even the most trivial architecture without using S3 buckets. In this recipe, you will create buckets in three ways – by using the web console, the command-line interface (CLI), and with CloudFormation. You will create buckets with different properties each time to give you a sample of the various configurations that are possible.

S3 provides a web-based service for hosting files. Files are referred to as objects and grouped in buckets. An object is effectively a key-value pair, similar to a document database. Keys are used like file paths, with / used as a separator and grouping character. Buckets can be accessed easily, like a website via an automatically generated domain name.

Due to being associated with a domain name, bucket names must be globally unique.

The following are some recommended use cases for S3:

  • Static website assets
  • Sharing large files
  • Short-term (that is, warm) backups

How to do it…

In this recipe, you will create different buckets using the web console, the CLI, and CloudFormation so that you are exposed to a variety of bucket configurations.

Using the web console to create a bucket with versioning enabled

Follow these steps to create a basic versioning-enabled bucket:

  1. Log in to your AWS account and go to the S3 dashboard.
  2. Click Create bucket:

Creating a bucket
  1. Give your bucket a globally unique name. 
  2. Click Next.
  3. Check the box to  Keep all versions of an object in the same bucket. This enables versioning so that you can revert objects to their former state if necessary:

Versioning a bucket
  1. Click Next.
  1. On the next screen, leave the defaults as they are – we don’t want our buckets to ever be made public:

Bucket access
  1. Click Next.
  1. Review the bucket settings on the following screen and click Create bucket:

Reviewing the bucket settings
  1. Once the bucket has been created, click on its name and take a look at the tools that are available to administer the new bucket:

Bucket administration
  1. Since we enabled versioning on this bucket, let’s test it out to see how it works.
  1. Create a text file on your desktop and add a single line to the file:

Creating a file to upload
  1. Click the Upload button and upload that file to the new bucket:

Uploading a file
  1. Click Next. Accept the defaults on the following screen and click Next again.
  2. On the next screen, inspect the various storage classes that are available. It’s worth spending some time learning the pros and cons of each class. Pick Standard:

Storage class
  1. Scroll down to the Encryption option. Select the  Amazon S3 master key. The object will be encrypted seamlessly, without requiring you to manually encrypt or decrypt it. Encryption is handled behind the scenes for you.
  1. On the final screen, click Upload:

File upload
  1. Click the object name to go to a screen dedicated to the object:

Object administration
  1. Note that there is an object URL at the bottom, but if you click it, you will get an error since we didn’t make this object public!
  2. Click the Download button to retrieve a copy of the object. Open it to confirm that it’s the same as what we uploaded.
  3. Edit the file on your desktop to add a new line. 
  4. Go back to the bucket administration and click Upload again. Upload a new copy of the file using the same procedure we outlined in the previous steps.
  5. Click the object name, and then click the Latest version link next to the name. Download each version to confirm that the content matches your expectations:

When versioning is enabled, all object versions are retained indefinitely

Using the CLI to create a bucket with cross-region replication enabled

In this recipe, we will do something slightly different than in the previous recipe. Cross-region replication can be a very important aspect of a solid disaster recovery plan, and it is often a requirement for various compliance certifications.

The first thing to understand about using S3 with the CLI is that there are two separate executables: s3 and s3api. These provide access to different tiers of functionality:

  • We use s3 for simple high-level commands that are similar to Unix shell commands such as cp and ls.
  • We use s3api to get complete access to the functionality offered by the entire S3 REST API.

In this recipe, you will use both tiers to create and manipulate buckets:

  1. List all the buckets in your account with aws s3:
$ aws s3 ls
2019-02-02 17:43:46 cf-templates-1llvkn4p8d3dr-us-east-1
2019-02-11 16:38:29 cf-templates-1llvkn4p8d3dr-us-west-1
2019-03-01 04:14:05 ezb-packt-admin1-console
2019-02-08 19:25:18 mycloudtrailbucketstack-mycloudtrailbucket-bf8yqwecopwv
  1. Create a new bucket in us-east-1. This bucket will be the source of cross-region replication (replace the following bucket name with a globally unique name of your choosing):
$ aws s3 mb --region us-east-1 s3://YOUR-SOURCE-BUCKET
make_bucket: YOUR-SOURCE-BUCKET
  1. Create a separate bucket in us-west-1 to act as the target of the replication:
$ aws s3 mb --region us-west-1 s3://YOUR-TARGET-BUCKET
make_bucket: YOUR-TARGET-BUCKET

  1. Enable versioning on the source and target buckets:
$ aws s3api put-bucket-versioning \
    --bucket YOUR-SOURCE-BUCKET \
    --versioning-configuration Status=Enabled
$ aws s3api put-bucket-versioning \
    --bucket YOUR-TARGET-BUCKET \
    --versioning-configuration Status=Enabled
  1. Create a role that will allow S3 to replicate objects on your behalf. Copy the following code into a file called cr-role.json:
{
   "Statement": [{
      "Effect": "Allow",
      "Principal": {
         "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
   }]
}
  1. Create the role:
$ aws iam create-role --role-name my-cr-role --assume-role-policy-document file://cr-role.json
  1. Attach a policy to the role that allows S3 access to the source and target buckets. Create a file called cr-policy.json, replacing the placeholders with your bucket names:
{
   "Statement":[
      {
         "Effect": "Allow",
         "Action": [
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl"
         ],
         "Resource": [
            "arn:aws:s3:::YOUR-SOURCE-BUCKET/*"
         ]
      },
      {
         "Effect": "Allow",
         "Action": [
            "s3:ListBucket",
            "s3:GetReplicationConfiguration"
         ],
         "Resource": [
            "arn:aws:s3:::YOUR-SOURCE-BUCKET"
         ]
      },
      {
         "Effect": "Allow",
         "Action": [
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags",
            "s3:GetObjectVersionTagging"

         ],
         "Resource": "arn:aws:s3:::YOUR-TARGET-BUCKET/*"
      }
   ]
}
  1. Attach the policy to the role:
$ aws iam put-role-policy --role-name my-cr-role --policy-name my-cr-role-policy --policy-document file://cr-policy.json 
  1. Configure cross-region replication on the first bucket using aws s3api. This is a complex command that is much easier to handle if you use a JSON file as input. Luckily, there is a way to output a template skeleton to get you started. In the following code, I used put-bucket-replication, along with the --generate-cli-skeleton parameter, and redirected the output to a file so that I could open it with a text editor:
$ aws s3api put-bucket-replication --generate-cli-skeleton > pbr.json
$ vim pbr.json
  1. Much of the skeleton is optional, so the file can be simplified. Use the following content for pbr.json. Make sure to replace the values for your role and your target bucket:
{
  "Role": "YOUR-ROLE-ARN",
  "Rules": [{
        "Status": "Enabled",
        "Prefix": "",
        "Destination": {
            "Bucket": "arn:aws:s3:::YOUR-TARGET-BUCKET"
        }
  }]
}
  1. Apply the replication configuration to the buckets:
$ aws s3api put-bucket-replication \
    --replication-configuration file://pbr.json \
    --bucket YOUR-SOURCE-BUCKET
  1. Copy an object into the source bucket:
$ touch hello.txt
$ aws s3 cp hello.txt s3://YOUR-SOURCE-BUCKET
  1. List the contents of the target bucket to confirm that the object was replicated successfully. Note that this might take a few seconds as the replication is eventually consistent:
$ aws s3 ls s3://YOUR-TARGET-BUCKET
2019-03-05 04:20:53 0 hello.txt

Using CloudFormation to create a bucket

In this recipe, you will use CloudFormation to create a bucket with resource tags that have been configured to enable cost-center reporting, and with a rule to move the objects to Glacier after 90 days:

  1. Create a file called 03-04-Glacier.yml with the following content:
Resources:
  MyCloudFormationBucket:
    Type: AWS::S3::Bucket
    Properties:
      LifecycleConfiguration:
        Rules:
        - Id: GlacierRule
          Prefix: glacier
          Status: Enabled
          Transitions:
            - TransitionInDays: '90'
              StorageClass: Glacier
      Tags: 
        - 
          Key: "COST_CENTER"
          Value: "Accounting"
        - 
          Key: "PROJECT_ID"
          Value: "FY2019"
Outputs:
  BucketName:
    Value: !Ref MyCloudFormationBucket
    Description: The bucket name
  1. Create the stack:
$ aws cloudformation create-stack \
    --stack-name 03-04-Glacier.yml \
    --template-body file://03-04-Glacier.yml

How it works…

You might be wondering why you have to give S3 buckets globally unique names when S3 is a strictly regional construct. The reason for this is that each bucket is given a URL that can be accessed from anywhere, regardless of the region. There are a variety of strategies you can use to make sure your buckets are named consistently, and one of them is to use a variation of your domain name. For example, if your account is associated with www.example.com, you could prefix each of your buckets with com-example-www-. This would give you a reasonable chance of avoiding name conflicts.

S3 is a great example of the kind of resiliency you can get from Amazon’s regional architecture that consists of multiple AZs, each of which a closely knit collection of data centers connected by high-speed data connections. Objects stored in S3 have 11 x 9s of durability, meaning they are 99.999999999% durable. The odds of permanently losing an object during our lifetimes are very slim.

There’s more…

Several interesting and powerful services are built on top of S3. The following are some examples.

Athena

Athena allows you to query data within S3 as if it were a relational database (with limitations). Athena queries are written in standard SQL, which allows data analysts to reach beyond the data warehouse to query structured data stored in S3, which is extremely cost-efficient.

S3 Select

S3 Select is similar to Athena, but it allows you to select fragments out of a single object in S3.

See alo

  • See the Backing up for compliance recipe in this chapter for a more detailed look at configuring Glacier.

Comments are closed.

loading...