AWS – Creating machine images with Hashicorp’s Packer

Creating or baking your own Amazon Machine Images (AMIs) is a key part of systems administration in AWS. Having a prebaked image helps you provision your servers faster, easier, and more consistently than configuring them by hand.

Packer is the de facto standard tool that helps you make your own AMIs. By automating the launch, configuration, and cleanup of your instances, it makes sure you get a repeatable image every time.

In this recipe, we will create an image with the Apache web server pre-installed and configured. While this is a simple example, it is also a very common use case.

By baking in your web server, you can scale up your web serving layer to dynamically match the demands on your websites. Having the software already installed and configured means you get the fastest and most reliable startup possible.

Getting ready

For this recipe, you must have the Packer tool available on your system. Download and install Packer from the project’s website at  https://www.packer.io/downloads.html.

Packer connects to your AWS account according to your current default AWS CLI configuration. Make sure that your CLI is configured properly before attempting this recipe.

How to do it…

  1. Create a new Packer template file and start by defining an amazon-ebs builder in the builders section. Note that the entire file, 13-01-Hashi.json, is available in the GitHub repository of this book https://github.com/PacktPublishing/AWS-SysOps-Cookbook-Second-Edition:
      "builders": [ 
        { 
          "type": "amazon-ebs", 
          "instance_type": "t2.micro", 
          "region": "us-east-1", 
          "source_ami": "ami-9be6f38c", 
          "ssh_username": "ec2-user", 
          "ami_name": "aws-linux-apache {{timestamp}}" 
        } 
      ],
The entire template file must be a valid JSON object. Remember to enclose the sections in curly braces: { ... }.
  1. Create a provisioners section and include the following snippet to install and activate Apache:
      "provisioners": [ 
        { 
          "type": "shell", 
          "inline": [ 
            "sudo yum install -y httpd", 
            "sudo chkconfig httpd on" 
          ] 
        } 
      ]
  1. Save the file with a specific name, such as 13-01-Hashi.json.
  1. Validate the configuration file you’ve created with the following packer validate command:
      packer validate 13-01-Hashi.json
  1. When valid, build the AMI with the following command:
      packer build13-01-Hashi.json
  1. Wait until the process is complete. While it is running, you will see an output similar to the following:

Packer build
  1. Take note of the AMI ID returned by Packer so that you can use it when launching instances in the future:

Packer build finished

Now, you can log in to your AWS account and go to the EC2 dashboard to verify the creation of the AMI in your default region. 

How it works…

While this is a very simple recipe, there is a lot going on behind the scenes. This is why we recommend you use Packer to create your machine images.

We will go over the following items in detail in the following subsections:

  • Template
  • Validating the template
  • Building the AMI

Template

In the builders section of the template, we define our build details.

We are using the most common type of AMI builder: amazon-ebs. There are other types of AWS builders available too, for instance, storage-backed instance types.

Next, we define the type of instance to use when baking.

Make sure that you can decrease the time it takes to bake your instance by using a larger instance size. Remember that the minimum price paid for an instance is one hour of billable time.

The source_ami property in this recipe is an AWS Linux AMI ID in the region we have specified. ssh_username allows you to set the username that’s used to connect and run provisioners on the instance. This will be determined by your operating system, which, in our case, is ec2-user.

Finally, the ami_name field includes the built-in packer variable called {{timestamp}}. This ensures the AMI you create will always have a unique name.

Validating the template

The packer validate command is a quick way to ensure your template is free of syntax errors before you launch any instances.

Building the AMI

Once you have created and validated your template, the packer build command does the following for you:

  1. First, it creates a one-time key pair for SSH access to the instance.
  2. Then, it creates a dedicated security group to control access to the instance.
  3. After completing the preceding step, it launches an instance.
  4. Then, it waits until SSH is ready to receive connections.
  5. Afterward, it runs the provisioner steps on the instance.
  6. Then, it stops the instance.
  7. After this, it generates an AMI from the stopped instance.
  8. Finally, it terminates the instance.
Check out the Packer documentation for more provisioners and functionality:  https://www.packer.io/docs/.

There’s more…

While Packer makes the administration of images much easier on AWS, there are still a few things to watch out for:

  • Debugging
  • Orphaned resources
  • Deregistering AMIs
  • Other platforms

Debugging

Obviously, with so many steps being automated for you, there are many things that can potentially go wrong. Packer gives you a few different ways to debug issues with your builds.

One of the most useful arguments to use with Packer is the -debug flag. This will force you to manually confirm each step before it takes place. Doing this makes it easy to work out exactly which step in the command is failing, which, in turn, usually makes it obvious what needs to be changed.

Another useful thing to do is to raise the level of logging output during a Packer command. You can do this by setting the PACKER_LOG variable to true. The easiest way to do this is by using PACKER_LOG=1 at the beginning of your Packer command line. This will mean you get a lot more information printed to the console (for example, SSH logs, AWS API calls, and so on) during the command. You may even want to run with this level of logging normally in your builds for auditing purposes.

Orphaned resources

Packer does a great job of managing and cleaning up the resource it uses, but it can only do that while it is running.

If your Packer job aborts for any reason (most likely network issues), then there may be some resources left orphaned, or unmanaged. It is good practice to check for any Packer instances (they will have Packer in their name) and stop them if there are no active Packer jobs running.

You may also need to clean up any leftover key pairs and security groups, but this is less of an issue as there is no cost associated with them (unlike instances).

Deregistering AMIs

As it becomes easier to create AMIs, you may find you end up with more than you need!

AMIs are made up of EC2 snapshots, which are stored in S3. There is a cost associated with storing snapshots, so you will want to clean them up periodically. Given the size of most AMIs (usually a few GBs), it is unlikely to be one of your major costs.

An even greater cost is the administrative overhead of managing too many AMIs. As your images improve and fixes are applied (especially security fixes), you may want to prevent people from using them.

To remove an AMI, you must first deregister it and then remove the underlying snapshots.

Make sure you do not deregister AMIs that are currently in use. For example, an auto-scaling group that references a deregistered AMI will fail to launch new instances!

You can easily deregister snapshots through the web console or using the AWS CLI tool.

Once an AMI is no longer registered, you can remove the associated snapshots. Packer automatically adds the AMI ID to the snapshot’s description. By searching your snapshots for the deregistered AMI ID, you can find which ones need to be deleted.

You will not be able to delete snapshots if the AMI has not been deregistered, or if deregistration is still taking place (it can take a few minutes).

Other platforms

It is also worth noting that Packer can build for more platforms than just AWS. You can also build images for VMWare, Docker, and many others.

This means you could build almost exactly the same machine image (for example, using Docker) that you have in AWS locally. This makes it much more convenient when setting up local development environments, for example.

Check the builders section of the Packer documentation for details.

Comments are closed.