AWS – Network logging and troubleshooting

How to configure nginx for Magento 2

One of the benefits of using a virtualized infrastructure is that you can get a level of introspection that is difficult or costly with physical hardware. Being able to quickly switch on logging at a network-device level is an extremely useful feature, especially when getting used to the interactions between VPCs, subnets, NACLs, routing, and security groups. A common use case would be figuring out why a specific user is not able to connect to an EC2 instance inside one of your VPCs.

In this recipe, we will turn on logging for our network resources by using VPC Flow Logs. You could do this all the time to give yourself another layer for monitoring and auditing, or you could selectively enable it during troubleshooting, saving yourself any additional data storage charges. VPC Flow Logs allow you to capture and analyze information (but not individual packets) about traffic that is flowing to and from network interfaces within your account.

Getting ready

For this recipe, you must have a VPC to log activity on.

How to do it…

Follow these steps to set up a flow-log:

  1. Start by defining the template version and description:
AWSTemplateFormatVersion: "2010-09-09" 
Description: Flow logs for networking resources
  1. Define Parameters for the template. In this case, it is just the VpcId where we will enable flow-logs:
   Type: String 
   Description: The VPC where we will enable flow logs

  1. Create the Resources section of the template and define the log group to use to send our flow logs to:
    Type: AWS::Logs::LogGroup 
      LogGroupName: LogGroup
  1. Next, we define the IAM role that will give the flow-logs service permission to write the logs:
    Type: AWS::IAM::Role 
        Version: "2012-10-17" 
            Effect: Allow 
            Action: sts:AssumeRole 
          PolicyName: CloudWatchLogsAccess 
            Version: "2012-10-17" 
                  - logs:CreateLogGroup 
                  - logs:CreateLogStream 
                  - logs:PutLogEvents 
                  - logs:DescribeLogGroups 
                  - logs:DescribeLogStreams 
                Effect: Allow 
  1. Finally, we define the flow-log itself:
    Type: AWS::EC2::FlowLog 
    DependsOn: LogGroup 
      DeliverLogsPermissionArn: !GetAtt IamRole.Arn 
      LogGroupName: LogGroup 
      ResourceId: !Ref VpcId 
      ResourceType: VPC 
      TrafficType: ALL        
  1. Save the template and give it a filename such as 05-02-NetworkLogging.yaml.
  2. Create the flow-logs and associated resources by creating the template with the following command:
      aws cloudformation create-stack \
        --stack-name VpcFlowLogs \
        --template-body file://05-02-NetworkLogging.yml \
        --capabilities CAPABILITY_IAM \
        --parameters ParameterKey=VpcId,ParameterValue=<your-vpc-id>

Once launched (and assuming you have network activity), you will be able to see your flow-log in the CloudWatch Logs console. If you don’t see any flow-logs, you may need to create a resource such as an EC2 instance and SSH into it.

How it works…

The only parameter that is required for this template is the VPC ID to target. We specifically target a VPC to turn on flow-logging for because it gives us the most bang for buck. While you can enable flow-logs for subnets and Elastic Network Interfaces (ENIs) individually, if you enable them on a VPC, you get flow-logs for all the networking resources contained in that VPC, which includes subnets and ENIs.

In the Resources section, we start by explicitly defining the log group to hold the flow-logs. If you don’t create the log group yourself (and specify it in your flow-log resource configuration), a log group will be created for you. This means that you will still be able to use flow-logs, but the log group won’t be managed by CloudFormation and will have to be maintained (for example, deleted) manually. We have also set a deletion policy of delete for our log group. This means that it will be deleted if the CloudFormation stack is deleted.

Next, we define the IAM role to use. Via the AssumeRolePropertyDocument value, we give the AWS flow-logs service permission to assume this role. Without this access, the flow-logs service cannot access the account. In the Policies property, we give the role permission to create and update log groups and streams.

Finally, now that we have created the dependent resources, we define the flow-log resource itself. You don’t need to define the resources in order of dependencies, but it is usually easier to read if you do. In the resource, we also define a DependsOn relationship to the log group that we defined earlier so that the log group is ready to receive the flow-logs when it is created.

The final step is to launch the template that you have created, passing the VPC ID as a parameter. As this template creates an IAM role to allow the VPC service to send logs to CloudWatch Logs, the command to create the stack must be given the CAPABILITY_IAM flag in order to signify that you are aware of the potential impact of launching this template.

There’s more…

Turning on logging is just the start of the troubleshooting process. There are a few other things that you should be aware of when using flow-logs:

  • Log format
  • Updates
  • Omissions

Log format

Once logging is enabled, you can view the logs in the CloudWatch Logs console. Here is a summary of the type of information that you will see in the flow-log (in order):

  • The VPC flow-logs version
  • The AWS account ID
  • The ID of the network interface
  • The source IPv4 or IPv6 address
  • The destination IPv4 or IPv6 address
  • The source port of the traffic
  • The destination port of the traffic
  • The Internet Assigned Numbers Authority (IANA) protocol number of the traffic
  • The number of packets transferred
  • The number of bytes transferred
  • The start time of the capture window (in Unix seconds)
  • The end time of the capture window (in Unix seconds)
  • The action associated with the traffic; for example, ACCEPT or REJECT
  • The logging status of the flow-log; for example, OK, NODATA, or SKIPDATA
To identify the protocol, check the protocol number field against the IANA protocol numbers list at .


You cannot update the configuration of an existing flow-log; you must delete it and recreate it if you want to change any of the associated settings. This is another reason why it is good to explicitly create and manage the associated log group.


Some traffic is not captured by the flow-logs service, as follows:

  • Traffic to the Amazon DNS server (x.x.x.2 in your allocated range)
  • Traffic for Amazon Windows license activation (only applicable to Windows instances).
  • Traffic to and from the instance metadata service (that is, IP address,, which gives the services that are running on the machine a way to get information on the instance’s configuration
  • DHCP traffic
  • Traffic to the reserved VPC IP address for the default VPC router (x.x.x.1 in your allocated range)

See also

  • The Virtual Private Cloud (VPC) recipe in Chapter 7, AWS Networking Essentials

Comments are closed.