Note: This is a repost of this blog post with the help of James Becker and reviewed by Milin Patel. The post was published on the company Rearc's blog page on August 18, 2022 during my summer internship and shared by Rearc on Linkedin.

Flow logs are the native network logging layer for AWS. These logs can be setup specifically for logging IP traffic on subnets, network interfaces, or VPCs. VPC flow logs in particular contain a vast amount of IP traffic information and data points for our resources that can be leveraged for:

• Monitoring boundaries for networks and AWS accounts

• Detecting anomolous network activity

• Catching unintentional cross-region data transfers early (to avoid unnecessary costs)

• Identifying system optimizations based on AZ distribution

• Performing various network traffic flow optimizations

In this blog post, we'll be learning how to:

• Deploy VPC flow logs to Amazon CloudWatch using AWS Cloud Development Kit (CDK) for Python

• Identify relevant data points with example use cases

• Query traffic logs for specific data points.

VPC flow log example architecture

In the following example, a flow log publishes all IP resource traffic in a VPC to a CloudWatch log group:

The flow log needs an IAM role with write-access for publishing the logs to CloudWatch.

How CloudWatch organizes VPC flow log data

The way VPC flow logs are published to CloudWatch is in three steps:

1. A log group is created for archiving all flow log data

2. A log stream is created for each resource being monitored

3. Log events are created within each log stream with custom data points for IP traffic

Basically, a log group consists of log streams which consist of log events.

Deploy VPC flow logs publishing to CloudWatch logs for near real-time analytics

Now that we have an idea about how flow logs work and how we can find our network data in CloudWatch, let's build some flow logs!

Deploying VPC flow logs with AWS CDK

AWS CDK allows us to write cloud application resources through code in a supported language (Typescript, Python, Go, etc.) which then gets provisioned/deployed by AWS CloudFormation in the background. We often use constructs, which are basic cloud components that can be made of one or more resources, in order to build our application.

There are a couple ways we can set up flow logs with AWS CDK for Python:

• The VPC construct has an intuitive add_flow_log method

• The default AWS CDK FlowLog construct is a high-level flow log resource we can add to a CDK stack by instantiating it

However, these options use the default log format and don't allow for setting a custom log format, which is a crucial feature for choosing specific data fields in our network traffic that we want the logs to output.

The solution is to build a custom AWS CDK construct with the lower-level construct CfnFlowLog since it includes a log_format attribute. By building a custom construct based on CfnFlowLog, we can:

• Customize the log format

• Modularize the construct into its own file

• Reuse the construct for any given VPC in an AWS CDK stack

Let's look at a custom FlowLog construct that implements this feature.

Custom FlowLog construct

from aws_cdk import aws_iam as iam, aws_logs as logs, aws_ec2 as ec2
from constructs import Construct


class FlowLog(Construct):
  def __init__(self, scope: Construct, id: str, *args, vpc: ec2.Vpc, **kwargs):
    super().__init__(scope, id, **kwargs)

    self.vpc = vpc

    self.role = iam.Role(
      self,
      "Role",
      assumed_by=iam.ServicePrincipal("vpc-flow-logs.amazonaws.com"),
    )

    self.log_group = logs.LogGroup(
      self, "LogGroup", retention=logs.RetentionDays.TWO_WEEKS
    )

    self.log_group.grant_write(self.role)

    self.flow_log = ec2.CfnFlowLog(
      self,
      "FlowLog",
      resource_id=self.vpc.vpc_id,
      resource_type="VPC",
      traffic_type="ALL",
      deliver_logs_permission_arn=self.role.role_arn,
      log_destination_type="cloud-watch-logs",
      log_group_name=self.log_group.log_group_name,
      log_format="${traffic-path} ${flow-direction} ${region} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${action} ${log-status}",
    )

FlowLog Construct Definition

class FlowLog(Construct):
  def __init__(self, scope: Construct, id: str, *args, vpc: ec2.Vpc, **kwargs):
    super().__init__(scope, id, **kwargs)

      self.vpc = vpc

• The FlowLog class inherits from Construct that can be reused in multiple stacks for deployment

• The parameter vpc is required to associate a new or existing VPC in the stack with its own flow logs

• The current instance's vpc is identified with self.vpc

VPC, IAM Role, Log Group, and Write Permissions

      self.role = iam.Role(
        self,
        "Role",
        assumed_by=iam.ServicePrincipal("vpc-flow-logs.amazonaws.com"),
      )

      self.log_group = logs.LogGroup(
        self, "LogGroup", retention=logs.RetentionDays.TWO_WEEKS
      )

      self.log_group.grant_write(self.role)

• The IAM role is attached to VPC flow logs through its service principal

• The IAM role grants write permissions for publishing to the CloudWatch log group

• The CloudWatch log group can have its logs last for a specified timeframe called a retention period, such as two weeks or indefinitely

CfnFlowLog Resource

This is where the flow log is actually created and connects with all the other components we've setup.

      self.flow_log = ec2.CfnFlowLog(
        self,
        "FlowLog",
        resource_id=self.vpc.vpc_id,
        resource_type="VPC",
        traffic_type="ALL",
        deliver_logs_permission_arn=self.role.role_arn,
        log_destination_type="cloud-watch-logs",
        log_group_name=self.log_group.log_group_name,
        log_format="${traffic-path} ${flow-direction} ${region} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${action} ${log-status}",
      )

• resource_id – Associates the flow log to a given VPC by ID

• deliver_log_permission_arn – Associates the IAM role by ARN (for granting write permission to the CloudWatch log group we created)

• log_group_name – Identifies the log group to write flow logs to

• log_format – Specifies the custom log format that appears on log events. Here is a full list of data fields you can customize the format with

Importing the FlowLog construct in a stack

A stack is a unit for deployment that is provisioned by AWS CloudFormation and can be added to an app for the stack to be deployed to AWS. You can imagine an app consisting of multiple stacks which consist of multiple resource constructs. We can import the custom flow log construct we just made into a stack to prepare it for deployment.

from aws_cdk import Stack, aws_ec2 as ec2
from flowlog import FlowLog


class MyStack(Stack)
  def __init__(self, scope: Construct, id: str)
    super().__init(scope, id)

    self.vpc = ec2.Vpc(self, "MyVPC")

    self.flow_log = FlowLog(self, "MyFlowLog", vpc=self.vpc)

• We can import the FlowLog construct from a local Python file (flowlog.py)

• A new VPC construct is instantiated and passed as an argument to the flow logs for specifying the VPC for generating log data

• The flow log construct is instantiated

Working with VPC flow log data fields

Identifying relevant data fields

There are plenty of data fields we can use for customizing our log format, as listed here. Recognizing the right fields depends on your use case, as some fields may be more useful than others.

Here is a collection of data fields you may find useful for network traffic monitoring and security with sample use cases:

An example of a log event using the data fields above as a custom log format is:

107530157253 eni-0c103a04bdb4e905c us-east-1 subnet-0cbf1673fe2 11.4.2.2 5.21.62.92 4213 80 egress 8 ACCEPT OK

Querying log records with CloudWatch Log Insights

CloudWatch Log Insights can be used to query CloudWatch log events with SQL-like syntax. VPC flow logs can aggregate CloudWatch log events very quickly, so querying can be very useful for specifying a log group's log events that we are interested in viewing based on their data points.

For example, let's say we want to see recent outgoing traffic from a specific user's resources. Let's look for the 20 most recent log events where the user's account ID is 107530157253 and the traffic is outgoing or egress. We can run the following query:

fields @timestamp, @message, accountId as ID, flowDirection
| sort @timestamp desc
| filter (
    ID = '107530157253' and
    flowDirection = "egress"
    )
| limit 20

• fields specifies the values that are imported from a log event, where @message is the log data

• accountId is a given value from the log event referenced in the query as ID

• flowDirection specifies whether traffic is incoming (ingress) or outgoing (egress)

• filter gets log events that match one or more conditions

Conclusion

Enabling VPC flow logs that publish to CloudWatch logs has a multitude of benefits with the various data fields provided. Being able to directly monitor resources in a VPC and query data through flow logs can be a valuable addition to your networking toolset.

References

• A full list of log event data fields can be found in the AWS Documentation here

• Further detail on CloudWatch log insights query syntax can be found here, along with sample queries here