AWS VPC Flow Logs for Security Monitoring: Threat Detection, Anomaly Alerting, and SIEM Integration
Retool's new app builder is where AI-generated code ships safely
Building apps with AI is easy. Getting them to production safely is another story.
VPC Flow Logs are enabled with a few console clicks, but most teams that enable them never build the queries or alert rules needed to extract security value from the data. The logs accumulate in S3 and are consulted only during incident investigation, at which point the investigation takes hours because there is no pre-built query library and no baseline to compare against.
The security value of flow logs comes from the work done before an incident: building the Athena query library, configuring GuardDuty to analyze the data automatically, and establishing the network communication baseline that makes anomalies visible. Enabling flow logs is the starting point, not the end goal.
Configuration: format and storage for security use
The format and field selection decisions made when enabling VPC Flow Logs determine whether the data is useful for security investigations weeks or months later when an incident occurs. Parquet format with all custom fields enabled takes less than a minute to configure but delivers 50-80% storage savings and 3-5x faster Athena query performance compared to default plain text format with standard fields. The pkt-srcaddr and pkt-dstaddr fields are the most commonly missing entries in default configurations, and their absence makes it impossible to trace traffic through NAT gateways back to originating instances. Creating the Athena table with partition projection immediately after enabling flow logs ensures that when an analyst needs to investigate an alert, the query infrastructure is already tested and ready rather than being set up under pressure during an incident.
Enable Parquet format and custom log fields at flow log creation
Parquet format reduces storage cost by 50-80% compared to plain text and makes Athena queries 3-5x faster because Athena can skip columns not referenced in the query. Include all custom fields in the log format, particularly pkt-srcaddr and pkt-dstaddr (the original source and destination IP for traffic traversing NAT gateways), flow-direction, and traffic-path. These fields are not included in the default format but are critical for investigating traffic that crosses NAT gateways or transit gateways, where the default srcaddr and dstaddr fields show the NAT gateway IP rather than the originating instance IP.
Create the Athena table with partition projection before your first investigation
AWS provides an Athena partition projection template for VPC Flow Logs that automatically maps S3 prefix paths to partition columns without requiring manual partition repair (MSCK REPAIR TABLE). Create this table immediately after enabling flow logs, before you need it for an investigation. During an incident is not the time to learn the Athena DDL syntax or troubleshoot S3 path mapping. Build and test the table with a simple query (SELECT COUNT(*) FROM vpc_flow_logs WHERE day = '2026/07/01') to confirm the table reads correctly and store the Athena query for the most common investigation patterns in named queries.
Detection: what to alert on and how to build queries
Effective VPC Flow Logs detection relies on two complementary layers: GuardDuty's automated ML analysis for real-time findings and Athena-based queries for threat hunting and investigation of patterns that fall below GuardDuty's anomaly detection threshold. GuardDuty handles the most common and high-confidence threat patterns including SSH brute force, cryptocurrency mining connections, and known malicious IP traffic, reducing the manual query burden for the highest-severity behaviors. The REJECT record queries built in Athena complement GuardDuty by catching low-and-slow reconnaissance, internal lateral movement probing, and custom detection logic specific to your network architecture that a general-purpose ML model cannot account for. Running these Athena queries on a weekly schedule and reviewing the results as a threat hunting exercise builds the baseline familiarity with normal traffic patterns that makes anomalies recognizable during an incident.
Enable GuardDuty as the first layer of automated flow log analysis
GuardDuty analyzes VPC Flow Logs using ML models that establish per-account baselines and detect deviations without requiring manual query construction. Enable GuardDuty in every AWS account and region where flow logs are enabled. GuardDuty findings from flow log data are the highest-confidence alerts because they are based on pattern matching against known threat techniques and comparison against account-specific baselines rather than static thresholds that generate high false positive rates. GuardDuty findings are the starting point for investigation; Athena queries on the raw flow log data provide the supporting evidence.
Build REJECT-based queries for probing detection as the manual complement
GuardDuty's ML coverage does not catch every threat pattern, particularly low-and-slow reconnaissance that stays below the anomaly detection threshold. Build Athena queries that run weekly on a schedule and report the top source IPs generating REJECT records against your subnets, the top destination ports being scanned, and any internal source IPs generating REJECT records against other internal subnets (which indicates compromised hosts or misconfigured applications). These weekly query results provide a threat hunting baseline that complements GuardDuty's real-time detection.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
The bottom line
AWS VPC Flow Logs are the network visibility layer for cloud threat detection, but the visibility they provide depends entirely on what you build on top of them. Enable flow logs in Parquet format with all custom fields, create the Athena table immediately for ad-hoc investigation capability, enable GuardDuty for automated ML-based detection, build REJECT-record queries for probing detection, and forward to your SIEM for correlation with CloudTrail and other log sources. The combination of GuardDuty automated analysis and Athena-based manual queries covers both real-time detection and deep-dive investigation when an alert requires evidence gathering.
Frequently asked questions
How do I enable AWS VPC Flow Logs and where should I store them?
Enable VPC Flow Logs from the VPC console by selecting your VPC, choosing Actions, then Create flow log. Configure the filter to capture All traffic (accepted and rejected) rather than just accepted or rejected, as REJECT records are critical for threat detection. For destination, choose S3 and specify a bucket with a path prefix per account and region for partitioning. Enable log file format as Parquet (not plain text) for significantly faster Athena queries and lower storage cost. Enable the custom log format and include all available fields including pkt-srcaddr and pkt-dstaddr for traffic that traverses NAT gateways. Set an S3 lifecycle rule to delete logs older than your retention requirement (commonly 90 days for security monitoring, longer for compliance).
How do I use Athena to query VPC Flow Logs for security investigations?
Create an Athena table using the partition projection pattern from the AWS documentation, specifying your S3 bucket path and the column definitions matching your log format. For security investigations, common queries include: finding all traffic to a specific destination IP, aggregating unique destination ports contacted by a source IP to detect port scanning, identifying the highest-volume external connections from each EC2 instance, and listing all REJECT records to a subnet to find probing attempts. Use WHERE start BETWEEN epoch1 AND epoch2 to bound queries to a time range and avoid full table scans. Create named Athena queries for frequently-used detection patterns so analysts can run them quickly during incidents.
What VPC Flow Log alert patterns detect port scanning?
Port scanning detection requires aggregating flow log data over a time window, typically 5 minutes, and identifying source IPs that generated REJECT records against more than a threshold number of distinct destination ports on the same destination IP or subnet. An effective Athena query for this groups by srcaddr and dstaddr, counts distinct dstport values, and filters for REJECT action with count above a threshold such as 20 distinct ports. CloudWatch Logs metric filters can trigger alarms for this pattern in near-real-time when flow logs are sent to CloudWatch Logs rather than S3, at higher cost but lower detection latency.
How do I send VPC Flow Logs to a SIEM for real-time alerting?
Two common architectures forward VPC Flow Logs to a SIEM. S3-based forwarding: configure the SIEM's S3 ingestion connector to poll the flow log bucket using SQS event notifications, which works for any SIEM that supports S3 source ingestion (Splunk, Elastic, Sumo Logic). Lambda-based forwarding: configure flow logs to CloudWatch Logs and use a Lambda function subscribed to the CloudWatch Logs stream to parse and forward records to the SIEM API in real time, providing lower ingestion latency at higher cost. For Elastic, the AWS integration in Elastic Agent handles both S3 and CloudWatch Logs VPC Flow Log sources. For Splunk, use the Splunk Add-on for AWS.
How do VPC Flow Logs help detect lateral movement?
Lateral movement within a VPC appears in flow logs as unexpected traffic between subnets that should not communicate directly. To detect it, define your expected network communication matrix (which security zones should talk to which) and build Athena queries that flag traffic crossing zone boundaries that do not match the expected pattern. Specifically, traffic from a public-facing subnet to a database subnet on a database port from an EC2 instance that is not an application server, or traffic between two isolated tenant subnets, are lateral movement indicators. REJECT records on cross-subnet security groups are particularly valuable because they indicate an attacker found the boundary and is probing it.
How does GuardDuty use VPC Flow Logs for threat detection?
GuardDuty ingests VPC Flow Logs as a data source and applies ML models to identify anomalous traffic patterns that indicate specific threat behaviors. Findings generated from flow log data include: UnauthorizedAccess:EC2/SSHBruteForce for high-rate SSH connection attempts, CryptoCurrency:EC2/BitcoinTool.B for EC2 instances making connections to known cryptocurrency mining pools, Trojan:EC2/BlackholeTraffic for connections to IPs known to be sinkholes, and Recon:EC2/PortProbeEMRUnprotectedPort for port probing against known EMR cluster ports. GuardDuty handles the baseline ML model training period automatically and adjusts baselines per account, reducing false positives compared to static threshold-based alerting.
What is the cost of enabling VPC Flow Logs and how do I reduce it?
VPC Flow Log cost has three components: the flow log data transfer charge, S3 storage, and Athena query scan charges. The data volume depends on your network traffic volume. Cost reduction strategies include enabling flow logs at the subnet level rather than the VPC level if you only need visibility into specific subnets, using Parquet format which reduces storage by 50-80% compared to plain text, configuring S3 Intelligent-Tiering or lifecycle rules to move logs older than 30 days to cheaper storage tiers, using Athena partition projection and partitioning by day to avoid scanning full log history on every query, and disabling flow logs for development VPCs with low security monitoring requirements if cost is a constraint.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.
Win a $2,495 Black Hat pass.
Full-access to Black Hat USA 2026 in Las Vegas. Subscribe free to enter.