AWS CloudTrail Log Analysis for Threat Hunting: Athena Queries, API Call Patterns, and Anomaly Detection
Retool's new app builder is where AI-generated code ships safely
Building apps with AI is easy. Getting them to production safely is another story.
Investigated a suspected credential compromise in an AWS account and found three weeks of completely clean CloudTrail — no suspicious API calls, no unusual activity, no indication of anything wrong. The investigation should have been straightforward from the CloudTrail evidence. It was not, because the organization's CloudTrail configuration only captured management events, not data events. The attacker had accessed the account using a legitimate IAM user's access key, enumerated S3 bucket contents using ListObjects and GetObject calls, and exfiltrated approximately 40GB of data from three S3 buckets. None of those S3 data plane operations appeared in the management event CloudTrail trail.
Enabling S3 data events retroactively does not recover historical events — they were never captured. The 90-day investigation concluded without definitive evidence of what was accessed, because the evidence was never recorded. The cost of enabling S3 data events for the three sensitive buckets would have been approximately $12 per month in CloudTrail logging costs. The cost of the investigation without that logging was 300 hours of security team time.
Query patterns: hunting specific attack techniques from CloudTrail data
Effective CloudTrail threat hunting requires a library of queries targeted at specific attack techniques rather than generic anomaly searches. Attackers using AWS accounts follow predictable patterns driven by the tools they use (Pacu, ScoutSuite, Prowler-style enumeration) and the objectives they pursue (credential validation, permission enumeration, data access, persistence). Queries that match these specific patterns produce higher signal-to-noise results than broad statistical anomaly detection.
Hunt for credential theft indicators using GetCallerIdentity and STS enumeration patterns
GetCallerIdentity is the canonical credential verification call that every attacker and every automated tool makes immediately after obtaining credentials to verify they are valid and identify the account ID, user/role ARN, and user ID. An attacker using stolen credentials will almost always make a GetCallerIdentity call within minutes of obtaining the credential. Build a threat hunting query that identifies GetCallerIdentity calls from source IPs not associated with any previous API calls in the account: compare the sourceIPAddress of recent GetCallerIdentity calls against the historical distribution of IPs for the same userIdentity.arn over the past 30 days. A GetCallerIdentity call from a brand-new IP for an IAM user who has been making API calls from the same corporate IP range for six months is a high-fidelity indicator of credential theft. Similarly, hunt for STS GetSessionToken and AssumeRoleWithSAML calls from external IPs — these API calls generate temporary credentials that provide time-limited access without further authentication, making them valuable to attackers who want to convert long-lived credentials into a set of temporary credentials with shorter validity that are less likely to be revoked during an ongoing investigation.
Detect infrastructure backdoor creation by hunting for new IAM users, access keys, and EC2 security group modifications outside change windows
Persistence techniques in AWS frequently involve creating new IAM users or access keys that survive revocation of the original compromised credentials, and modifying EC2 security groups to allow ongoing inbound access. Build a daily threat hunt query that returns all CreateUser, CreateAccessKey, CreateLoginProfile, and AttachUserPolicy events from the past 24 hours, filtered for events that occurred outside the organization's defined change windows (typically business hours in the primary time zone) or were made by identities that do not normally manage IAM. For security group modifications, hunt for AuthorizeSecurityGroupIngress calls that add rules allowing SSH (port 22) or RDP (port 3389) from 0.0.0.0/0 or ::/0 — these rules provide unrestricted remote access to EC2 instances and are a common persistence mechanism after initial access. CloudTrail captures the specific CIDR and port being authorized in the requestParameters field, making it possible to build a precise query that returns only security group rules with external-facing access rather than all security group changes.
Detection engineering: converting hunt queries into production alerts
Threat hunting queries that identify real incidents need to be converted into production detection rules so that the same pattern triggers automated alerting rather than requiring manual re-execution on a schedule. The conversion process involves validating that the query has an acceptable false positive rate, parameterizing the threshold values, and integrating with the incident response workflow so that a triggered alert becomes an actionable notification with sufficient context for the responding analyst.
Build CloudTrail Insights rules for write API call volume anomalies that indicate automated tooling
Enable CloudTrail Insights and configure EventBridge rules to route Insights events to the security team for automatic detection of write API call volume anomalies without building custom statistical queries. CloudTrail Insights analyzes the rolling 7-day baseline of write API call rates per API and per IAM identity and generates an Insights event when the current call rate deviates significantly from the baseline — the statistical model accounts for time-of-day and day-of-week patterns that would cause manual threshold-based approaches to generate excessive false positives. Create an EventBridge rule matching source: aws.cloudtrail and detail-type: AWS Insight via CloudTrail with a Lambda target that formats the Insights event into a Slack alert including the specific API, the baseline call rate, the anomalous call rate, and the contributing user identity. CloudTrail Insights events for unusual write API rates are particularly valuable for detecting mass resource creation (an attacker spinning up EC2 instances for cryptomining), mass resource deletion (a destructive attack targeting the account), and credential harvesting operations that involve many CreateAccessKey calls.
Create a CloudTrail detective control baseline by documenting normal API patterns for each IAM role
Document the normal API call patterns for each IAM role in the environment as a detective control baseline that enables statistical comparison in threat hunting queries. For each role, record the typical services accessed (EventSource values), the typical API calls made (EventName values), the typical source IP ranges, and the typical time patterns (weekday business hours vs. after hours). Export 30 days of CloudTrail history per role using an Athena query grouped by userIdentity.arn, eventsource, and eventname with counts, producing a profile of normal behavior. Store the baseline in a DynamoDB table or S3 file that Athena can join against in hunting queries: a query that returns API calls made by a role today that do not appear in the role's 30-day baseline immediately highlights novel API usage that warrants investigation. Update the baseline monthly to incorporate legitimate changes in role usage patterns as applications evolve, preventing baseline drift that causes the comparison to generate false positives on legitimately new API usage.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
The bottom line
AWS CloudTrail threat hunting requires complete trail configuration (management and data events, multi-region, CloudWatch Logs delivery, Insights enabled) before the first security incident — retroactive data event capture is impossible. Set up Athena with partition projection over the CloudTrail S3 bucket for efficient historical analysis, or use CloudTrail Lake for managed query capability without table management overhead. Build a library of Athena queries targeting specific attack techniques: GetCallerIdentity from new IPs for credential theft detection, AssumeRole chain analysis for privilege escalation, CreateUser and CreateAccessKey outside change windows for persistence detection, and S3 GetObject from external IPs for data exfiltration detection. Convert validated hunting queries into CloudWatch Logs metric filters with alarms for real-time alerting on the highest-priority event types. Enable CloudTrail Insights for automated write API anomaly detection. Integrate CloudTrail with SIEM via Kinesis Data Firehose for sub-minute event delivery and cross-source correlation with GuardDuty and VPC flow logs.
Frequently asked questions
How do I configure CloudTrail to capture all events needed for threat hunting?
Configure CloudTrail for comprehensive threat hunting coverage by enabling management events, data events for critical services, and CloudTrail Insights on the primary trail. Start with a multi-region trail (not single-region) so that API calls in any AWS region are captured — attackers often use regions outside the organization's primary region to avoid detection by security tools that only monitor primary regions. Enable management events for both read and write operations — read events (DescribeInstances, ListBuckets, GetCallerIdentity) are valuable for detecting reconnaissance activity from compromised credentials. Enable data events for S3 on buckets containing sensitive data (select specific buckets by ARN rather than all buckets to manage log volume and cost), Lambda for function invocations that handle sensitive data, and DynamoDB for tables containing PII or financial records. Enable CloudTrail Insights for Write Management Events to detect unusual API call volume patterns. Configure CloudTrail to deliver logs to both S3 (for long-term storage and Athena analysis) and CloudWatch Logs (for real-time metric filter alerting). Enable CloudTrail log file validation to detect log tampering — an attacker who gains S3 write access to the log bucket can attempt to delete or modify CloudTrail records. Enable S3 MFA delete and Object Lock on the CloudTrail log bucket for tamper resistance.
How do I set up Athena to query CloudTrail logs for threat hunting?
Set up Athena to query CloudTrail logs by creating an Athena table over the CloudTrail S3 bucket with partition projection for efficient time-based queries without loading all log files. In the AWS Athena console, create a new database for security analysis and run the CloudTrail table creation DDL that AWS provides in the CloudTrail documentation: CREATE EXTERNAL TABLE cloudtrail_logs with the CloudTrail JSON log format schema and LOCATION pointing to the S3 path where CloudTrail delivers logs. Add partition projection to avoid full table scans: define the account, region, and date fields as partition keys with projection.enabled = true and the date range in the table properties. With partition projection, an Athena query filtered to a specific date range (WHERE DATE(eventtime) BETWEEN '2026-07-01' AND '2026-07-14') scans only the S3 objects for that date range rather than all CloudTrail logs in the bucket, reducing query cost and time dramatically for historical analysis. Alternatively, use AWS CloudTrail Lake which provides a managed SQL query interface over CloudTrail events without requiring Athena table creation or S3 management — CloudTrail Lake stores events in its own managed data store with query pricing per GB scanned.
What Athena SQL queries are most useful for CloudTrail threat hunting?
The most actionable CloudTrail Athena threat hunting queries cover four attack categories: credential misuse, privilege escalation, persistence, and data exfiltration. Credential misuse — console logins from new geographic locations: SELECT useridentity.username, sourceipaddress, eventsource, eventname, eventtime FROM cloudtrail_logs WHERE eventname='ConsoleLogin' AND DATE(eventtime) > DATE_ADD('day', -7, CURRENT_DATE) AND json_extract_scalar(additionaleventdata, '$.MFAUsed') = 'No' ORDER BY eventtime DESC. Privilege escalation — IAM role assumption chains: SELECT useridentity.sessioncontext.sessionissuer.arn, requestparameters, eventtime FROM cloudtrail_logs WHERE eventname='AssumeRole' AND useridentity.type='AssumedRole' AND DATE(eventtime) > DATE_ADD('day', -1, CURRENT_DATE). Persistence — new IAM user or access key creation: SELECT useridentity.arn, eventname, requestparameters, eventtime FROM cloudtrail_logs WHERE eventname IN ('CreateUser','CreateAccessKey','CreateLoginProfile','AttachUserPolicy') AND DATE(eventtime) > DATE_ADD('day', -7, CURRENT_DATE). Data exfiltration — S3 GetObject from external IPs: SELECT requestparameters, sourceipaddress, useridentity.arn, eventtime FROM cloudtrail_logs WHERE eventname='GetObject' AND sourceipaddress NOT LIKE '10.%' AND sourceipaddress NOT LIKE '172.%' AND DATE(eventtime) > DATE_ADD('day', -1, CURRENT_DATE).
How do I detect compromised AWS credentials using CloudTrail?
Detect compromised AWS credentials in CloudTrail by looking for specific behavioral patterns that indicate credential theft: API calls from geographic locations inconsistent with the user's history, error spikes from invalid API calls during reconnaissance, API calls from Tor exit nodes or known malicious IPs, and enumeration API call patterns (rapid succession of Describe, List, and Get calls across multiple services). The GetCallerIdentity API call is a reliable indicator of credential testing — an attacker who obtains credentials typically calls GetCallerIdentity first to verify the credential is valid and identify the account. Query CloudTrail for GetCallerIdentity calls from external IPs: SELECT sourceipaddress, useridentity.arn, useridentity.accountid, eventtime FROM cloudtrail_logs WHERE eventname='GetCallerIdentity' AND sourceipaddress NOT IN (SELECT known_internal_ips FROM your_ip_reference_table) AND DATE(eventtime) > DATE_ADD('hour', -24, CURRENT_DATE). For rapid enumeration detection, look for a single identity making more than 50 unique API calls across more than 5 different AWS services within a 60-minute window — this pattern is consistent with automated cloud enumeration tools (Pacu, ScoutSuite, CloudMapper) and inconsistent with normal human or application API usage patterns.
How do I build CloudWatch Logs metric filters for real-time CloudTrail alerting?
Build CloudWatch Logs metric filters that extract specific CloudTrail event patterns and trigger CloudWatch Alarms for real-time alerting on high-priority security events. First, ensure CloudTrail is configured to deliver logs to CloudWatch Logs by adding a CloudWatch Logs log group in the trail configuration. Create metric filters on the log group that match specific event patterns: for root account usage, the filter pattern is { $.userIdentity.type = 'Root' && $.userIdentity.invokedBy NOT EXISTS && $.eventType != 'AwsServiceEvent' }. Create a CloudWatch metric filter with this pattern, define a custom metric (RootAccountUsage, value 1), then create a CloudWatch Alarm that triggers when the metric sum exceeds 0 in a 5-minute period. Connect the alarm to an SNS topic that delivers to the security team Slack channel and PagerDuty. Implement the same pattern for other high-priority events: IAM policy changes ({ $.eventName = CreatePolicy || $.eventName = DeletePolicy || $.eventName = AttachUserPolicy }), CloudTrail configuration changes ({ $.eventName = StopLogging || $.eventName = DeleteTrail || $.eventName = UpdateTrail }), and console logins without MFA ({ $.eventName = ConsoleLogin && $.additionalEventData.MFAUsed != 'Yes' }). These four metric filters cover the most critical CloudTrail event categories that warrant immediate investigation.
How do I detect lateral movement and privilege escalation in CloudTrail logs?
Detect lateral movement and privilege escalation in CloudTrail by hunting for AssumeRole chains where a compromised role assumes a second role with higher privileges, and for IAM policy modifications that expand the compromised identity's access. AssumeRole chains appear in CloudTrail as sequential AssumeRole events where the userIdentity.arn of each event is the role ARN from the previous event's responseElements. Build a query that detects role chains with more than two hops: SELECT eventtime, useridentity.sessioncontext.sessionissuer.arn as source_role, json_extract_scalar(requestparameters, '$.roleArn') as target_role FROM cloudtrail_logs WHERE eventname='AssumeRole' AND DATE(eventtime) > DATE_ADD('hour', -24, CURRENT_DATE) ORDER BY eventtime. A role assuming another role that then assumes a third role is a pattern characteristic of privilege escalation through role chaining that would not appear in normal application or developer activity. For IAM privilege escalation without role chaining: look for PutUserPolicy, AttachUserPolicy, AttachRolePolicy, or CreatePolicyVersion events made by identities that do not normally manage IAM resources. A Lambda execution role making IAM API calls is a strong indicator of SSRF-to-IAM privilege escalation — the application running in Lambda obtained its own execution role credentials and used them to expand permissions.
How do I integrate CloudTrail with a SIEM for automated detection and correlation?
Integrate CloudTrail with a SIEM by streaming CloudTrail events through Kinesis Data Firehose to the SIEM's ingest endpoint, providing real-time event delivery with sub-minute latency compared to the S3-delivery approach that batches events every 5-15 minutes. Configure CloudTrail to deliver to a CloudWatch Logs log group, create a Kinesis Data Firehose delivery stream with the SIEM's HTTP endpoint as the destination, and configure a CloudWatch Logs subscription filter to forward all CloudTrail events from the log group to the Kinesis stream. In the SIEM (Splunk, Elastic SIEM, Microsoft Sentinel), create source type or data stream mappings that parse CloudTrail JSON and extract the key fields: eventSource, eventName, userIdentity.arn, userIdentity.type, sourceIPAddress, awsRegion, errorCode. Build SIEM detection rules for CloudTrail events that benefit from correlation with other data sources: a GuardDuty finding for a specific EC2 instance correlated with CloudTrail API calls from that instance's IAM role within the same time window provides a much higher-confidence incident than either source alone. Use the cloudtrail.sourceIPAddress and userIdentity.arn fields as correlation keys to join CloudTrail events with VPC flow logs, GuardDuty findings, and AWS Config configuration change events in the SIEM.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.
Win a $2,495 Black Hat pass.
Full-access to Black Hat USA 2026 in Las Vegas. Subscribe free to enter.