Security Data Lake: A Cheaper Alternative to Full SIEM for Log Storage and Investigation

Traditional SIEM pricing models were designed when enterprise log volumes were measured in gigabytes per day. Modern infrastructure generates logs in the terabytes-per-day range for organizations of modest size: cloud provider infrastructure logs, container orchestration platform logs, endpoint detection telemetry, network flow data, identity provider authentication events, and application-level audit logs all generate continuous high-volume data streams. The same SIEM pricing model that was manageable at 50GB/day becomes the largest security tool budget line item at 1TB/day.

The per-GB pricing structure means that SIEM cost scales linearly with log volume, while detection value does not scale linearly with log volume. The authentication events from your identity provider are high-value logs that should go into the SIEM. The verbose debug logs from a development application server are low-value for detection purposes. Most organizations end up in a position where they need to actively manage SIEM ingestion to control costs, making filtering decisions that are difficult to reverse if a later investigation reveals the filtered data was relevant.

Retention pressure compounds the ingestion cost problem. Compliance frameworks require log retention of 12-24 months in many regulated sectors; SIEM costs for that retention volume are prohibitive. The common compromise is to retain only the most recent 30-90 days in the SIEM and export older logs to cold storage (S3 Glacier or equivalent) with no tooling for querying them effectively. The cold storage logs satisfy compliance checkbox requirements but are practically useless for investigation because extracting and searching them requires manual effort that makes investigators avoid the data.

SIEM vendors have responded to the cost pressure with tiered storage models (hot, warm, cold within the SIEM platform) and consumption-based pricing alternatives, but the fundamental structural issue remains: SIEM platforms are built around an ingestion and indexing model where every byte of ingested data carries processing and indexing cost. Moving to an architecture that separates cheap storage from query capability addresses this structural issue rather than optimizing within it.

A security data lake uses cloud object storage as the durable, cheap foundation for all log data, with separate query engines applied on-demand rather than maintaining a continuously running indexed database. The core AWS architecture is S3 as storage, Kinesis Data Firehose or a log pipeline agent for ingestion, and Amazon Athena for interactive SQL queries against the stored data. AWS Security Lake, launched in 2022, is a managed implementation of this architecture that handles normalization to the OCSF schema and integrations with common log sources.

The storage layer is straightforward: S3 buckets organized by log source type and date partitioning. Partition structure matters for query cost and performance: a structure of s3://security-logs/source=cloudtrail/year=2026/month=05/day=22/ allows Athena queries to scan only the partitions relevant to a time window rather than the entire dataset. Without date partitioning, every query scans the entire log history, which is both slow and expensive at scale.

File format selection significantly affects query performance and storage cost. Raw JSON or text log files stored in S3 work but are inefficient: Athena must parse each file on every query, and text format does not compress as efficiently as columnar formats. Parquet or ORC columnar formats store data in columns rather than rows, which means a query looking for specific fields (source IP, username, event type) scans only the relevant column data rather than every field in every record. Converting logs to Parquet at ingest time reduces Athena query cost by 60-80% and query time by a comparable margin. The tradeoff is a conversion step in the pipeline that adds latency and computational cost at ingest time.

AWS Security Lake uses the Open Cybersecurity Schema Framework (OCSF) as its normalized schema, which is a significant operational advantage if your log sources support it. OCSF provides a consistent field naming convention across different log sources: instead of mapping AWS CloudTrail's userIdentity.userName and Azure AD's properties.userPrincipalName to the same semantic concept in custom ETL code, OCSF provides a standard actor.user.name field that both source-specific parsers write to. This normalization enables cross-source queries without join complexity.

The query layer determines how useful the data lake is for actual security investigation work. An S3 bucket full of logs with no effective query tool is a compliance archive, not an investigation capability. Three architectures for the query layer serve different use cases and organizational contexts.

Amazon Athena is the lowest-cost, lowest-operational-overhead option for SQL-based investigations. Athena is serverless: there are no clusters to manage, no indexes to maintain, and no cost when queries are not running. You pay $5 per terabyte of data scanned. On Parquet-formatted data with date partitions, a typical investigation query scanning one day of logs for a 1TB/day environment costs roughly $0.50-2.00 per query. Athena is excellent for investigation tasks where an analyst can write SQL: finding all login events for a specific user in a 90-day window, querying all outbound connections from a specific IP range, correlating process execution events with network connections on the same host.

Amazon OpenSearch Service (formerly Elasticsearch) provides full-text search and Kibana visualization capabilities. OpenSearch can index S3 data on demand using the Ultra Warm tier, which stores data in S3 and indexes it when queried rather than maintaining persistent indexes for all data. This provides a search experience closer to a traditional SIEM (keyword search, timeline visualization, dashboard support) at a cost between pure Athena and a full SIEM. OpenSearch is the better choice for organizations where analysts are more comfortable with search interfaces than SQL, or where the use case includes frequent free-text searching across log content rather than structured field queries.

Databricks and Snowflake are the enterprise data platform options, appropriate for organizations that have already standardized on these platforms for business analytics and want to bring security data into the same environment. Both support structured and semi-structured data, provide strong SQL query capabilities, and offer fine-grained access controls suitable for security data segregation. The licensing cost is higher than Athena for pure security use cases, but the operational advantage of using a platform the data engineering team already manages and the ability to join security data with business data (for context enrichment) can justify the cost in large enterprises.

The practical hybrid for most organizations is Athena as the primary investigation query layer (low cost, SQL accessible, no operational overhead) with a SIEM covering the most recent 30-90 days of hot data for real-time detection. Analysts investigating an incident use the SIEM for recent events and Athena for historical context, with both surfaces available in the investigation workflow.

The ingestion pipeline is the operational component that determines whether the data lake is actually populated with current, complete, and queryable data. A reliable ingestion pipeline requires source connectors for each log type, normalization logic to produce consistent schemas, format conversion to Parquet for efficient querying, and delivery to S3 with appropriate partitioning.

Kinesis Data Firehose is the managed AWS option for streaming log delivery to S3. Firehose accepts data via the AWS API or Kinesis Data Streams, buffers it, optionally transforms it via Lambda, converts it to Parquet using a Glue schema reference, and delivers it to S3 with configurable partitioning. For AWS-native log sources (CloudTrail, VPC Flow Logs, GuardDuty, S3 access logs), the integration is direct: enable the log source delivery to Kinesis and configure Firehose as the destination. For external log sources, Firehose is less convenient because it requires building an API ingestion layer.

Vector is an open-source data pipeline tool (from Datadog, but fully open source) that handles log collection from diverse sources with a consistent configuration model. Vector runs as an agent on log-generating systems or as a centralized aggregator, accepts input from syslog, file tailing, Kafka, HTTP, and dozens of other sources, transforms logs using VRL (Vector Remap Language) for field normalization, and delivers to S3 (and other destinations). Vector's strength is flexibility: the same configuration syntax handles transforming AWS CloudTrail events, Linux audit logs, and application JSON logs into a consistent Parquet output. For multi-cloud or hybrid environments, Vector is often more practical than the cloud-provider-native ingestion options.

Fluentd and Fluent Bit are the CNCF-graduated alternatives, widely used in Kubernetes environments where the Fluent Bit DaemonSet pattern for container log collection is a standard pattern. Fluent Bit is the lightweight agent version; Fluentd is the heavier aggregator with richer plugin ecosystem. For Kubernetes-centric environments, Fluent Bit to Fluentd to S3 is a well-understood and well-documented pattern.

Timestamp normalization deserves specific attention in any multi-source log pipeline. Different log sources use different timestamp formats, different timezones, and different levels of precision. A security investigation correlating events across sources requires consistent UTC timestamps with millisecond precision. Building timestamp normalization into the ingestion pipeline (at the Firehose transformation Lambda, the Vector VRL transform, or the Fluentd parser) is significantly easier than handling inconsistencies at query time. The OCSF metadata.event_time field is the standard location for the normalized timestamp in OCSF-compliant pipelines.

Athena's value for security investigations is proportional to the analyst's ability to construct targeted SQL queries that surface relevant events efficiently. Several query patterns cover the majority of incident investigation use cases and are worth having as a team reference.

Authentication anomaly investigation queries filter identity provider or Windows Security Event Logs for authentication events, join against a user baseline (average login count, typical source IP ranges), and surface accounts with significant deviations. A basic Athena query for AWS CloudTrail authentication events might group by userIdentity.userName and sourceIPAddress, filter for ConsoleLogin and AssumeRole events, and compare the source IP to a known-good IP range list stored as a reference table in Athena.

Process execution searches against endpoint telemetry (Sysmon events, EDR process logs) can identify specific executable names, command-line patterns, or parent-child process relationships across an entire fleet for a defined time window. The query follows the pattern: select from the process execution table where timestamp is between investigation start and end, filter by process name or command line pattern, join against the asset table to get hostname and business unit context, order by timestamp. Without date partitioning, this query on 90 days of endpoint telemetry at 1TB/day scale would be prohibitively expensive; with partitioning, it scans only the relevant date range.

Network connection analysis queries against VPC Flow Logs, firewall logs, or proxy logs can identify unusual outbound connections, internal lateral movement patterns, or data exfiltration indicators. A common investigation pattern is: identify all external destinations contacted by a specific internal host, aggregate by destination IP and port, and join against a threat intelligence table of known-bad IPs to surface confirmed malicious connections. The threat intelligence join requires maintaining a reference table in Athena or S3 that is periodically refreshed from your TI feed.

User behavior timeline reconstruction is a common investigation task: given a compromised user account, build a chronological timeline of everything that account did across all log sources in the 30 days before and after the compromise indicator. In a SIEM, this is a built-in feature. In Athena, it requires a UNION query across tables for each log source (authentication events, file access events, email gateway events, web proxy events), filtered by username and time range, ordered by timestamp. The result is the same comprehensive timeline; the mechanism is SQL rather than a search interface.

A security data lake is not a SIEM replacement for organizations that require real-time detection. Understanding what the data lake architecture does not provide is as important as understanding what it does provide, because the hybrid architecture decision depends on an accurate assessment of the gaps.

Real-time correlation rules do not exist in a data lake without additional tooling. A SIEM's correlation engine evaluates incoming events against rules like "five failed logins followed by a successful login from the same IP within 10 minutes" in real-time as events arrive. In a data lake, this pattern detection requires a streaming analytics layer (Amazon Kinesis Data Analytics, Apache Flink, or Kafka Streams) running continuously against the data stream, which adds cost and operational complexity. If real-time correlation is required, the SIEM provides this capability with far less engineering effort than building it on top of a data lake.

Alerts and SOAR integration depend on real-time rule evaluation. A SIEM generates alerts that feed into a SOAR platform for automated response playbooks. A data lake generates query results when queried; it does not proactively generate alerts. Organizations that have invested in SOAR integration for automated response workflows need a SIEM layer for the alert generation that drives those workflows.

Built-in content (detection rules, dashboards, use case content) is a significant operational value in commercial SIEM platforms. Vendors like Splunk, Microsoft Sentinel, and CrowdStrike Falcon LogScale ship with hundreds of curated detection rules for common attack patterns. Building equivalent detection logic in Athena requires writing it from scratch or using community-maintained rule libraries that require translation to SQL.

The hybrid architecture that works in practice: SIEM retains 30-90 days of hot data for real-time detection and alerting, handles SOAR integration, and provides the built-in detection content the SOC team uses daily. The data lake retains 12-24 months of all log data for compliance and investigation, accessed via Athena for deep-dive investigations when the SIEM retention window does not cover the relevant timeframe. Logs are written to both destinations simultaneously at ingest time, or the SIEM exports to S3 daily as the data ages out of hot retention.

The cost math is the primary driver of the data lake adoption decision, and it is worth working through concretely rather than in general terms. For an organization generating 1TB of logs per day, the annual storage is approximately 365TB. S3 Standard storage for 365TB costs roughly $8,395 per year ($0.023/GB). S3 Intelligent-Tiering, which automatically moves less-accessed data to cheaper storage tiers, would reduce this further to approximately $5,000-6,000 per year with typical access patterns.

Athena query costs at 1TB/day scale depend heavily on query frequency and efficiency. An investigation team running 50 queries per day on Parquet-formatted data with date partitions, where each query scans approximately 1TB, costs roughly $250 per day or $91,000 per year in Athena query fees. This sounds significant but is not a fixed cost: Athena costs only accrue when queries run, and in practice, investigations are not run at full volume daily. More realistic heavy investigation usage costs $5,000-20,000 per year in query fees.

Comparison to SIEM: at $0.50/GB for SIEM ingestion (a reasonable mid-market estimate), 1TB/day costs $500/day in ingestion alone, or $182,500 per year. For 90 days of retention, the ongoing SIEM storage cost adds to this. Total SIEM cost for a 1TB/day environment with 90-day retention is commonly $200,000-400,000 per year depending on the vendor and contract.

The hybrid architecture achieves the best cost outcome: route 30-90% of log volume to S3 directly (the compliance and investigation logs that do not drive real-time detection), route 10-30% of high-value detection logs to the SIEM. SIEM ingestion cost drops by 70-90%, data lake storage covers long-term retention, and investigation capability is preserved. The total cost for this architecture at 1TB/day is commonly $30,000-80,000 per year versus $200,000-400,000 for full SIEM ingestion, with the difference funding additional security tooling or headcount.

The data lake replaces rather than supplements the SIEM only in specific organizational contexts: mature security teams with the SQL skills to write investigation queries, organizations where real-time detection is provided by EDR and cloud-native detection tools (GuardDuty, Defender for Endpoint) rather than a SIEM correlation engine, and organizations where budget constraints make a full SIEM operationally untenable. For these organizations, a data lake with Athena plus cloud-native detection tools can provide acceptable security coverage at a fraction of SIEM cost.