Why Your Splunk Bill Tripled After Moving to Cloud (And How to Cut It Without Losing Coverage)

On-prem Splunk hid your true volume problem because disk was cheap and nobody tracked it carefully. When you move to Cloud, Splunk starts billing per GB of ingest and suddenly every log source has a dollar figure attached to it.

The three most common surprises when migrating to Splunk Cloud:

Firewall and network device logs. These are almost always the largest single contributor. A mid-size organization with 200 network devices easily generates 50-150 GB/day from firewall traffic logs alone. At the default verbosity settings, you are logging every permitted connection, every dropped packet, every NAT translation. That is not a security requirement. It is a habit.

Windows Security Event logs. The default Windows Security audit policy generates thousands of events per host per hour. A 2,000-seat Windows environment can easily push 80-120 GB/day. Most of that volume is logon noise (Event ID 4624), object access events for shares nobody cares about, and authentication chatter between machine accounts.

Endpoint telemetry duplication. Organizations that installed both an EDR agent and a Sysmon instance are often sending overlapping process creation, network connection, and file events to Splunk twice. The EDR vendor also runs their own cloud SIEM, so sometimes that data is going three places.

Before you filter or drop anything, you need to know where your bytes are actually coming from.

Run this search in Splunk to get a 7-day ingest breakdown by source type:

index=* earliest=-7d latest=now | eval gb_per_event=len(_raw)/1073741824 | stats sum(gb_per_event) as total_gb by sourcetype | sort -total_gb | eval daily_avg_gb=round(total_gb/7, 2) | eval monthly_est_gb=round(daily_avg_gb30, 0) | eval annual_cost_usd=round(monthly_est_gb30*150, 0) | table sourcetype, daily_avg_gb, monthly_est_gb, annual_cost_usd

Adjust the $150 figure to your actual contract rate. Run this for each index if you use multiple.

You will almost certainly find that 5-10 source types account for 75-85% of your total volume. Everything else is noise in the cost sense. Focus your optimization effort entirely on those top sources.

Firewall logs are the single highest-leverage target in most environments. The key insight is that permitted traffic logs are almost never your detection source. Your detections come from denied traffic, suspicious allowed traffic to known-bad IPs, and traffic pattern anomalies.

What you can almost always drop:

• Permitted inbound traffic from RFC 1918 (internal) addresses to internal destinations. You have this in your network flow data already.
• Permit logs for common benign services (DNS to your resolvers, NTP to your time servers, ICMP between monitoring systems).
• High-volume low-value log entries like Microsoft NPS RADIUS accounting logs that are just kept alive chatter.

What you must keep:

• All denied traffic (inbound and outbound).
• All traffic to/from external IPs on non-standard ports.
• Any traffic involving your internet-facing servers.
• Outbound connections on unusual ports (anything that is not 80, 443, 25, 587, 53, 123).

Implement this at the forwarder level using Splunk's props.conf and transforms.conf to drop events before they are transmitted to the indexer. Dropping at the forwarder means you never pay ingest cost for those events.

Example transforms.conf entry to drop routine permitted internal traffic:

[drop_internal_permit] REGEX = action="allow".*src_ip="(10.|172.1[6-9].|172.2[0-9].|172.3[01].|192.168.)".*dst_ip="(10.|172.1[6-9].|172.2[0-9].|172.3[01].|192.168.)" DEST_KEY = queue FORMAT = nullQueue

This pattern alone routinely cuts firewall ingest by 40-60% in environments with heavy internal East-West traffic.

The Windows Security event log is a minefield of high-volume, low-value events when left at default audit settings.

The events you need for real detection coverage:

The most impactful change is filtering 4624 Type 3 (network logon) events. In an Active Directory environment, domain-joined machines generate thousands of Type 3 logon events per hour for normal operations like file share access, printer connections, and scheduled task authentication. For most detection use cases these are noise.

Implement filtering at the Windows Event Forwarding (WEF) level or in Splunk Heavy Forwarder's inputs.conf whitelist. WEF is preferable because it stops events at the source before any transmission.

Expected volume reduction: 30-50% of Windows Security event log ingest, depending on your environment.

If you are running an EDR alongside Sysmon and sending both to Splunk, audit for overlap. Most EDRs (CrowdStrike, SentinelOne, Microsoft Defender for Endpoint) collect:

• Process creation with command line arguments
• Network connection events
• File write events
• Registry modifications
• DNS queries

Sysmon collects exactly the same events. If you need Sysmon for historical data before your EDR contract, that is reasonable. If you have had your EDR for over a year and are still running Sysmon in parallel, you are paying for the same telemetry twice.

Decision matrix:

• EDR cloud console handles your threat hunting and incident response? Disable Sysmon on endpoints.
• EDR does not export raw telemetry to Splunk, or exports with a lag? Keep Sysmon for real-time detection.
• You have specific Sysmon event types the EDR does not expose (e.g., DNS query logging)? Keep only those specific Sysmon event IDs.

Splunk Cloud has three storage tiers that differ in search speed and cost:

Hot/Warm: Fastest search performance. Most expensive for long retention. Cold/Frozen (SmartStore): Slower search, but significantly cheaper per GB for long-term retention. Summary Indexes: Pre-aggregated data at low volume for long-term trending without storing raw events.

For compliance-driven retention requirements (PCI DSS requires 12 months, many internal policies require 2-3 years), the difference between keeping raw logs in hot/warm versus moving them to SmartStore after 90 days can cut long-term storage costs by 60-70%.

For log sources where you need trending data but not raw event search (e.g., authentication volume by user, failed login counts by source IP), use a saved search that runs every 15 minutes and writes aggregated summary data to a summary index. Then delete the raw events after 30 days instead of 12 months.

Example summary index pattern for authentication volume:

index=windows EventCode=4625 | stats count as failed_logons by user, src_ip | collect index=summary_auth sourcetype=auth_summary

This compresses 30 days of authentication failure data into a fraction of the raw volume while preserving the ability to trend by user and source.

Splunk Cloud's native Ingest Actions feature lets you route, filter, and transform events at ingestion time without requiring a Heavy Forwarder. This is the preferred approach in fully Splunk Cloud environments.

Key capabilities:

• Drop: Discard matching events entirely (never counted toward your ingestion quota).
• Route: Send high-volume low-value data to a cheaper cold index instead of your main security index.
• Mask: Redact sensitive fields (PII, credentials) before indexing.

Build an Ingest Action pipeline for each high-volume source type. Start with the top 3 by volume from your Step 1 analysis. Test each filter against a representative sample before enabling in production.

The fastest path to meaningful cost reduction:

Week 1: Run the volume analysis. Identify your top 5 source types by GB/day. Do not change anything yet.

Week 2: For each of those 5 source types, document what detections currently use it. Check your detection inventory (if you have one) or search your saved alerts for references to those source types. This tells you what you cannot safely drop.

Week 3: Implement drops at the forwarder for events with no detection dependency. Start conservative. Drop only events that are definitively not referenced in any alert or saved search.

Week 4: Measure. Run the volume analysis again. Calculate the reduction percentage. If coverage held (re-run your detections on the same test data and confirm they still fire), expand the filtering scope.

Most teams see 25-35% reduction in the first pass. The second pass, done 30-60 days later with more confidence in what is safe to filter, gets the rest.

Dropping entire source types without a detection audit. You will eventually discover that a Tier 1 alert you relied on for 2 years silently stopped working because the source type it queried no longer exists.

Filtering at the indexer instead of the forwarder. Events filtered at the indexer still count toward your ingest quota. You only save storage, not ingestion cost.

Applying global drops to compliance-required sources. PCI DSS, HIPAA, and SOC 2 auditors will ask for specific log types. Make sure your filter list has been reviewed against your compliance obligations before enabling.

Not documenting what you dropped and why. Six months from now, an incident will happen and someone will ask why a specific log source is missing. Document every drop decision with the date, the detection audit that was performed, and the business justification.

Organizations that apply the firewall, Windows event, and endpoint deduplication strategies typically see:

• Firewall logs: 40-60% reduction
• Windows Security events: 30-50% reduction
• Endpoint telemetry (with EDR in place): 50-100% reduction if Sysmon is disabled
• Overall bill: 35-55% reduction for a standard enterprise environment