Splunk Universal Forwarder Not Sending Data: Fixing Blocked Queues
Reach 50,000+ security professionals
Sponsor a newsletter slot and put your brand in front of CISOs, engineers, and IT decision-makers.
The ticket usually reads the same way: a source that has reported into Splunk reliably for months has gone quiet, the host is up, the Universal Forwarder process is running, and nothing in the obvious places explains it. The instinct is to check the forwarder's configuration files first, but a stopped-sending forwarder with an unchanged configuration is far more often a blocked internal queue than a broken inputs.conf or outputs.conf stanza.
Splunk's data pipeline moves events through a fixed sequence of internal queues, and any one of them filling up creates backpressure that eventually reaches the forwarder itself. Diagnosing the problem means finding which queue is blocked, on which host, and mapping that back to one of a small number of root causes: the indexer can't keep up with disk writes, the network path between the forwarder and its target is broken or timing out, or the forwarder's own throughput ceiling is capping it below what it needs to send. This guide walks through that diagnosis in order, from confirming the block exists, to isolating its cause, to fixing it, to proving the fix actually worked.
Causes: the four things that actually block a Universal Forwarder's queue
A blocked queue is a symptom, not a diagnosis by itself. The fix depends entirely on which of a small number of underlying conditions is producing it, and treating the wrong one wastes time while data keeps piling up. The full four-stage pipeline (parsing queue, aggregator or merging queue, typing queue, indexing queue) runs on the parsing tier, meaning an indexer or a heavy forwarder that does full event processing. A plain Universal Forwarder does not parse, type, or aggregate locally; it holds a lighter parsing queue and an output queue (tcpout) before handing bytes to the network, so when people say a UF's queue is blocked, they usually mean either that light local queue or, more often, that the indexer on the other end of the connection has stopped draining fast enough and the backpressure has traveled back to the forwarder.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
Diagnostics: finding which queue is blocked and confirming why
A plain Universal Forwarder has no search head or web interface of its own, so diagnosis usually happens from wherever its internal logs land, typically the indexing tier it reports to, or by reading files directly on the forwarder's host. Work outward from confirming the forwarder is connecting at all, to identifying the specific queue and host that is blocked, to matching that against a plausible cause before changing any configuration.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
Fixes: resolving the block once the cause is confirmed
Each of the four causes above has a distinct fix, and applying the wrong one, such as raising the forwarder's thruput limit when the indexer's disk is the actual bottleneck, does not resolve the block and can make indexer-side contention worse by letting more data arrive at a tier that is already behind.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
Validation: proving ingestion has actually resumed
Clearing the immediate symptom is not the same as confirming the fix worked. Blocked queues can intermittently unblock and reblock under load, and a forwarder that appears to be sending again may still be running behind on a backlog it built up during the outage.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
Failure cases: when the fix doesn't actually restore ingestion
Not every blocked queue resolves cleanly on the first attempted fix, and a few patterns come up often enough to check for before concluding the wrong root cause was addressed.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
Escalation criteria: when to involve the indexing tier owner or Splunk Support
Most blocked-queue incidents resolve at the forwarder or single-indexer level using the steps above. A smaller set of situations warrant escalating beyond forwarder-level troubleshooting rather than continuing to iterate alone.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
The bottom line
A Universal Forwarder that has stopped sending data is almost always waiting on a blocked internal queue rather than failing outright, and the fix depends on correctly identifying which queue is blocked and why before changing anything. Confirm the forwarder is connecting with a tcpin_connections search, then identify the specific blocked host and queue name with a group=queue blocked=true search against the indexing tier's _internal data, and cross-check with TcpOutputProc entries in splunkd.log on the forwarder itself if its own internal logs are not making it through. Match what is found against the four common causes: indexer disk I/O overload, a broken network path, a forwarder thruput ceiling left at its 256 KBps default, or useACK enabled without a correspondingly sized maxQueueSize. Fix the specific cause rather than the nearest configuration knob, then validate that blocked=true has actually stopped recurring and that indexed event volume has returned to baseline rather than declaring victory the moment data starts flowing again. Escalate to the indexing tier owner or Splunk Support when the cause is a capacity problem beyond configuration, when data loss is confirmed without a matching explanation, or when the pattern is showing up across many forwarders at once rather than a single host.
Frequently asked questions
What does blocked=true mean in Splunk's metrics.log?
blocked=true means that the named processing queue was full at the moment Splunk wrote that metrics.log entry, and whatever tried to push data into it had to wait rather than being accepted immediately. Metrics.log entries for the queue stat group look like group=queue, name=parsingqueue, blocked=true, along with max_size_kb and current_size_kb fields showing the queue at capacity. A handful of scattered blocked entries on a busy indexer is normal flow control and self-corrects within seconds. A queue that stays blocked=true across repeated metrics.log intervals, several minutes apart, is the queue actually responsible for a Universal Forwarder appearing to stop sending data, because backpressure from a stuck queue propagates upstream through the pipeline and eventually stalls the forwarder's own output queue.
Why would a Universal Forwarder stop sending data if the indexer looks healthy?
A Universal Forwarder can stop sending data even when the indexer's CPU and process list look fine because the bottleneck is often disk I/O on the indexer, not CPU. An indexer that is falling behind on writing to its index queue backs up into the typing queue, then the aggregator queue, then the parsing queue, and that backpressure eventually reaches every forwarder connected to it through TCP flow control, since the indexer stops reading from the socket as fast as the forwarder is writing to it. From the forwarder's side, this shows up as a WARN TcpOutputProc entry in splunkd.log reporting that forwarding to the indexer group has been blocked for a growing number of seconds, even though the indexer process itself has not crashed and basic connectivity checks like a port test still succeed.
What is the difference between a blocked queue and a saturated queue in Splunk?
A saturated queue is one that is running close to full but is still accepting and draining data at a sustainable rate, which is common on a busy indexer and does not by itself mean data has stopped moving. A blocked queue is one that has actually hit its capacity limit and is refusing new items until space opens up, recorded explicitly as blocked=true in metrics.log. Splunk's own guidance is to look at the queue fill pattern over time rather than a single saturated reading: a queue that spikes to near-full and recovers quickly is healthy flow control, while a queue that sits blocked across consecutive metrics.log intervals is the one to investigate as the actual cause of a forwarder that has stopped shipping data.
How do I check whether a Universal Forwarder is even connecting to its indexer?
Confirm the forwarder is establishing a TCP connection before assuming the problem is a blocked queue, since a forwarder that cannot reach its indexer at all produces a different failure mode. Run index=_internal source=*metrics.log* tcpin_connections | stats count by sourceIp on the indexer or search head to see whether the forwarder's IP address appears with an active, ongoing connection rather than repeated connect and disconnect cycles. A plain Universal Forwarder has no search head of its own, so this search has to run against wherever that forwarder's internal logs land, typically the indexing tier it reports to, using host=<forwarder_hostname> to scope the results if the forwarder's own _internal index events are being ingested normally.
Does a blocked queue on the indexer cause the Universal Forwarder to drop data?
A blocked queue delays data rather than dropping it in the typical case, because the forwarder's own output queue backs up and the file monitor input stops advancing its read position until room opens up downstream, so the same bytes get retried rather than discarded. Data loss becomes a real risk when the blockage lasts long enough that a monitored file rotates or is deleted before the forwarder has forwarded everything it read, or when persistent queueing is not enabled and the forwarder process itself restarts while backed up. This is why validating that ingestion has resumed after a fix should include checking for a gap in indexed event volume during the blocked window, not just confirming that blocked=true entries have stopped appearing.
When should I raise the thruput limit on a Universal Forwarder instead of fixing the indexer?
Raise the thruput limit only after confirming that the forwarder's own [thruput] maxKBps setting in limits.conf, which defaults to 256 KBps on a Universal Forwarder, is the actual bottleneck rather than a symptom of an overloaded indexer pushing back on the connection. If metrics.log shows the forwarder's own queues blocking at a throughput ceiling well below what the indexing tier can absorb, and the indexer's queues are not themselves blocked, increasing or removing maxKBps in the [thruput] stanza is the correct fix. If the indexer-side queues (aggregator, typing, or indexing queue) are the ones showing blocked=true, raising the forwarder's thruput limit will not help and can make indexer-side contention worse by letting more data arrive at a tier that is already falling behind.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Daily briefings + Google Top Stories in one step.
Enter your email and we'll subscribe you to the daily briefing and add Decryption Digest to your Google Search feed at the same time.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.