Cribl Stream Backpressure and Persistent Queue Data Loss: Diagnosis and Fix
Why security log data goes missing between sources and the SIEM, and how to size queues and alerts so the next drop doesn't happen silently
Proactive Security for the AI Era
NodeZero continuously and autonomously pentests infrastructure, identity, cloud, and now web applications, chaining weaknesses across every domain the way real attackers do. Every finding ships with replayable proof showing exploitable business impact, not theoretical risk.
A detection that should have fired and did not is one of the harder problems to root-cause in a SOC, because the absence of an alert looks identical to the absence of malicious activity. When the actual cause is that the underlying log data never reached the SIEM, the pipeline sitting between the source and the destination is where that investigation needs to start, and for a growing number of environments that pipeline is Cribl Stream.
Cribl Stream's backpressure mechanism exists to protect the pipeline from being overwhelmed: when a destination cannot keep up, Cribl can block the upstream sender, drop new events, or buffer overflow to a persistent queue on local disk. Each of those three responses is a reasonable engineering tradeoff on its own. The problem is that a misconfiguration in queue sizing, protocol handling, or destination monitoring turns any of them into silent, sustained data loss that nobody notices until a detection gap or an audit turns it up.
This guide covers three documented ways that happens: a persistent queue's max size configured without regard to the disk space actually available, source-side persistent queue "Smart" mode engaging backpressure before the queue's configured limit is actually reached, and destination-side slowness or unavailability propagating backward through the pipeline in ways that are easy to miss without dedicated destination health alerting. It also covers why UDP sources are the most exposed to this problem, since UDP itself has no retry or backpressure mechanism to fall back on.
Why This Happens: How Backpressure Turns Into Data Loss
Cribl's own documentation frames backpressure simply: it happens when production outpaces consumption, either because a source sends a sudden burst of data Cribl cannot immediately process, or because a destination does not have the capacity to receive what Cribl is trying to send it. Troubleshooting backpressure correctly means starting at the destination and working backward, since Cribl can only send as fast as the slowest connection to a given destination type allows; a destination that throttles its TCP receive window causes Cribl to stop sending and periodically retry, and that slowdown then ripples upstream.
Once backpressure is detected, Cribl applies one of three configured responses. Block, the default, propagates a block signal back to every source sending to the affected destination type, which is safe from a data-loss standpoint but can stall ingestion broadly. Drop discards new events outright once backpressure triggers, trading data completeness for pipeline responsiveness. Persistent Queue (PQ) writes overflow events to local disk instead of blocking or dropping immediately, with blocking or dropping only beginning once the queue itself reaches its configured capacity.
Persistent queues are where the sizing problem shows up. A queue's max size setting controls how much disk space it is allowed to consume, and that setting is independent of how much disk space is actually free on the host running that worker process. If the configured max queue size exceeds available disk, the queue can fail to write cleanly once the disk actually fills, which behaves as a silent drop rather than a clean, visible block. Cribl's own guidance on persistent queues frames a continuously saturated PQ as a signal of insufficient destination capacity in the first place, not something to solve by growing the queue indefinitely; queue size should be sized against a specific, deliberate question, how long a specific destination outage needs to be covered, not against whatever disk happens to be available.
Source-side persistent queue Smart mode introduces a second, more subtle failure. Smart mode is designed to engage PQ only when a source actually detects backpressure from Cribl's processing engine, rather than running the queue continuously. Cribl's own documentation notes that when Smart mode is combined with a large max file size setting, source-side PQ can engage backpressure well before the queue's overall configured capacity is reached, because Smart mode's trigger condition is tied to filling the current queue file, not the queue's total configured limit. When that happens on a UDP source specifically, the consequence is worse than on a TCP or HTTP source: UDP has no built-in retry or acknowledgment mechanism of its own, so when Cribl needs to signal a sender to slow down, there is no lower-level protocol mechanism to carry that signal. A TCP-based sender can have its connection throttled at the window level; a UDP sender simply keeps sending, and packets arriving faster than Cribl's UDP listener can hand them to the processing pipeline are dropped rather than queued or retried. Cribl's own known-issues documentation for versions through 4.4.4 separately confirms that ingesting high rates of UDP events could cause dropped data in Cribl.Cloud specifically under sustained load, independent of the Smart mode interaction, which underscores that UDP ingestion is a documented, ongoing risk area rather than a one-off edge case.
Diagnostics: Finding Where Data Is Actually Dropping
Work backward from the destination, per Cribl's own troubleshooting guidance, rather than starting at the source. The destination is where slowness originates even when the symptom shows up as a source-side queue filling up.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
Fixes: Remediation Per Cause
Apply the fix matching the specific cause confirmed in diagnostics. Increasing queue size or disk allocation without first confirming the actual bottleneck usually just delays the same failure to a larger scale.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
Validation: Confirming the Fix Worked
A configuration change to queue sizing or destination behavior needs to be validated against actual data completeness, not just against the absence of a new error in the Cribl UI.
Subscribe to unlock Remediation & Mitigation steps
Free subscribers unlock full IOC lists, Sigma detection rules, remediation steps, and every daily briefing.
Failure Cases: When the Standard Fix Doesn't Resolve It
If reconciled event counts still show a gap after resizing persistent queues and confirming disk headroom, check whether more than one worker process on the same node shares the disk being sized against; a per-queue calculation that looks correct in isolation can still overcommit disk once every PQ-enabled pipeline on that node is accounted for simultaneously.
If switching source-side PQ from Smart mode to Always On does not eliminate the premature backpressure, confirm the change was applied and committed on every worker process in the group, not just on the leader or a single node in a distributed deployment; persistent queue settings are per-worker-process configuration and a partial rollout will show inconsistent behavior across the fleet.
If UDP loss continues after moving a source to TCP, confirm the sending device or forwarder itself was reconfigured to send TCP and that any intermediate network path (a load balancer, a relay, or a security appliance in between) is not silently converting or dropping the TCP stream; a partial migration where the sender configuration was not fully updated will look identical to the original UDP loss from the Cribl side.
If setting queue-full behavior to Block resolves the silent loss but now causes broader pipeline stalls during genuine destination outages, that is expected behavior, not a new bug; it means the destination's actual outage frequency and duration were underestimated when the persistent queue was originally sized, and the queue capacity itself needs to be revisited rather than reverting back to Drop.
Escalation Criteria: When to Open a Cribl Support Case
Handle internally when the cause traces to a configuration issue you can directly change: queue sizing against disk, Smart mode file size, queue-full behavior, or a destination-side capacity limit under your own control.
Open a case with Cribl support when a persistent queue shows abnormal behavior that does not match documented Smart mode or sizing behavior, such as a queue that does not drain even after the destination confirms healthy throughput and adequate capacity; when a Kafka-based or other destination type is suspected of not reporting backpressure promptly (a documented issue in earlier Stream versions that could cause upstream stalls or delayed persistent-queue engagement); or when UDP data loss persists at a rate inconsistent with expected load even after applying the buffer-size and worker-placement mitigations, since sustained high-rate UDP ingestion loss has been an acknowledged, version-specific issue in Cribl's own known-issues documentation.
Before opening a case, gather: the Cribl Stream version in use, the specific source and destination types involved, current and configured max queue size for every affected PQ, whether Smart mode or Always On is configured on the affected source, event count comparisons across the pipeline showing the specific gap, and destination-side health metrics covering the same window. For teams evaluating whether Cribl's queue and backpressure model fits their scale versus alternative pipeline tools, our Cribl vs. DataBahn vs. Observo AI comparison covers that tradeoff, and our guide on building a log pipeline into a self-hosted OpenSearch SIEM covers destination-side sizing considerations relevant to the capacity fixes above.
The bottom line
Cribl Stream backpressure is a designed safety mechanism, but three specific configuration patterns turn it into silent security log data loss: a persistent queue's max size configured without regard to actual available disk, source-side Smart mode persistent queue triggering backpressure before its configured limit is genuinely reached, and destination-side slowness propagating upstream without dedicated health alerting to surface it. UDP sources carry the highest risk in all three cases, since UDP has no retry or backpressure mechanism of its own and simply drops packets once Cribl's listener falls behind. Diagnose backward from the destination, reconcile event counts across the pipeline rather than trusting the absence of an error, and size persistent queues against a specific outage-duration target and confirmed disk headroom rather than a default.
Frequently asked questions
Why is Cribl Stream silently dropping log data before it reaches my SIEM?
The most common causes are a persistent queue whose configured max size exceeds actual available disk, source-side Smart mode persistent queue triggering backpressure before its configured limit is reached, or a destination that is slow or unavailable without dedicated health alerting to surface the resulting drop.
Why are UDP sources more likely to lose data during Cribl Stream backpressure than TCP sources?
UDP has no built-in retry or acknowledgment mechanism, so when Cribl needs to signal a sender to slow down, there is no protocol-level way to do it. A TCP sender can be throttled through the connection window; a UDP sender keeps sending regardless, so packets arriving faster than Cribl's listener can process are dropped.
What does persistent queue 'Smart' mode do, and why can it cause unexpected drops?
Smart mode engages the persistent queue only when a source detects backpressure from Cribl's processing engine. When combined with a large max file size, Smart mode can trigger backpressure before the queue's overall configured capacity is actually reached, since its trigger condition is tied to filling the current queue file rather than the total queue limit.
Should I just increase my persistent queue's max size to fix backpressure and data loss?
Only after confirming the queue's max size against actual available disk on the worker host. A queue continuously near its configured limit is Cribl's own documented signal that destination capacity is insufficient; growing the queue without fixing that just delays the same failure at a larger scale.
What is the difference between Block and Drop queue-full behavior in Cribl Stream?
Block, the default, propagates a signal upstream to pause senders once the queue is full, which is visible and easier to alert on. Drop New Data discards new events outright once the queue fills, silently, with no automatic upstream signal, which is why security-relevant pipelines should generally use Block instead.
How do I confirm a Cribl Stream data loss fix actually worked?
Compare event counts between the source and the destination over the same time window, ideally replicating the load pattern that originally caused the loss, and confirm persistent queue disk usage returns to a stable baseline between load events rather than staying elevated.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.