Cribl Stream Worker Process Crashes: Diagnosing Out-of-Memory Failures in the Security Data Pipeline

A Worker Process has two separate memory ceilings that can be exceeded, and telling them apart matters for diagnosis. The first is the Node.js heap, which Cribl allocates a default of 2048 MB for per Worker Process; when a process's JavaScript object allocations exceed that limit, the Node.js runtime itself terminates the process with a heap-exhaustion error rather than waiting for the OS to intervene. The second is total system memory: a Worker Process consumes heap, Node.js runtime overhead, and additional memory that scales with configuration, including memory that sits entirely outside the heap and is not governed by the heap setting at all. When total memory across all Worker Processes on a host exceeds what the host actually has, the Linux OOM killer terminates a process to protect the system, and it does not care whether that process's own heap setting had room to spare.

Several specific pipeline and configuration patterns drive Worker Processes toward one of these two ceilings under real production load:

Large lookup tables loaded entirely into memory. Cribl's Lookup function reads the full lookup file into memory on each Worker Process rather than querying it on disk per event. A lookup table that is small in a lab environment but grows over time, an asset inventory, a threat intel IOC list, an identity-to-department mapping, consumes that same block of memory on every single Worker Process in the group, not once per host. A lookup that has quietly grown from a few megabytes to several hundred megabytes multiplies that growth by the Worker Process count on every node in the fleet.

Regex catastrophic backtracking against high-cardinality field values. A regex with nested quantifiers or ambiguous alternation can, against the right input string, cause the regex engine to explore an exponential number of matching paths before failing. Cribl Pipeline functions that evaluate regex per event, Regex Extract, Mask, Eval expressions using match or replace, run on Node.js's single-threaded JavaScript engine per Worker Process, so a pathological match blocks that process's event loop. While the event loop is blocked, new events keep arriving and queuing in memory faster than the process can drain them, and that backlog is what actually inflates memory usage and can push the process toward its heap limit, even though the underlying trigger was a CPU-bound regex match rather than a memory-bound one.

Unbounded Aggregations function state. Cribl's Aggregations function maintains running state per unique group-by key until its configured time window closes and flushes. If the group-by expression is built on a high-cardinality field, a source IP address, a user identifier, a full URL, across an entire network's worth of traffic, and the window is long or the flush interval is misconfigured, the function accumulates state for every distinct key simultaneously rather than for a small, bounded set. That state lives in the Worker Process's heap for the full duration of the window, and cardinality that looked reasonable in testing can be an order of magnitude higher against full production traffic.

Oversized batching and buffering ahead of a slow destination. When a destination cannot keep up, Cribl's persistent queue writes overflow to disk, which is the behavior covered in our backpressure guide. But before that mechanism engages, events sitting in a destination's in-flight send buffer or configured batch also consume memory, and a destination configured with a large max batch size or a high number of concurrent connections holds proportionally more data in memory while waiting to flush. Cribl's own engineering blog documents a specific, well-known version of this problem with Load Balancing Destinations such as Splunk LB: each Worker Process allocates a minimum of roughly 2 MB of buffer per configured indexer under normal conditions, and up to roughly 4 MB per indexer once that destination is under backpressure, and this buffer memory sits outside the Node.js heap entirely, which means it is not governed by the heap size setting and does not show up as heap usage. Cribl's blog gives a real customer example of 1,300 indexers configured across 4 Splunk LB Destinations consuming roughly 2.6 GB of this external memory per Worker Process; duplicate indexer lists defined redundantly across multiple Destination configurations make this worse without adding any actual throughput.

Worker Group or Fleet memory settings sized against CPU alone, not against host RAM. Cribl's default sizing guidance scales Worker Process count primarily against available CPU cores, but each additional Worker Process also carries its own default 2048 MB heap allocation plus Node.js runtime overhead plus any external memory from the causes above. A host sized for CPU headroom but not for the memory footprint of every Worker Process running on it simultaneously can be memory-constrained even when CPU utilization looks healthy, and that gap only shows up under real load, not during initial deployment testing at low volume.

A smaller number of memory-related Worker Process crashes are documented, version-specific product defects rather than configuration issues. Cribl's own Known Issues page for Stream 4.7 lists several: events with timestamps before January 1, 1970 processed through C.Time functions can drive a Worker Process to high CPU and unresponsiveness (CRIBL-26719); the jemalloc memory allocator library becoming unlinked from the Cribl binary on certain on-premises x86_64 deployments can cause memory spikes (CRIBL-25702); malformed Syslog messages over UDP were documented as capable of causing a Worker to run out of memory across versions 4.0.0 through 4.7.1 (CRIBL-25572); a large volume of logs written in a short window caused memory spikes in versions 4.3.0 through 4.4.1, fixed in 4.4.2 (CRIBL-20607); and the File Monitor Source in Manual Discovery mode leaked memory and a file descriptor on every file rediscovery in versions 4.2.0 through 4.2.1, fixed in 4.2.2 (CRIBL-19154). These are stated here as documented facts from Cribl's own known-issues page for version 4.7 specifically; whether any of them apply depends entirely on the version actually running, and the correct step is checking the known-issues page for that exact version rather than assuming a fix landed just because it is not the version installed here.

Start by determining which of the two ceilings was actually hit. That single fact changes where the rest of the investigation looks.

Apply the fix that matches the specific cause confirmed in diagnostics. Raising the heap size setting without addressing an unbounded growth pattern underneath it does not fix the crash, it only buys time before the same process hits the new, larger ceiling.

A configuration change needs to be validated against sustained production load over time, not just against a clean restart, since most of these causes only manifest once memory has had time to accumulate.

If Worker Processes continue crashing after fixing the cause identified in diagnostics, check whether more than one of the causes above was present simultaneously; a large lookup and an unbounded Aggregations function on the same pipeline can each individually be within tolerance while together exceeding available memory, and fixing only one leaves the other to eventually reach the same ceiling on its own.

If increasing the per-process heap size stopped the crash temporarily but it recurs weeks later at a similar point in a growth curve, the underlying cause is very likely a genuine unbounded accumulation, a growing lookup, uncontrolled Aggregations cardinality, or a memory leak matching a known product defect, not a one-time sizing gap. Raising the heap setting again treats the symptom and should not be treated as the fix.

If reducing destination batch size or Load Balancing connection counts to control memory stops the crash but introduces new queue growth or throughput delay, that is expected: the fix traded in-memory buffering for either slower throughput or increased reliance on persistent queue and disk. If that persistent queue behavior itself starts showing signs of silent data loss rather than a visible slowdown, that is the distinct symptom covered in our Cribl Stream backpressure and persistent queue data loss guide, not a continuation of this OOM issue.

If crashes continue on one specific host in a Worker Group while other hosts running the same configuration remain stable, suspect a host-level cause outside Cribl entirely: another process on a shared host consuming memory Cribl's own sizing did not account for, or a hardware or hypervisor-level memory constraint specific to that host.

Handle internally when the cause traces to something directly under your own control: a lookup size, a regex pattern, an Aggregations configuration, a destination's batch and connection settings, or Worker Process count against host RAM.

Open a case with Cribl Support when the crash signature matches a specific known issue listed on the Known Issues page for the version actually installed, since a version-specific defect needs a vendor fix or an official workaround rather than a pipeline-level remediation; reference the specific issue identifier from the vendor's documentation when opening the case. Also escalate when a Worker Process continues crashing after every cause identified in this guide has been ruled out through the diagnostics above, when memory usage climbs in a pattern that does not correlate to any identifiable pipeline function or destination configuration, or when the external, non-heap memory growth pattern documented for Load Balancing Destinations appears with an indexer count and Destination configuration that does not match the pattern described in Cribl's own guidance.

Before opening a case, gather: the exact Cribl Stream version in use, the affected Worker Group's Worker Process count and per-process heap setting, host RAM and current memory utilization at the time of the crash, the OOM killer or heap exhaustion log entries with timestamps, and a list of the pipelines, lookups, Aggregations functions, and destinations active on the crashed Worker Process. For teams whose Worker Process memory issues are recurring often enough to raise questions about whether Cribl's architecture fits their scale, our Cribl vs. DataBahn vs. Observo AI comparison covers that broader tradeoff.