Self-Hosted SIEM with OpenSearch: An Honest Build Guide

The case for self-hosting is strongest under four conditions, in roughly this order. First, your team has demonstrated operational maturity running stateful distributed systems. If you do not have engineers who can debug a JVM heap dump, diagnose a stuck shard relocation, or restore a snapshot under pressure, the savings will evaporate into incident toil. Second, your licensing math has crossed the line where the savings genuinely pay for the headcount, generally above the one to two million dollar annual SIEM line item depending on region. Below that threshold the math is rarely favorable once you fully load engineer time. Third, you have a compliance or data residency requirement that constrains where data can land; self-hosting in your own VPC sidesteps a long vendor review. Fourth, you have a customization need the commercial tool genuinely cannot meet: extreme retention, unusual data shapes, or programmatic integration with internal platforms. Conversely, if your team is small, your detection content is mostly vendor-provided, your retention needs are modest, and you would rather buy outcomes than build them, a commercial SIEM is the rational choice. The worst outcome is a self-hosted build owned by a team that wanted to buy and was told to build.

A production-grade architecture has five layers. Collection: Fluent Bit on every endpoint or syslog forwarder, with Logstash reserved for sources that need heavy parsing. Fluent Bit is dramatically lighter and easier to operate at fleet scale. Buffering: Kafka between collection and indexing, partitioned by source type, with retention of at least 72 hours. Kafka is non-negotiable for any cluster taking more than a few thousand events per second; it absorbs ingest spikes, decouples failure domains, and gives you replay capability when a parsing change requires reprocessing. Indexing: a hot tier of three or more data nodes on NVMe with dedicated master nodes, a warm tier on cheaper storage for older indices, and optional cold or frozen tiers backed by S3 for compliance retention. Visualization: OpenSearch Dashboards, with role-based access tied to your identity provider via SAML or OIDC. Detection: OpenSearch Alerting plugin for per-document and bucket-level monitors, with ElastAlert2 layered on for use cases the native plugin handles poorly such as cardinality and spike rules. Wrap the whole thing in infrastructure as code, ideally Terraform for cluster provisioning and a configuration management tool for cluster settings. Treat the cluster as cattle, not pets; you will rebuild it during major version upgrades and you do not want to rediscover its configuration each time.

Index design is where most self-hosted SIEMs silently degrade. Use data streams or rollover aliases rather than time-based index names; the alias indirection lets you change shard counts and mappings without breaking queries. Target shard size between 10 and 30 gigabytes for search performance; smaller wastes overhead, larger slows recovery and search. Calculate primary shard count as expected daily volume divided by target shard size; do not copy default settings. Configure Index State Management policies to roll over on size or age and to transition through hot, warm, and cold phases. Force merge on rollover to reduce segment count and free heap. Disable the _source field only if you genuinely never need to retrieve original events, which is rarely true for security data because investigations require raw context. Map fields explicitly; dynamic mapping will create fields like user.name.keyword and user.name in parallel, doubling storage and creating subtle query inconsistencies. Use ECS field names consistently from day one. Plan capacity with the formula: raw daily ingest times replication factor times 1.2 for overhead times retention days, then add 25 percent headroom. Cluster pain almost always traces back to one of three failures: unbounded shard count, unbounded field count from dynamic mapping, and undersized heap relative to data.

Detection engineering at scale requires that source.ip means the same thing in every index. The Elastic Common Schema gives you that vocabulary; adopt it without modification. Parse at the collector when possible and at Logstash when not. Fluent Bit handles common formats natively and supports Lua for custom logic; Logstash grok filters remain the most flexible option for unstructured logs but are computationally expensive. For Windows Event Log, use Winlogbeat or the native XML ingestion to preserve event structure; flattening XML into key-value loses context you will need later. For CEF and LEEF, parse to ECS at ingestion rather than at query time; query-time parsing destroys performance and prevents efficient aggregation. Normalize timestamps to UTC at the collector, never at the indexer. Add an event.ingested timestamp separate from the source timestamp; the gap between them is your end-to-end pipeline latency metric and one of the most useful health signals you have. Document every parser in a repository alongside test fixtures; a parsing change without a regression test will eventually break a detection rule and you will not notice until the rule fails to fire on a real attack. Treat parsing as production code, not configuration.

OpenSearch Alerting supports per-document monitors that evaluate each indexed event against a query and bucket-level monitors that aggregate over time windows. For SIEM use, you want both. Per-document monitors are right for atomic detections such as Sigma single-event rules; bucket-level monitors are right for rate, cardinality, and threshold logic. Convert Sigma rules with the pySigma OpenSearch backend; expect to hand-tune perhaps 20 percent of rules where the conversion does not map cleanly, particularly rules that depend on Sysmon-specific fields or correlation across events. Build a detection-as-code repository with rule sources in YAML, a CI pipeline that converts and validates against OpenSearch DSL, and a deployment step that pushes monitors via the API. Tag every rule with ATT&CK technique, data source, severity, and ownership. Implement throttling on every monitor to prevent alert storms during incidents. Schedule monitors with awareness of cluster load; running 500 monitors every minute on a small cluster will eat your search capacity and slow analyst queries. Stagger schedules and use bucket-level monitors with longer intervals where the detection logic permits.

Plan for 0.5 to 1.0 FTE of platform engineering time per production cluster, indefinitely. The work includes JVM heap tuning when GC pauses cause cluster instability, certificate rotation across nodes and clients quarterly or more often, version upgrades that are non-trivial across major versions and occasionally require reindexing, shard rebalancing intervention when the cluster gets stuck, snapshot and restore testing on a real schedule because untested backups are not backups, capacity forecasting as ingest grows, and the inevitable midnight pages when a node dies under load. Add another 0.5 FTE of detection engineering time for rule development, tuning, and content curation; that cost exists regardless of platform but is more visible here because you cannot blame the vendor. Budget for hardware refresh on a three-year cycle, license costs for non-free plugins if you choose Elasticsearch over OpenSearch, and observability tooling for the cluster itself, which is non-negotiable. The savings versus a commercial SIEM are real but they are paid in headcount and operational risk rather than software licensing. Going in with that framing produces a successful build; going in expecting free produces a project that quietly degrades until someone proposes migrating back.