SIEM Architecture from Scratch: Building a Security Monitoring Platform with Elastic Stack or Wazuh
Retool's new app builder is where AI-generated code ships safely
Building apps with AI is easy. Getting them to production safely is another story.
The gap between 'we installed Elasticsearch and we are collecting logs' and 'we have a functional SIEM' is significant. A log collection platform that stores events but has no detection rules, no normalization, and no alert routing is an expensive searchable log archive. A SIEM requires the additional components that make it a security monitoring tool: detection rules that generate alerts on suspicious patterns, normalization that makes disparate log sources queryable with the same field names, and alert routing that ensures analysts see findings promptly.
Both Elastic Security and Wazuh provide these components in their respective ecosystems. The choice between them is primarily a question of deployment complexity versus feature scope — Wazuh delivers faster time-to-detection for endpoint events; Elastic Stack handles broader log source diversity and more complex correlations at scale.
Architecture decisions that determine SIEM success or failure
Three architectural decisions made in the first week of a SIEM build determine whether the platform becomes a functional security tool or an underutilized log archive. First, the normalization strategy: using Elastic Agent integration packs for supported sources and building Ingest Pipelines for custom sources ensures that every detection rule can use ECS field names without per-source modification. Second, the ILM policy: configuring index lifecycle management before the first production index reaches 50 GB prevents runaway SSD storage costs and the retroactive ILM migration complexity that follows. Third, the log source sequence: onboarding Windows Event Logs and an identity provider first delivers prebuilt detection rule coverage immediately, while deferring high-volume low-signal sources like full DNS logs until the hot tier is tuned. Getting these three decisions right at the start avoids the technical debt that grows with every additional log source onboarded into an unnormalized, untiered, undetected archive.
Decide on ECS normalization before onboarding more than two log sources
The normalization decision compounds: every log source onboarded without ECS normalization creates technical debt that grows with each subsequent detection rule that needs per-source field references. Use Elastic Agent with integration packs for all sources that have a published integration — integration packs apply ECS normalization automatically. For custom log sources without a published integration: create an Ingest Pipeline that maps vendor fields to ECS before the data is indexed. The normalization investment for a new log source is 30-120 minutes — the payoff is every future detection rule can use ECS field names without per-source modification.
Configure ILM before the first production log index reaches 50 GB
ILM policy must be applied to new indices — it cannot be applied retroactively to indices that already exist. Configure the ILM policy and index template before the first production log data starts flowing. If the SIEM has already been running without ILM: create the ILM policy, apply it to new indices via index template, and manually move the existing hot indices to warm/cold by updating their ILM phase using the Kibana Index Lifecycle Management interface. For most organizations with under 50 GB/day log ingest: a single Elasticsearch node with SSD storage is sufficient for the hot tier. At 100 GB/day or more: plan for a multi-node cluster with explicit hot/warm/cold node allocation.
Week-one value: getting to first alert as fast as possible
A SIEM that has been running for two weeks with no alerts is not a sign the environment is clean — it is a sign the detection pipeline is not working. Enabling prebuilt rules immediately after onboarding the first log sources is the fastest way to validate that logs are flowing, normalized correctly, and producing signal. Elastic Security's 700+ prebuilt rules cover the MITRE ATT&CK techniques most relevant to Windows endpoints and cloud environments, so the first alert will typically appear within hours of enabling Credential Access and Lateral Movement rules on a production Windows estate. Running those rules on 24 hours of historical data during the initial configuration session both tests the pipeline and begins the analyst team's familiarity with the alert interface. The alert volume from prebuilt rules in the first historical run drives the exception-tuning work that makes ongoing alert volume manageable.
Enable prebuilt detection rules immediately after onboarding the first log sources
In Elastic Security: after deploying Elastic Agent to Windows servers and confirming Windows Event Logs are being ingested, immediately enable the Elastic prebuilt rules for Windows under MITRE ATT&CK: Credential Access, Defense Evasion, and Lateral Movement. Run the rules on 24 hours of historical data to see the initial alert volume. Most environments see 10-50 alerts in the first historical run — investigate each one manually to establish which are true positives, which are false positives, and which require exception tuning. This investigation session both validates the rules and trains the analyst team on the SIEM's alert interface. Target: at least one true positive detection from prebuilt rules within the first week of deployment, confirming end-to-end pipeline function.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
The bottom line
Building a functional SIEM requires five components beyond log storage: ECS normalization for consistent field names across sources, an ILM policy that controls storage costs before they become unmanageable, prebuilt detection rules enabled on the first log sources rather than after all log sources are onboarded, exception tuning that reduces false positive volume to a level the team can handle, and alert routing that gets findings to analysts within minutes of detection. Elastic Security and Wazuh both provide these components — Elastic Stack with broader log source integrations and more powerful query capabilities, Wazuh with faster time-to-endpoint-detection and built-in host-based security features. Choose the platform that fits your primary log source profile and team's operational capacity, then focus on the detection and alerting configuration that converts log data into security signal.
Frequently asked questions
Should I build my SIEM on Elastic Stack or Wazuh, and what are the key differences?
Platform comparison: Elastic Stack SIEM (Elasticsearch + Kibana + Elastic Security): Strengths: best-in-class query performance for large log volumes, powerful KQL query language, 700+ prebuilt MITRE ATT&CK detection rules, excellent dashboards, cloud-native log source integrations (AWS CloudTrail, GCP Audit Logs, Azure Monitor). Deployment complexity: moderate — requires deploying Elastic Agent or Filebeat to each log source, configuring ingest pipelines for normalization. License: the SIEM features are in the free Basic tier of Elastic Cloud or self-hosted. Scaling: Elasticsearch scales horizontally and handles billions of events per day in large deployments. Wazuh: Strengths: purpose-built endpoint security agent that provides HIDS, file integrity monitoring, rootkit detection, and log forwarding from a single agent install; active response (automatically block IP, kill process when a rule fires); faster time-to-value for endpoint monitoring. Deployment complexity: lower for endpoint monitoring — deploy the Wazuh agent, and it immediately starts sending events, applying built-in detection rules. License: fully open source (GPL). Scaling: Wazuh uses Elasticsearch as its backend; at very high scale the indexing architecture requires similar tuning. Recommendation: Wazuh for organizations that want faster time-to-endpoint-detection-value and simpler deployment. Elastic Stack for organizations with diverse log sources (cloud infrastructure, network devices, SaaS APIs) that need the most powerful query capability and broadest integration library.
What log sources should I onboard first for fastest security value?
Log source prioritization by security value and onboarding effort: Week 1 (highest value, lowest effort): (1) Windows Security Event Log: Event IDs 4624 (login success), 4625 (login failure), 4688 (process creation), 4698 (scheduled task creation), 4732 (group membership change). Deploy Elastic Agent or Winlogbeat to all Windows servers. (2) Authentication log from your identity provider (Okta, Entra ID, Google Workspace): authentication events reveal suspicious logins, impossible travel, and MFA fatigue attacks. Most IdPs have native Elastic or SIEM integrations. Week 2: (3) Firewall logs: perimeter firewall deny logs, VPN connection logs. Provides network context for security events detected from endpoint logs. (4) Cloud trail logs: AWS CloudTrail, GCP Cloud Audit Logs, Azure Monitor — essential for cloud environments. Week 3-4: (5) DNS logs: identify malware command-and-control via unusual domain resolution patterns. (6) Endpoint EDR telemetry: if you have CrowdStrike, Defender for Endpoint, or SentinelOne, export alerts and/or raw telemetry to the SIEM for centralized correlation. (7) Application logs for critical applications (web server access logs, database audit logs). Deferred: network flow logs (very high volume, high storage cost), full packet capture (not practical for most SIEM deployments).
How do I configure index lifecycle management to control Elasticsearch storage costs?
Index Lifecycle Management (ILM) configuration in Elasticsearch: (1) Define a lifecycle policy (in Kibana > Stack Management > Index Lifecycle Policies): Phase Hot: rollover when index reaches 50 GB or 7 days old. Phase Warm: move to warm tier after 7 days. Force merge to 1 segment and set replica to 0. Phase Cold: move to cold tier (object storage via S3 Snapshot or Elastic's frozen tier) after 30 days. Phase Delete: delete after 90 days (or your compliance requirement). (2) Configure hot/warm/cold node tiers in your Elasticsearch cluster: hot nodes use SSDs, warm nodes use HDDs, cold nodes use object storage (AWS S3, GCS) via the Elasticsearch S3 snapshot or ILM frozen tier. (3) Create an index template that applies the ILM policy to all new security log indices: PUT /_index_template/security-logs with index.lifecycle.name set to your ILM policy name. (4) Estimate cost: at 1 TB/day ingest, 7-day hot retention requires ~7 TB hot storage, 23 more days warm requires ~23 TB warm (HDD, 60-70% cheaper than SSD per GB). (5) Use Elastic Cloud with autoscaling if self-managing node tiers is too complex — the managed service handles tier transitions automatically.
How do I configure and tune detection rules in Elastic Security?
Elastic Security detection rule configuration: (1) Enable prebuilt rules: in Kibana > Security > Rules > Add Elastic rules. Filter by log source (Windows, Linux, AWS) and enable rules relevant to your deployed log sources. Start with rules tagged MITRE ATT&CK: Defense Evasion, Lateral Movement, and Credential Access — these cover the most common post-initial-access techniques. (2) Evaluate the enabled rules for your environment: after enabling rules, run them on the last 24 hours of historical data to see how many alerts each rule generates. Rules generating more than 10 alerts per day should be reviewed for false positive rate before being treated as operational. (3) Add exception lists for known-safe patterns: in each noisy rule, add exception entries for: legitimate administrative processes that trigger the rule (IT management tools, scheduled tasks), known-good IP addresses that trigger network-based rules, specific user accounts whose behavior is expected to match the rule pattern. (4) Create custom rules for organization-specific scenarios: KQL rule example for impossible travel: event.dataset:'okta.system' AND okta.event_type:'user.session.start' — add the impossible travel correlation by comparing source IP geolocation between consecutive login events for the same user. (5) Set alert severity and routing: Critical rules should trigger immediate PagerDuty alert; High should create a Jira ticket; Medium/Low go to a daily digest.
How do I normalize logs from different sources to use consistent field names?
Log normalization with the Elastic Common Schema (ECS): ECS defines standard field names for common data types — source IP is source.ip, destination port is destination.port, username is user.name — across all log sources. Without normalization: a detection rule for 'failed login from unusual IP' requires different field names for Windows Event Logs (SubjectUserName vs EventData.IpAddress), Okta logs (actor.alternateId vs client.ipAddress), and Linux auth logs (user vs host). With ECS: all of these are mapped to user.name and source.ip, enabling a single rule. Implementation: (1) Elastic Agent with integration packs: Elastic's integration packs (available in Fleet > Integrations) automatically apply ECS normalization for 200+ log sources when used with Elastic Agent — no manual pipeline configuration required. (2) Manual Ingest Pipeline normalization: for custom log sources, create an Elasticsearch Ingest Pipeline that uses the rename processor to map vendor field names to ECS field names. Example: {type: rename, field: src_ip, target_field: source.ip}. (3) Logstash normalization: use the mutate filter in Logstash configurations to rename fields before indexing. (4) Verify normalization: after configuring a new log source, run: GET /logs-custom-*/_search?q=source.ip:* — if results return, the IP field is correctly mapped to ECS.
How do I deploy Wazuh agents to Windows and Linux endpoints?
Wazuh agent deployment: (1) Deploy the Wazuh manager first: Wazuh provides a single-command installation script for the manager + OpenSearch + Kibana stack on a dedicated Linux server. curl -sO https://packages.wazuh.com/4.7/wazuh-install.sh && sudo bash ./wazuh-install.sh -a. (2) Deploy the Wazuh agent to Windows endpoints: download the MSI from the Wazuh documentation, deploy via Intune/SCCM: msiexec /i wazuh-agent-4.7.x-1.msi /q WAZUH_MANAGER=YOUR_MANAGER_IP WAZUH_REGISTRATION_SERVER=YOUR_MANAGER_IP WAZUH_AGENT_NAME=%COMPUTERNAME%. (3) Deploy to Linux endpoints: add the Wazuh repository and install: curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add - echo 'deb https://packages.wazuh.com/4.x/apt/ stable main' | tee /etc/apt/sources.list.d/wazuh.list; apt-get update && apt-get install wazuh-agent. Configure: /var/ossec/etc/ossec.conf: set the Wazuh manager address. systemctl enable wazuh-agent && systemctl start wazuh-agent. (4) Verify enrollment: in the Wazuh dashboard (Kibana) > Agents, newly deployed agents appear within 1-2 minutes of starting the agent service. (5) Deploy at scale: use Ansible, Chef, or Puppet to deploy Wazuh agents across large fleets. Wazuh provides example Ansible playbooks in their documentation.
What alert routing and escalation should I configure for a small security team?
Alert routing for small security teams (1-5 security analysts): (1) Tier the alerts by severity and action required: Critical (CVSS equivalent 9.0+): automatic page to on-call analyst via PagerDuty or Opsgenie, create incident in ticketing system, notify security team Slack channel. High: create ticket in Jira/ServiceNow, notify Slack security channel. Medium: add to daily digest email. Low/Informational: searchable in SIEM, no active notification. (2) Configure routing in Elastic Stack: Security > Rules > each rule has a notification action. Add actions: Slack webhook for High+ severity, PagerDuty webhook for Critical severity. (3) Deduplicate alerts: configure alert deduplication so that the same detection rule firing on the same host multiple times in 1 hour creates one alert (not one per event). In Elastic Security: use the 'Alert suppression' feature on rules to group repeated alerts by host.name with a 1-hour suppression window. (4) Define investigation SLAs: Critical alerts must be acknowledged within 15 minutes and triaged within 1 hour. High: acknowledged within 2 hours. This SLA should be documented and monitored (how many Critical alerts were acknowledged within the SLA?). (5) Use a runbook for each detection rule: each alert should have a linked runbook that tells the analyst what the alert means and the first three investigation steps — this reduces analyst cognitive load during incidents and ensures consistent response.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.
Win a $2,495 Black Hat pass.
Full-access to Black Hat USA 2026 in Las Vegas. Subscribe free to enter.