FortiGate HA Split-Brain: Diagnosing and Forcing a Safe Resync

FortiGate HA elects a primary using heartbeat interfaces, typically two dedicated ports (commonly labeled ha1 and ha2) that exchange hello packets and synchronize session and configuration state between cluster members. As long as at least one heartbeat link stays up, the cluster's arbitration logic is designed to prevent both units from claiming primary at once. Split-brain shows up specifically when that protection breaks down, and in practice that happens for a small number of recurring reasons.

The most common cause is total heartbeat link loss on both interfaces simultaneously, usually because both heartbeat cables run through the same switch, the same switch stack member, or the same power circuit, so a single upstream failure takes down both links at once rather than just one. A close second is heartbeat link congestion or a bad NIC or transceiver on one member: if heartbeat hello packets are delayed or dropped intermittently rather than the link going fully down, both units can still misjudge peer state and each decide it's the only one left.

A less obvious but well-documented cause is two separate HA clusters on the same broadcast domain sharing the same HA Group ID. FortiGate uses the Group ID (along with the group name/password) to distinguish which units belong together; if a second, unrelated cluster on the same VLAN is configured with the same Group ID, both clusters' heartbeat traffic can interfere with each other and produce exactly the same symptoms as a genuine heartbeat failure. Finally, mismatched firmware between the two units, one member running a different FortiOS build than the other after an incomplete or failed upgrade, will also block clean synchronization and can present as persistent out-of-sync status even without a full split-brain event.

Start by running get system ha status on both units, not just one. This command reports the HA mode (active-passive or active-active), the Group ID, and which unit each member currently believes is primary. If both units report themselves as primary, you have confirmed split-brain rather than a false alarm from a monitoring tool. If the Group ID differs between what you expect and what's configured, or if you find a second cluster nearby sharing your Group ID, that's your root cause without needing to go further.

Next, check heartbeat interface health directly rather than trusting link lights. diagnose sys ha status (the extended form) reports heartbeat packet loss counters (hb_lost_cnt) per interface; a counter that is climbing, rather than flat at zero, points to a congested or failing heartbeat link even when the interface shows as physically up. Where available, heartbeat statistics commands that show sent, received, and lost packet counts per heartbeat interface are the fastest way to tell the difference between a link that's fully dead and one that's degraded enough to break arbitration without dropping.

Once you know which link failed, compare configuration state between the two units with diagnose sys ha showcsum, run separately on each unit. This reports checksums for the debug zone and data zone of the running configuration, broken into global and per-VDOM sections, each ending in a sum value. If the sum values match between primary and secondary, the units agree on their configuration and you are dealing with a heartbeat problem only. If the sums differ, you have real configuration drift on top of the heartbeat issue, meaning changes were made independently on one or both units while they were split, and you need to identify which side holds the config you actually want before you force anything. Also pull diagnose sys ha history read on both units to see the timestamped record of role changes; a cluster that has flipped primary role multiple times in a short window confirms an intermittent, not a one-time, heartbeat problem, which changes how urgently you need to fix the physical link rather than just resync configuration.

Fix the physical or logical cause of the heartbeat loss before you force anything. If both heartbeat links share a switch, a power circuit, or a cable run, separate them, ideally onto genuinely independent physical paths, or direct-cable them between the two units if your rack layout allows it. If you found a duplicate Group ID collision with a neighboring cluster, change one cluster's Group ID and password to a value that doesn't overlap. If firmware is mismatched, bring both units to the same FortiOS build; HA will not synchronize reliably across differing versions no matter what else you fix.

Only after the heartbeat path is stable should you address configuration drift. Do not force a sync in the direction of whichever unit happens to currently be primary without checking first; if the secondary independently applied a legitimate emergency change (say, a temporary firewall rule opened during the split window), forcing a sync from the wrong side silently discards that change with no warning. Compare the showcsum output and your own change log or configuration backups from both units to determine which one holds the configuration you actually want to keep going forward, and back up both configs before you touch anything further.

Once you've confirmed the source of truth, run execute ha synchronize start from the unit holding the correct configuration to force it to push its full running configuration out to the other cluster member. This is a configuration-level synchronization, not a manual failover, so it does not by itself force a role change or interrupt existing sessions on a stable cluster. That said, forcing a full sync while the heartbeat link is still flapping can itself trigger another false failover, which is exactly why the link has to be stable first. If checksums still don't converge after a forced sync, and you've already ruled out an active heartbeat problem, the next step is removing the out-of-sync unit from the cluster, resetting it to a clean HA state, and rejoining it, which forces a full initial sync rather than an incremental one.

Run get system ha status on both units again and confirm only one reports itself as primary, with the other correctly showing as secondary and both agreeing on the same Group ID and cluster membership. Then re-run diagnose sys ha showcsum on both units and confirm the sum values now match for every zone; a match here is the actual signal that configuration is synchronized, not just that the heartbeat link is up.

Check diagnose sys ha history read again after the fix and watch it over a real observation window, at least 15 to 30 minutes of normal traffic, rather than declaring victory immediately. A cluster that flipped once during your fix and then stays stable is healthy; one that continues flipping roles even after you've addressed the heartbeat link means you haven't found the actual root cause yet. Also verify heartbeat packet loss counters stay flat rather than continuing to climb, and check with your network team that upstream switches have stopped logging MAC flapping or duplicate IP/ARP warnings for the cluster's virtual MAC and shared IPs, since those L2 symptoms can persist briefly even after the FortiGate side is healthy again while switch MAC and ARP tables age out old entries.

A handful of patterns show up repeatedly when a fix that looked complete doesn't actually stick. Checksums that converge after a forced sync but drift again within minutes usually mean something outside your manual process is still writing configuration to only one unit, commonly a centralized management tool such as FortiManager pushing a policy update to a single device rather than through the HA-aware workflow, or an API/automation script targeting one unit's management IP directly instead of the cluster's HA management interface.

Heartbeat links that test clean in isolation but the cluster still occasionally re-elects primary point to an intermittent problem you haven't actually found yet, commonly a heartbeat interface sharing a switch with other high-bandwidth traffic that occasionally saturates the port, or a failing SFP/transceiver that passes basic link tests but drops packets under load. Don't close this out until you've watched the packet loss counters under real traffic conditions, not just at idle.

Downstream network symptoms, duplicate MAC or IP warnings, ARP flapping alerts, occasionally outlive the FortiGate-side fix because adjacent switches and routers cached the duplicated virtual MAC address during the split window. If L2 symptoms persist after both units report a clean, single-primary HA status, the fix is clearing stale ARP and MAC table entries on the affected switches, not further troubleshooting the FortiGate cluster itself. Finally, watch for a forced sync that silently overwrote a legitimate change made during the split; if something that worked before the split-brain event stops working right after your resync, check whether it depended on a configuration change that existed only on the unit whose config you discarded.

Escalate to Fortinet support rather than continuing to iterate on the CLI when any of the following are true: configuration checksums still fail to converge after you've fixed the heartbeat link, confirmed matching firmware on both units, and run a forced sync at least twice; the cluster re-enters split-brain repeatedly with no identifiable heartbeat root cause after a genuine physical-layer investigation, which can indicate a hardware fault in a heartbeat NIC or a deeper HA process issue rather than a cabling problem; or the two units are running mismatched FortiOS firmware and you cannot safely bring them to a common version without a maintenance window, since HA diagnostics are not fully reliable across differing builds and this needs to be resolved through planned change management rather than live troubleshooting.

Also escalate immediately, without attempting further live fixes, if the cluster is actively passing production traffic and you cannot get a safe maintenance window to force a resync; in that case collect diagnose sys ha showcsum output from both units, diagnose sys ha history read, and the heartbeat packet loss counters first, and hand that evidence to Fortinet support rather than experimenting further on a live, traffic-carrying cluster.