F5 BIG-IP Device Service Clustering: Fixing a Cluster Stuck 'Out of Sync'

A device group's sync status reflects whether every member holds the same configuration. F5's own documentation is direct about what a yellow status means: one or more devices in the group has recent configuration changes that haven't yet propagated to the others. In the ordinary case, that's just timing, someone made a change on one device and either automatic sync hasn't run yet or manual sync hasn't been triggered, and it clears on its own or with a single manual sync. A persistent version of that status, one that doesn't clear after a sync attempt, has a smaller set of likely causes.

The first is a genuine device-trust break. BIG-IP devices in a DSC group communicate configuration state over an iQuery channel that depends on an established trust relationship between them. If that trust relationship is damaged, commonly after a device is rebuilt, re-licensed, has its management IP changed, or has a certificate expire, the devices can no longer reliably exchange sync data even if the network path between them is otherwise fine. F5's documentation describes this failure mode explicitly as an 'Error in trust domain' condition, distinct from ordinary configuration drift.

The second, and the one most likely to be misread as a stuck sync rather than a hard block, is a software-version mismatch. F5 states plainly that to synchronize configuration data among device group members, all members must be running the same version of BIG-IP system software. If one device was upgraded and its peer wasn't, or an upgrade completed on one unit but rolled back or failed on the other, config-sync does not partially succeed or silently skip incompatible objects. It refuses to sync, and the group sits at 'Changes Pending' indefinitely no matter how many times someone forces it, because the underlying operation is blocked at a level above ordinary configuration comparison.

A third, less common cause is a communication path problem that isn't a trust break at all: the iQuery channel between devices can also be disrupted by a network path issue, a firewall change between the devices' self IPs, or a management-plane outage, producing a 'Disconnected' status that looks similar to a trust failure but has a different fix. Distinguishing between these three (ordinary drift, trust break, version mismatch, or connectivity loss) is the entire point of the diagnostic step, because the fix for each is different and the wrong fix can make things worse.

Start in the GUI under Device Management > Overview, which shows the sync status for the device group and, expanded, the status of each individual member. The status colors and labels are specific and worth reading literally rather than glancing past: 'In Sync' (green) means every device holds the current configuration. 'Changes Pending' (yellow) means unsynced changes exist somewhere in the group. 'Awaiting Initial Sync' means a device has never completed a first sync with the group. 'Disconnected' means the iQuery channel between devices has been terminated or disrupted, typically from a trust domain problem or a network access issue. 'Not All Devices Synced' and 'Sync Failure' (the latter specifically indicating a validation error rejected an attempted sync) are both signals that something is actively blocking synchronization rather than it simply being overdue.

Before you touch sync in either direction, confirm software-version parity across every device in the group. This is the single fact that determines whether config-sync can succeed at all, so check it first. From the command line, tmsh show sys version on each device in the group reports the installed BIG-IP version and build; run it on every member and compare output directly rather than trusting change records, since an upgrade that was documented as completed can still have failed or rolled back on one unit. If versions differ, you already have your root cause and can stop chasing trust or connectivity issues until the versions are aligned.

If versions match, check device-trust state next. In the GUI, Device Management > Device Trust shows the trust domain membership and certificate status for each device; a device shown outside the expected trust domain, or with a certificate error, confirms a trust break rather than ordinary drift. From tmsh, tmsh show cm device lists the devices BIG-IP currently recognizes and their state, and tmsh show cm device-group (with the device group name) reports the group's configured members and sync-leader settings; a device missing from the list it should be in, or showing as unreachable, points to trust or connectivity rather than a simple pending change. tmsh show cm sync-status surfaces the same status detail available in the GUI at the command line, useful for scripting or when GUI access to the affected device isn't available.

Only once you've confirmed matching versions and intact trust should you treat this as ordinary sync drift and move on to picking a sync direction. If you found a version mismatch or a trust error, the fix is different (and comes first) from forcing a sync, covered below.

If diagnostics turned up a version mismatch, do not attempt to force a config-sync as a workaround. It will not partially succeed, and repeated attempts waste a maintenance window without changing the outcome. The fix is bringing every device in the group to the same BIG-IP version through a planned upgrade or rollback, following F5's documented upgrade path for your platform. Only after tmsh show sys version reports matching output across every member should you move on to sync.

If diagnostics turned up a broken trust relationship instead, the fix is resetting device trust rather than forcing sync against a channel that can't reliably carry it. F5's documentation covers this under Device Management > Device Trust: remove the affected device from the trust domain and re-add it, which reestablishes the certificate-based trust relationship the iQuery channel depends on. Do this deliberately and one device at a time in a group larger than two, confirming trust is restored before moving to the next, rather than resetting trust across the whole group at once.

With versions aligned and trust confirmed intact, the remaining step is picking which device is the safe sync source and forcing sync in that direction. F5 describes this as a directional choice: syncing 'to group' pushes the configuration of the device you're currently working from out to every other member, while syncing 'from device' (sometimes described as a pull) pulls the most recent configuration from a specified peer device onto the one you're working from. These are opposite operations, and running the wrong one overwrites the device that actually holds the configuration you need to keep. Before forcing anything, identify which device has the changes you want to preserve, typically the one where a legitimate recent change was made and never propagated, and confirm that against your own change log or recent configuration backups rather than assuming the currently active device is automatically correct.

From the GUI, the sync direction is chosen directly on the Device Management > Overview page by selecting the source device and choosing 'Sync Device to Group' or the equivalent pull action. From tmsh, tmsh run cm config-sync to-group <device-group-name> pushes the local device's configuration to the rest of the named group, and tmsh run cm config-sync from-device <device-name> to-group <device-group-name> pulls configuration from the named peer device into the group instead. Run whichever direction matches the source device you've already confirmed, not whichever one is more convenient from where you happen to be logged in.

After a forced sync, don't rely on the initial status change alone; refresh the Device Management > Overview page and confirm every device in the group now shows 'In Sync,' not just the two devices most recently involved in the sync operation, since a group larger than two can have one member catch up while another remains behind. Cross-check with tmsh show cm sync-status on more than one device in the group, since a status read from only the device you just synced from can look clean even if a different member is still lagging.

Spot-check actual configuration objects, not just the status indicator. Pick a handful of virtual servers, pools, or iRules that you know were part of the pending change and confirm they now match across devices, either through the GUI or with a targeted tmsh list comparison. The sync status reflects F5's own internal comparison of configuration state, which is generally reliable, but confirming the specific objects that mattered gives you direct evidence rather than trusting the summary indicator alone, particularly right after resolving a trust or version issue where the underlying mechanism was broken until minutes ago.

Finally, if you had to reset device trust as part of the fix, verify HA failover itself still works as expected in a controlled window rather than assuming a clean sync status means the whole DSC relationship is healthy. A trust reset that restores config-sync doesn't automatically guarantee that failover-triggering health monitors and connection mirroring are also fully reestablished; a brief, planned failover test during a maintenance window is the only way to confirm that.

A sync that reports success but reverts to 'Changes Pending' within minutes usually means something outside your manual process is still writing configuration to only one device, commonly a centralized management tool, an orchestration script, or an administrator with an open session making a change directly on one unit's management IP rather than through a workflow that accounts for the device group. Check for other active sessions and automation touching the device group before assuming the sync mechanism itself is unreliable.

A device-trust reset that appears to succeed in the GUI but sync still won't complete often points to a certificate or time-sync issue rather than trust domain membership itself; BIG-IP devices rely on certificate validity for the trust relationship, and a significant clock skew between devices can cause certificate validation to fail in ways that look like a trust problem even after trust was reset. Check NTP configuration and current time on both devices if a trust reset doesn't resolve the issue on its own.

A version mismatch that was supposedly resolved by an upgrade but config-sync still refuses to run usually means the upgrade didn't fully complete on one unit, active modules or a boot location the device isn't currently running from can report a mismatched version even though the visible dashboard suggests otherwise. Re-run tmsh show sys version after any upgrade and confirm the currently active boot location matches what you intended, not just that an upgrade process reported success.

And the failure mode with the highest cost: an admin forces a sync in the wrong direction, overwriting the device that held the correct, more recent configuration, and doesn't discover it until a failover happens later and traffic starts hitting objects that no longer exist or were reverted. There is no undo for a completed config-sync beyond restoring from a configuration backup (a UCS archive) taken before the sync, which is why confirming the sync direction against an actual record of what changed, not a guess about which device seems more likely to be current, is worth the extra few minutes every time.

Escalate to F5 Support rather than continuing to force syncs when any of the following are true: config-sync continues to fail with a validation error after you've confirmed matching software versions and reset device trust, since a persistent 'Sync Failure' status under those conditions usually indicates a deeper configuration corruption issue that needs vendor-level diagnostics rather than another manual sync attempt; the device group won't reach 'In Sync' after two or more careful, direction-confirmed sync attempts and you've ruled out version mismatch, trust break, and active automation still writing to one device; or you're facing a version mismatch that can't be resolved through a straightforward upgrade or rollback, for example when the only available upgrade path for one device isn't validated against modules or configuration in active use on the other.

Also escalate before attempting any further forced sync if you're not confident which device holds the correct configuration and no reliable change log or recent UCS backup exists to confirm it; guessing wrong in a live, traffic-carrying pair is the single most damaging mistake in this workflow, and it's worth F5 Support's time to help identify the safe source device from configuration timestamps and internal diagnostics before you commit to a direction. Take a UCS backup of both devices before opening the case and before making any further changes, so a rollback path exists regardless of what the case turns up.