Infoblox Grid Members Stuck Syncing: Diagnosing Replication and Join Failures

Infoblox Grid communication, the join handshake, ongoing replication, and HA heartbeat between members, all rides over an OpenVPN-based tunnel. That tunnel depends on UDP 2114 and UDP 1194 being open, bidirectionally, between the joining member and the Grid Master. If either port is blocked by an intermediate firewall, ACL, or security group (a common gap when a virtual appliance is provisioned in a cloud VPC or a segmented on-prem network that wasn't updated for the new member), the appliance never completes the VPN key exchange that the join process depends on, and the member sits stuck rather than failing cleanly with an obvious error.

An MTU mismatch on that same OpenVPN tunnel is the second common cause, and it is easy to miss because it does not always block the join outright. The tunnel defaults to an MTU around 1450 bytes. If a network segment in the path, a VPN gateway, a cloud provider's virtual network, or an intermediate hop with a lower supported MTU, can't carry packets that size, path MTU discovery can fail and the appliance may report a PATH-MTU error during the join's key exchange, or complete the join and then stall during the larger data transfers that full replication requires. This produces the specific pattern reported in Infoblox's community forum: a member that connects but never finishes synchronizing, alternating between connecting and synchronizing without ever reaching green.

A subnet mask mismatch between the appliance's own network configuration and what Grid Manager has recorded for that member is a third cause, and it is specific to how the member was provisioned. If the subnet mask (or, for some cloud vNIOS platforms, the gateway) was left unset or entered incorrectly during initial setup, because certain cloud vNIOS images do not auto-populate these fields, the appliance and the Grid Master disagree about the member's own addressing, which breaks the network-level handshake underneath the VPN tunnel before OpenVPN itself is even the issue.

A wrong Member Type is the fourth cause, and it is specific to virtual appliances. Grid Manager needs the member type set to Virtual NIOS for a vNIOS deployment (VMware, KVM, Nutanix, and the major cloud platforms all use this same setting); if the member was added to Grid Manager as a physical Infoblox appliance type by mistake, a common slip when a template or an existing member entry gets cloned during provisioning, the mismatch between what Grid Manager expects and what the appliance actually is will block the join even though ports, MTU, and subnet are all correct. Infoblox's community forum documents this as one of the two leading causes of virtual appliance join failures, alongside a missing or invalid license.

Start in Grid Manager rather than on the member itself. Under Grid, Grid Manager, Members, toggle the Replication Status view to see per-member state: green means the member is synchronized with the Grid Master, yellow means a connection or synchronization is actively in progress, and red means the member is offline, unlicensed, or in a bad state. The panel also shows Send Queue, Last Send, Receive Queue, and Last Receive timestamps for each member; a queue that keeps growing without the Last Send or Last Receive timestamp advancing tells you the tunnel came up but replication traffic isn't actually moving, which points toward an MTU problem rather than a hard port block.

On the network path between the member and the Grid Master, run tcpdump filtered to UDP 2114 and UDP 1194. If you see no traffic at all on those ports in either direction, you're looking at a port block, a firewall, ACL, or security group between the two appliances is dropping the handshake before OpenVPN ever gets a chance to negotiate. If you see traffic on those ports but it consists of packets that are all the same size at regular intervals rather than the varying packet sizes a healthy, actively transferring OpenVPN tunnel produces, that's consistent with a tunnel that's up but not successfully carrying the larger frames the initial data sync needs, which points toward an MTU mismatch. Watch specifically for a PATH-MTU error in the appliance's console output or logs during the join attempt; this is Infoblox's own signal that path MTU discovery failed partway through the key exchange.

For a join that fails outright rather than stalling mid-sync, check the appliance's own console screen rather than relying on Grid Manager. Because the member cannot yet send logs to the Grid Master if it hasn't joined, join failures related to network settings, license validation, or a member type mismatch often only appear on the local console, not centrally. Cross-reference what you see there against the member's configuration in Grid Manager: confirm the subnet mask (and gateway, for cloud vNIOS platforms that don't auto-populate it) matches what the appliance itself is configured with, and confirm the Member Type recorded for that member is Virtual NIOS if it's a virtual appliance rather than a physical Infoblox unit.

If tcpdump shows no traffic at all on UDP 2114 or UDP 1194 between the member and Grid Master, open both ports bidirectionally on every firewall, ACL, or security group between them. Treat this as the first thing to rule out for any new member in a segmented network or a cloud VPC, since a missing rule at provisioning time is the single most common reason a join times out with no other symptom.

If diagnostics point to an MTU mismatch, lower the VPN tunnel MTU on the appliance from its default of around 1450 down toward 1000. Infoblox exposes this as a per-appliance VPN MTU setting specifically for links that can't support the default size and can't otherwise join the Grid; you don't need to touch MTU on every member in the Grid, only the one hitting the problem, and you don't need to guess at a value, work down from 1450 in a maintenance window until the join or sync completes cleanly rather than searching for an exact minimum.

If the appliance's subnet mask or gateway doesn't match what Grid Manager has recorded, correct it on the appliance itself so it matches the addressing Grid Manager expects, and for cloud vNIOS platforms that don't auto-populate these fields, explicitly set both the subnet mask and gateway rather than assuming a default will be applied.

If the Member Type is wrong, edit the member's entry in Grid Manager and change Member Type to Virtual NIOS for any vNIOS deployment; this is a configuration correction on the Grid Master side, not something you fix from the appliance console, and it should be done before you retry the join rather than after, since retrying with the wrong type set will only reproduce the same failure.

After applying a fix, don't treat a join that completes as proof the underlying cause is resolved; confirm the member reaches and holds green in the Replication Status panel through at least one full replication cycle, not just the initial handshake. Watch the Send Queue and Receive Queue values specifically: they should drain to zero (or a small, actively decreasing number) rather than sitting flat, and Last Send and Last Receive should keep advancing over time rather than freezing at the moment the join completed.

Run a second tcpdump capture on UDP 2114 and UDP 1194 after the fix and compare it against your original capture: a healthy tunnel shows varying packet sizes and continuous back-and-forth traffic, not the uniform, regularly spaced packets that indicated a stalled tunnel before. If you changed the VPN tunnel MTU, let the member run through at least one scheduled Grid backup or a manually triggered full sync (not just the lightweight incremental updates) before considering the fix validated, since a larger data transfer is what originally exposed the MTU problem and is the best test that it's actually resolved.

A member that opens both Grid ports cleanly in tcpdump but still fails to join is not a port problem; move directly to checking Member Type and subnet mask configuration rather than re-checking firewall rules a second time.

A member that reaches green briefly after a fix and then drops back to yellow or red under load (during a scheduled backup, a large zone transfer, or a DHCP lease table sync) suggests the MTU value you set is still too high for the path, not that the fix failed outright; lower it further rather than reverting.

A member that fails to join with no traffic visible on either Grid port even after firewall rules are confirmed open often indicates a routing problem rather than a filtering one, verify the member actually has a route to the Grid Master's IP and isn't attempting to reach it over an interface or VLAN that doesn't have connectivity at all.

Repeated join attempts that each fail identically without any of the four causes above being present, correct ports, correct MTU, correct subnet mask, correct Member Type, are the point at which retrying stops being useful and the problem needs escalation rather than another attempt.

Escalate to Infoblox Technical Support when a member fails to join after ports, MTU, subnet mask, and Member Type have all been verified correct and diagnostics don't clearly point to one of these four causes; when the Grid Master itself shows signs of a licensing or capacity issue that's blocking new members rather than the joining appliance being at fault; when a previously healthy, synchronized member suddenly drops to red with no configuration change on either side, which can indicate a Grid Master-side issue rather than a member-side one; or when a fix resolves the join but the member repeatedly loses sync afterward in a pattern that doesn't correlate with any of the causes in this guide. Also escalate before making MTU or Member Type changes on a member that is currently in production actively serving DNS or DHCP, since both changes typically require a rejoin and a brief service interruption that's worth planning as a maintenance window rather than troubleshooting live.