Helient Systems : ESXi Upgrades Can Break Your Citrix ADC (NetScaler) Virtual Appliances

by Daniel Ruiz
Senior Solutions Architect

With the latest VMware vCenter Server & Cloud Foundation Vulnerabilities many organizations are rushing to upgrade the VMware infrastructure, and as a result breaking the Citrix ADC (NetScaler) Virtual Appliances.

Before preforming an upgrade on your vSphere ESXi infrastructure, it is very important to check the latest Support matrix and usage guidelines from Citrix which details a list of the different hypervisors, features supported, and limitations on a Citrix ADC (NetScaler) VPX instance.

Issue #1:
After upgrading your VMware ESXi hosts, the Citrix ADC virtual appliance(s) can display the error “vmx0: device enable command failed“, which will disable all network services on the appliances upon booting up.

Issue #2:
HA pair reports the error “Unable to connect to primary, please check the network connectivity from secondary to primary” after upgrading the firmware. This problem will disable HA synchronization between the hosts.

Helient Testing:
During Helient’s investigation after performing ESXi upgrades, it was observed that the Citrix ADC VMXNET3 network adapter driver fails immediately after network services are enabled.

In some instances, replacing the virtual network adapter with an E1000 driver will allow the virtual appliances to boot.

*Note however that the MAC address will change if the E1000 driver is installed and will disable all licensed features of the appliance(s) until they are re-licensed.

For more information, please read this article on  how to allocate and apply for a license on your Citrix ADCs.

Helient does not recommend installing the E1000 Driver in production, as it runs an emulated version of the Intel 82545EM Gigabit Ethernet NIC versus the VMXNET 3, which is designed for performance.

For additional information on the different ESXi network adapter drivers, please visit this article from VMware which discusses the different network adapter options available for virtual machines.

Regarding Issue #2, after firmware upgrades, this issue arises, where the Citrix ADC HA synchronization broke when running the Citrix ADCs as an HA pair.

The Citrix ADC HA process relies on a set of services which in turn use specific ports, heartbeats are sent over UDP port 3003 between the nodes, and propagation and synchronization use TCP ports 3008 and 3009 or 3010 and 3011.

After upgrading your Citrix ADC appliances, the secondary appliance in a HA pair can report the error “Unable to connect to primary, please check the network connectivity from secondary to primary“, which means that any changes on the primary node will not be replicated to the secondary; which in the case of a failover, will likely cause a loss of any configuration changes.

The root cause of this is due to the new firmware automatically turning on Secure HA using TCP 3008/3009, and because the older firmware is set up to utilize TLS 1.0 or 1.2, the nodes are unable to communicate to one another.

To fix the problem it will require for all internal services to run TLS 1.2, which will reestablish HA communication.

Please visit how to upgrade and downgrade a Citrix ADC appliance, which provides information about upgrading and downgrading a Citrix ADC appliance (MPX and VPX) firmware by using the Citrix ADC GUI or CLI.

Below is a screenshot of the Support matrix and usage guidelines

If you would like more information or assistance from Helient’s industry leading team of Citrix experts to plan and execute the upgrade, please contact us at service@helient.com.