ADC

View PDF

Optimize Citrix ADC VPX performance on VMware ESX, Linux KVM, and Citrix Hypervisors

September 29, 2021

Contributed by:

C S

The Citrix ADC VPX performance greatly varies depending on the hypervisor, allocated system resources, and the host configurations. To achieve the desired performance, first follow the recommendations in the VPX data sheet, and then further optimize it using the best practices provided in this document.

Citrix ADC VPX instance on VMware ESX hypervisors

This section contains details of configurable options and settings, and other suggestions that help you achieve optimal performance of Citrix ADC VPX instance on VMware ESX hypervisors.

Recommended configuration on ESX hosts

To achieve high performance for VPX with E1000, VMXNET3, SR-IOV, and PCI passthrough network interfaces, follow these recommendations:

The total number of virtual CPUs (vCPUs) provisioned on the ESX host must be less than or equal to the total number of physical CPUs (pCPUs) on the ESX host.
Non-uniform Memory Access (NUMA) affinity and CPU affinity must be set for the ESX host to achieve good results.
– To find the NUMA affinity of a Vmnic, log in to the host locally or remotely, and type:
```
#vsish -e get /net/pNics/vmnic7/properties | grep NUMA Device NUMA Node: 0 
```
- To set NUMA and vCPU affinity for a VM, seeVMware documentation.

Citrix ADC VPX with E1000 network interfaces

Perform the following settings on the VMware ESX host:

On the VMware ESX host, create two vNICs from one pNIC vSwitch. Multiple vNICs create multiple Rx threads in the ESX host. This increases the Rx throughput of the pNIC interface.
Enable VLANs on the vSwitch port group level for each vNIC that you have created.
To increase vNIC transmit (Tx) throughput, use a separate Tx thread in the ESX host per vNIC. Use the following ESX command:
- For ESX version 5.5:
```
esxcli system settings advanced set –o /Net/NetTxWorldlet –i 
```
- For ESX version 6.0 onwards:
```
esxcli system settings advanced set -o /Net/NetVMTxType –i 1 
```
为了进一步增加vNIC Tx吞吐量,使用separate Tx completion thread and Rx threads per device (NIC) queue. Use the following ESX command:
```
esxcli system settings advanced set -o /Net/NetNetqRxQueueFeatPairEnable -i 0 
```

Note:

Make sure that you reboot the VMware ESX host to apply the updated settings.

Two vNICs per pNIC deployment

The following is a sample topology and configuration commands for theTwo vNICs per pNICmodel of deployment that delivers better network performance.

Two vNICs per pNIC deployment

Citrix ADC VPX sample configuration:

To achieve the deployment shown in the preceding sample topology, perform the following configuration on the Citrix ADC VPX instance:

On the client side, bind the SNIP (1.1.1.2) to network interface 1/1 and enable the VLAN tag mode.
```
bind vlan 2 -ifnum 1/1 –tagged bind vlan 2 -IPAddress 1.1.1.2 255.255.255.0 
```
On the server side, bind the SNIP (2.2.2.2) to network interface 1/1 and enable the VLAN tag mode.
```
bind vlan 3 -ifnum 1/2 –tagged bind vlan 3 -IPAddress 2.2.2.2 255.255.255.0 
```

Add an HTTP virtual server (1.1.1.100) and bind it to a service (2.2.2.100).

                add lb vserver v1 HTTP 1.1.1.100 80 -persistenceType NONE -Listenpolicy None -cltTimeout 180 add service s1 2.2.2.100 HTTP 80 -gslb NONE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -sp ON -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO bind lb vserver v1 s1 
               

Note:

Make sure that you include the following two entries in the route table:

1.1.1.0/24 subnet with gateway pointing to SNIP 1.1.1.2

2.2.2.0/24 subnet with gateway pointing to SNIP 2.2.2.2

Citrix ADC VPX with VMXNET3 network interfaces

To achieve high performance for VPX with VMXNET3 network interfaces, do the following settings on the VMware ESX host:

Create two vNICs from one pNIC vSwitch. Multiple vNICs create multiple Rx threads in the ESX host. This increases the Rx throughput of the pNIC interface.
Enable VLANs on the vSwitch port group level for each vNIC that you have created.
To increase vNIC transmit (Tx) throughput, use a separate Tx thread in the ESX host per vNIC. Use the following ESX commands:
- For ESX version 5.5:
```
esxcli system settings advanced set –o /Net/NetTxWorldlet –i 
```
- For ESX version 6.0 onwards:
```
esxcli system settings advanced set -o /Net/NetVMTxType –i 1 
```

On the VMware ESX host, perform the following configuration:

On the VMware ESX host, create two vNICs from 1 pNIC vSwitch. Multiple vNICs create multiple Tx and Rx threads in the ESX host. This increases the Tx and Rx throughput of the pNIC interface.
Enable VLANs on the vSwitch port group level for each vNIC that you have created.
To increase Tx throughput of a vNIC, use a separate Tx completion thread and Rx threads per device (NIC) queue. Use the following command:
```
esxcli system settings advanced set -o /Net/NetNetqRxQueueFeatPairEnable -i 0 
```
配置ure a VM to use one transmit thread per vNIC, by adding the following setting to the VM’s configuration:
```
ethernetX.ctxPerDev = "1" 
```

For more information, seeBest Practices for Performance Tuning of Telco and NFV Workloads in vSphere

Note:

Make sure that you reboot the VMware ESX host to apply the updated settings.

You can configure VMXNET3 as aTwo vNICs per pNICdeployment. For more information, seeTwo vNICs per pNIC deployment.

Citrix ADC VPX with SR-IOV and PCI passthrough network interfaces

To achieve high performance for VPX with SR-IOV and PCI passthrough network interfaces, seeRecommended configuration on ESX hosts.

Citrix ADC VPX instance on Linux-KVM platform

This section contains details of configurable options and settings, and other suggestions that help you achieve optimal performance of Citrix ADC VPX instance on Linux-KVM platform.

性能设置s for KVM

Perform the following settings on the KVM host:

Find the NUMA domain of the NIC using thelstopocommand:

Make sure that memory for the VPX and the CPU is pinned to the same location. In the following output, the 10G NIC “ens2” is tied to NUMA domain #1.

NUMA domain #1

Allocate the VPX memory from the NUMA domain.

Thenumactlcommand indicates the NUMA domain from which the memory is allocated. In the following output, around 10 GB RAM is allocated from NUMA node #0.

NUMA node #0

To change the NUMA node mapping, follow these steps.

Edit the .xml of the VPX on the host.
```
/etc/libvirt/qemu/.xml 
```
Add the following tag:
```
   This is the NUMA domain name  
```
Shut down the VPX.
Run the following command:
```
virsh define /etc/libvirt/qemu/.xml 
```
This command updates the configuration information for the VM with the NUMA node mappings.
Power on the VPX. Then check thenumactl –hardwarecommand output on the host to see the updated memory allocations for the VPX.

Pin vCPUs of VPX to physical cores.

To view the vCPU to pCPU mappings of a VPX, type the following command
```
virsh vcpupin  
```
The vCPUs 0–4 are mapped to physical cores 8–11.
To view the current pCPU usage, type the following command:
```
mpstat -P ALL 5 
```
In this output, 8 is management CPU, and 9–11 are packet engines.
To change the vCPU to pCPU pinning, there are two options.
- Change it at runtime after the VPX boots up using the following command:
```
virsh vcpupin    virsh vcpupin NetScaler-VPX-XML 0 8 virsh vcpupin NetScaler-VPX-XML 1 9 virsh vcpupin NetScaler-VPX-XML 2 10 virsh vcpupin NetScaler-VPX-XML 3 11 
```
- To make static changes to the VPX, edit the.xmlfile as before with the following tags:
  1. Edit the .xml file of the VPX on the host
    /etc/libvirt/qemu/.xml
  2. Add the following tag:
    4
  3. Shut down the VPX.
  4. Update the configuration information for the VM with the NUMA node mappings using the following command:
    virsh define /etc/libvirt/qemu/ .xml
  5. Power on the VPX. Then check thevirsh vcpupin command output on the host to see the updated CPU pinning.

Eliminate host interrupt overhead.

Detect VM_EXITS using thekvm_statcommand.
At the hypervisor level, host interrupts are mapped to the same pCPUs on which the vCPUs of the VPX are pinned. This might cause vCPUs on the VPX to get kicked out periodically.
To find the VM exits done by VMs running the host, use thekvm_statcommand.
```
[root@localhost ~]# kvm_stat -1 | grep EXTERNAL kvm_exit(EXTERNAL_INTERRUPT) 1728349 27738 [root@localhost ~]# 
```
A higher value in the order of 1+M indicates an issue.
If a single VM is present, the expected value is 30–100 K. Anything more than that can indicate that there are one or more host interrupt vectors mapped to the same pCPU.
Detect host interrupts and migrate host interrupts.
When you run theconcatenatecommand for the “/proc/interrupts” file, it displays all the host interrupt mappings. If one or more active IRQs map to the same pCPU, its corresponding counter increments.
Move any interrupts that overlap with your Citrix ADC VPX’s pCPUs to unused pCPUs:
```
echo 0000000f > /proc/irq/55/smp_affinity 0000000f - - > it is a bitmap, LSBs indicates that IRQ 55 can only be scheduled on pCPUs 0 – 3 
```
Disable IRQ balance.
Disable IRQ balance daemon, so that no rescheduling happens on the fly.
```
service irqbalance stop service irqbalance show - To check the status service irqbalance start - Enable if needed 
```
Make sure you run thekvm_statcommand to ensure that there are not many counters.

Citrix ADC VPX with PV network interfaces

You can configure para-virtualization (PV), SR-IOV, and PCIe passthrough network interfaces as aTwo vNICs per pNICdeployment. For more information, seeTwo vNICs per pNIC deployment.

For optimal performance of PV (virtio) interfaces, follow these steps:

Identify the NUMA domain to which the PCIe slot/NIC is tied to.
The Memory and vCPU for the VPX must be pinned to the same NUMA domain.
Vhost thread must be bound to the CPUs in the same NUMA domain.

Bind the virtual host threads to the corresponding CPUs:

Once the traffic is started, run thetopcommand on the host.
Identify the virtual host process (named asvhost-) affinity.
Bind the vHost processes to the physical cores in the NUMA domain identified earlier using the following command:
```
taskset –pc   
```
Example:
```
taskset –pc 12 29838 
```
The processor cores corresponding to the NUMA domain can be identified with the following command:
```
(root@localhost ~) # virsh能力| grep cpu                        
```

Bind the QEMU process to the corresponding physical core:

Identify the physical cores on which the QEMU process is running. For more information, see the preceding output.
Bind the QEMU process to the same physical cores to which you bind the vCPUs, using the following command:
```
taskset –pc 8-11 29824 
```

Citrix ADC VPX with SR-IOV and Fortville PCIe passthrough network interfaces

For optimal performance of the SR-IOV and Fortville PCIe passthrough network interfaces, follow these steps:

Identify the NUMA domain to which the PCIe slot/NIC is tied to.
The Memory and vCPU for the VPX must be pinned to the same NUMA domain.

Sample VPX XML file for vCPU and memory pinning for Linux KVM:

 NetScaler-VPX 138f7782-1cd3-484b-8b6d-7604f35b14f4 8097152 8097152 4

Citrix ADC VPX instance on Citrix Hypervisors

This section contains details of configurable options and settings, and other suggestions that help you achieve optimal performance of Citrix ADC VPX instance on Citrix Hypervisors.

性能设置s for Citrix Hypervisors

Find the NUMA domain of the NIC using the “xl” command:

xl info -n

Pin vCPUs of VPX to physical cores.

xl vcpu-pin

Check binding of vCPUs.

xl vcpu-list

Allocate more than 8 vCPUs to Citrix ADC VMs.

For configuring more than 8 vCPUs, run the following commands from the Citrix Hypervisor console:

              xe vm-param-set uuid=your_vms_uuid VCPUs-max=16 xe vm-param-set uuid=your_vms_uuid VCPUs-at-startup=16 
             

Citrix ADC VPX with SR-IOV network interfaces

For optimal performance of the SR-IOV network interfaces, follow these steps:

识别NUMA作为PCIe插槽或域NIC is tied to.
Pin the Memory and vCPU for the VPX to the same NUMA domain.
绑定域为0 vCPU to the remaining CPU.

Citrix ADC VPX with para-virtualized interfaces

For optimal performance, two vNICs per pNIC and one vNIC per pNIC configurations are advised, as in other PV environments.

To achieve optimal performance of para-virtualized (netfront) interfaces, follow these steps:

识别NUMA作为PCIe插槽或域NIC is tied to.
Pin the memory and vCPU for the VPX to the same NUMA domain.
绑定域为0 vCPU to the remaining CPU of the same NUMA domain.
Pin host Rx/Tx threads of vNIC to Domain-0 vCPUs.

Pin host threads to Domain-0 vCPUs:

Find Xen-ID of the VPX by using thexl listcommand on the Citrix Hypervisor host shell.
Identify host threads by using the following command:
```
ps -ax | grep vif  
```
In the following example, these values indicate:
- vif5.0- The threads for first interface allocated to VPX in XenCenter (management interface).
- vif5.1- The threads for second interface assigned to VPX and so on.
Pin the threads to Domain-0 vCPUs using the following command:
```
taskset –pc   
```
Example:
```
taskset -pc 1 29189 
```

The official version of this content is in English. Some of the Cloud Software Group documentation content is machine translated for your convenience only. Cloud Software Group has no control over machine-translated content, which may contain errors, inaccuracies or unsuitable language. No warranty of any kind, either expressed or implied, is made as to the accuracy, reliability, suitability, or correctness of any translations made from the English original into any other language, or that your Cloud Software Group product or service conforms to any machine translated content, and any warranty provided under the applicable end user license agreement or terms of service, or any other agreement with Cloud Software Group, that the product or service conforms with any documentation shall not apply to the extent that such documentation has been machine translated. Cloud Software Group will not be held responsible for any damage or issues that may arise from using machine-translated content.

Optimize Citrix ADC VPX performance on VMware ESX, Linux KVM, and Citrix Hypervisors

September 29, 2021

Contributed by:

C S

September 29, 2021

Contributed by:

C S

Optimize Citrix ADC VPX performance on VMware ESX, Linux KVM, and Citrix Hypervisors

Citrix ADC VPX instance on VMware ESX hypervisors

Recommended configuration on ESX hosts

Citrix ADC VPX with E1000 network interfaces

Two vNICs per pNIC deployment

Citrix ADC VPX with VMXNET3 network interfaces

Citrix ADC VPX with SR-IOV and PCI passthrough network interfaces

Citrix ADC VPX instance on Linux-KVM platform

性能设置s for KVM

Citrix ADC VPX with PV network interfaces

Citrix ADC VPX with SR-IOV and Fortville PCIe passthrough network interfaces

Citrix ADC VPX instance on Citrix Hypervisors

性能设置s for Citrix Hypervisors

Citrix ADC VPX with SR-IOV network interfaces

Citrix ADC VPX with para-virtualized interfaces

In this article