Citrix ADC

How to record a packet trace on Citrix ADC

This troubleshooting article explains how an administrator can record a network packet trace using the Citrix ADC GUI.

Points to remember

  • Citrix recommends you to use the recent Wireshark version from the “automated build section” available in the following webpage:http://www.wireshark.org/download/automated.

  • In Citrix ADC version 10.5 or later, to decrypt the capture and ensure ECC (Elliptic Curve Cryptography), Session Reuse and DH parameters are disabled from the virtual server. You must do before you capture a trace.

Record packet trace on NetScaler version 11.1

  1. Navigate toSystem>Diagnosticspage.
  2. click theStart new tracelink in theDiagnosticpage, as shown in the following screenshot.

    Accessing Diagnostic page

  3. Update the packet size to 0 in thePacket sizefield.

    Packet size

  4. ClickStartto start recording the network packet trace.
  5. ClickStop and Downloadto stop recording the network packet trace after the test is complete.

    Stop and Download trace

  6. Select the required file and clickSelectand clickDownload.

    Download packet trace

  7. Open the network packet trace file with the Wireshark utility to display the content of the file.

Record packet trace on NetScaler 10.5 appliance

  1. Navigate toSystem>Diagnosticspage.

    Access diagnostic page

  2. Click theStart new tracelink underTechnical Support Tools as shown in the following screenshot.
  3. Update the packet size to 0 in thePacket Sizefield.

    Packet size

    Note: If appliance headers are not required then select Capture trace in .pcap format.

  4. ClickStartto start recording the network packet trace.
  5. ClickOKto stop recording the network packet trace after the test is complete.

    Stop trace recording

    An nstrace.cap file is generated, which contains the network packet trace.

  6. Highlight the required file and clickDownload.

    Download the file

  7. Specify a destination and save the packet trace.
  8. Open the network packet trace file with the Wireshark utility to display the content of the file.

    Note: Select Decrypted SSL packets (SSLPLAIN) to decrypt the packet trace without the private key.

    Decrypted SSL packets

Capture SSL master keys

在11.0,11.1版本以上option to capture the session keys which is valid for only for that particular session/nstrace and this option can be used if you do not want to share the private key or use SSLPLAIN mode. For more information, seehttps://support.citrix.com/article/CTX135889.

Export Session Keys without sharing Private key

In most of the scenarios the private key is not available or shared. In such scenarios we can suggest exporting theSSL sessionkeys instead of the private key. Read, [How to Export and Use SSL Session Keys to Decrypt SSL Traces Without Sharing the SSL Private Key, seehttps://support.citrix.com/article/CTX135889.

Filters

Also, it is always recommended to add IP based filters while taking traces. The process ensures that you capture only interested traffic which eases your troubleshooting. Adding filters also decreases the load on the appliance while taking traces.

Filter section

Simple IP-based filters are enough to get the right captures. For more information aboutnstracefilters and examples, seeCitrix Documentationpage.

Use case to capture a packet trace with virtual server IP filter (both front-end and back end)

Using a filter of the virtual server IP address and enabling the option “–link” in CLI or selecting the option “Trace filtered connection peer traffic” in GUI (available 10.1 and above), you can capture both the front-end and back-end traffic for the IP address.

start nstrace -size 0 -filter "CONNECTION.IP.EQ(1.1.1.1)" -link ENABLED show nstrace State: RUNNING Scope: LOCAL TraceLocation: "/var/nstrace/24Mar2017_16_00_19/..." Nf: 24 Time: 3600 Size: 0 Mode: TXB NEW_RX Traceformat: NSCAP PerNIC: DISABLED FileName: 24Mar2017_16_00_19 Filter: "CONNECTION.IP.EQ(1.1.1.1)" Link: ENABLED Merge: ONSTOP Doruntimecleanup: ENABLED TraceBuffers: 5000 SkipRPC: DISABLED Capsslkeys: DISABLED InMemoryTrace: DISABLED 

Merge

Capturing cyclic traces

It is always challenging to troubleshoot an intermittent issue. Cyclic tracing is best suited for issues which are intermittent. The traces can be run over a span of few hours or days before the issue occurs. Also, you can use a specific filter and evaluate the size of the trace files that are generated before you run it for a longer time.

Run the following command from the CLI:

start nstrace -nf 60 -time 30 -size 0 This particular trace will create 60 files each of them for 30 sec. This means the files will start getting overwritten after 60 trace files or 30 mins Show nstrace à To check the status of the nstrace Stop nstrace à To stop the nstrace. 

Best Practices

On a unit handling GB of traffic per second, capturing traffic is a very resource intensive process. The impact to resources is mainly in terms of the CPU and the disk space. Disk space impact can be reduced by using filtering expressions. However, the impact on the CPU remains and sometimes causes a slight increase as the appliance now needs to process packets according to the filter before capturing them.

The best practice with regard to tracing is:

  1. The duration for which the trace is run must be as limited as possible when you still ensure the packets of interest are captured.
  2. Schedule the tracing activity to happen at a time when the number of users (and hence the traffic) is greatly reduced, such as during off hours.

More resources

Disable session reuse on virtual server from the GUI

Session reuse is disabled when you capture a trace to complete an SSL handshake in the trace. When it is enabled, you can capture a partial handshake in the trace. Ensure you enable the option after the trace collection. Do not disable an SSL session reuse when the persistence method is sslsession, as it breaks the persistence for existing connections. For more information refer tohttps://support.citrix.com/article/CTX121925.

  1. Open the virtual server and navigate to SSL Parameters.
  2. Disable Enable Session Reuse if enabled.

    Enable session reuse

Disable session reuse on virtual server from the CLI

  1. SSH to the appliance console.
  2. Run the following command to disable DH Param from the virtual server:

    set ssl vserver "vServer_Name" -sessReuse DISABLED

Disable DH parameter on virtual server from the GUI

Refer tohttps://support.citrix.com/article/CTX213335To understand about DH Parameter.

  1. Open the virtual server and navigate to SSL Parameters.
  2. Disable DH Param if enabled.

    SSL Parameters

Disable DH parameter on virtual server from the CLI

  1. SSH to the appliance console.
  2. Run the following command to disable DH Param from the virtual server:

    set ssl vserver "vServer_Name" -dh DISABLED

Disable ECC curve on virtual server from the GUI

ECC curve is disabled to decrypt the captured SSL trace with private key. You must not disable the keys if the related SSL ciphers are used. For more information about the ECC curve, seehttps://support.citrix.com/article/CTX205289

  1. Open the virtual server and navigate to ECC Curve.

    ECC Curve

  2. 如果没有绑定到虚拟服务ECC曲线er then no other action is required.

    No CC Curve

  3. If any ECC Curve is bound to the virtual server then click the ECC Curve and Unbind it from the virtual server.

Disable ECC curve on virtual server from the CLI

  1. SSH to the appliance console.
  2. Run the following command for each ECC Curve bound to the virtual server:

    unbind ssl vserver "vServer_Name" -eccCurveName "ECC_Curve_Name"

How to record a packet trace on Citrix ADC