High CPU
Following are some of the functionality and high CPU related debugging issues encoutered and the best practices to follow when working with Web App Firewall:
•Check Policy hits, Bindings, Network configuration, Web App Firewall configuration
Identify misconfiguration
Identifyvserverthat is serving the affected traffic
•Inspect logs in the following log files for security violations and recent configuration changes
/var/log/ns.log
/var/nslog/import.log
/var/nslog/aslearn.log
tail -f /var/log/ns.log | grep APPFW_SIGNATURE_MATCH
Example:
Jun 13 01:11:09 10.217.31.98 CEF:0|Citrix|NetScaler|NS11.0|APPFW| APPFW_SIGNATURE_MATCH|6|src=10.217.253.62 spt=61141 method=GET request= http://aaron.stratum8.net/FFC/wwwboard/passwd.txt msg=Signature violation rule ID 807: web-cgi /wwwboard/passwd.txt access cn1=140 cn2=841 cs1=pr_ffc cs2=PPE0 cs3=OyTgjbXBqcpBFeENKDlde3OkMQ00001 cs4=ALERT cs5=2015 cs6=web-cgi act=not blocked
•Isolate the traffic that is effected
Isolate the profile
Isolate the security check
Isolate the URL, vserver and traffic parameters
•Conditional profile level trace helps identify the traffic and violation records
set appfw profile
-trace ON start nstrace -mode APPFW -size 0
stop nstrace
Note: Ensure that the trace is collected with -size 0 option.
•Check appfw, dht, IP reputation activity counters
nsconmsg -g as_ -g appfwreq_ -g iprep -d current
•Monitor window size for resets in connection
- Appfw sets the window size to 9845 when Citrix ADC resets the connection due to an invalid http message.
Examples:
- Malformed request received - connection reset
- High CPU related issues
- Check data sheets for system limits
- Inspect for cpu usage, appfw, DHT and memory related activity. Monitor appfw sessions
- nsconmsg -g cc_cpu_use -g appfwreq -g as -g dht -g mem_AS_OBJ -g mem_AS_COMPONENT -d current
• Monitor memory allocated and freed from Web App Firewall components and objects during the target time period. It helps in isolating the protection leading to high CPU usage.
- Profiler output
- Observe logs
•Isolate appfw check leading to high CPU
- startURLClosure
- Formfiledconsistency
- CSRF
- Cookie protections
- Referer header check
Ascertain that autoupdate of signatures is not leading to high CPU (Disable to confirm).