Field formats check
The Field Formats check verifies the data that users send to your websites in web forms. It examines both the length and type of data to ensure that it is appropriate for the form field in which it appears. If the Web App Firewall detects inappropriate web form data in a user request, it blocks the request.
By preventing an attacker from sending inappropriate web form data to your website, the Field Formats check prevents certain types of attacks on your website and database servers. For example, if a particular field expects the user to enter a phone number, the Field Formats check examines the user-submitted input to ensure that the data matches the format of a phone number. If a particular field expects a first name, the Field Formats check ensures that the data in that field is of a type and length appropriate for a first name. It does the same thing for each form field that you configure it to protect.
This check applies to HTML requests only. It does not apply to XML requests. You can configure Field Format Checks in HTML profiles or Web 2.0 profiles to inspect HTML payload for protecting your applications. The Web App Firewall also supports Field Format Check protection for Google Web Toolkit (GWT) applications.
The Field Formats check requires that you enable one or more actions. The Web App Firewall examines the submitted inputs and applies the specified actions.
Note
Field format rules are tightening rules. Adding them to relaxation list from learned data acts as a blocking rule.
To relax field format rules, please remove particular “fieldname” from the fieldformat relaxations list.
You have the option to set the default field formats to specify Field Type and the minimum and maximum length of data expected in each form field on each web form that you want to protect. You can deploy relaxation rules to configure a Field Format for an individual field of a specific form. Multiple rules can be added to specify the field name, the action URL, and Field Formats. Specify Field Formats to accept different types of inputs in different form fields. The learning feature can provide recommendations for the relaxation rules.
Field Format Actions—You can enable Block, Log, Stats, and Learn actions. At least one of these actions must be enabled to engage the Field Format Check protection.
- Block.如果你启用块,块行动是触发ed if the input does not conform to the specified Field Format. If a rule was configured for the target field, the input is checked against the specified rule. Otherwise, it is checked against the default field format specification. Any mismatch in the Field Type or min/max length specification results in blocking the request.
- Log.If you enable the log feature, the Field Format check generates log messages indicating the actions that it takes. You can monitor the logs to determine whether responses to legitimate requests are getting blocked. A large increase in the number of log messages can indicate malicious attempts to launch an attack.
- Stats.If enabled, the stats feature gathers statistics about violations and logs. An unexpected surge in the stats counter might indicate that your application is under attack, or you might have to revisit the configuration to see if the specified field format is too restrictive.
- Learn.If you are not sure which Field Types or minimum and maximum length values might be ideally suited for your application, you can use the learn feature to generate recommendations based on the learned data. The Web App Firewall learning engine monitors the traffic and provides field format recommendations based on the observed values. To get optimal benefit without compromising performance, you might want to enable the learn option for a short time to get a representative sample of the rules, and then deploy the rules and disable learning. Note: The Web App Firewall’s learning engine can distinguish only the first 128 bytes of the name. If a form has multiple fields with names that match for the first 128 bytes, the learning engine might not be able to distinguish between them. Similarly, the deployed relaxation rule might inadvertently relax all such fields.
Default Field Format—In addition to configuring the actions, you can configure the default Field Format to specify the type of data expected in all the form fields for your application. A Field Type can be selected as the Field Format type. Minimum length and Maximum length parameters can be used to specify the length of the allowed inputs. As an alternative to Field Types, you can use Character Maps to specify what’s allowed in a field (except in cluster deployments).
Field Type—Field Types are named expression to which you assign assigned priority values. Field Type expressions specify the allowed inputs and are matched against the submitted data to determine whether the received values are consistent with the allowed values. The Field Types are checked in the order of their priority numbers. A lower number indicates a higher priority. The Web App Firewall gives you the option to add your own Field Types and assign them the priorities you want. The priority value can range from 0 through 64000. The following built-in Field Types are provided to help simplify the configuration process:
> sh appfw fieldtype 1)名称:整数正则表达式:“(+ - ^]?[0-9]+$" Priority: 30 Comment: Integer Builtin: IMMUTABLE 2) Name: alpha Regex: "^[a-zA-Z]+$" Priority: 40 Comment: "Alpha characters" Builtin: IMMUTABLE 3) Name: alphanum Regex: "^[a-zA-Z0-9]+$" Priority: 50 Comment: "Alpha-numeric characters" Builtin: IMMUTABLE 4) Name: nohtml Regex: "^[^&<>]*$" Priority: 60 Comment: "Not HTML" Builtin: IMMUTABLE 5) Name: any Regex: "^.*$" Priority: 70 Comment: Anything Builtin: IMMUTABLE Done >
Note:The built-in Field Types are IMMUTABLE. They cannot be modified or removed. Any Field Types that you add are MODIFIABLE. You can edit them or remove them.
Configuring a Field Type as a default Field Format might be useful when you have a PCRE expression that can identify the valid inputs in all or most of the form fields for your application and exclude the invalid inputs. For example, if all the inputs in your application forms are expected to contain only numbers and letters, you might want to use the built-in Field Type alphanum as the default Field Type. Any non-alphanumeric character such as a backslash () or semicolon ; in the input will trigger a violation. You can also add your own customized Field Types and use them to configure default Field Formats. For example, if you want to make the lowercase “x”, “y”, and “z” the only allowed alpha characters, you can configure a customized Field Type with regular expression “^[x-z]+$”. You can assign it a higher priority (lower priority number) then the built-in Field Types and use it as the default Field Type.
Minimum Length— The default minimum data length assigned to form fields in web forms that do not have an explicit setting. This parameter is set to 0 by default, which allows the user to leave the field blank. Any higher setting forces users to fill in the field.
Caution: If the minimum length value is 0 but the Field Type is integer, alpha, or alphanum, a request is blocked if any input field is left empty, despite the minimum length setting. That is because the regEx for these Field Types contains a + character, which means one or more characters. Distinguishing an integer from an alpha character requires at least one character.
Maximum Length—The default maximum data length assigned to form fields in web forms that do not have an explicit setting. This parameter is set to 65535 by default.
Note:Characters vs bytes. The minimum and the maximum lengths for the field formats represent the number of bytes, not the number of characters. Languages that have greater than one-byte character representation can cause the limit to be exceeded with fewer characters than the number configured for the maximum value. For example, with double-byte character representation, the maximum value of 9 allows no more than 4 characters.Tip:The GUI allows you to cut and paste UTF-8 characters directly into the GUI without having to convert them to hex.
Character Maps: In addition to recommending the Field Types, the Web App Firewall learning engine offers you an additional option, Use Character Maps, to deploy the Format Check rules. A Character Map is a set of all the characters allowed in a particular form field. You can fine tune the Field format specification to allow or disallow specific characters by using Character Maps. A separate Character Map is generated for each form field. The alpha and numeric characters are treated differently in Character Maps. If any alpha character is seen in the input, all alpha characters [A-Za-z] will be allowed by the recommended PCRE expression in the Character Map. Similarly, if any digit is included, all digits [0-9] will be allowed. Non-printable characters are specified by using the x construct. Only single Byte characters with values between 0-255 are considered for Character Map recommendations.
A Character Map can be more specific than the corresponding Field Type recommendation. In some situations, Character Maps might be a better option, because they give you tighter control over the set of characters allowed as inputs. The deployed character maps are displayed as strings that start with prefix “CM” followed by digits. The priority for the Character Maps starts at 10000. As with user-added Field Types, you can add, edit or remove a Character Map. Character Maps that are currently used in deployed rules cannot be modified or removed.
Note: Character Maps are not supported in cluster deployments.
Note
When you add a field formats rule with any built-in Field Type and use character map instead of Field Type and save it, the changes do not get saved and rule still shows with Field Type.
When the character map matches one of the built-in type, the field type is reused instead of creating a new character map.
Using the command line to configure the field format check
命令行接口,您可以使用addappfw fieldtype command to add a new Field Type. You can use either the set appfw profile command or the add appfw profile command to configure the Field Format check and specify which actions to perform. You can use the unset appfw profile command to revert the configured settings back to their defaults. To specify a Field Format rule, use the bind appfw command to bind a Field Type to a form Field and the action URL, along with the minimum and maximum length specifications.
To add, remove or view a Field Type by using the command line:
Use the add command to add a Field Type. You must specify the Name, Regular expression and Priority when adding a new Field Type. You also have the option to add a Comment. You can use the show command to display the configured Field Types. You can also delete a Field Type by using the remove command, which requires only the Name of the Field Type.
add [appfw] fieldType
where:
is a regular expression
< >优先
is a positive_integer
Example:
add fieldtype "Cust_Zipcode" "^[0-9]{5}[-][0-9]{4}$" 4 - show [appfw] fieldType [] Example: sh fieldtype sh appfw fieldtype sh appfw fieldtype cust_zipcode - `rm [appfw] fieldType ` Example: rm fieldtype cusT_ziPcode `rm appfw fieldtype cusT_ziPcode`
Note:As shown above, use of “appfw” in the command is optional. For example, “Add FieldType” or “Add appfw fieldType” are both valid options. The names of the Field Types are case insensitive due to normalization. As shown in the above examples, Cust_Zipcode, cust_zipcode, and cUsT_ziPcode refer to the same Field Type.
To configure a Field Format check by using the command line
Use either the set appfw profile command or the add appfw profile command, as follows:
set appfw profile
-fieldFormatAction (([block] [learn] [log] [stats]) | [none]) set appfw profile
-defaultFieldFormatType set appfw profile
-defaultFieldFormatMinLength set appfw profile
-defaultFieldFormatMaxLength
To configure a Field Format relaxation rule by using the command line
bind appfw profile (-fieldFormat [-fieldFormatMinLength ] [-fieldFormatMaxLength ] [-isRegex ( REGEX | NOTREGEX )])
Example:
bind appfw profile pr_ffc -fieldFormat "login_name" ".*/login.php" integer -fieldformatMinLength 3 -FieldformatMaxlength 6
Using the GUI to configure the field formats security check
In the GUI, you can manage the Field Types. You can also configure the Field Formats security check in the pane for the profile associated with your application.
To add, modify or remove a Field Type using the GUI
- Navigate to Application Firewall node. In the Settings, clickManage Field Typesto display the Configure Application Firewall Field Type dialogue box.
- ClickAddto add a new Field Type. Follow the instructions in this pane and click Create. You can also edit or delete any user-added Field Type if it is currently not being used by a deployed rule.
To add or modify the Field Formats security check by using the GUI
Navigate toApplication Firewall>Profiles, highlight the target profile, and clickEdit.
In theAdvanced Settingspane, clickSecurity Checks.
安全检查表显示当前有限公司nfigured action settings for all the security checks. You have 2 options for configuration:
- If you just want to enable or disableBlock,Log,Stats, andLearnactions for Field Formats, you can select or clear check boxes in the table, clickOK, and then clickSave and Closeto close the Security Check pane.
- If you want to configure additional options for this security check, double click Field Formats, or select the row and click Action Settings, to display the following options forDefault Field Format:
Field Type—Select the Field Type that you want to configure as the default Field Type. You can select the built-in and user-defined Field Types. The deployed Character Maps are also included in the list and can be selected.
Minimum Length—Specify the minimum number of characters that must be in each field. Possible values: 0-65535.
Maximum Length— Specify the maximum number of characters that must be in each field. Possible values: 1-65535.
You can also edit theBlock,Log,StatsandLearnactions in the Field Formats Settings pane.
After making any of the above changes, clickOKto save the changes and return to the Security Checks table. You can proceed to configure other security checks if needed. ClickOKto save all the changes you have made in the Security Checks section, and then clickSave and Closeto close the Security Check pane.
To configure a Field Formats relaxation rule by using the GUI
Navigate toApplication Firewall>Profiles, highlight the target profile, and clickEdit.
In theAdvanced Settingspane, clickRelaxation Rules.The Relaxation Rules table has a Field Formats entry. You can double click, or select this row and click the Edit button, to access the Field Formats Relaxation Rules dialogue. You can performAdd,Edit,删除,Enable, orDisableoperations for relaxation rules.
For a consolidated view of all the relaxation rules, you can highlight the Field Formats row and click Visualizer. The visualizer for deployed relaxations offers you the option to Add a new rule or Edit an existing one. You can also Enable or Disable a group of rules by selecting a node and clicking the corresponding buttons in the relaxation visualizer.
Using the learn feature with the Field Formats Check
When the learn action is enabled, the Web App Firewall learning engine monitors the traffic and learns the triggered violations. You can periodically inspect these learned rules. After due consideration, you can deploy the learned rule as a Field Format relaxation rule.
Field formats learning enhancement—An Web App Firewall learning enhancement was introduced in release 11.0. In the previous releases, once the learned field format recommendation is deployed, the Web App Firewall learning engine stops monitoring the valid requests for the purpose of recommending new rules on the basis of the new data points. This limits the configured security protection, because the learning database does not include any representations of the new data seen in the valid requests processed by the security check.
Violations are no longer coupled with learning. The learning engine learns and makes recommendations for the field formats regardless of the violations. In addition to checking the blocked requests to determine whether the current field format is too restrictive and needs to be relaxed, the learning engine also monitors the allowed requests to determine whether the current field format is too permissive, and allows elevating the security by deploying a more restrictive rule.
Following is a summary of the Field Formats learning behavior:
No Field format is bound—The behavior remains unchanged in this scenario. All learn data is sent to the aslearn engine. The learning engine suggests a field format rule based on the data set.
Field format is bound: In the previous releases, observed data is sent to the aslearn engine only in the case of a violation. The learning engine suggests a field format rule based on the data set. In the 11.0 release, all data is sent to aslearn engine even if no violation is triggered. The learning engine suggests a field format rule based on the entire data set of all received inputs.
Use Case for learning enhancement:
如果最初的字段格式学习规则是基础d on a small sample of data, a few non typical values might result in a recommendation that is too lenient for the target field. The ongoing learning allows the Web App Firewall to observe data points from every request to collect a representative sample for the learned recommendations. This is helpful in further tightening the security to deploy the optimal input format with an adequate range value.
The field format learning makes use of the priority of the Field Types as well as the configured settings of the following learning thresholds:
- FieldFormatMinThreshold—Minimum number of times a specific form field must be observed before a learned relaxation is generated. Default: 1.
- FieldFormatPercentThreshold—Percentage of times a form field matched a particular Field Type, before a learned relaxation is generated. Default: 0.
The field format rule recommendations are based on the following criteria:
- Field Type recommendations—The Field Type recommendations are determined by the assigned priorities of the existing Field Types and the specified Field Format thresholds. The priorities determine the order in which the Field Types are matched against the inputs. A lower number specifies a higher priority. For example, Field Type integer has the higher priority (30) and is therefore evaluated before Field Type alphanum (50). The thresholds determine the number of inputs evaluated to collect a representative sample for the data point. Assigning the right priority to the configured Field Types, and configuring an appropriatelearningsettingvalue for thefieldFormatPercentThresholdandfieldFormatMinThresholdparameters, is essential for getting the correct Field Format recommendation. The Field Type with the highest priority, based on the configured thresholds, is matched first against the inputs. If there is a match, this Field Type is suggested without considering the other Field Types. For example, three default Field Types, integer, alphanum, and any will match if all the inputs contain only numbers. However, integer will be recommended since it has the highest priority.
- Minimum and Maximum length recommendations—Calculations for the minimum and maximum lengths for the Field Format are done independently of the determination for the Field Type. The field format length calculations are based on the average length of all the observed inputs. Half of this calculated average is suggested as the min value, and twice the value of this average is suggested as the max value. The range for the Minimum Length is 0-65535 and the range for the Maximum Length is 1-65535. The configured value for the minimum length cannot exceed the maximum length.
- Handling of space character—The Field Format check counts every space character when checking for the Field Formats length. Leading or trailing spaces are not stripped, and multiple consecutive spaces in the middle of the input string are no longer consolidated to a single space during input processing.
Example to illustrate the Field Format recommendations:
Total requests: 100 Number of Req with Field Type: Int : 22 (22 int values) – 22% Alpha : 44 (44 alpha values) – 44% Alphanum: 14 (14 + 44 + 22 = 80 alphanum values) = 80% noHTML: 10 (80 + 10 = 90 noHTML values) = 90% any : 10 (90 + 10 = 100 any values) = 100% % threshold Suggested Field Type 0-22 int 23-44 alpha 45-80 alphanum 81-90 noHTML 91-100 any
To view or use learned data by using the command line interface
show appfw learningdata FieldFormat rm appfw learningdata -fieldFormat export appfw learningdata FieldFormat
To view or use learned data by using the GUI
Navigate toApplication Firewall>Profiles, highlight the target profile, and clickEdit.
In theAdvanced Settingspane, clickLearned Rules.You can select the Field Formats entry in the Learned Rules table and double-click it to access the learned rules. You can deploy the learned rules or edit a rule before deploying it as a relaxation rule. To discard a rule, you can select it and click theSkipbutton. You can edit only one rule at a time, but you can select multiple rules to deploy or skip.
You also have the option to show a summarized view of the learned relaxations by selecting the Field Formats entry in the Learned Rules table and clicking Visualizer to get a consolidated view of all the learned violations. The visualizer makes it very easy to manage the learned rules. It presents a comprehensive view of the data on one screen and facilitates taking action on a group of rules with one click. The biggest advantage of the visualizer is that it recommends regular expressions to consolidate multiple rules. You can select a subset of these rules, based on the delimiter and Action URL. You can display 25, 50, or 75 rules in the visualizer, by selecting the number from a drop-down list. The visualizer for learned rules offers the option to edit the rules and deploy them as relaxations. Or you can skip the rules to ignore them.
Using the log feature with the field formats check
When the log action is enabled, the Field Formats security check violations are logged in the audit log as APPFW_FIELDFORMAT violations. The Web App Firewall supports both Native and CEF log formats. You can also send the logs to a remote syslog server.
To access the log messages by using the command line
Switch to the shell and tail the ns.logs in the /var/log/ folder to access the log messages pertaining to the Field Formats violations:
Shell
tail -f /var/log/ns.log | grep APPFW_FIELDFORMAT
To access the log messages by using the GUI
The Citrix GUI includes a very useful tool (Syslog Viewer) for analyzing the log messages. You have multiple options for accessing the Syslog Viewer:
Navigate to theApplication Firewall>Profiles, select the target profile, and clickSecurity Checks.Highlight the Field Formats row and clickLogs.When you access the logs directly from theField Formats securitycheck of the profile, it filters out the log messages and displays only the logs pertaining to these security check violations.
You can also access the Syslog Viewer by navigating toCitrix ADC>System>Auditing.In theAudit Messagessection, click theSyslog messageslink to display theSyslog Viewer, which displays all log messages, including other security check violation logs. This is useful for debugging when multiple security check violations might be triggered during request processing.
Navigate toApplication Firewall>policies>Auditing.In theAudit Messagessection, click the Syslog messages link to display the Syslog Viewer, which displays all log messages, including other security check violation logs.
The HTML based Syslog Viewer provides various filter options for selecting only the log messages that are of interest to you. To access Field Formats security check violation log messages, filter by selecting APPFW in the dropdown options for Module. The Event Type displays a rich set of options to further refine your selection. For example, if you select theAPPFW_FIELDFORMATcheck box and click theApplybutton, only log messages pertaining to the Field Formats security check violations appear in the Syslog Viewer.
If you place the cursor in the row for a specific log message, multiple options, such as Module and EventType, appear below the log message. You can select any of these options to highlight the corresponding information in the logs.
Example of a Native format log message when the request is not blocked
Jun 10 22:32:26 10.217.31.98 06/10/2015:22:32:26 GMT ns 0-PPE-0 : default APPFW APPFW_FIELDFORMAT 97 0 : 10.217.253.62 562-PPE0 x1MV+YnNGzQFM3Bsy2wti4bhXio0001 pr_ffc http://aaron.stratum8.net/FFC/login_post.php Field format check failed for field passwd="65568888sz-*_" Example of a CEF format log message when the request is blocked Jun 11 00:03:51 10.217.31.98 CEF:0|Citrix|Citrix ADC|NS11.0|APPFW|APPFW_FIELDFORMAT|6|src=10.217.253.62 spt=27076 method=POST requet=http://aaron.stratum8.net/FFC/maxlen_post.php msg=Field format check failed for field text_area="" cn1=108 cn2=644 cs1=pr_ffc cs2=PPE0 cs3=GaUROfl1Nx1jJTvja5twH5BBqI0000 cs4=ALERT cs5=2015 act=blocked
Statistics for the field formats violations
启用统计数据操作时,corresponding counter for the Field Formats check is incremented when the Web App Firewall takes any action for this security check. The statistics are collected for Rate and Total count for Traffic, Violations, and Logs. The increment of the log counter can vary depending on the configured settings. For example, if the block action is enabled, the request for a page that contains 3 Field Format violations increments the stats counter by one, because the page is blocked as soon as the first Field Formats violation is detected. However, if block is disabled, processing the same request increments the statistics counter for violations and the logs by 3, because each Field Formats violation generates a separate log message.
To display Field Formats statistics by using command line
At the command prompt, type:
sh appfw stats
To display stats for a specific profile, use the following command:
stat appfw profile
To display Field Formats statistics by using GUI
- Navigate toSystem>Security>Application Firewall.
- In the right pane, access the Statistics Link.
- Use the scroll bar to view the statistics about Field Formats violations and logs. The statistics table provides real-time data and is updated every 7 seconds.
Deployment Tip
- Enable Field format actions log, learn and stats.
- After running a representative sample of the traffic to your application, review the learned recommendations.
- If a Field Type is recommended by most of the learned rules, configure that Field Type as the Default Field Type. For minimum and maximum lengths, use the widest range suggested by these rules.
- Deploy rules for other fields for which different Field Types or different minimum/maximum lengths are better suited.
- Enable blocking and disable learning.
- Monitor stats and logs. If a significant number of violations are still being triggered, you might want to review the log messages to confirm that the violations represent malicious requests that must have been blocked. If valid requests are being flagged as violations, you can either edit the configured Field Format rule to further relax it or enable learning again to get recommendations based on the new data points.
Note:You can fine tune your configuration by getting new learning recommendations.
Highlights
Note the following points about the Field Format security check:
- Protection—By configuring optimal field format rules, you can protect against many attacks. For example, if you specify that a field can only have integers, hackers will not be able to launch SQL Injection or cross-site scripting attacks by using this field, because the inputs required to launch such attacks will not meet the configured field format requirement.
- Performance你可以限制最小和最大允许的length for the inputs in the field format rules. This can prevent a malicious user from entering excessively large input strings in an attempt to add processing overhead to the server, or worse, cause the server to dump core because of stack overflow. By limiting the input size, you can shorten the time required for processing legitimate requests.
- Configuring Field Formats—You must enable one of the actions (block, log, stats, learn) to engage the field Format protection. You can also specify the Field format rules to identify the allowed inputs in your form fields.
- Selecting Character Maps vs. Field Types—Both Character Maps and Field Types use regular expressions. However, a Character Map provides a more specific expression by narrowing down the list of allowed characters. For example, for an input such as janedoe@citrix.com, the learning engine might recommend the Field Type nohtml but the Character Map [.@-Za-z] might be more specific, because it narrows down the allowed set of non-alpha characters. The Character Map option allows, in addition to alpha characters, only two non-alpha characters: period (.) and at (@).
- Ongoing Learning—The Web App Firewall monitors and takes into account all the incoming data (violations as well as allowed inputs) to build a learning table for recommending rules. The rules are revised and updated as new incoming data arrives. New field format rules are suggested for a field even if it already has a bound field format rule. If the configured Field Formats are too restrictive and are blocking the valid requests, you can deploy a more relaxed Field Format. Similarly, if the current Field Formats are too generic, you can further refine and tighten the security by deploying a more restrictive Field Format.
- Overwriting Rules如果一个规则已经被部署为一个字段/ URL combination, the GUI allows the user to update the field format. A dialog box asks for confirmation to replace the existing rule. If you are using the command line interface, you have to explicitly unbind the previous binding and then bind the new rule.
- Multiple match—If multiple field formats match a given field name and its action URL, the Web App Firewall arbitrarily selects one of them to apply.
- Buffer boundary—If a field value extends across multiple streaming buffers, and the format for these two parts of the field value is different, a field format corresponding to “any” is sent to the learn database.
- Field Format vs. Field Consistency Check—Both the Field Format check and the Field Consistency check are form-based protection checks. The Field Formats check provides a different type of protection than does the Form Field Consistency check. The Form Field Consistency check verifies that the structure of the web forms returned by users is intact, that data format restrictions configured in the HTML are respected, and that data in hidden fields has not been modified. It can do this without any specific knowledge about your web forms other than what it derives from the web form itself. The Field Formats check verifies that the data in each form field matches the specific formatting restrictions that you configured manually, or that the learning feature generated and you approved. In other words, the Form Field Consistency check enforces general web form security, while the Field Formats check enforces the specific rules for the allowed inputs for your web forms.