URL Categorization
URL Categorization restricts user access to specific websites and website categories. As a subscribed service in collaboration withNetSTAR
, the feature enables enterprise customers to filter web traffic using a commercial categorization database. TheNetSTAR
database has a vast number (billions) of URLs classified into different categories, such as social networking, gambling, adult content, new media, and shopping. In addition to categorization, each URL has a reputation score kept up to date based on the site’s historical risk profile. We can useNetSTAR
data to filter the traffic by configuring advanced policies based on categories, category groups (such as Terrorism, Illegal drugs), or site-reputation scores.
For example, you might block access to dangerous sites, such as sites infected with malware, or selectively restrict access to adult content or entertainment streaming media.
How URL Categorization Works
The following figure shows how the Citrix ADC URL Filtering service is integrated with a commercial URL Categorization database and cloud services for frequent updates.
The components interact as follows:
- Client sends internet bound URL request.
- A Citrix ADC policy attempts to evaluate the request in terms of categorization details (such as category, category group, and site-reputation score) retrieved from the URL categorization database. If the database returns the category details, the process jumps to step 5.
- If the database does not return categorization details, the request is sent to a cloud-based lookup service maintained by a URL categorization vendor. However, the appliance does not wait for a response. Instead, it marks the URL as Uncategorized and jumps to step 5. However, it continues to monitor the cloud query feedback and uses it to update the cache so that future requests can benefit from the cloud lookup.
- The Citrix ADC appliance receives the URL category details (category, category group, and reputation score) from the cloud-based service and stores it in the cloud cache.
- If the policy allows the URL, the request is sent to the origin server. Otherwise, the appliance drops or redirects the request, or responds with a custom HTML page.
- The origin server responds with the requested data to the Citrix ADC appliance.
- The appliance sends the response to the client.
You can use the URL filtering feature to detect sites that violate safe internet usage mandates issued by the government and implement policies to block these sites. Sites that host adult content, streaming media, or social networking identified as unsafe for children or banned as illegal.
Prerequisites
The feature works on Telco platforms with the purchase of a basic CBM license and CBM Premium license and for other Citrix ADC platforms, the feature works with the purchase of a CNS Premium license.
Note: In addition to a Basic CBM license and a CBM Premium license, the appliance must have a URL Threat Intelligence license with a subscription service for 1 year or 3 years. Before enabling and configuring the feature, you must install the following licenses:
License support for Telco platforms:
- CBM_TXXX_SERVER_Retail.lic
- CBM_TPRE_SERVER_Retail.lic
- CNS_WEBF_SSERVER_Retail.lic
Where XXX is the throughput, for example, Citrix ADC T1000.
License support for other Citrix ADC platforms:
- CNS_XXX_SERVER_PLT_Retail.lic
Where XXX is the throughput.
URL Categorization Policy Expressions
The following table lists the different URL categorization policy expressions for identifying incoming URLs and applies a configured action.
Expression | Operation |
---|---|
|
Returns a URL_CATEGORY object. Reputation score is a number from 1 to 4. To get objects all reputation scores use 0.0 as and |
Returns the category string for this object. If the URL does not have a category, or if the URL is malformed, the returned value is “Uncategorized”. | |
Returns a string identifying the object’s category group. This is a higher level grouping of categories, which is useful in operations that require less detailed information about the URL category. If the URL does not have a category, or if the URL is malformed, the returned value is “Uncategorized”. | |
Returns the reputation score as a number from 1 to 4, where 4 indicates the riskiest reputation. If the category is “Uncategorized,” the reputation value is 2. |
Sample Policy Expressions
Policy | Policy Expressions |
---|---|
政策选择请求的urlSearch Engine category | add responder policy p1 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). CATEGORY.EQ(“Search Engine”) |
政策选择请求的urlAdult category group | add responder policy p1 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). GROUP.EQ(“Adult”)’ |
Policy to select requests for Search Engine URLs with a reputation score equal to 4. | add responder policy p2 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(4,0). CATEGORY.EQ(“Search Engine”)’ |
Policy to select requests for Search Engine and Shopping URLs | add policy patset good_categories; bind policy good_categories “Search Engine”; bind policy good_categories “Shopping”; add responder policy p3 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). CATEGORY .EQUALS_ANY(“good_categories”) |
Policy to select requests for Search Engine URLs with a reputation score equal to 4. | add responder policy p5 ‘CLIENT.SSL.DETECTED_DOMAIN.URL_CATEGORIZE(4,0). CATEGORY.EQ(“Search Engine”) |
URL Categorization Policy Actions
A URL filtering policy evaluates the traffic to identify requests belonging to a particular category. The following table lists the actions that you can assign to a URL filtering policy.
Policy Action | Policy Group | Description |
---|---|---|
ALLOW | Responder | Allow the incoming request to access the target URL |
REDIRECT | Responder | Redirect the incoming request to the URL specified as the target. |
DENY | Responder | Deny incoming request. |
RESET | Responder, VideoOptimization | Reset connection. |
DROP | Responder, VideoOptimization | Drop connection. |
Note
For encrypted traffic, the VideoOptimization policy includes actions that implement the URL Filtering actions.
Configuring URL Categorization
To configure URL categorization, begin by enabling the URL Filtering feature. You must then configure the cache memory limits, categorization policy, and virtual servers for HTTP and HTTPS traffic. Configuring URL Categorization by using the CLI.
To use the CLI configure URL categorization on a Citrix ADC appliance, do the following:
- Set up URL Categorization.
- Enable the URL Filtering feature.
- Configure shared memory to limit cache memory.
- Configure URL categorization parameters.
- Configure URL categorization for HTTP traffic.
- Add URL categorization actions.
- Add URL categorization policies.
- Add a load balancing virtual server for HTTP traffic.
- Bind URL categorization policies to the load balancing virtual server.
- Configure URL categorization for HTTPS traffic.
- Add URL categorization policies.
- Add a SSL-Bridge load balancing virtual server.
- Bind URL categorization policies to the load balancing virtual server.
Setting up URL Categorization
To set up the feature, you must enable the URL Categorization feature, configure the filtering parameters and set the shared memory limit.
To enable URL Filtering feature
At the command prompt, type:
enable ns feature URLFiltering VideoOptimization Responder IC SSL AppFlow
To configure shared memory limit
At the command prompt, type:
set cache parameter [-memLimit ]
Where memLimit is the memory limit for caching.
Example:
set cache parameter -memLimit 10
To configure URL categorization parameters
At the command prompt, type:
set urlfiltering parameter [-HoursBetweenDBUpdates ] [-TimeOfDayToUpdateDB ]
*Example:
set urlfiltering parameter -HoursBetweenDBUpdates 3 -TimeOfDayToUpdateDB 03:00
Configuring URL categorization for HTTP traffic
To configure the URL categorization feature for HTTP traffic, you must configure a loading balancing virtual server, add URL categorization policies and bind the policies to the virtual server. By doing so, the virtual server receives the HTTP traffic and based on policy evaluation, the system assigns a filtering action.
To add URL categorization action for HTTP traffic
At the command prompt, type:
add responder action
Example:
add responder action act_url_categorize respondwith "\"HTTP/1.1 200 OK\r\n\r\n\" + HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0).CATEGORY + \"\n\""
To add URL categorization policy for HTTP traffic
At the command prompt, type:
add responder policy
Example:
add responder policy pol_url_categorize_http "HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0).GROUP.EQ(\"Adult\") || HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0).GROUP.EQ(\"Gambling\")" RESET
To add an HTTP load balancing virtual server
如果一个virtual server for HTTP traffic is not already configured, at the command prompt, type:
add lb vserver
Example:
add lb vserver vsrv-HTTP HTTP * 80 -persistenceType NONE -cltTimeout 120
To bind URL categorization policy with load balancing virtual server
At the command prompt, type:
bind lb vserver
Example:
bind lb vserver vsrv-HTTP -policyName pol_url_categorize_http -priority 10 -gotoPriorityExpression END -type REQUEST
Configuring URL categorization for HTTPS traffic
To configure the URL categorization feature for HTTPS traffic, you must configure a SSL-bridge loading balancing virtual server, add URL categorization policies and bind the policies to the SSL-bridge virtual server. By doing so, the server receives the HTTPS traffic and based on policy evaluation, the system assigns a filtering action.
To add URL categorization policy for HTTPS traffic
At the command prompt, type:
add videooptimization detectionpolicy
Example:
add videooptimization detectionpolicy pol_url_categorize_https_block_adult –rule "CLIENT.SSL.DETECTED_DOMAIN.URL_CATEGORIZE(0,0).CATEGORY.EQ("Adult")' –action RESET
To add SSL-Bridge load balancing virtual server
At the command prompt, type:
add lb vserver
Example:
add lb vserver vsrv-HTTPS SSL_BRIDGE * 443 -persistenceType NONE -cltTimeout 180
To bind categorization policy with SSL-Bridge virtual server
At the command prompt, type:
bind lb vserver
Example:
bind lb vserver vsrv-HTTPS -policyName pol_url_categorize_https_block_adult -priority 20 -type REQUEST
Configuring URL Categorization by using the GUI
The GUI enables you to:
- Enable the URL Categorization feature.
- Add URL Categorization actions for HTTP traffic.
- Add URL Categorization policies for HTTP traffic.
- Add URL Categorization policies for HTTPS traffic.
- Add a load balancing virtual server for HTTP traffic.
- Add an SSL bridge load balancing virtual server for HTTPS traffic.
- Bind URL Categorization policies to the load balancing virtual server.
- Bind URL Categorization policies to the SSL-Bridge load balancing virtual server.
- Configure shared memory limit.
- Configure URL categorization parameters.
To enable URL categorization
- In the navigation pane, expandSystemand then clickSettings。
- On theSettingspage, clickConfigure Advanced Featureslink.
- On theConfigure Advanced Featurespage, selectURL Filteringcheck box.
- ClickOKandClose。
To add a URL categorization action
- In the navigation pane, expandAppExpert>Responder>Action。
- In the details pane, clickAdd。
- On theCreate Responder Actionpage, set the following parameters.
- Name。Name of the URL categorization policy action.
- Type。希利ct an action type.
- Expression。Use the expression editor to create the policy expression.
- Comments。A short description of the policy action.
- ClickCreateandClose。
To add a URL categorization policy for HTTP traffic
- In the navigation pane, expandAppExpert>Responder>Policies。
- On the details pane, clickAdd。
- On theCreate Responder Policypage, set the following parameters.
- Name。Name of the URL categorization policy action.
- Action。希利ct the URL Categorization action that you prefer to associate with the policy.
- Log Action。希利ct the log action.
- AppFlow。希利ct an AppFlow action.
- Expression。Use the expression editor to create the policy expression.
- Comments。A short description about the policy action.
- ClickCreateandClose。
To add a categorization policy for HTTPS traffic
- Log on to the Citrix ADC appliance and navigate toConfiguration>Optimization>VideoOptimization>Detection。
- On theDetectionpage, click theVideo Optimization Detection Policieslink.
- On the Video Optimization Detection Policies page, clickAdd。
- On theCreate Video Optimization Detection Policypage, set the following parameters.
- Name。Name of the optimization policy
- Expression。配置策略使用自定义表达式。
- Action。Optimization action associated with the policy to handle the incoming video traffic.
- UNDEF Action。Undefined event if the incoming request does not match the optimization policy.
- Comment。A short description about the policy.
- Log Action.希利ct an audit log action that specifies the action to be performed for the log messages.
- ClickCreateandClose。
To add a load balancing virtual server for HTTP traffic
- Navigate to theTraffic Management>Load Balancing>Virtual Serverspage.
- In the details pane, clickAdd。
- On theLoad Balancing Virtual Serverpage set the following parameters:
- Name。Name of the load balancing virtual server.
- Protocol。Choose protocol type as HTTP.
- IP Address Type.IPv4 or IPv6.
- IP Address.IPv4 or IPv6, VIP address assigned to the virtual server.
- Port。Port number of the virtual server.
- ClickOKto continue with the configuration of other, optional, parameters.
- ClickCreateandClose。
To add an SSL-bridge load balancing virtual server
- Navigate to theTraffic Management>Load Balancing>Virtual Serverspage.
- On the details pane, clickAdd。
- On theLoad Balancing Virtual Serverpage, set the following parameters:
- Name。Name of the load balancing virtual server.
- Protocol。希利ct protocol type as SSL-bridge.
- IP Address Type。IP addressable type.
- IP Address。IP 4 or IP6 IP address assigned to the virtual server.
- Port。Port number of the virtual server.
- ChooseOKto continue configuration other optional parameters.
- ClickCreateand thenClose。
To bind a URL categorization policy to the HTTP load balancing virtual server
- Navigate toTraffic Management>Load Balancing>Virtual Serverspage.
- On the details pane, select the load balancing virtual server and clickEdit。
- In theAdvanced Settingsection, clickPolicies。
- In thePoliciessection, click the+icon to access thePoliciesslider.
- Set the following parameters.
- Choose Policy。希利ct URL categorization policy from the drop-down list.
- Choose Type。希利ct the policy type as Request.
- ClickContinue。
- 希利ct the URL categorization policy from the list and clickClose。
To bind a categorization policy to the SSL-bridge load balancing virtual server
- Navigate toTraffic Management>Load Balancing>Virtual Serversscreen.
- In the details pane, select the SSL-bridge load balancing virtual server and clickEdit。
- In theAdvanced Settingsection, clickPolicies。
- In thePoliciessection, click+icon to access thePoliciesslider.
- In thePoliciessection, set the following parameters.
- Choose Policy。希利ct video detection policy from the drop-down list.
- Choose Type。希利ct the policy type as Request.
- ClickContinue。
- 希利ct the video detection policy from the list and clickClose。
To configure the shared memory limit
- Sign on to the appliance and navigate toOptimization>Integrated Caching。
- In the details pane, clickChange cache settingslink.
- On theCache Global Settingspage, set the following parameters.
- Memory Usage Limit (MB).
- Active memory Usage Limit。
- Via Header。
- 要缓存最大身体后长度
- Global Undefined-Result Action
- Enable HA Object Persist
- Verify Cached Object Persist
- Prefetches
- ClickOKandClose。
To configure URL categorization parameters
- Sign to the appliance and navigate toSecurity。
- On the details pane, clickChange URL filtering settingslink.
- In theConfiguring URL Filtering Paramspage, set the following parameters.
- Hours Between DB Updates. URL Filtering hours between database updates. Minimum value: 0 and Maximum value: 720.
- Time of Day to Update DB. URL Filtering time of day to update the database.
- ClickOKandClose.
Configuring Audit Log Messaging
When a Citrix ADC appliance receives an incoming URL, if the responder policy has a URL Filtering expression, the audit log feature collects categorization information and displays it as log messages to any target audit log server that is configured. The info is logged.
- Source IP address (the IP address of the client that made the request).
- Destination IP address (the IP address of the requested server).
- Requested URL containing the schema, the host, and the domain name (
http://www.example.com
). - URL category that the URL filtering framework returns.
- URL category group that the URL filtering framework returned.
- URL reputation number that the URL filtering framework returned.
- Audit log action taken by URL Categorization policy.
To configure audit logging for the URL List feature, you must complete the following tasks:
- Enable Audit Log.
- Create Audit Log message action.
- Set URL List responder policy with Audit Log message action.
For more information, seeAudit Loggingtopic.
Storing Failure Errors Using SYSLOG Messaging
At any stage of the URL Filtering process, if there is a system-level failure, the Citrix ADC appliance uses the audit log mechanism to store logs in the ns.log file. The errors are stored as text messages in SYSLOG format so that, an administrator can view it later in a chronological order of event occurrence. These logs are also sent to an external SYSLOG server for archival. For more information, seearticle CTX229399。
For example, if a failure occurs when you initialize the URL Filtering SDK, the error message is stored in the following messaging format.
Oct 3 15:43:40
ns URLFiltering[1349]: Error initializing NetStar SDK (SDK error=-1). (status=1).
The Citrix ADC appliance stores the error messages under four different failure categories:
- Download failure. If an error occurs when you try to download the categorization database.
- Integration failure. If an error occurs when you integrate an update into the existing categorization database.
- Initialization failure. If an error occurs when you initialize the URL Categorization feature, set categorization parameters, or end a categorization service.
- Retrieval failure. If an error occurs when the appliance retrieves the categorization details of the request.
URL Reputation Score
The URL Categorization feature provides policy-based control to restrict blacklisted URLs. You can control access to websites based on URL category, reputation score, or URL category and reputation score. If a network administrator monitors a user accessing highly risky websites, he or she can use a responder policy bound to the URL reputation score to block such risky websites.
Upon receiving an incoming URL request, the appliance retrieves the category and reputation score from the URL categorization database. Based on the reputation score returned by the database, the appliance assigns a reputation rating for websites. The value can range from 1 to 4, where 4 is the riskiest type of websites, as shown in the following table.
URL Reputation Rating | Reputation Comment |
---|---|
1 | Clean site. |
2 | Unknown site. |
3 | Potentially dangerous or affiliated to dangerous site. |
4 | Malicious site. |
In this article
- How URL Categorization Works
- Prerequisites
- URL Categorization Policy Expressions
- URL Categorization Policy Actions
- Configuring URL Categorization
- Setting up URL Categorization
- Configuring URL categorization for HTTP traffic
- Configuring URL categorization for HTTPS traffic
- Configuring URL Categorization by using the GUI
- Configuring Audit Log Messaging
- Storing Failure Errors Using SYSLOG Messaging
- URL Reputation Score