Skip to main content

What is URL Filtering? A Guide About URL Filtering

Users are spending more time on the internet, reading their favorite sites, clicking on email links, and using a range of web-based SaaS apps for personal and corporate purposes. While this type of unrestricted web activity is extremely valuable for increasing company productivity, it also exposes firms to a variety of security and business concerns, including the spread of malware, data loss, and more.

Companies have traditionally employed URL filtering to prevent employees from visiting non-productive websites. Firms can enable safe web access and protection from more sophisticated dangers, such as malware and phishing sites, with today's URL filtering.

What is the URL?​

URL stands for Uniform Resource Locator. A URL is the address of a specific unique resource on the Internet. Each valid URL, in principle, refers to a single resource. An HTML page, a CSS document, an image, and so on are examples of such resources. There are some exceptions in practice, the most common of which is a URL pointing to a resource that no longer exists or has migrated. Because the Web server manages both the resource represented by the URL and the URL itself, it is up to the web server's owner to carefully manage that resource and its associated URL.

A URL is made up of several elements, some of which are required and others which are optional. On the URL below, the most significant components are underlined

URL structure

Figure 1. URL structure

You can parse any url to understand its components by using unfurl.

Parsing an URL using unfurl

Figure 2. Parsing an URL using unfurl

What is URL Filtering?​

URL filtering is a form of technology that allows organizations to limit their users' and visitors' access to specific web material.

URL filtering is based on filtering databases that categorize URLs by topic. Each topic is either β€œblocked” or β€œallowed” in this system. This system was created primarily for the purpose of increasing productivity by allowing companies to ban websites that were unsuitable for work or were designed for entertainment .

URL filtering aids in the following aspects of network behavior control:

  • Access to some types of websites, such as gambling and pornographic sites, is restricted.
  • During the specified period, you will have access to a specific category of websites. For example,disallowing access to IM (Instant Messaging) websites during office hours.
  • Control of access to the website whose URL contains the terms given. For example, you can't go to a URL that has the word "game" in it.

URL filtering has several major advantages, including:

Minimize Security Threats​

The first and most obvious benefit is that you'll harden your company's security. You'll be less vulnerable to dangers from the internet.

Improve Compliance​

You can also stay compliant while avoiding liabilities.Depending on your sector, you may be required to follow more stringent cybersecurity and operations standards.

Increasing Efficiency​

URL filtering has the extra benefit of allowing you to regulate which websites your employees access, resulting in increased overall efficiency.

What Does a URL Filter Do?​

  • URL filtering aids productivity by ensuring that employee time is not wasted on non-productive activities during working hours.
  • URL filtering allows you to restrict access to specified websites by allowing or disallowing access depending on information included in a URL list.
  • URL filtering can also aid in the prevention of malicious code/spyware, phishing, and other threats to the organization.
  • It also aids in the blocking of peer-to-peer software and instant messaging, both of which consume more resources, waste time, and pose a security risk.

What are the Features of URL Filter?​

IT administrators can use the URL filtering feature to prevent users from visiting unapproved websites and web applications. The URL filter compares URLs entered into the search engine with an accept or deny list supplied by IT administrators. URLs will be allowed or banned based on the policy provided. The main features of URL Filtering are listed below.

  • Automated web categorization engine blocks non-productive websites
  • URL filtering for HTTP & HTTPS Protocols
  • MIME Type and file extension blocking
  • User, Group-based URL filtering and Download size restriction policies
  • Time-based access schedule
  • Prevents downloads of streaming media, images, pop-up ads
  • Blocks malware, phishing and pharming URLs
  • Blocks Java Applets, Cookies and Active X
  • URL Exempt/White list
  • Automatic updates on URL categories

How Does URL Filtering Work?​

The working principle of URL filtering is to compare the URL contained in a URL request to entries in a URL database or URL list. If a match is found, the device takes the appropriate action (permit or deny). If a user enters a URL manually or clicks on a search engine link to the URL and the URL is blocked, the browser will redirect the user to a web page similar to the one shown below.

Blocked web page by url filtering solution

Figure 3. Blocked web page by URL filtering solution

If we want to illustrate how it works we can give an example from the Palo Alto URL filtering solution.How URL filtering works

Figure 4. How URL filtering works

PAN-DB, or the URL Filtering cloud database of PaloAlto, categorizes websites based on their content, features, and security. A URL can have up to four URL categories, including risk categories (high, medium, and low) that indicate how likely the site is to expose you to threats.

Firewalls with URL Filtering enabled can use the information gathered by PAN-DB to enforce your organization's security policies in real time. Advanced URL Filtering enables real-time analysis using machine language to guard against new and unknown threats, in addition to the security provided by the PAN-DB database.

After taking a look at how url filtering works in Palo Alto, url filtering can be configured in Palo Alto by following the next steps:

  1. Create a profile for URL Filtering.
  2. For each URL category, define site access.
  3. Set up the URL Filtering profile to identify corporate credential submissions to websites in the permitted URL categories.
  4. To combat credential phishing, allow or prohibit users from entering corporate credentials to sites based on URL categorization.
  5. Create an exception category for URLs. Websites that should always be prohibited or allowed, regardless of URL category, are specified in lists.
  6. Safe Search Enforcement should be enabled.
  7. For URL filtering events, only Container Pages should be logged.
  8. Enable HTTP Header Logging for one or more of the HTTP header fields that are supported.
  9. Save and commit your modifications to the URL Filtering profile.
  10. Enable hold-client-request to block client requests while the firewall executes URL category lookups
  11. Set how long a URL category lookup will take before it times out.

What is the Inspection Order for URL Filtering?​

What happens when a URL matches multiple patterns within a URL filtering profile (multiple custom URL filtering categories and allow/block-list)? What is the order of URL Filtering actions?

When a URL matches multiple categories, the category with the most severe action defined below is chosen (block being most severe and allow least severe).

Starting with the different sources of URL Filtering Data, the precedence is from the top down - First Match Wins:

The priority for URL filtering is:

  1. block list
  2. allow list
  3. custom categories
  4. cached
  5. predefined categories

How Does URL Filtering Prevent Malware and Attacks?​

Using URL filtering and anti-malware tools, you can prevent malicious destinations from connecting to your systems. Advanced URL filtering and anti-malware solutions can detect attacker infrastructure that has been set up in preparation for current and emerging threats to your system. You can prevent a threat by identifying it before it has a chance to fully manifest. You can stop phishing, malware, and ransomware attacks before they start.

By blocking access to known malware and phishing sites, the chance for a security breach is greatly reduced.

Is URL Filtering Same as Web Filtering?​

Web filtering includes URL filtering. The term "web filtering" refers to a variety of methods for limiting the content that users on a network can access through the Internet. Another common approach for limiting web material is DNS filtering.

What is the Difference Between URL Filtering and DNS Filtering?​

DNS filtering and URL filtering serve the same purpose. The primary distinction is that URL filtering prevents access to URLs, whereas DNS filtering prevents access to DNS queries. URL filtering prevents access to web pages, whereas DNS filtering prevents access to domains. Other differences between DNS filtering and URL filtering are as follows.

  • URL filtering is a more granular type of web filtering than DNS filtering, which uses DNS queries to ban or allow material across whole online domains.
  • DNS filtering is best used to block out entire domains linked to highly harmful activities and with little probability of hosting legitimate information. Meanwhile, URL filtering allows businesses to block specific web pages or stored files from a domain while enabling users to access other valid pages hosted on the same domain without restriction.

Should You Enable URL Filter?​

Yes. You should enable URL Filtering because URL filtering assists organizations in increasing productivity by ensuring that employee time is not wasted on unnecessary activities during office hours. URL filtering can also aid in the prevention of malicious code/spyware, phishing, and other potentially harmful activities to the organization. Some vendors also assist in the blocking of Peer-to-Peer software and instant messaging, which consume more resources such as internet bandwidth, and waste time.

Although URL filtering has a huge positive impact on your web surfing security it also has some drawbacks. Disadvantages of URL filtering are given below.

  • The most significant disadvantage of URL filtering is overblocking. Over-blocking can cause problems for users (Example, some commercial spyware needs to be installed for certain applications to work and blocking them might deny access to those applications).
  • Over-blocking can also result in an increase in the number of help-desk tickets that need to be addressed and resolved by the support team. If this occurs frequently, both the user's and the support team's time is being wasted.
  • Another problem can be misclassification .There are times when certain websites that have already been classified become threat sites/avoidable sites at a later stage.

What is URL Filtering in Firewall?​

There are two types of url filtering solutions that are integrated and standalone ones. Organizations may choose to consider an integrated approach instead of standalone solutions. Threat analytics, cloud service, endpoint, and network security, and threat intelligence understanding that stops known and undiscovered attacks are all included. URL filtering in a firewall is the example of an integrated solution.

A firewall URL filtering mechanism compares a user's request for a URL to a database or list of sites that have been restricted or allowed for use. This usually restricts employees from viewing websites that could disrupt the organization's normal operations, such as those that include unlawful or improper information, are unrelated to work, or are high-risk, harmful, or linked to phishing attacks.

How to Do OPNSense URL Filtering?​

OPNsense uses the built-in proxy and one of the freely accessible or commercial blacklists to perform category-based web filtering.

One of the best url filtering solutions on OPNsense is the Sensei (ZENARMOR) plugin. Sensei (ZENARMOR) lets administrators create predefined or customizable web filtering profiles and policies based on a cloud-based web categorization of 140+ Million web sites under 60+ categories.

Enable or disable web and URL filtering controls by navigating to the Web Controls tab in the Policies section. Preset profiles allow for permissive to custom controls.

Popular lists that are expected to work well with OPNsense are:

  • Shallalist.de Free for personal usage and partly for commercial usage
  • URLBlacklist.com Commercial paid service
  • Squidblacklist.org Commercial paid service

How to Do pfSense URL Filtering?​

URL filtering is a technique for restricting access to specific websites based on their URL. There are several commercial products available for URL or content filtering, but SquidGuard and pfSense can be used to build a very robust system on your own. SquidGuard is a very useful plugin for the widely used Squid proxy server that can be used to block or redirect web requests on your network.

SquidGuard has a long list of features that can be tailored to your specific requirements. It's also lightning fast and won't slow down your consumers' internet connection. SquidGuard can help you ban access to a list of unwanted websites or just allow access to specific websites.

SquidGuard is extremely adaptable, making it simple to integrate into a variety of applications. SquidGuard is capable of performing basic URL filtering on your home network as well as creating complex rules for a large public network.

Also, pfBlockerNG is a widespread pfSense package that has an effective DNS filtering capability. You may find more information about pfBlockerNG on pfBlockerNG Guide written by Sunny Valley Networks.

Lastly, you can try Sensei (ZENARMOR) on your pfSense firewall for URL filtering too.

What is Palo Alto URL Filtering?​

Palo Alto Networks URL Filtering enables you to monitor and control the sites that users can access, to prevent phishing attacks by limiting the sites to which users can submit valid corporate credentials, and to enforce safe search for search engines such as Google and Bing.

URL Filtering from Palo Alto Networks protects against web-based threats by allowing you to safely enable web access while controlling how your users interact with online content. You can define policy rules to restrict site access based on URL categories, users, and groups.

What is Linux URL Filtering?​

On your Linux environment you can use DansGuardian software for Url filtering.

DansGuardian is an open source web content filter that operates on Linux, FreeBSD, OpenBSD, NetBSD, Mac OS X, HP-UX, and Solaris. It uses a variety of methods to filter page content, including phrase matching, PICS filtering, and URL filtering. It does not filter solely on the basis of a banned list of websites, as do some other, purely commercial filters.

DansGuardian is designed to be completely adaptable, allowing you to tailor the filtering to your specific requirements. You can make it as strict or as lax as you wish. The default settings are designed for a primary school, but DansGuardian gives you complete flexibility over what you wish to ban.

Also, Sensei (ZENARMOR) supports most widespread Linux distributions such as, Ubuntu (18.04 and 20.04), CentOS 7/8, Debian 10 and AlmaLinux 1. You can install and configure Sensei (ZENARMOR) on one of these supported Linux distirbutions for URL filtering to protect your network against cyber threats such as malware or phishing. Sensei (ZENARMOR), provides you a centralized cloud management portal with an intuitive web interface to manage your next generation firewall.

What is Check Point URL Filtering?​

Check Point URL Filtering restricts access to millions of websites by category, users, groups, and computers, ensuring that users are protected from harmful websites and that they can use the Internet safely. UserCheck technology is used by URL Filtering to educate users on online usage policies in real time.

How Can I Bypass Blocked by URL Filter Database?​

We explain it for educational purposes, to make the subject more clearer. Do not use the methods explained in this article for illegal purposes. Sunny Valley Networks takes no responsibility for use of this tool for illegitimate purposes.

To get around url filtering, use the following generic methods.

  • Try to use HTTPS instead of HTTP, for example, https://mail.google.com rather than http://mail.google.com, because some companies forget to filter HTTPS sites.
  • SSL VPNs will achieve this for you without fiddling with your settings (e.g. Express VPN which is SSL based OpenVPN, or try hide.me).
  • Try replacing the IP address with a domain name, such as http://142.250.187.174 instead of google.com in a browser's address bar. Using nslookup from the command line, you can simply send DNS requests.
  • In your browser, try using one of the free public proxies.
  • Install Tor Browser if possible.