Recently, I had the experience of assisting a friend of mine with standing up a new Ubiquiti UniFi network, consisting of hardware such as the UniFi Dream Machine Pro, UniFi Switch 48, and a handful of UAP-AC-Pros. With this network, they were configuring multiple VLANs as well as tightening down the security compared to the existing network, implementing IPS/IDS detection, Content Filtering for Adult Content, and utilizing services such as Ubiquiti’s Identity Service to assist with remote access. Also as part of the network rebuild, the ISP for the network, Comcast Business, was upgraded to a 2Gbps down, 300Mbps service plan delivered via DOCSIS 3.1, with a handful of Static IPs. Everything changed pretty much all at once.
All was going fine and dandy. But then a problem was encountered which ended up revealing a flaw with Ubiquti’s implementation of DNS Shield and Content Filtering, thanks to Comcast Business. As part of the network configuration, the DNS for the Dream Machine Pro was pointed to use Quad9’s DNS servers in the WAN Settings. Additionally, DNS over HTTPS (DoH) was configured in the Dream Machine under Settings –> Security –> DNS Shield to use Quad9’s DNS over HTTPS services. Yes, the IPv6 DNS settings in the Dream Machine on the WAN side were also configured to use Quad9’s DNS services, with each VLAN set to use the Dream Machine’s local DNS resolver for all queries, on both IPv4 and IPv6. This means for all networks, all DNS queries should be going to Quad9. This was true, on the networks without content filtering.
For the VLANs with Content Filtering, Ubiquiti partners with NextDNS for DNS-based Web Filtering and Security. Whenever the Content Restrictions are enabled, the Ubiquti Dream Machine will modify the Dream Machine’s Local DNS Resolver settings to point all DNS traffic from VLANs with filtering enabled, towards NextDNS, rather than use what is configured globally on the Dream Machine for DNS on the WAN side. This is no different from how other services such as Cisco’s OpenDNS work to protect networks from unwanted and malicious content. We used the “Family” setting in the controller. In theory, after making these changes to the UniFi network, content blocking should be coming from NextDNS, Adult Content should be blocked (based on the Filtering settings we chose in the UniFi Controller) with YouTube and Google in Family mode, and normal, everyday applications should just work fine. That isn’t what happened.
Shortly after implementing the Content Restrictions, we found a few web applications that were needed for day to day business were no longer working. HTTPS errors were being thrown. On VLANs without Content Filtering, the applications loaded up just fine. Thinking this was a false positive on the NextDNS Side, I looked around for a mechanism to report a false positive to NextDNS. Upon having difficulties with this, either because it doesn’t exist, or my search-fu is weak, I enabled Content FIltering with the same settings on my personal UniFi network at home just as a test… and the applications continued to work without an issue. With some bickering back and forth between myself and my friend, we decided to start gutting some of the security measures implemented on the network, only to find that nothing worked except for disabling the Content Filtering feature for a VLAN.
Confused about the fact that the Content Filtering setting was breaking access to the needed web applications only on that specific location, and fully aware that the Content FIltering option would point DNS traffic to NextDNS, I decided to double check that NextDNS was in fact, the DNS provider being used. A handy tool for this called the DNS Leak Test exists. This tool is commonly used to ensure VPN Clients and certain network configurations aren’t causing DNS traffic to leak to an undesired server. In essence, it proves that your DNS queries are being handled by the provider you choose.
On the networks without Content Filtering enabled, I performed a DNS Leak Test, and found for all servers detected, the provider was PCH.net (Packet Clearing House), which is Quad9’s network. When the same test was performed from a VLAN with Content Filtering enabled, every server was coming up with a domain, which, after running a WHOIS query against, shows up as Barracuda Networks. Wait… Barracuda? The provider is supposed to show up as NextDNS. So I try the same test from another network with the Content Filtering enabled, and NextDNS does indeed show up as the DNS provider.
At this point, everything started to point at Comcast. Gathering some more information from the business where the new network was installed, the applications seemed to break right around the same time that the Comcast Business service was upgraded to the 2Gbps tier. A bit of research later revealed a few major problems!
- Comcast enabled something called Comcast SecurityEdge by default, when the service plan was upgraded. The business claims they weren’t aware of this being tacked on, and it wasn’t mentioned to them. Whether this is true or not, since these things tend to be slipped in by means of product advertising, I’m not sure, as I am not in a Comcast market. Comcast’s SecurityEdge feature works by inserting a firewall rule within the provided Comcast modem (which was being used as a result of having Static IP service) which hijacks and rewrites all UDP/53 and possibly TCP/53 traffic passing through the modem to point to Barracuda-controlled DNS Servers, regardless of what the user’s device has configured as a DNS provider. There is no DMZ for Static IP Addresses to bypass this; it is everything or nothing.
- Ubiquiti’s network controller does not call out that the Content Filtering feature is changing a VLAN’s DNS provider to NextDNS. My friend initially believed that the Dream Machine was running the filtering lists/lookups locally, and continuing to use Quad9 as the DNS provider.
- Ubiquiti’s network controller does not point out that it disables DNS over HTTPS resolution at the Dream Machine end for the VLANs with Content Filtering enabled. Because of the ISP tampering with DNS requests sent over Port 53, the content filtering was not working as intended, AND DNS queries were being sent to an unexpected third party, without any knowledge of it happening (DNS over HTTPS would fail if the DNS query is hijacked, by design).
- Because the network was set up to use Quad9 DNS over HTTPS, queries were being sent to the correct place. However, as DNS over HTTPS can be configured in an opportunistic fashion much like DNS over TLS can on many routers (On UniFi, DoH is strict rather than opportunistic), fallback to regular Port 53 DNS would cause the network’s DNS queries to be sent to DNS Servers other than what was configured, without any alerts being thrown. This means DNS behavior could become erratic, and there are privacy and security implications when DNS Traffic is being hijacked.
What was done next…
First thing’s first, we needed to fix the DNS hijacking problem. To do this, a call to Comcast Business support was needed to disable the SecurityEdge feature. Once disabled, the web applications that were attempting to be used all started to work again. The DNS Leak Test properly showed NextDNS as the DNS provider for networks with Content Filtering enabled, and Quad9 DNS appeared for networks without. Sites like YouTube and Google were in Family mode, and Adult content on the Internet was being rightfully blocked, all as expected.
For the Ubiquiti issues, we submitted a report via HackerOne regarding the defects in Ubiquiti’s product design. In our mind, this was the right path to take, as a rogue or malicious network provider (for example, nation-state owned ISPs) could perform trivial DNS manipulation or inspection, as Comcast’s SecurityEdge feature did. This means that any content filtering that was configured and desired by the network administrator with “Smart DNS” solutions like OpenDNS or NextDNS, whether for security, for compliance reasons, for safety, for privacy, or all of the above, could be easily negated. An attacker with control of an ISP’s CPE, or any upstream equipment, could cause plenty of privacy and security issues, by leaking DNS traffic to their controlled servers, modifying responses to point resources to rogue endpoint addresses, and by blocking access to Software Update servers for Windows. Additionally, because the Content Filtering feature does not have DNS over TLS or DNS over HTTPS implemented, and therefore ignores the “DNS Shield” Setting set in the Security section of the UniFi Controller, there is no mechanism for networks using the Content Filtering feature to “fail shut” or detect DNS Tampering occurring. Once again, leading to privacy, security, and compliance issues. The UniFi Network Controller as previously mentioned, neither states that Content Filtering switches the VLAN to use NextDNS, and does not state that “DNS Shield” or DNS over HTTPS does not apply to networks with Content Filtering enabled.
Ubiquiti unfortunately closed the HackerOne report stating that this is intended design, and there are no issues. I disagree with this assessment, after having spent an hour or so battling content filtering troubles that were caused by the Internet Service Provider, Comcast Business, and caused by the lack of clarity on how certain features were implemented. Hopefully Ubiquiti will see this post and take action to fix the problem, and hopefully won’t ask for a takedown of this otherwise valid post with the concern. The HackerOne report was submitted over two months ago as of this writing.
The takeaway to this post is:
- If you use Comcast Business and have no intention to use Comcast’s DNS, make sure SecurityEdge is disabled on your account. They enable it by default.
- Content Filtering on Ubiquiti Dream Machines uses NextDNS, and all traffic is sent over plain old Port 53 DNS by the UniFi Gateway.
- DNS Shield (DNS over HTTPS) only applies to networks without Content Filtering enabled on UniFi networks, with no option in the GUI to enable it for Content Filtering enabled networks. It might be possible to implement it with some command line work.
- Use tools like the DNS Leak Test to ensure your DNS queries are going to where you want them to go. Run tools like that as part of your network assessments regularly.
- Ubiquiti acknowledged but currently refuses to fix a valid security-related concern with how DNS traffic is handled out of the UniFi Gateways, and has not fixed the verbiage in the UniFi Controllers to add clarity on what filtering changes, and its limitations around the DNS Shield feature.
