I’ve been experimenting a bit with pfSense as a firewall for >1Gbps networking. Comcast provides (in limited areas) speeds of 2Gbps (full duplex…ie. both up and down). Per my research, it seems they terminate this with two connections:
- RJ-45 1Gbps ethernet port
- SFP+ interface (not sure if it’s blank or if it has a MM SR, SM LR, or something else entirely)
Additionally, from their documentation and people documenting it online, to get the full 2Gbps in a single flow, you have to use the SFP+ interface (duh, that one is a given since GigE is limited to 1Gbps full duplex, while SFP+ can handle 10Gbps full duplex). That said, if you want the full gigglebits (my terminology for gigabit as it’s more fun), you need to use something that has SFP+ (or higher…but let’s face it QSFP, QSFP+, zQSFP, and SFP28 are all too expensive right now).
Based on current pricing of 10G, 25G, 40G, and 100G…the most affordable to the common consumer is going to be 10G. You can pick up a nice intel 82599 dual-port (yes, the same chip used in AWS’s enhanced networking) from Amazon (or any other retailer really) for about $160. Also, 10G switches are coming down in price as well…I picked mine up from unix surplus for $240 for a used 24-port switch. You can pick up a 16-port SFP+ ubiquiti switch for around $500 (brand new).
So, the cost of implementing 10G-based networking is rather cheap. Thus, the attempt to set up my own firewall with 10G networking (see…I eventually got to my point…Comcast+cheaper SFP+ tech = custom pfSense build with 10Gtek intel 82599 dual-port SFP+ adapter for WAN and LAN connections).
In the process of all this, I’ve run into several snags. The most annoying being the Ring video doorbell not working…but I’ve just accepted that as a loss at this time until I can revisit down the road. There seem to be a few other hiccups…all of which seem to be relatively related to the same issue…incompatibility with UDP or poor handling of UDP packets.
Diagnosing the ring video doorbell was/is tough as they don’t really give you a whole lot of information with which to troubleshoot, but after seeing enough patterns, it seems that something is going wrong in handling UDP via wifi over my unifi-AP-AC pro that pfSense is not correcting for. Somehow, I think my Asus router was doing some sort of magic to repair the packets or something because the problems are not there in AsusWRT/merlin build.
I noticed that my VPN was disconnecting every 20 minutes (client-site for work). What I discovered was that my VPN client was sending about 10:1 control packets…it would send 10 and receive one back…so, if the control packet back did not come in, it would drop the connection. This all happened around the same time that packets would stop sending over the network (ie. no control packet when expected and about the same time, ping would stop working over VPN). After some more testing, I found that this ONLY happened when I was on wifi (ie. using my ubiquiti AP-AC-PRO). When I swapped over to wired ethernet, the connection would go for hours without dropping the connection, with the same rate of control packets.
So, I came to the conclusion that one of a few things was happening:
- Bad network cable (easy fix)
- Ubiquiti AP-AC-PRO does something screwy with rx packets and doesn’t tx them over the eth0 link (very not good if it’s the issue and difficult to fix sans removing the AP)
- Some funky OS-level packet drop due to flags, states, or bad checksum
- weird wifi action with duplicate packets due to the nature of wifi and that it expects certain levels of loss and essentially double transmits as a safety precaution
Since I saw the behavior in more than just ring, it’s safe to conclude that it’s not an app-level problem (per-se) with ring…while I’m sure that the VPN client and Ring could probably handle packets better by leveraging TCP more (ie. for sending things like control packets and checking connectivity/packet counters on each end and use some fallback or workaround), generally this points to a different problem.
Anyway, only time will tell how much of a headache this is going to be moving forward and what the actual RCA ends up being, but there are a lot of cool things that you can set up with pfSense that outweighs the issues of dealing with wireless and apps that don’t thrive in non-standard environments (ie. ring as they clearly did not test a similar use-case with their doorbell otherwise there would be a knowledge article, video, etc. to help make sure that you have configured pfSense to work optimally with the Ring doorbell).
I’d like to point out I’m not knocking Ring…I’m just pointing out that not all edge cases have been fleshed out, which is to be expected when you’re trying to build a product for specific demographics. You’ll only target the largest majority you can feasibly address, which generally doesn’t include custom home-built router solutions like pfSense, VyOS, ipfire, ClearOS, etc.
The target audience will likely be using AsusWRT, Netgear, D-Link, etc. and won’t care about getting 2Gbps+ of throughput to the internet. So, at this point, I have a stable-ish firewall (we’ll see how long I can get the uptime), with a cool and relatively easy to configure ipsec+bgp configuration (though it takes some tweaks to keep it from crashing) for things like connecting your home network into your AWS account (or your corporate network) using a CGW and VGW to set up IPsec tunnels and overlay that with BGP routing to allow dynamic updating to your VPC’s route tables AND your internal network to update it’s route tables as well (like if you were to connect a second VPC to the first VPC, which you then connect back to the “home” network or if you connect in a hub-and-spoke style and propagate routes for each VPC to one another via the pfSense box).
IPv6 works pretty quickly out of the box as dual-stack and does not take much work to configure. It has lots of packages that can be installed to enhance functionality, AND it can be installed on consumer hardware (or enterprise), which allows you the flexibility to put as much horsepower behind your router as you may require. I personally have 16GB of ram with room to grow to 32GB, 1 X520-DA2 (intel 82599), an AMD Ryzen 7 1700x (turned off SMT for better pps), AIO water cooling, 250GB ssd for storing logs, and a low power GFX card for serial/console output so that I can configure the system via keyboard should networking or other components fail. I could’ve gone and bought another rack server, but I wanted this to be able to go in the closet of my bedroom and so I wanted this to be quiet…thus the AIO water cooling.
All this said, I don’t have too many complaints about this now that I’ve worked out some kinks in the system. I just need to finish wiring my house for ethernet and fiber so that I can put this in the closet and call it a day instead of having it sit downstairs (my cable management at home is a mess right now).
…well, it’s getting late. I’ll try to write another tech blog post a little later down the line…probably will be related to docker, chef, ansible, or some other automation-like thing…maybe another post about OpenStack…just depends on what the next project is that I get some work done on.