Why?

A few months ago I realized it is rather hard to tell if any devices on a home network are compromised. A typical wireless modem / router combo is simply too weak to do serious traffic analysis, and probably can’t be implemented anyway because then ISPs would have to do something about all the alerts from their provided modem-router-wireless access point combo devices.

Some projects like pihole do a good job at providing some basic security by recording DNS lookups and blocking malicious ones. Another application that did basic scans of the network for vulnerable devices and basic analysis of traffic received by the Pi would provide decent basic alerting of compromised devices.

However, as far as I could tell no such application exists for the Pi and the closest we get are aimed at small businesses and need at least a decent x86 processor and 6+ GB of RAM. I researched the options and tested SecurityOnion, SELKS, pfsense’s suricata and ntop plugins, and Alienvault OSSIM - more on those in a future post.

But before you can hook up an older laptop or desktop or a VM in your homelab for network traffic analysis, you have to be able to get the traffic analysis to the device - a surprisingly difficult task. The very devices that have access to all the traffic (wireless routers, pfsense boxes, SOHO router/firewalls like edgerouter) are usually much too weak to process it.

So, I had to take on the task of copying all my home network traffic to either a VM or a spare computer. In the past with 10 / 100Mb connections this could be as easy as inserting a hub between your router and a proper switch, but 1000Mb ethernet is more complicated. I needed to port mirror, or in Cisco terms set up a SPAN.

Privacy

If you do this at home, please tell anyone whose traffic you will be intercepting first. I made sure to explain to the members of my household that I had theoretical access to their traffic, but that everyting important is encrypted and I frankly don’t care or have the time to snoop on them; my goal is to quickly find out if any of our devices shows signs of malicious activity.

Quick note on Promiscuous Mode

Something I should have realized earlier was that every interface, and I mean EVERY interface, needs to be promiscuous on the way from your traffic source to the traffic analyzer. If you are running a security appliance in a VM this can mean as many as 3 interfaces to be put into promiscuous mode: - the actual interface for the NIC

• the bridge interface that the VM (note that Linux bridges are essentially switches, they are not at all like bridges in the physical networking sense). Also making a bridge promiscuous is very different from normal interfaces.
• The interface inside the VM that connects to the bridge

Existing hardware

I already had pfsense running on an old laptop with a USB adapter for WAN. Additionally, I ordered an HP NC365T quad-gigabit network adapter for about 50 Canadian dollars for my HP Z400 VM host. A bit pricy but I knew it would be useful beyond this project, and if you love yourself, don’t buy anything other than Intel-based ethernet adapeters. Here’s what that looks like:

First attempt at port spanning: pfsense

My first attempt at port mirroring was to buy another RTL8153 based USB ethernet adapter (~$15, but I needed another one in the future anyway).

After that, I essentially configured a bridge from the existing WAN and LAN interface to the new “span” interface. This sorta worked, but I think at the time I thought it was only one direction, but in fact it was bidirectional, so devices started getting DHCP addresses from my ISP modem-router-WAP but then after accepting them couldn’t communicate as the pfsense firewall was understandably not having it with all these devices outside its subnet range.

Essentially, it may be possible to do L2 port mirroring with the L3 oriented pfsense, but it’s a giant pain that probably won’t work like you expect.

Second attempt: Cisco 2960s

Proper port mirroring means using a proper switch - there’s no way around it. And easy to use port mirroring (for example, mirror multiple ports traffic to a single port rather than just one to one mirrors) means getting a quality enterprise switch.

After much research, and a decision to pay the premium for used Cisco rather than Dell, HP, or Nortel on the basis that the extra $50 is overwhelmingly worth it in the long run to gain Cisco experience, I bought a used Cisco Catalyst 2960s for about $87 on ebay. The 2960s runs at about 30W while some of the cheaper switches run at 60-80W so that’s nice as well.

Here’s what it looked like after being “racked” and “cabled”:

Linux bridge workaround for promiscuous behaviour in Proxmox

A very big thanks to this blog post for discussing in short and in detail how to make Linux bridges promiscuous and also specifically what to do for Proxmox.

After choosing a NIC to use as your receiver of promiscuous traffic, and a bridge to connect the traffic analyzing VM(s) to:

Here’s the basics:

brctl setageing vmbr1 0
brctl setfd vmbr1 0

If you are sure you will only use one traffic analyzer you could also try to directly passthrough the NIC to that VM, but this has its own complications and requirements. In my case it probably would have meant passing the whole quad port NIC through to the one VM.

Final working setup

So in the end what worked for me was pfsense continuing as normal as a router and firewall, a 48 port enterprise switch mirroring traffic to a port on the quad port adapter, and a bunch of promiscuous intefaces on the way passing the traffic to my SecurityOnion, SELKS, and Alienvault OSSIM VMs. So now these VMs have full access to all my home network traffic.

Some notes on testing and debugging

Install wireshark on your laptop and on the ultimate target VM / computer. If you see nothing, you’re probably not port mirroring at all or there’s an unplugged cable. If you see only broadcast traffic, then your switch is misconfigured or you need to enable promiscuous mode somewhere. If you see all the traffic, and you will know from the incredible amount of background communication with the internet that happens now, then you see communication to and from the internet from devices that are not your wireshark host.