Detecting hosts that cross connect two networks...

There are different scenarios in which you can have a situation where some of your clients are connected in the same time to two different networks of different trust levels. This is dangerous because effectively they can be used as a staging point for potential attackers or malware to "jump" from less trusted network to more trusted one.

For example, suppose that you have protected internal wired LAN, and in the same time you have wireless LAN that is used for guests and allowed unrestricted access to the Internet, as shown in the figure below. Someone, from your internal and protected network, might intentionally or accidentally connect to the wireless network too, and in that case he/she will shortcut the two networks. If you thought that you can use MAC addresses to detect such hosts, you can not, for a simple reason that in one network host is connected using wired ethernet card with one MAC address, and in the second network it is connected using WLAN network card with another MAC address.

Hypothetical situation to illustrate how internal host might shortcut two networks of different trust levels

So, how to detect those hosts that shortcut two networks? Actually, there is an easy and elegant way. Namely, you can send ARP requests on one network for IP addresses on another network. ARP module, by default, doesn't know or care for routing tables so that it knows on which interface it should respond with specific addresses. If we look again on the figure above, this means that you can run the following command from the proxy host (the host above firewall):

arping -I eth1 10.0.0.250

I assume in that command that eth1 is the interface connected to AP. What will happen is that broadcast will be sent on the wireless network and the client will respond with its wireless MAC address even though that address is not used on the wireless network.

So, I hope the idea is clear now. To detect if there is some host cross connecting two networks I send arp request on a host (i.e. proxy host) for each possible IP address used on the the other network (i.e. local protected network in the figure above).

Note that it is possible to disable such behavior on Linux machines using sysctl variable /proc/sys/net/ipv4/conf/*/arp_filter. You can find more information, for example, here.

nmap games

Now, there is another problem. How to scan the whole network without manually trying each possible IP address. The first solution is, obviously, to use nmap. Nmap is a great tool for network scanning, but in this case it has a problem, I tried to run it in the following way, but unsuccessfuly:

# nmap -PR -Pn -e eth1 10.0.0.0/24
Starting Nmap 5.51 ( http://nmap.org ) at 2013-03-21 10:07 CET
nexthost: failed to determine route to 10.0.0.0
QUITTING!

Option -PR requires ARP scan, -Pn disables ping scan, and -e eth1 tells nmap to send packets via interface eth1. The problem is that I'm trying to scan network 10.0.0.0/24 on interface eth1 and there is no route in routig tables that tells kernel/nmap that network is really connected on interface eth1. So, nmap refuses to scan those addresses. One solution is to temporarily add that route:

ip route add 10.0.0.0/24 dev eth1

But this isn't an option if the route already exists in order for the hosts on the protected network to be able to access proxy, i.e. if there is route similar to the following one:

# ip ro sh
...
10.0.0.0/24 via 172.16.1.1 dev eth0
...

Again, I'm making a lot of assumptions here (network between proxy and firewall, IP addresses and interfaces) but I hope you understand the point. The given route is in the routing tables and removing it isn't an option.

Next try was using -sn switch, i.e. to disable port scan:

# nmap -PR -Pn -sn -e eth1 10.0.0.0/24
Starting Nmap 5.51 ( http://nmap.org ) at 2013-03-21 10:07 CET

Well, now nmap worked, sort of, because it showed all the hosts are up. Using tcpdump I found that it didn't send anything to the network. Reason: it thinks this is remote network, pings are disabled, arp cannot be used, and finally, because of -Pn, assumes all the hosts are up. So, I was again at the beginning.

Simple arping solution

Until I figure out how to force nmap to send arp probes without worrying  about routing tables here is a simple solution using arping command:

#!/bin/bash

TIMEOUT=4

for i in {1..254}
do
if arping -q -f -w \$TIMEOUT -I eth2 10.0.0.\$i
then
echo "10.0.0.$i is up"
fi
done

There are three problems with this solution:

1. In case your network has netmask other than /24 you'll have to change this script, i.e. it is a bit more complicated. How much, depends on the network mask.
2. The problem with this solution is that it is slow. For example, to scan 254 addresses and with timeout of 4 seconds, it will take about 17 minutes to scan the whole address range assuming no address is alive (what is actually desired state on the network).
3. Finally, timeout value is a bit tricky to determine. Majority of the responses are really quick, i.e. under a second. But some devices respond slower, i.e. when they entered some kind of a sleep state.

Still, it is a satisfactory solution until I find a way to use nmap for that purpose. If you know how, please, leave a comment.

Cisco's bug in ARP behavior - A story of firewall configuration oddisey...

Well, a very interesting situation happened to me. I was changing an old firewall with a new one and after switching I added secondary IP address to a firewall's public interface with the intention that all the traffic that was comming to that secondary IP address is redirected to one internal server. All nice except that it didn't work! I created such configuration numerous times and it simply had to work, but it didn't! The similar functionality for the primary IP address worked as expected, but secondary didn't! This was driving me nuts, until I finally figured out what was happening. This is a story of how I finally resolved this problem.

In cases when something isn't working as expected I use Wireshark, or better yet, tcpdump, and start to watch what's happening with packets on interfaces. I use tcpdump as it is more ubiquitous than Wireshark, i.e. available with default install on many operating systems, Unix like I mean. Now using this tool has one drawback, at least on Linux. The point where it catches packets is before PREROUTING chain and there is no way (at least I don't now how) to see packet flows between different chains. This is actually a restriction of Linux kernel so any other tool (Wireshark included) will behave the same.

Additional complication in this particular case was that to debug firewall I had to run it in production. This makes things quite complicated because when you run something in production that doesn't (fully) work as expected there will be many unhappy people and in the end you don't have much time to experiment, you have to revert old firewall so that people can continue to work. In the end this translates into longer debug period as you have relatively short time windows in which you can debug. Oh yeah, and it didn't helped that this was done at 8pm, outside of usual working hours, for various reasons I won't go into now.

So, using tcpdump I saw that packets with secondary address were reaching the firewall interface and they mysteriously disappeared within a box! Naturally, based on that I concluded that something strange is happening within Linux itself.

I have to admit that usually this would be a far reached hypothesis as it would mean that there is a bug in a relatively simple NAT configuration and it had to be due to the bug which would certainly be known. Quick googling revealed nothing at all and added a further confirmation that this hypothesis isn't good. But what kept me going in that direction was that I decided to use Firewall Builder as a tool to manage firewall and firewall rules. This was my first use of this tool ever (very good tool by the way!). The reason I selected that tool was that this firewall is used by one customer for which I intended to allow him to change rules by himself (so that I have less work to do :)). I wasn't particularly happy with the way rules are generated by that tool, and so I suspected that maybe it messed something, or I didn't define rules as it expects me to. To see if this is true, I flushed all the rules on the firewall and quickly generated a test case by manually entering iptables rules. Well, it turned out that it doesn't work either.

The next hypothesis was that FWBuilder somehow messed something within /proc filesystem. For example, it could be that I told him to be overlay restrictive. But trying out different combinations and poking throughout /proc/sys/net didn't help either, the things were not working and that was it!

Finally, at a moment of despair I started again tcpdump but this time I requested it to show me MAC addresses too. And then I noticed that destination MAC address doesn't belong to firewall! I rarely use this mode of tcpdump operation as L2 simply works, but now I realized what the problem was. The core of the problem was that the router (which is some Cisco) didn't change MAC address assigned to secondary IP address when I switched firewall. What I would usually do in such situation is to restart Cisco. But, since this router was within cabinet that I didn't have access to, and also I didn't have access to its power supply, it was not an option. Yet, it turned out that it is easy to "persuade" some device to change MAC address, just send it a gratuitous ARP response:

arping -s myaddr -I eth0 ciscoaddr

Both addresses are IP addresses, with myaddr being the address for which I want to change MAC address and ciscoaddr is device where I want this mapping to be changed. And that was it! Cisco now had correct MAC address and thing worked as expected. The primary address worked correctly because firewall itself sent a packet to Cisco and in that way changed MAC address belonging to primary IP address.

To conclude, this is a short version of everything that happened as I also used iptables' logging functionality (that obviously didn't help, as there was nothing to log for a start :)). Finally, there's left only one question to answer, i.e. How did I saw packets with "wrong" MAC address it tcpdump output? First, switch shouldn't send it to me, and second, interface should drop them before OS sees them (and by extension tcpdump). Well, switch was sending those frames to me because it's entry for MAC address expired and it didn't know where the old MAC address is, so it sent every frame to all the outputs/ports (apart from the receiving one, of course). The reason input interface didn't drop the packet was that sniffing tool places interface into promiscuous mode, and so it sees every frame that reaches it. Easy, and interesting how things combine to create problem and false clues. :)

arping command

arping command is a very useful Linux command. It's purpose is to send ARP request on the local network and to print received responses. It's a sort like ping command, but it works on layer 2 (data link layer) instead on a network layer like ping. Unlike ping command you have to specify egress interface to this command since otherwise it doesn't know where to send request, and the command itself is programmed to send it to some predetermined interface, e.g. eth0.

Most frequently, you'll use it like this:

arping -I wlan0 192.168.1.1

which, in this particular case, tells arping command to send requests asking for a link layer address corresponding to IP address 192.168.1.1. Requests should go via wlan0 interface.

Now you may wonder what's so special about this command. Well, the special part is that there is no way you can block it, unlike ping command. For example, Windows 7 by default blocks protocol packets used by ping command and thus you wont be able to check if host is alive using ping. With arping you can definitely determine if it is alive or not. But there is a restriction, you can use it only on a local, Ethernet-like, network! So there is no way it can be used over the network, at least not without some heavy tricks. Additional restriction is that you need administrative (root) privileges on a host, unlike ping command that can be run by any user.