Who's listening on an interface...

While I was trying to deduce whether arpwatch honors multiple -i options and listens on multiple interfaces I had a problem of detecting on which interface exactly does it listen? To determine that, I started arpwatch in the following way:

arpwatch -i wlan0 -i em1

but that didn't report to me which interface did the command bound to. Using -d option (debug) didn't help either. So, the first attempt was looking into files open by the command. It can be done using lsof command, or by directly looking into aprwatch's proc directory. So, I found out PID (in this particular case that was 23833) of the command (use ps for that) and then I went into directory /proc/PID/fd. In there, I saw the following content:

# cd /proc/23833/fd
# ls -l
total 0
lrwx------. 1 root root 64 Feb 12 12:32 0 -> socket:[27121497]
lrwx------. 1 root root 64 Feb 12 12:32 1 -> socket:[27121498]

This wasn't so useful! Actually, it tells me that arpwatch closed it's stdin and stdout descriptors and opened only appropriate sockets to get arp frames. lsof command also didn't show anything direct:

# lsof -p 23833 -n
COMMAND    PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
arpwatch 23833 root  cwd    DIR              253,2     4096   821770 /var/lib/arpwatch
arpwatch 23833 root  rtd    DIR              253,2     4096        2 /
arpwatch 23833 root  txt    REG              253,2    34144  2672471 /usr/sbin/arpwatch
arpwatch 23833 root  mem    REG              253,2   168512  2228280 /lib64/ld-2.14.90.so
arpwatch 23833 root  mem    REG              253,2  2068608  2228301 /lib64/libc-2.14.90.so
arpwatch 23833 root  mem    REG              253,2   235944  2675058 /usr/lib64/libpcap.so.1.1.1
arpwatch 23833 root  mem    REG                0,6          27121497 socket:[27121497] (stat: No such file or directory)
arpwatch 23833 root    0u  pack           27121497      0t0      ALL type=SOCK_RAW
arpwatch 23833 root    1u  unix 0xffff8802e3024ac0      0t0 27121498 socket

But it did show that there is a raw socket in use, but not anything more than that. So, the next step was to find out how to list all raw sockets? netstat can list all open and listening sockets, so, looking into man page it turns out that the option --raw or -w is used to show raw sockets, i.e.

# netstat -w
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State     

Well, neither this was very useful, but by default netstat doesn't show listening sockets so I repeated command adding -l option:

# netstat -w -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State     
raw        0      0 *:icmp                      *:*                         7

So, this is definitely not what I'm looking for. This RAW socket listens for ICMP messages, and arpwatch definitely isn't capturing those. In man page it also says that netstat looks for information about raw sockets from /proc/net/raw file, so I looked into its content:

# cat /proc/net/raw
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
   1: 00000000:0001 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 47105 2 ffff880308e18340 0

Also not useful! There was inode listed (47105) but how to find out information about particular inode? I looked throughout /proc file system but didn't find anything. I also checked lsof manual but wasn't able to find something useful (though I didn't read manual from start to finish, I just search word inode!).

Then, I remembered that there is a ss command that is specific to Linux, and that is used to provide information about sockets! So, I looked into man page and there it says that the option -0 (or -f link) is used to show PACKET sockets, so I tried:

# ss -f link
Netid  State      Recv-Q Send-Q   Local Address:Port       Peer Address:Port

Again nothing, but it quickly occured to me that it doesn't show listening sockets by default, so I tried with -l (and -n to avoid any resolving):

# ss -f link -ln
Netid  State      Recv-Q Send-Q     Local Address:Port       Peer Address:Port
p_raw  UNCONN     0      0                      *:em1                    *    
p_dgr  UNCONN     0      0                [34958]:wlan0                  *

Woohoo, I was on something, finally! I see raw socket bound to em1 interface (note that I started arpwach with the intention that it listens on wlan0 and em1 interfaces!) Only, I still don't know who is exactly using it. I only see that the other socket is datagram type, meaning network layer, and probably not used by arpwatch. man page for ss again helped, it says to use -p option to find out which process owns a socket, so I tried:

# ss -f link -lnp
Netid  State      Recv-Q Send-Q     Local Address:Port       Peer Address:Port
p_raw  UNCONN     0      0                      *:em1                    *      users:(("arpwatch",23833,0))
p_dgr  UNCONN     0      0                [34958]:wlan0                  *      users:(("wpa_supplicant",1425,9))

Wow!! That was it! I found out that arwatch is listening only to a single interface, and later I confirmed it by looking into the source! I also saw that the other socket is used by wpa_supplicant, i.e. for a wireless network management purposes.

One final thing bothered me. From where does ss take this information? But it's easy to find out that, use strace! :) So, using strace I found out that ss is using /proc/net/packet file:

# cat /proc/net/packet
sk               RefCnt Type Proto  Iface R Rmem   User   Inode
ffff880308cec000 3      3    0003   2     1 0      0      27121497
ffff880004358800 3      2    888e   7     1 0      0      25292685

Maybe I would get to that earlier if I had looked more closely into available files in /proc/net when /proc/net/raw turned out to be wrong file! But it doesn't matter, this search was fun and educative. :)