Showing posts with label arpwatch. Show all posts
Showing posts with label arpwatch. Show all posts

Tuesday, March 12, 2013

Storing arpwatch output into database

arpwatch is very useful tool which logs its output via syslog and also sends mail alerts. Unfortunately, this isn't configurable, i.e. arpwatch, out-of-the-box, doesn't support any other way of logging.  One approach is to modify arpwatch to be able to log into some SQL database, but this isn't straightforward way, i.e. not an easy one. Namely, arpwatch is written in C, and besides, it's hard to know if this would be accepted by upstream (who ever that migh be).

So, I decided to go with a different approach. I configured arpwatch to log its output into log file and wrote a Python script that executes via cron and transfers all the data into the database. Here is how I did it along with all the scripts.

Configuring logging

The first step is to configure arpwatch to log its output into a separate file. This isn't possible to do in arpwatch itself, but it is possible to achieve it by configuring syslog, or rsyslog to be more precise. On CentOS 6 rsyslog is used that allows just that. All you have to do is to place a file named (for example) arpwatch.conf in directory /etc/rsyslog.d with the following content:
if $programname == 'arpwatch' then /var/log/arpwatch.log
&~
Don't forget to restart rsyslog after that. This will write anything logged by arpwatch binary into /var/log/arpwatch.log file. All the different log lines that can appear are documented in arpwatch's manual page so I won't replicate them here.

Configuring database

In my case I created a single table using the following SQL statement:
CREATE TABLE arpwatch (
  macaddr char(17) NOT NULL,
  ip_addr int(10) unsigned NOT NULL,
  state varchar(8) NOT NULL,
  timestamp datetime NOT NULL,
  oldmac char(17) DEFAULT NULL
)
I think it's pretty obvious what goes where. Only thing that might be strange is that I'm using INT(10) for IP address. But that is because SNORT also stores IP addresses in such a way so in order to be compatible with it I used it also. Also, what is missing is primary key, but for the time being I'm not using it.

Script

Here is the script that should be started from the cron. For example, store it in /usr/local/sbin directory and to start it every 20 minutes add the following line (as root user) to cron using 'crontab -e' command:
*/20 * * * * /usr/local/sbin/arpwatchlog2sql.py
Note that the script expects configuration file. Here is a sample configuration file you'll have to modify. The script expects configuration file to be in its current directory, but you can place it into /usr/local/etc and modify the line CONFIGFILE in script accordingly.

Log rotation

Finally, you should be certain that logs are properly handled, i.e. rotated along with other logs. Since arpwatch is logging via syslog, that means that you have to modify rsyslog's log configuration file, i.e. /etc/logrotate.d/syslog. In there you'll see that logfiles maintained by rsyslog are enumerated, one per line. Just add arpwatch.log to that list and that should be it.

Thursday, August 2, 2012

arpwatch: too short requests error

Today, I got a lot of errors from arpwatch (actually arpwatch-NG) that look like this:
arpwatch: short (want 42)
There is no point in googling this, even though I tried. :) There is one post that mentions some Debian bug, or something like that.

Anyway, the problem is that something on the network has a bug, or is wrongly configured, and arpwatch doesn't log sending MAC address from frames in case it receives something erroneous that is supposed to be ARP. Of course it could be discussed as to how much this information would be useful, but, it would at least give some clue who's sending this.

In the end, the only thing that could be done in this case is to sniff network and wait for the error to repeat.

Update

I figured out what was causing this errors. It was nmap that I put to regularly scan the network! :)

Sunday, February 12, 2012

Who's listening on an interface...

While I was trying to deduce whether arpwatch honors multiple -i options and listens on multiple interfaces I had a problem of detecting on which interface exactly does it listen? To determine that, I started arpwatch in the following way:
arpwatch -i wlan0 -i em1
but that didn't report to me which interface did the command bound to. Using -d option (debug) didn't help either. So, the first attempt was looking into files open by the command. It can be done using lsof command, or by directly looking into aprwatch's proc directory. So, I found out PID (in this particular case that was 23833) of the command (use ps for that) and then I went into directory /proc/PID/fd. In there, I saw the following content:
# cd /proc/23833/fd
# ls -l
total 0
lrwx------. 1 root root 64 Feb 12 12:32 0 -> socket:[27121497]
lrwx------. 1 root root 64 Feb 12 12:32 1 -> socket:[27121498]
This wasn't so useful! Actually, it tells me that arpwatch closed it's stdin and stdout descriptors and opened only appropriate sockets to get arp frames. lsof command also didn't show anything direct:
# lsof -p 23833 -n
COMMAND    PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
arpwatch 23833 root  cwd    DIR              253,2     4096   821770 /var/lib/arpwatch
arpwatch 23833 root  rtd    DIR              253,2     4096        2 /
arpwatch 23833 root  txt    REG              253,2    34144  2672471 /usr/sbin/arpwatch
arpwatch 23833 root  mem    REG              253,2   168512  2228280 /lib64/ld-2.14.90.so
arpwatch 23833 root  mem    REG              253,2  2068608  2228301 /lib64/libc-2.14.90.so
arpwatch 23833 root  mem    REG              253,2   235944  2675058 /usr/lib64/libpcap.so.1.1.1
arpwatch 23833 root  mem    REG                0,6          27121497 socket:[27121497] (stat: No such file or directory)
arpwatch 23833 root    0u  pack           27121497      0t0      ALL type=SOCK_RAW
arpwatch 23833 root    1u  unix 0xffff8802e3024ac0      0t0 27121498 socket
But it did show that there is a raw socket in use, but not anything more than that. So, the next step was to find out how to list all raw sockets? netstat can list all open and listening sockets, so, looking into man page it turns out that the option --raw or -w is used to show raw sockets, i.e.
# netstat -w
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State   
  
Well, neither this was very useful, but by default netstat doesn't show listening sockets so I repeated command adding -l option:
# netstat -w -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State     
raw        0      0 *:icmp                      *:*                         7
So, this is definitely not what I'm looking for. This RAW socket listens for ICMP messages, and arpwatch definitely isn't capturing those. In man page it also says that netstat looks for information about raw sockets from /proc/net/raw file, so I looked into its content:
# cat /proc/net/raw
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
   1: 00000000:0001 00000000:0000 07 00000000:00000000 00:00000000 00000000     0        0 47105 2 ffff880308e18340 0
Also not useful! There was inode listed (47105) but how to find out information about particular inode? I looked throughout /proc file system but didn't find anything. I also checked lsof manual but wasn't able to find something useful (though I didn't read manual from start to finish, I just search word inode!).

Then, I remembered that there is a ss command that is specific to Linux, and that is used to provide information about sockets! So, I looked into man page and there it says that the option -0 (or -f link) is used to show PACKET sockets, so I tried:
# ss -f link
Netid  State      Recv-Q Send-Q   Local Address:Port       Peer Address:Port
Again nothing, but it quickly occured to me that it doesn't show listening sockets by default, so I tried with -l (and -n to avoid any resolving):
# ss -f link -ln
Netid  State      Recv-Q Send-Q     Local Address:Port       Peer Address:Port
p_raw  UNCONN     0      0                      *:em1                    *    
p_dgr  UNCONN     0      0                [34958]:wlan0                  *
Woohoo, I was on something, finally! I see raw socket bound to em1 interface (note that I started arpwach with the intention that it listens on wlan0 and em1 interfaces!) Only, I still don't know who is exactly using it. I only see that the other socket is datagram type, meaning network layer, and probably not used by arpwatch. man page for ss again helped, it says to use -p option to find out which process owns a socket, so I tried:
# ss -f link -lnp
Netid  State      Recv-Q Send-Q     Local Address:Port       Peer Address:Port
p_raw  UNCONN     0      0                      *:em1                    *      users:(("arpwatch",23833,0))
p_dgr  UNCONN     0      0                [34958]:wlan0                  *      users:(("wpa_supplicant",1425,9))
Wow!! That was it! I found out that arwatch is listening only to a single interface, and later I confirmed it by looking into the source! I also saw that the other socket is used by wpa_supplicant, i.e. for a wireless network management purposes.

One final thing bothered me. From where does ss take this information? But it's easy to find out that, use strace! :) So, using strace I found out that ss is using /proc/net/packet file:
# cat /proc/net/packet
sk               RefCnt Type Proto  Iface R Rmem   User   Inode
ffff880308cec000 3      3    0003   2     1 0      0      27121497
ffff880004358800 3      2    888e   7     1 0      0      25292685
Maybe I would get to that earlier if I had looked more closely into available files in /proc/net when /proc/net/raw turned out to be wrong file! But it doesn't matter, this search was fun and educative. :)

Tuesday, January 31, 2012

arpwatch on multiple interfaces

I'm regularly using arpwatch on all servers I install in order to track MAC changes and to notice potential MAC spoofings. But the problem is that on CentOS 6.2 the startup script shipped with arpwatch (package arpwatch-2.1a15-14.el6.x86_64) doesn't support multiple interfaces. More specifically, I can tell arpwatch on which interface to listen by modifying OPTIONS variable in /etc/sysconfig/arpwatch file and inserting -i <interface> option. But, I'm still restricted to a single interface. That is, it is possible to specify multiple -i options, but arpwatch still listens only on a single interface. I checked that in the source (version 2.1a15), and the last -i command is in effect, the previous one's are ignored.

So, I modified startup script so that it now accepts INTERFACES variable within /etc/syconfig/arpwatch configuration file and starts arpwatch on each specified interface. If this variable isn't defined then it behaves as before. For example, to start it on interfaces eth0 and eth1 you should add the following line in /etc/syconfig/arpwatch:
INTERFACES="eth0 eth1" 
The basic idea behind this change is to start arpwatch tool multiple times, once per each specified interface. Also, to each instance I give different database (arp.dat) so that multiple instances don't overwrite each other data.

Note that the script is a bit rough on edges, i.e. it properly behaves during startup phase, but not on shudown. Also, I embedded fixed path to data files. I'll improve this script in a due course when I find more time, or when it turns out that it's necessary to do so. :)

[20120203] Update: I had a an error in script because of which database files were placed in wrong directory and, as a consequence, arpwatch couldn't write database when it was exiting. Now, the script is updated and it works, furthermore, I tested stoping arpwatch using that script and it also works

About Me

scientist, consultant, security specialist, networking guy, system administrator, philosopher ;)

Blog Archive