Getting Libreswan to connect to Cisco ASA 5500

Here are some notes about problems I had while trying to make Libreswan connect to Cisco ASA. Note that I'm using certificate based authentication in a roadwarrior configuration. The setup is done on Fedora 19.

The procedure, at least in theory, is quite simple. Edit, /etc/ipsec.conf file, modify /etc/ipsec.d/ipsec.secrets file, import certificates into NSS database and then start libreswan daemon. Finally, activate the connection. Well, it turned out that the theory is very far from the practice. So, here we go.

Preparation steps

First, I used the following ipsec.conf file when I started to test connection to the ASA:

version 2.0     # conforms to second version of ipsec.conf specification
# basic configuration
config setup
    nat_traversal=yes
    nhelpers=1
    protostack=netkey
    interfaces=%defaultroute

conn VPN
    # Left side is RoadWarrior
    left=%defaultroute
    leftrsasigkey=%cert
    leftcert=SGROS
    leftca=ROOTCA
    leftid=%fromcert
    leftsendcert=always
    # Right side is Cisco
    right=1.1.1.1 # IP address of Cisco VPN
    rightrsasigkey=%cert
    rightcert=CISCO
    rightca=%same
    rightid=%fromcert
    # config
    type=tunnel
    keyingtries=2
    disablearrivalcheck=no
    authby=rsasig
    auth=esp
    keyexchange=ike
    auto=route
    remote_peer_type=cisco
    pfs=no

Note few things about this configuration. First, my client machine (roadwarrior) is a left node, while Cisco is the right one. Next, I didn't arrive to this configuration immediately, I had to experiment with the value of interfaces and left statements. The reason was that I'm assigned dynamic NATed address. So, those settings will cause openswan to automatically select appropriate values (interface and IP address) at the time I'm connecting to VPN. Also, certificate related stuff also took me some time to figure it out.

Into ipsec.secrets file I added the following line:

: RSA SGROS

this will cause openswan to use RSA key for authentication. Finally, I had to import certificates and keys into the NSS database. Note that NSS database is already precreated in /etc/ipsec.d directory. More specifically, the database consists of files cert8.db, key3.db and secmod.db. To see imported certificates (if there are any), use the following command:

# certutil -L -d /etc/ipsec.d/
Certificate Nickname               Trust Attributes
                                   SSL,S/MIME,JAR/XPI
ROOTCA                             CT,C,C
SGROS                              u,u,u
CISCO                              P,P,P

In my case there are three certificates in the database. Mine (SGROS), VPN's (CISCO) and CA that signed them (ROOTCA). Note that I'm referencing those certificates in ipsec.conf file. If you are configuring this database for the first time, it will be empty and you'll have to import all the certificates.

To import CA into your database, use the following command:

certutil -A -i rootca.pem -n ROOTCA -t "TC,TC,TC" -d /etc/ipsec.d/

Note that I'm assuming you have PEM version of certificate stored in the current directory (argument to the -i option). For the rest options, and their meaning, please consult man page.

To import your certificate, with a private key, use the following command:

certutil -A -i certkey.pfx -n SGROS -t "u,u,u" -d /etc/ipsec.d/

Note that this time certificate and private key are stored in PKCS#12 file (named certkey.pfx).

Finally, to import certificate of Cisco ASA, use the following command:

certutil -A -i rootca.pem -n ROOTCA -t "P,P,P" -d /etc/ipsec.d/

Note that the command is very similar to the one used to import ROOTCA, but the trust attributes (option -t) are different. Namely, you don't want ASA's certificate to be CA, i.e. to be able to issue new certificates.

Starting and debugging

To start Libreswan daemon, I used the following command:

ipsec pluto --stderrlog --config /etc/ipsec.conf --nofork

That way I was forcing it not to go to the background (--nofork) and to log to stderr (--stderrlog). Then in another terminal I would trigger VPN establishment using the following command:

ipsec auto --up VPN

The first problem was that Libreswan said it can not determine which end of the connection it is, i.e. I was receiving the following error message.

022 "VPN": We cannot identify ourselves with either end of this connection.

That error message took me some time to resolve. I tried everything possible to let Libreswan knows if it is left or right part in the configuration, which included changing roles several times, changing different parameters and other stuff. In the end, it turned out that that has nothing to do with the configuration file, namely the problem actually was missing kernel module!? NETKEY wasn't loaded, and libreswan couldn't access IPsec stack within the kernel. To be honest, it could be inferred from the log by nothing the following lines:

No Kernel XFRM/NETKEY interface detected
No Kernel KLIPS interface detected
No Kernel MASTKLIPS interface detected
Using 'no_kernel' interface code on 3.10.4-300.fc19.x86_64

But then again, some more informative error message would actually help a lot! In the end, the following command solved that problem:

modprobe af_key

and then, in the logs I saw the following line:

Using Linux XFRM/NETKEY IPsec interface code on 3.10.4-300.fc19.x86_64

that confirmed NETKEY was now accessible, and the error message about not knowing which end of the connection it is, disappeared.

Next, I had problem with peer's public key. The error message I received was:

003 "VPN" #1: no RSA public key known for 'C=HR, L=Zagreb, O=Some ORG, CN=Cisco ASA'

Again, I lost a lot of time trying to figure out why it can not access public key even though it is in the certificate. I also tried to extract public key and write it directly into configuration file. Nothing helped. Then, when I turned on debugging of x509 in ipsec.conf, I found some suspicious messages, like the following one:

added connection description "VPN"
| processing connection VPN
|   trusted_ca called with a=CN=ROOTCA b=
|   trusted_ca returning with failed
|   trusted_ca called with a=CN=ROOTCA b=\001\200\255\373
|   trusted_ca returning with failed

Note the garbage as the second argument of the function trusted_ca!? Googling around for something about this didn't reveal anything useful. But then, out of desperation I tried removing leftca and rightca parameters from ipsec.conf, and guess what! Everything started to work. Checking again at the logging output I saw that now b parameter has the same value as a.

Yet, it still didn't work and after some tinkering I suspected that on the Cisco side XAuth is enabled and required. This I concluded based on the log output where Libreswan says what it received from Cisco:

"VPN" #1: received Vendor ID payload [Cisco-Unity]
"VPN" #1: received Vendor ID payload [XAUTH]
"VPN" #1: ignoring unknown Vendor ID payload [...]
"VPN" #1: ignoring Vendor ID payload [Cisco VPN 3000 Series]

At first, I thought that Libreswan will support XAuth, but obviously, if it is not configured Libreswan can not use it. Also, looking into manual page it says there that Xauth is disabled by default. So, after adding the following statements into ipsec.conf file:

leftxauthclient=yes
leftxauthusername=sgros

and adding appropriate line into ipsec.secrets file:

@sgros : XAUTH "mypassword"

I managed to get further. Yet, it still didn't work. Looking again in the log output I realised that something was wrong with client configuration. Also, I got segfaults there that I didn't report to upstream for a simple fear that I might send some secret information. But, after adding the following statements into ipsec.conf segmentation fault was solved:

modecfgpull=yes
leftmodecfgclient=yes

In the logging output of Libreswan I saw that configuration parameters were properly obtained, i.e.:

"VPN" #1: modecfg: Sending IP request (MODECFG_I1)
"VPN" #1: received mode cfg reply
"VPN" #1: setting client address to 192.168.2.33/32
"VPN" #1: setting ip source address to 192.168.2.33/32
"VPN" #1: Received subnet 192.168.0.0/16, maskbits 16
"VPN" #1: transition from state STATE_MODE_CFG_I1 to state STATE_MAIN_I4

Now, it seemed like everything was connected, but ICMP probes were not going through. Using setkey command I checked that policies and associations are correctly installed into the kernel, which they were. I quickly realised that the problem was that Libreswan didn't assign IP address to my local interface, nor did it assign routes. That was easy to check by just listing interface's IP addresses. To see if this is really problem, I manually assigned address and route:

# ip addr add 192.168.2.33/32 dev wlp3s0
# ip ro add 192.168.0.0/22 src 192.168.2.33 via 192.168.1.1

and after that I was able to reach addresses within a destination network. Note that IP address given as argument to via keyword (in my case 192.168.1.1) isn't important since XFRM will change it anyway. So, the problem was why this address isn't added in the first place.

After some poking around I found that the script /usr/libexec/ipsec/_updown.netkey is called to setup all the parameters, and also, looking into that script, I found that it didn't do anything when pluto calls it using up-client parameter! So, no wonder nothing happened. I also found on the Internet post about that problem. The fix is simple, as shown in the post I linked, but it messes something with routes. After some further investigation I discovered that when adding locally assigned IP address, the script messes up netmask. To cut the story, I changed the following line:

-it="ip addr add ${PLUTO_MY_SOURCEIP%/*}/${PLUTO_PEER_CLIENT##*/} dev ${PLUTO_INTERFACE%:*}"
+it="ip addr add ${PLUTO_MY_CLIENT} dev ${PLUTO_INTERFACE%:*}"

and also, I changed the following lines for IP address removal:

-it="ip addr del ${PLUTO_MY_SOURCEIP%/*}/${PLUTO_PEER_CLIENT##*/} dev ${PLUTO_INTERFACE%:*}"
+it="ip addr del ${PLUTO_MY_CLIENT} dev ${PLUTO_INTERFACE%:*}"

You can get the complete patch here.

Some random notes

It might happen that Libreswan suddenly stops working and that you can not access network, i.e. you can only ping you local address, but not local router. In that case try to clear XFRM policy using setkey command:

setkey -FP

you can also check if there is anything with:

setkey -DP

Periodically scan network with nmap...

I think that it is a good idea to periodically scan network using nmap in order to take a snapshot of a current state, and to be able to track changes in the network. For that purpose I wrote the following quick and dirty bash script:

#!/bin/bash

# Interface on which scan should be performed. Multiple interfaces
# should be separated with spaces!
SCAN_INTERFACES="eth1"

# Network that should be scanned. If empty, or undefined, automatically
# deduce network attached to interface. Note that if you specified
# multiple interfaces than this variable should be undefined!
SCAN_NETWORKS=

#######################################################################
# THERE ARE NO MORE CONFIGURABLE PARTS AFTER THIS LINE
#######################################################################

TIMESTAMP=`date +%Y%m%d%H%M`
START=`date +%Y%m%d%H%m%S.%N`

cd /var/log/nmap || exit 1

for if in \$SCAN_INTERFACES
do
    # Find network to scan if it isn't specified...
    [ -z "\$SCAN_NETWORKS" -o "\$if" != "\$SCAN_INTERFACES" ] && SCAN_NETWORKS=`/sbin/ip ro sh dev \$if | grep -v via | cut -f1 -d" "`

    # Find addresses on the output interface so that we don't scan them
    EXCLUDE_LIST=`/sbin/ip addr sh dev \$if | awk '/inet / {print "--exclude ", substr(\$2, 1, index(\$2, "/")-1)}'`
    [ -z "\$SCAN_NETWORKS" ] && continue

    # Start scanning
    nmap -n -Pn -sS -O -sV -T4 -vv \${EXCLUDE_LIST} -oA nmap-\$if-\${TIMESTAMP} -e \$if ${SCAN_NETWORKS} >& nmap-scan-\$if-\${TIMESTAMP}.log
done

echo "START \$START END `date +%Y%m%d%H%m%S.%N`" >> /var/log/nmap-scan.log

exit 0

Note that some lines are wrapped due to the shortage of space. This script assumes several things in order to run properly:

1. You have a directory /var/log/nmap where all the result files will be placed.
2. nmap is version 6, but definitely not 4 because version 4 has some weaknesses.
3. You want to scan networks assigned to your interfaces.
4. The script is run under root user.

Now, after each run of this script you'll have four files left in /var/log/nmap each with the following extension:

1. nmap - this is a standard nmap output file
2. gnmap - greppable nmap output
3. xml - XML output file
4. log - Log file into which stdout and stderr were redirected during nmap's run.

It is also necessary to configure script to be run periodically. cron is ideal for that purpose. To achieve that, you can add the following entry to root's crontab:

0 */2 * * * full_path_and_name_to_your_script

Obviously, you'll have to change full_path_and_name_to_your_script with exact path and filename. In this case, you'll get the script to be run every two hours.

Detecting hosts that cross connect two networks...

There are different scenarios in which you can have a situation where some of your clients are connected in the same time to two different networks of different trust levels. This is dangerous because effectively they can be used as a staging point for potential attackers or malware to "jump" from less trusted network to more trusted one.

For example, suppose that you have protected internal wired LAN, and in the same time you have wireless LAN that is used for guests and allowed unrestricted access to the Internet, as shown in the figure below. Someone, from your internal and protected network, might intentionally or accidentally connect to the wireless network too, and in that case he/she will shortcut the two networks. If you thought that you can use MAC addresses to detect such hosts, you can not, for a simple reason that in one network host is connected using wired ethernet card with one MAC address, and in the second network it is connected using WLAN network card with another MAC address.

Hypothetical situation to illustrate how internal host might shortcut two networks of different trust levels

So, how to detect those hosts that shortcut two networks? Actually, there is an easy and elegant way. Namely, you can send ARP requests on one network for IP addresses on another network. ARP module, by default, doesn't know or care for routing tables so that it knows on which interface it should respond with specific addresses. If we look again on the figure above, this means that you can run the following command from the proxy host (the host above firewall):

arping -I eth1 10.0.0.250

I assume in that command that eth1 is the interface connected to AP. What will happen is that broadcast will be sent on the wireless network and the client will respond with its wireless MAC address even though that address is not used on the wireless network.

So, I hope the idea is clear now. To detect if there is some host cross connecting two networks I send arp request on a host (i.e. proxy host) for each possible IP address used on the the other network (i.e. local protected network in the figure above).

Note that it is possible to disable such behavior on Linux machines using sysctl variable /proc/sys/net/ipv4/conf/*/arp_filter. You can find more information, for example, here.

nmap games

Now, there is another problem. How to scan the whole network without manually trying each possible IP address. The first solution is, obviously, to use nmap. Nmap is a great tool for network scanning, but in this case it has a problem, I tried to run it in the following way, but unsuccessfuly:

# nmap -PR -Pn -e eth1 10.0.0.0/24
Starting Nmap 5.51 ( http://nmap.org ) at 2013-03-21 10:07 CET
nexthost: failed to determine route to 10.0.0.0
QUITTING!

Option -PR requires ARP scan, -Pn disables ping scan, and -e eth1 tells nmap to send packets via interface eth1. The problem is that I'm trying to scan network 10.0.0.0/24 on interface eth1 and there is no route in routig tables that tells kernel/nmap that network is really connected on interface eth1. So, nmap refuses to scan those addresses. One solution is to temporarily add that route:

ip route add 10.0.0.0/24 dev eth1

But this isn't an option if the route already exists in order for the hosts on the protected network to be able to access proxy, i.e. if there is route similar to the following one:

# ip ro sh
...
10.0.0.0/24 via 172.16.1.1 dev eth0
...

Again, I'm making a lot of assumptions here (network between proxy and firewall, IP addresses and interfaces) but I hope you understand the point. The given route is in the routing tables and removing it isn't an option.

Next try was using -sn switch, i.e. to disable port scan:

# nmap -PR -Pn -sn -e eth1 10.0.0.0/24
Starting Nmap 5.51 ( http://nmap.org ) at 2013-03-21 10:07 CET

Well, now nmap worked, sort of, because it showed all the hosts are up. Using tcpdump I found that it didn't send anything to the network. Reason: it thinks this is remote network, pings are disabled, arp cannot be used, and finally, because of -Pn, assumes all the hosts are up. So, I was again at the beginning.

Simple arping solution

Until I figure out how to force nmap to send arp probes without worrying  about routing tables here is a simple solution using arping command:

#!/bin/bash

TIMEOUT=4

for i in {1..254}
do
if arping -q -f -w \$TIMEOUT -I eth2 10.0.0.\$i
then
echo "10.0.0.$i is up"
fi
done

There are three problems with this solution:

1. In case your network has netmask other than /24 you'll have to change this script, i.e. it is a bit more complicated. How much, depends on the network mask.
2. The problem with this solution is that it is slow. For example, to scan 254 addresses and with timeout of 4 seconds, it will take about 17 minutes to scan the whole address range assuming no address is alive (what is actually desired state on the network).
3. Finally, timeout value is a bit tricky to determine. Majority of the responses are really quick, i.e. under a second. But some devices respond slower, i.e. when they entered some kind of a sleep state.

Still, it is a satisfactory solution until I find a way to use nmap for that purpose. If you know how, please, leave a comment.

Storing arpwatch output into database

arpwatch is very useful tool which logs its output via syslog and also sends mail alerts. Unfortunately, this isn't configurable, i.e. arpwatch, out-of-the-box, doesn't support any other way of logging.  One approach is to modify arpwatch to be able to log into some SQL database, but this isn't straightforward way, i.e. not an easy one. Namely, arpwatch is written in C, and besides, it's hard to know if this would be accepted by upstream (who ever that migh be).

So, I decided to go with a different approach. I configured arpwatch to log its output into log file and wrote a Python script that executes via cron and transfers all the data into the database. Here is how I did it along with all the scripts.

Configuring logging

The first step is to configure arpwatch to log its output into a separate file. This isn't possible to do in arpwatch itself, but it is possible to achieve it by configuring syslog, or rsyslog to be more precise. On CentOS 6 rsyslog is used that allows just that. All you have to do is to place a file named (for example) arpwatch.conf in directory /etc/rsyslog.d with the following content:

if $programname == 'arpwatch' then /var/log/arpwatch.log
&~

Don't forget to restart rsyslog after that. This will write anything logged by arpwatch binary into /var/log/arpwatch.log file. All the different log lines that can appear are documented in arpwatch's manual page so I won't replicate them here.

Configuring database

In my case I created a single table using the following SQL statement:

CREATE TABLE arpwatch (
  macaddr char(17) NOT NULL,
  ip_addr int(10) unsigned NOT NULL,
  state varchar(8) NOT NULL,
  timestamp datetime NOT NULL,
  oldmac char(17) DEFAULT NULL
)

I think it's pretty obvious what goes where. Only thing that might be strange is that I'm using INT(10) for IP address. But that is because SNORT also stores IP addresses in such a way so in order to be compatible with it I used it also. Also, what is missing is primary key, but for the time being I'm not using it.

Script

Here is the script that should be started from the cron. For example, store it in /usr/local/sbin directory and to start it every 20 minutes add the following line (as root user) to cron using 'crontab -e' command:

*/20 * * * * /usr/local/sbin/arpwatchlog2sql.py

Note that the script expects configuration file. Here is a sample configuration file you'll have to modify. The script expects configuration file to be in its current directory, but you can place it into /usr/local/etc and modify the line CONFIGFILE in script accordingly.

Log rotation

Finally, you should be certain that logs are properly handled, i.e. rotated along with other logs. Since arpwatch is logging via syslog, that means that you have to modify rsyslog's log configuration file, i.e. /etc/logrotate.d/syslog. In there you'll see that logfiles maintained by rsyslog are enumerated, one per line. Just add arpwatch.log to that list and that should be it.

OSSEC active response...

This is still work in progress (I need to add more about configuration part). But since OSSEC is so badly documented and I don't know when this will be finished I'm publishing it now.

Prompted by the problem caused by the OSSEC active response, I decided to try to debug why an error is occurring in logs and fine tune it. But in order to do that I had first to understand how it works. There is a section in OSSEC manual about active response, but what wasn't immediately clear to me is who runs active response and where this is configured. But soon I found out that the active response is run on the client via agent (actually this part wasn't problematic) but that active response is configured on the server and the server is the one that instructs agents to run active response.

On server you'll find active response configuration in $OSSEC/etc/ossec.conf. In that file you have several parts of the configuration:

1. Set of <command> blocks. Each one defines a command for active response and the arguments it expects. Note that those commands have to exist on agents.
2. Set of <active-response> blocks that define circumstances under which there will be active response and which command will be executed.

Scripts for active response

Scripts for active response receive five arguments. The first argument is either add or delete. If add is specified, given IP address should be added to a ban list, in case delete is specified address should be removed from the ban list. The second argument is a user name. The third argument is offending IP address. Fourth argument is Unix timestamp (microsecond resolution) when the script was called. Final argument is rule number that triggered active response.

firewall-drop.sh script

This script ships with OSSEC and it adds or removes IP addresses from firewall. It is a relatively simple shell script that accepts command line arguments as specified in the introduction of this section and installs IP address in the ban list (or removes it from there, depending on the command line arguments). The script could be run on Linux, FreeBSD, NetBSD, Solaris and AIX. The following description is specific to Linux behavior even though some things will be common across different platforms.

This script logs its activity into active-responses.log file (in my case in /var/ossec/logs directory). For each invocation one line is emitted into that file. Here is an example of one log entry:

Thu Jan 19 13:52:29 CET 2012 /var/ossec/active-response/bin/firewall-drop.sh add - 193.41.36.141 1326977549.1358625 3301

First group of fields is timestamp when the log entry was generated. Next is a full path and name of the script itself. Finally, all five arguments given to script are also recorded.

Ocasionally, you'll also see error messages like the following one:

Thu Jan 19 06:17:19 CET 2012 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - 208.115.236.82 1326949601.551727 3301

This log entry is a bit misleading. What happened is that iptables command returned exit code 1 (judging from the log entry it could be interpreted as if it returned something else and 1 was expected, but that's not true). What is important to note is that you'll usually see multiple log entries like the previous one grouped together and the only thing that will differ between them is the number 1 (shown in bold above). What basically happens is that in case of an error returned by iptables command the script tries to run it five times, so, you'll usually see five records and each of those records is numbered.

There are two places where this error might occur. The first one is when the IP address is removed from INPUT chain, while the other is when it is removed from FORWARD chain.

The only reason this error can occur is because someone or something already removed IP address (or added it). But, this should not happen. Still, it happened to em but I don't know the reason for that.

Looking into this script it was clear that it could be improved from the logging perspective. Currently, if you manually run this command from the command line it will write part of error messages to stdout and some other to log file.

Manual control of scripts

Scripts for active response can be started from the server using agent_control tool. Note that the help message of this tool isn't updated to reflect real arguments (bug?) so I had to look into source to infer how to call it. Let me give you several examples of its use.

To list all available agent use this command in the following way:

# ./agent_control -l
   ID: 000, Name: agent0.somedomain.local (server), IP: 127.0.0.1, Active/Local
   ID: 001, Name: agent1.somedomain.local, IP: 192.168.1.2, Active
   ID: 002, Name: agent2.somedomain.local, IP: 192.168.1.3, Active
   ID: 003, Name: agent3.somedomain.local, IP: 192.168.1.4, Disconnected

The output of the command is a list of known agents, either active or non-active. In case you want only active agents use -lc option instead.

Next, if you want to find out some information about a certain agent, you can query it in the following way:

# ./agent_control -i 002

OSSEC HIDS agent_control. Agent information:
   Agent ID:   002
   Agent Name: agent2.somedomain.local
   IP address: 192.168.1.3
   Status:     Active

   Operating system:    Linux agent2.somedomain.local 2.6.32-131.17.1.el6.x86_64..
   Client version:      OSSEC HIDS v2.5-SNP-100907
   Last keep alive:     Thu Jan 19 13:26:01 2012

   Syscheck last started  at: Thu Jan 19 12:10:16 2012
   Rootcheck last started at: Thu Jan 19 06:33:06 2012

To activate active response on a certain agent use the following form of the agent_control command:

./agent_control -b 1.1.1.1 -f firewall-drop600 -u 002

Here, the IP address to be blocked is argument of the -b option (in this case 1.1.1.1). There could be more responses available (defined in ossec.conf on server) and the option -f selects which one to run. To see which responses are available use the option -L, like this:

# ./agent_control -L

OSSEC HIDS agent_control. Available active responses:

   Response name: host-deny600, command: host-deny.sh
   Response name: firewall-drop600, command: firewall-drop.sh

Agent on which this command should initiate active response is specified via ID given as the parameter to option -u. Note that, if you look into help output of the agent_control, this option is not listed, at least not in the version 2.6.0. There is a bit of inconsistency here, as some commands use agent ID as a parameter, while this one requires separate parameter. It would be more uniform if all command would instead use -u option.

Note that when you manually initiate active response then fourth argument to the script will be '(from_the_server)' and the fifth argument will be '(no_rule_id)'.

Also, the rule that was added can not be removed with agent_control, you have to wait for it to timeout when it will be automatically removed.