Getting Libreswan to connect to Cisco ASA 5500

Here are some notes about problems I had while trying to make Libreswan connect to Cisco ASA. Note that I'm using certificate based authentication in a roadwarrior configuration. The setup is done on Fedora 19.

The procedure, at least in theory, is quite simple. Edit, /etc/ipsec.conf file, modify /etc/ipsec.d/ipsec.secrets file, import certificates into NSS database and then start libreswan daemon. Finally, activate the connection. Well, it turned out that the theory is very far from the practice. So, here we go.

Preparation steps

First, I used the following ipsec.conf file when I started to test connection to the ASA:

version 2.0     # conforms to second version of ipsec.conf specification
# basic configuration
config setup
    nat_traversal=yes
    nhelpers=1
    protostack=netkey
    interfaces=%defaultroute

conn VPN
    # Left side is RoadWarrior
    left=%defaultroute
    leftrsasigkey=%cert
    leftcert=SGROS
    leftca=ROOTCA
    leftid=%fromcert
    leftsendcert=always
    # Right side is Cisco
    right=1.1.1.1 # IP address of Cisco VPN
    rightrsasigkey=%cert
    rightcert=CISCO
    rightca=%same
    rightid=%fromcert
    # config
    type=tunnel
    keyingtries=2
    disablearrivalcheck=no
    authby=rsasig
    auth=esp
    keyexchange=ike
    auto=route
    remote_peer_type=cisco
    pfs=no

Note few things about this configuration. First, my client machine (roadwarrior) is a left node, while Cisco is the right one. Next, I didn't arrive to this configuration immediately, I had to experiment with the value of interfaces and left statements. The reason was that I'm assigned dynamic NATed address. So, those settings will cause openswan to automatically select appropriate values (interface and IP address) at the time I'm connecting to VPN. Also, certificate related stuff also took me some time to figure it out.

Into ipsec.secrets file I added the following line:

: RSA SGROS

this will cause openswan to use RSA key for authentication. Finally, I had to import certificates and keys into the NSS database. Note that NSS database is already precreated in /etc/ipsec.d directory. More specifically, the database consists of files cert8.db, key3.db and secmod.db. To see imported certificates (if there are any), use the following command:

# certutil -L -d /etc/ipsec.d/
Certificate Nickname               Trust Attributes
                                   SSL,S/MIME,JAR/XPI
ROOTCA                             CT,C,C
SGROS                              u,u,u
CISCO                              P,P,P

In my case there are three certificates in the database. Mine (SGROS), VPN's (CISCO) and CA that signed them (ROOTCA). Note that I'm referencing those certificates in ipsec.conf file. If you are configuring this database for the first time, it will be empty and you'll have to import all the certificates.

To import CA into your database, use the following command:

certutil -A -i rootca.pem -n ROOTCA -t "TC,TC,TC" -d /etc/ipsec.d/

Note that I'm assuming you have PEM version of certificate stored in the current directory (argument to the -i option). For the rest options, and their meaning, please consult man page.

To import your certificate, with a private key, use the following command:

certutil -A -i certkey.pfx -n SGROS -t "u,u,u" -d /etc/ipsec.d/

Note that this time certificate and private key are stored in PKCS#12 file (named certkey.pfx).

Finally, to import certificate of Cisco ASA, use the following command:

certutil -A -i rootca.pem -n ROOTCA -t "P,P,P" -d /etc/ipsec.d/

Note that the command is very similar to the one used to import ROOTCA, but the trust attributes (option -t) are different. Namely, you don't want ASA's certificate to be CA, i.e. to be able to issue new certificates.

Starting and debugging

To start Libreswan daemon, I used the following command:

ipsec pluto --stderrlog --config /etc/ipsec.conf --nofork

That way I was forcing it not to go to the background (--nofork) and to log to stderr (--stderrlog). Then in another terminal I would trigger VPN establishment using the following command:

ipsec auto --up VPN

The first problem was that Libreswan said it can not determine which end of the connection it is, i.e. I was receiving the following error message.

022 "VPN": We cannot identify ourselves with either end of this connection.

That error message took me some time to resolve. I tried everything possible to let Libreswan knows if it is left or right part in the configuration, which included changing roles several times, changing different parameters and other stuff. In the end, it turned out that that has nothing to do with the configuration file, namely the problem actually was missing kernel module!? NETKEY wasn't loaded, and libreswan couldn't access IPsec stack within the kernel. To be honest, it could be inferred from the log by nothing the following lines:

No Kernel XFRM/NETKEY interface detected
No Kernel KLIPS interface detected
No Kernel MASTKLIPS interface detected
Using 'no_kernel' interface code on 3.10.4-300.fc19.x86_64

But then again, some more informative error message would actually help a lot! In the end, the following command solved that problem:

modprobe af_key

and then, in the logs I saw the following line:

Using Linux XFRM/NETKEY IPsec interface code on 3.10.4-300.fc19.x86_64

that confirmed NETKEY was now accessible, and the error message about not knowing which end of the connection it is, disappeared.

Next, I had problem with peer's public key. The error message I received was:

003 "VPN" #1: no RSA public key known for 'C=HR, L=Zagreb, O=Some ORG, CN=Cisco ASA'

Again, I lost a lot of time trying to figure out why it can not access public key even though it is in the certificate. I also tried to extract public key and write it directly into configuration file. Nothing helped. Then, when I turned on debugging of x509 in ipsec.conf, I found some suspicious messages, like the following one:

added connection description "VPN"
| processing connection VPN
|   trusted_ca called with a=CN=ROOTCA b=
|   trusted_ca returning with failed
|   trusted_ca called with a=CN=ROOTCA b=\001\200\255\373
|   trusted_ca returning with failed

Note the garbage as the second argument of the function trusted_ca!? Googling around for something about this didn't reveal anything useful. But then, out of desperation I tried removing leftca and rightca parameters from ipsec.conf, and guess what! Everything started to work. Checking again at the logging output I saw that now b parameter has the same value as a.

Yet, it still didn't work and after some tinkering I suspected that on the Cisco side XAuth is enabled and required. This I concluded based on the log output where Libreswan says what it received from Cisco:

"VPN" #1: received Vendor ID payload [Cisco-Unity]
"VPN" #1: received Vendor ID payload [XAUTH]
"VPN" #1: ignoring unknown Vendor ID payload [...]
"VPN" #1: ignoring Vendor ID payload [Cisco VPN 3000 Series]

At first, I thought that Libreswan will support XAuth, but obviously, if it is not configured Libreswan can not use it. Also, looking into manual page it says there that Xauth is disabled by default. So, after adding the following statements into ipsec.conf file:

leftxauthclient=yes
leftxauthusername=sgros

and adding appropriate line into ipsec.secrets file:

@sgros : XAUTH "mypassword"

I managed to get further. Yet, it still didn't work. Looking again in the log output I realised that something was wrong with client configuration. Also, I got segfaults there that I didn't report to upstream for a simple fear that I might send some secret information. But, after adding the following statements into ipsec.conf segmentation fault was solved:

modecfgpull=yes
leftmodecfgclient=yes

In the logging output of Libreswan I saw that configuration parameters were properly obtained, i.e.:

"VPN" #1: modecfg: Sending IP request (MODECFG_I1)
"VPN" #1: received mode cfg reply
"VPN" #1: setting client address to 192.168.2.33/32
"VPN" #1: setting ip source address to 192.168.2.33/32
"VPN" #1: Received subnet 192.168.0.0/16, maskbits 16
"VPN" #1: transition from state STATE_MODE_CFG_I1 to state STATE_MAIN_I4

Now, it seemed like everything was connected, but ICMP probes were not going through. Using setkey command I checked that policies and associations are correctly installed into the kernel, which they were. I quickly realised that the problem was that Libreswan didn't assign IP address to my local interface, nor did it assign routes. That was easy to check by just listing interface's IP addresses. To see if this is really problem, I manually assigned address and route:

# ip addr add 192.168.2.33/32 dev wlp3s0
# ip ro add 192.168.0.0/22 src 192.168.2.33 via 192.168.1.1

and after that I was able to reach addresses within a destination network. Note that IP address given as argument to via keyword (in my case 192.168.1.1) isn't important since XFRM will change it anyway. So, the problem was why this address isn't added in the first place.

After some poking around I found that the script /usr/libexec/ipsec/_updown.netkey is called to setup all the parameters, and also, looking into that script, I found that it didn't do anything when pluto calls it using up-client parameter! So, no wonder nothing happened. I also found on the Internet post about that problem. The fix is simple, as shown in the post I linked, but it messes something with routes. After some further investigation I discovered that when adding locally assigned IP address, the script messes up netmask. To cut the story, I changed the following line:

-it="ip addr add ${PLUTO_MY_SOURCEIP%/*}/${PLUTO_PEER_CLIENT##*/} dev ${PLUTO_INTERFACE%:*}"
+it="ip addr add ${PLUTO_MY_CLIENT} dev ${PLUTO_INTERFACE%:*}"

and also, I changed the following lines for IP address removal:

-it="ip addr del ${PLUTO_MY_SOURCEIP%/*}/${PLUTO_PEER_CLIENT##*/} dev ${PLUTO_INTERFACE%:*}"
+it="ip addr del ${PLUTO_MY_CLIENT} dev ${PLUTO_INTERFACE%:*}"

You can get the complete patch here.

Some random notes

It might happen that Libreswan suddenly stops working and that you can not access network, i.e. you can only ping you local address, but not local router. In that case try to clear XFRM policy using setkey command:

setkey -FP

you can also check if there is anything with:

setkey -DP

SSH VPNs: Bridged connection to LAN using tap

In the previous post I showed how to create SSH tunnel that ends on a network  layer (i.e. ppp and point-to-point based vpns) and on a link layer (ethernet type). Now I'm continuing with a series of scenarios that do configuration on a network layer (and few on a link layer too).

This is the first scenario in which I want remote host to look like it is directly attached to a local network. The network layout for this scenario is shown in the following figure:

As you can see there is a local network that has address 192.168.0.0/24. On that local network there is a gateway (192.168.0.1) as well as remote server (192.168.0.30) that we are going to use as the end point of a tunnel. What we want to achieve is that our laptop, that is somewhere on the Internet and has IP address 10.2.4.60, behaves as if it is attached directly on a local network and to use IP address 192.168.0.40. This kind of setup when a single remote machine connects to some network remotely via VPN is offten called road-warrior scenario. To accomplish this we are going to use bridging built into the Linux kernel. In the future post I'll describe variant of this setup that uses forwarding (i.e. routing) to achieve the same thing.

Preparation steps

The idea (for this scenario) is to use briging. But since manipulating bridge and active interface might (and very probably will) disconnect you, it is better to first configure brigde with only a single interface, the one that connects remote computer to a local network. In case you are using CentOS (RHEL/Fedora or some derivative) then do the following on your remote server:

1. Go into directory /etc/sysconfig/network-scripts.
2. Copy file ifcfg-eth0 into ifcfg-br0 (I'm assuming that your active interface is eth0, if it is something else change the name accordingly). Also make a copy of that file in case you want to revert changes! Remove UUID line if there is any.
3. Edit file ifcfg-eth0. Add line BRIDGE=br0, remove all lines that specify IP address and related parameters (broadcast, netmask, gateway, DNS, ...). Also be certain that BOOTPROTO parameter is set to none. Remove UUID line if there is any.
4. Modify ifcfg-br0. Change type from Ethernet into Brige, also change name from eth0 into br0 and finally add line STP=off.

Now, restart machine and see if everything is OK. First, you should be able to connect to the machine, and second, there should be new interface, br0,  which has to have IP address of interface eth0. Interface eth0 itself should work (UP, LOWER_UP flags) and also it must not have IP address attached.

One other thing you have to take care of, firewall. In case you have firewall configured (which you should!), during this test it's best to disable it and enable it later. To disable firewall on CentOS/Fedora (and similar distributions) do the following:

service iptables stop

This will turn off firewall until you turn it on again, or until you restart machine. To turn it off permanently (strongly not advised!) do the following:

chkconfig iptables off

SELinux might get into your way. In case that you at some point receive the following error message:

channel 0: open failed: administratively prohibited: open failed

it's the indication that tap device hasn't been created because of missing privileges. The simplest way is to turn off temporarily SELinux (again, I do not advise to do that in production environment!):

setenforce 0

If, at any point, something doesn't work three things you can use to debug problems. The first one is option -v to ssh command (or -vv) that will make it verbose and it will print out what's happening. The next thing is to look into log files on the remote machine (for sshd messages). Finally, you should know how to use tcpdump tool.

Creating and configuring tunnel

Ok, for a start, open two terminals on a laptop and in each one of them switch to root user (su command). Then, in one of them execute ssh command like this:

ssh -w any -o Tunnel=ethernet remote_machine

This will connect you to the remote machine (after successful authentication) and create two tap devices, one on the local machine and the other on the remote machine. I'll assume that their name is tap0, on both sides. Now, switch temporarily to other terminal and execute the following command:

ip link set tap0 up
tcpdump -nni tap0

The first command will start interface and the second will run tcpdump command. You'll be able to see traffic from remote network as soon as we attach the other part of the tunnel to bridge br0. Options n instruct tcpdump not to print symbolic names of anything, while option i binds tcpdump to interface tap0. When you want to stop tcpdump, use Ctrl+C keys.

Now, on the remote host execute the following commands:

brctl addif br0 tap0
ip link set tap0 up

The first command will add tap device to bridge (more colloquially switch), and the second one will activate the interface. The moment you execute the other command you'll notice that tcpdump command, running in the second windows, starts to print some output. This output is the traffic from the local network, transferred to the laptop. Of course, you'll see traffic if there is any traffic on the local network.

One final step, to add IP address from the local network to laptop, i.e. 192.168.0.40. But before that, we have to add explicit route for remote host or otherwise things will lock up. The laptop will think that remote host is now directly attached, while it is not and nothing will work. So, execute the following command on laptop (terminate tcpdump or open new windows and switch to root):

ip route add 192.168.0.30/32 via your_default_route

Change the string your_default_route to your default router (check what it is using 'ip route sh' command, it is an IP address in the same line as the word default). Finally, we are ready to add IP address, also on laptop, in the terminal where you added explicit route, execute the following command:

ip addr add 192.168.0.40/24 dev tap0

From that moment on, when you communicate with any host on local network this communication will go through the tunnel to the remote host that will send traffic to the local network for you. Hosts on the network 192.168.0.0/24 won't notice that you are actually somewhere else on the Internet.

One final thing. The question which destinations you want to reach via this tunnel. You can select all, or only a subset of destinations. In any case, you use routing to achieve that, i.e. you use the following command:

ip route add destination via gateway_on_local_network

Change destination to whatever you want to access via tunnel. In case you want everything than use word default (it could happen that you first need to remove existing route for default!). Or, you can set network, or IP address. Let's suppose that you want to reach google via local network. In that case find out IP address (or network) fo google and use that instead of the word destionation.

In place of gateway_on_local_network use gateway on the local network. In our case that is 192.168.0.1.

Finally, to tear down connection, just kill ssh.

Automating access

As a last thing, I'll describe how to automate the whole procedure. If you want fully automated solution, then first you have to configure passwordless login for ssh. Then, create two scripts. The first one will be called vpn_start.sh, will be placed on remote machine in a directory /usr/local/sbin and will contain the following lines:

#!/bin/bash
ip link set tap0 up
brctl addif br0 tap0

Lets call the second script also vpn_start.sh and let it be placed in the /usr/local/sbin/ directory. The content of that script should be:

#!/bin/bash
ssh -f -w any -o Tunnel=ethernet remote_machine /usr/local/sbin/vpn_start.sh
ip route add 192.168.0.30/32 via your_default_route
ip addr add 192.168.0.40/24 dev tap0
ip route add destination via gateway_on_local_network

Repeat the last command as many times as necessary and change all the parameters accordingly. Don't forget to make both scripts executable! Now, to run the configuration just execute the script on the laptop:

/usr/local/sbin/vpn_start.sh

And that's it! Of course, those scripts might be a lot fancier, but this will do just good!

Tunneling everything with SSH... or how to make VPNs...

In the previous three posts I described how to use OpenSSH to tunnel traffic for different applications. What all those three techniques had in common was that they tunneled only TCP traffic and that every time the connection was initiated from the local machine, i.e. in a way it was not possible for the machine on the other side to transfer data to us (actually, it is possible to circumwent some of those restrictions, but more about those techniques in some other post!). Furthermore, the implicit assumption was that there is only one TCP connection from your host to remote host. In case there are multiple connections opened on different ports, you'll need to run ssh as many times as there are those connections.

In this post I'm going to describe how to tunnel all traffic, regardless of it's type, from one machine to some other. In a way, I'll show how to create VPN networks using SSH. Actually, there are three ways to do that:

• Tunneling using ppp protocol on top of SSH
• Tunneling using tun devices natively supported by newer versions of ssh on, at least, Linux.
• Tunneling using Ethernet-like tap devices, also supported on a Linux OS.

In all of those cases you'll need to have administrative privileges in order to implement them. In the end, whichever path you decide to take (i.e. ppp or tun/tap) you'll end up configuring network parameters and firewall. So, I'm going to break down the description into more posts. The first post I'll deal with link layer setup (ppp/tun/tap) and the following posts I'll describe network layer configuration for different scenarios.

Link-layer setup

The basic goal of this step is to provide network device on which network layer will work.

Using ppp program and protocol

For this method there is a very good small howto document. Here I'll only repeat some relevant bits in case you don't want to read the whole document. First, you have to configure passwordless authentication to a remote host. This is easy to do and there are a plenty of references on the Internet. Later, maybe I write one, too. :) Anyway, in the following text I'll assume that you are using root account on both machines, i.e. you are root on a local machine and you are connecting to a remote machine under the username root. Beware that this is very bad security practice, but for a quick test or in a lab environment it'll do.

Ok, after you configured passwordless login for a root user, run the following command (note: this is a single line up to and including IP addresses) but it is broken into more lines because of a formatting in a browser!):

# pppd updetach noauth passive pty 'ssh REMOTE_HOST -o Batchmode=yes /usr/sbin/pppd nodetach notty noauth' 10.0.0.1:10.0.0.2
Using interface ppp0
Connect: ppp0 <--> /dev/pts/12
Deflate (15) compression enabled
local  IP address 10.0.0.1
remote IP address 10.0.0.2
#

As you can see, you got some information about interface and your prompt is immediately back. You need to have ppp package installed on both machines, if it is not then there will be an error message, something like command not found, and no connection will be established. Anyway, in this case everything was successful and we are notified that the remote side has address 10.0.0.2 and local side has address 10.0.0.1. To verify that everything works, try to ping remote side:

# ping -c 3 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=19.3 ms
64 bytes from 10.0.0.2: icmp_req=2 ttl=64 time=17.5 ms
64 bytes from 10.0.0.2: icmp_req=3 ttl=64 time=18.7 ms

--- 10.0.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 17.503/18.534/19.380/0.777 ms

If you see the output like the one above, that means that the link is working and the next thing to do is to setup network layer parameters (apart from IP addresses).

So, what happened? SSH created a communication channel between the two machines. This channel is then used by pppd processes on both ends to create ppp interfaces (one per each side). Then, this interface is used to transfer IP packets via protocol PPP. What we are (mis)using here is that pppd requires direct link between two points and through this link it then sends IP packets. Direct link is some sort of a serial interface, for example direct cable, UMTS/EDGE or similar connection, etc. The exact means by which two endpoints communicate is unimportant from pppd's perspective, what only matters is that whatever is sent on one end is received by other, and vice versa. So, because of this we could place SSH between two pppd processes.

The way the things are exectuted is as follows:

1. Local pppd starts (the first pppd in a line), without authentication phase (option noauth), in a passive mode (option passive) meaning it is waiting for someone to connect. It also allocates one master/slave pty (option pty) pair in which he controls master side, while slave side is connected to ssh proces (argument to pty option). There are also two ip addresses at the end of the command line that are actually arguments to this instance of PPP. They instruct local ppp process to assign to itself address 10.0.0.1 and to the remote end 10.0.0.2 address.
2. ssh process connects to REMOTE_HOST in batch mode (option -o BatchMode=yes). Basically batch mode tells ssh not to use password/passphrase because there is no user to enter necessary parameters. The rest of the command until closing single quote is the command that ssh has to execute after successfully connecting to remote host.
3. ssh that connected to the remote machine runs there second pppd process. The options instruct that pppd process to use stdin/stdout instead of a termina (option notty), to not detach from the controlling terminal (option nodetach), and to not require authentication (option noauth).

And that's it. Quite simple as you can see. But as I said, it is not good security practice to allow root login from the network! Locally, you can run this command directly from the root account! So, for some production deployment you'll need to three two additional things:

1. Create separate account that will be used to connect to a remote machine.
2. Configure sudo so that it allows that new account to run pppd binary without entering the password.
3. Modify invocation of ssh command so that remote user is specified and pppd program is executed via sudo command.

Using tun with Ethernet like functionality

For this mode you need to have the following directive in the server configuration (/etc/ssh/sshd_config):

Tunnel any

After the change don't forget to reload ssh configuration. What this configuration option tells ssh deamon is to allow tunneling using tun and tap devices. In other words, we can make our tunnel looks like ethernet interface, or like point-to-point interface on the 3rd (network) layer. Option any allows both. In case you want to restrict to a certain kind use either ethernet or point-to-point, depending on which one you need.

In this case we want Ethernet-like functionality, so assuming that you provided either any or ethernet parameter to Tunnel option, as a root user run ssh client as follows:

ssh -f -N -w any -o Tunnel=ethernet root@remotehost

change remotehost part with the host name (or IP address) you are connecting to. It is necessary to specify Tunnel option in ssh command because default value is point-to-point. The -w option requests from ssh to do tunneling using tap or tun device (depending on the value of Tunnel option) and in our case it is tap. After successfully logging to a remote machine, check interfaces with ip command (of ifconfig). You should see on both hosts that there is tap0 interface. If you already had tap interfaces, than the new ones will probably have highest number. Options -f and -N cause ssh to go into background after performing authentication, since command line is not necessary for tunneling.

To stop tap device, send a SIGTERM signal to ssh (using kill command, of course).

Using tun with only network layer functionality

As with the ethernet like functionality, you have to enable this mode in sshd configuration file, and also, you need to do this as a root on both sides of the connection.

The procedure to create point-to-point tunnels is similar to creating the Ethernet ones, only the argument is point-to-point instead of ethernet. Since point-to-point type is default, you don't have to specify Tunnel option, i.e.

ssh -f -N -w any root@remotehost

After logging into the remote host, ssh will go into background and you'll see new tun device created.