NetworkManager and OpenVPN - How it works?

I spent a lot of time trying to figure out how NetworkManager works. Very early in the process I came to conclusion that NM is a very complex piece of a software while in the same time documentation is lacking. What adds to the complexity is GObject mechanism that tries to fit object oriented programming into C programming language. So, in the end, I decided to write everything I managed to learn. First, in that way I'm leaving notes for myself later. In addition, I hope I'll help someone and thus save someone's time.

NetworkManager

As a first goal to understand NetworkManager I set to understand how NM manages VPN connections. The following sequence of steps is a result of a research to answer the given question. Note that the flow isn't complete, but is enough to understand mechanics of VPN establishment. In addition, note that all the given file pathnames are relative to NetworkManager git repository.

So, everything starts when user activates VPN. At that moment a message is sent via DBus by nm-applet, nmcli or some other mechanism the following happens:

1. Message to activate VPN sent via the DBus ends up in the function src/nm-manager.c:impl_manager_activate_connection().
2. The function src/nm-manager.c:impl_manager_activate_connection() calls function src/nm-manager.c:_new_active_connection().
3. Function _new_active_connection() function creates a new object of a type NM_TYPE_ACTIVE_CONNECTION, i.e. new active connection. To create new active connection object a method src/vpn-manager/nm-vpn-connection.c:nm_vpn_connection_new() in this particular case is used. This triggers a chain of initialization events described later.
4. After the object is created a control returns to impl_manager_activate_connection() where asynchronous authorization check is initiated. Callback function to call when authorization check finishes is src/nm-manager.c:_activation_auth_done().
5. After authorization is done the callback function _activation_auth_done() is called which in turn, if everything is OK, calls function src/nm-manager.c:_internal_activate_generic() which in turn calls function src/nm-manager.c:_internal_activate_vpn().
6. The function _internal_activate_vpn() calls the function src/vpn-manager/nm-vpn-manager.c:nm_vpn_manager_activate_connection() is called.
7. The function src/vpn-manager/nm-vpn-manager.c:nm_vpn_manager_activate_connection() calls function src/vpn-manager/nm-vpn-connection.c:nm_vpn_connection_activate().
8. The function nm_vpn_connection_activate() connects asynchronously to DBus and when the connection is made a callback function on_proxy_acquired() is called.
9. The function on_proxy_acquired() connects to signal "notify::g-name-owner" and then calls src/vpn-manager/nm-vpn-connection.c:nm_vpn_service_daemon_exec(). The purpose of the function nm_vpn_service_daemon_exec() is to start plugin binary (nm-openvpn-service).
   
   The goal of connecting to the signal "notify::g-name-owner" is to receive notification when the service appears, i.e. when it is initialized and available over the DBus so that appropriate signals can be registered. The most important registered signals, in our case, are Ip4Config and Ip6Config. When the function on_proxy_acquired() connects callback functions to DBus signals the process of establishing VPN can be continued so it also calls the function src/vpn-manager/nm-vpn-connection.c:get_secrets().
10. The function get_secrets() calls function nm_settings_connection_get_secrets() and registers a callback src/vpn-manager/nm-vpn-connection.c:get_secrets_cb().
11. The function src/vpn-manager/nm-vpn-connection.c:get_secrets_cb() sends secrets to VPN plugin via DBus. The send process is asynchronous and when finished callback src/vpn-manager/nm-vpn-connection.c:plugin_need_secrets_cb() is called.
12. The callback function src/vpn-manager/nm-vpn-connection.c:plugin_need_secrets_cb() calls function src/vpn-manager/nm-vpn-connection.c:really_activate().
13. The function src/vpn-manager/nm-vpn-connection.c:plugin_need_secrets_cb() calls Connect method on VPN plugin via DBus interface. Since DBus call is asynchronous callback function is registered, src/vpn-manager/nm-vpn-connection.c:connect_cb(). Callback function just calls src/vpn-manager/nm-vpn-connection.c:connect_success() that clears all timers.

The call to Connect method mentioned in the last step above initiates the chain of events in the VPN plugin described in the next section that, in the end, results in real connection establishment. When VPN connection is established, the VPN plugin will send three signals: Config, Ip4Config and Ip6Config. Those signals are caught by callbacks config_cb(), ip4_config_cb() and ip6_config_cb() respectively. Those signals, when activated, will be caught in nm-vpn-connection.c (look at step 9 in the previous list).

OpenVPN plugin

VPN plugins for NetworkManager are written so that they inherit some base classes from Network Manager and they only have to implement code specific to a particular VPN, while generic parts are implemented by NetworkManager  and placed in the base class. For example, DBus interface that each plugin has to implement is common to all of them and thus implemented in the base class. This is classic OO programming paradigm. But, in this case OO paradigm is emulated in C so when you start to study the code of some specific plugin, at first sight nothing will make sense and it will be hard to grasp what happens, where and when. So, before I describe sequence of events, I'll describe a code structure first as it aids a lot in understanding the code.

Static code structure and plugin initialization

The main part of the NetworkManager OpenVPN plugin is in the file src/nm-openvpn-service.c. This file inherits a class defined in libnm/nm-vpn-service-plugin.c from NetworkManager. Additionally, VPN DBus interface is defined in NetworkManager source in file introspection/nm-vpn-plugin.xml.

So, when you build OpenVPN plugin, a new binary is created, nm-openvpn-service. When NetworkManager executes this plugin its main method is invoked. In main method, the most important line is the following one:

plugin = nm_openvpn_plugin_new (bus_name);

That line causes new object of type NM_TYPE_OPENVPN_PLUGIN to be instantiated. Looking at the code of the function nm_openvpn_plugin_new() nothing special can be seen at a first glance. Basically, there is only the following line:

plugin = (NMOpenvpnPlugin *)
    g_initable_new (NM_TYPE_OPENVPN_PLUGIN,
                    NULL, &error,
                    NM_VPN_SERVICE_PLUGIN_DBUS_SERVICE_NAME,
                    bus_name,
                    NULL);

But, because GObject type system is in background, actually a lot of things happen. First, initialization methods/constructors of OpenVPN plugin class and objects are called (nm_openvpn_plugin_init() and nm_openvpn_plugin_class_init()). Also, initialization methods/constructors of base class are called (nm_vpn_service_plugin_class_init() and nm_vpn_service_plugin_init()).

In addition, base class nm-vpn-service-plugin defines an interface that is initialized separately from class and object. The mechanism used is described here.

Note that after the plugin is initialized, NetworkManager receives information about this via a signal notify::g-name-owner in src/vpn-manager/nm-vpn-connection.c. This causes src/vpn-manager/nm-vpn-connection.c to connect to DBus interface and starts the sequence of steps described in the following subsection.

VPN activation sequence of steps

In the following code, when I write base class I mean on the code in the file libnm/nm-vpn-plugin-service.c in NetworkManager. When I write OpenVPN class or OpenVPN service then I'm thinking on file src/nm-openvpn-service in network-manager-openvpn code.

The following sequence of steps is initiated by calling method Connect on VPN plugin via DBus:

1. DBus signal initiates a function impl_vpn_service_plugin_connect() in base class. This function is registered as a handler for DBus Connect method serviced by the plugin in the function init_sync() in the base class. If you look at the code of the given function you'll notice that it only transfers control to the function _connect_generic() in the base class.
2. The function _connect_generic() does some checks and transfers function to OpenVPN class, i.e. the method real_connect() is called in the file src/nm-openvpn-service.c.
3. The method real_connect() just redirects control to the function _connect_common().
4. The method _connect_common() does some sanity check, obtains some parameters, and calls method nm_openvpn_start_openvpn_binary().
5. The method nm_openvpn_start_openvpn_binary() searches for openvpn binary on a local file system, constructs command line options based on preferences set for the VPN connection and from environment variables, and finally starts openvpn binary (call to function g_spawn_async()). One important part of the command line construction is that openvpn binary is told not to assign parameters itself, but to call helper scripts and pass it all the parameters. This helper script is nm-openvpn-service-openvpn-helper contained in the src directory of networkmanager-openvpn-plugin. So, when OpenVPN binary establishes VPN connection it calls helper script.
   
   It also registers two callbacks. The first one monitors process using a g-child-watch-source-new() function. A callback function is openvpn_watch_cb() that, if called, assumes an error occurred or openvpn binary just finished. The second callback is timer that, when fires, calls function nm_openvpn_connect_timer_cb(). It only tries to connect to OpenVPN's management socket. So, in other words, notification about successful VPN establishment to OpenVPN plugin isn't done using GLib or DBus notification system, but just by waiting and checking. As a final note, timer isn't used in every case. Actually, it seems that it is used rarely, only when authentication type is static key.
6. The nm-openvpn-service-openvpn-helper script, in its main function, collects all network related data from the environment (set by openvpn binary) and sends it via DBus to nm-openvpn-service using methods SetConfig, SetIp4Config and SetIp6Config.
7.  The configuration is caught by functions impl_vpn_service_plugin_set_config(), impl_vpn_service_plugin_set_ip4_config(), and impl_vpn_service_plugin_set_ip6_config() in VPN plugin base class. Those function, in turn, call functions nm_vpn_service_plugin_set_config(), nm_vpn_service_plugin_set_ip4_config() and nm_vpn_service_plugin_set_ip6_config(). They also call finish DBus proxies so that helper script can continue executing after each call.
8. Each of the functions nm_vpn_service_plugin_set_config(), nm_vpn_service_plugin_set_ip4_config() and nm_vpn_service_plugin_set_ip6_config() does two things. It emits signal ("config", "ip4-config" and "ip6-config") from base class and OpenVPN class. [Note: I don't fully understand this mechanism yet]

Literature

It seems to me that it is very hard to come up with a good documenation that describes this topic. So, here are some better references I used:

1. For signals the best reference I could find was from Maemo 4 documentation, available on the following link: http://maemo.org/maemo_training_material/maemo4.x/html/maemo_Platform_Development_Chinook/Chapter_04_Implementing_and_using_DBus_signals.html

Fedora 23 and VMWare Workstation 12.1

In the post about VMWare 11 I wrote about the problem that newest Fedoras are compiled using gcc5 which breaks C++ ABI thus making VMWare unrunnable. Several commenters suggested workarounds of which the simplest and most elegant solution is by user Andy who suggested that before running vmware command the environment variable LD_PRELOAD should be set to /usr/lib/vmware/lib/libglibmm-2.4.so.1/libglibmm-2.4.so.1, i.e. you should execute the following command:

export LD_PRELOAD=/usr/lib/vmware/lib/libglibmm-2.4.so.1/libglibmm-2.4.so.1 /usr/bin/vmware

This works for VMWare 12 as it did for VMWare 11.

Now, at some point it will happen that VMWare notifies you about newer version and offers you to download it and install it. But, you won't be able to run update and you'll receive the following error message:

/tmp/vmis.lLCMq3/install/vmware-installer/vmware-installer: line 56: 14595 Aborted (core dumped) VMWARE_INSTALLER="$VMWARE_INSTALLER" VMISPYVERSION="$VMISPYVERSION" "$VMWARE_INSTALLER"/vmis-launcher "$VMWARE_INSTALLER"/vmware-installer.py "$@"

What you have to do in that case is to manually download update and start it without defining LD_PRELOAD variable. Note that if you changed /usr/bin/vmware after upgrade you'll have to change it again.

Research paper: "Development of a Cyber Warfare Training Prototype for Current Simulations"

One of my research directions I'm taking is simulation of security incidents and cyber security conflicts.  So, I'm searching for research papers that present work about that particular topic and one of them is the paper "Development of a Cyber Warfare Training Prototype for Current Simulations". I found out for this paper via announcement made on SCADASEC mailing list. The interesting thing is that the given paper couldn't be found on Google Scholar at the time this post was written. Anyway, it was presented on Fall 2014 Simulation Interoperability Workshop organized by Simulation Interoperability Standards Organization (SISO). All papers presented on the Workshop are freely available on SISO Web pages. The given workshop is, according to papers presented, mainly oriented towards military applications of simulation. Note that cybersecurity simulations only started to appear but the use of simulations in military are old thing.

Reading the paper Development of a Cyber Warfare Training Prototype for Current Simulations was valuable experience because I met for the first time a number of new terms specific to military domain. Also, there are references worth taking a look at, what I'm going to do.

In the end, I had the following conclusions about the paper:

1. The paper talks about integrating cyber domain into  existing combat simulation tools. So, they are not interested in having a cybersecurity domain specific/isolated simulation tool. It might be extrapolated that this is based on the US military requirements.
2. When the authors talk about cyber warfare training what they are basically describing is a cyber attack on command and control (C&C) infrastructure used on a battlefield.
3. The main contribution of the paper is a description of requirements gathering phase based on use cases (section 3) and proposed component that would allow implementation of proposed scenarios (section 4).

SCADA/ICS security conferences in 2016

On SCADASEC mailing list there was a question about security conferences in 2016 worth attending. I find this question very interesting so I decided to list here all responses received in this thread. The result is shown in the following table:

List of ICS/SCADA conferences in 2016

Important dates	Conference name	Venue	Comment
Conference date: January 12-14, 2016	S4x16 Week	Miami South Beach	Probably the best in the US for a heavy research focus, Dale and team do an excellent job on trying to do a "what's next" approach usually as well as a lot of flash/flair (fun time)
Conference date: February 7-11, 2016	Kaspersky Security Analyst Summit	Tenerife, Spain
Conference date: February 9-11, 2016	DistribuTech	Orange County Convention Center, West Halls A3-4 & B, Orlando, FL	Focused on power grid, very OT centric and not security focused gives a unique look at broader industry
Conference date: February 16-23, 2016	ICS Security Summit	Orlando, FL	Great 2 day conference with opportunity to take training classes if you want, hands-on challenges, live demos, and ~200 strong IT/OT mixed audience
Conference date: April 26-28, 2016	ICS Cyber Security	London, United Kingdom
Conference date: May 3-5, 2016	ICSJWG 2016 Spring Meeting	Scottsdale, AZ	Multiple times a year in different locations): Definitely one to go for anyone new to the ICS community, it's free and the ICSCERT folks are always very kind/professional/awesome.
Conference date: May 30, 2016	ACM Cyber-Physical System Security Workshop (CPSS 2016)	Xi’an, China
Conference date: October 25-27, 2016	4SICS	Stockholm, Sweden	Great IT/OT mix (50% practitioners this past year) with a very similar vibe to S4 in terms of flair/research
Conference date: November 10-11, 2016	11th Annual API Cybersecurity Conference & Expo	Westin Houston Memorial City Houston, Texas	Focused on Oil/Gas, very diverse group of speakers with a lot of vendor interaction. Note: I'm extrapolating in this case as when and where the next conference will take place.

The column titled Comment is taken from this mail message, so I'm crediting the original author as I don't have any first-person experience with the mentioned conferences. Also, you can find additional list of conferences here and here.

Intercepting and redirecting traffic from VMWare Workstation

After some time I decided to try again sslstrip (note that there is enhanced version of sslstrip on GitHub). First, I tried to do everything on a single host, i.e. sslstrip is running on a same machine as is browser whose traffic I wanted to intercept. It turned out that this isn't so easy because there is no rule for iptables that would allow me to differentiate traffic of Firefox and SSLStrip.  Because of that fact, if I were to redirect traffic that tries to go the port 80 on the Internet so that Firefox's traffic goes to SSLStrip, I would also redirect traffic from SSLStrip to itself creating infinite loop. For some reason filtering based on PID, a.k.a. process identified, isn't possible. It used to be possible, but then it was removed. There are two solutions to overcome the problem of running everything within a single OS:

1. Run SSLStrip as a separate user (or Firefox as a separate user). IPTables allows match on UID, or
2. Run Firefox or SSLStrip in a separate network namespace.

Instead, I decided to use VMWare Workstation (I had a version 11.1.2 when I wrote this post) and to intercept its traffic. The reason I chose VMWare was that I had one virtual machine already running at hand. But it turned out that it isn't so easy, either. In the end, the problem was caused by my lack of understanding on how VMWare Workstation works. Additional problems were caused by delays that happen after changing network parameters, i.e. it takes time that the changes take effect and thus to be observable.

So, the main problem was that traffic generated by the virtual machine isn't intercepted by standard Netfilter rules when default configuration is used along with NAT networking. To see why, look at the following figure which show logical network topology:

The figure shows that traffic from Guest OS goes to virtual switch (vswitch) and then to the external network bypassing Host's netfilter hooks. So, even though there is vmnet8 in the host OS and traffic can be seen on the given interface, it doesn't go through standard NETFILTER hooks. Actually, vmnet8 is Host's interface to virtual switch. Also, if you take look at how the network is configured within Guest OS you'll note that the gateway address is set to x.y.z.2 while IP address of vmnet8 is x.y.z.1.

The behavior when the gateway is x.y.z.2 is:

1. If you try to filter packets going from Guest OS to the Internet using iptables you won't succeed. I tried to put DENY rule in filter table of all chains and it didn't work. Actually, nothing will work as netfilter hooks aren't called at all.

So, the trick, in the end, is to change default GW on Guest OS so that traffic is routed to HostOS where you can then manipulate it. In that case you'll have to also:

1. enable IP forwarding (/proc/sys/net/ipv4/ip_forward) on Host OS.
2. Activate NAT or masquerade for outgoing traffic if you want guest machine to communicate with the outside world!

Note that you can observe some delay between setting some parameter/behavior (like adding NAT rule) and the time the behavior starts to be observable, i.e. it works. Anyway, at this point you can redirect network traffic, in other words, you can then use netfilter hooks.

So, to finish this post, you should start SSLStrip and add the following rule on Host OS:

iptables -A PREROUTING -t nat -i vmnet8 -p tcp \
        --dport 80 -j REDIRECT --to-port 10000

And then try to access, e.g. http://www.facebook.com. Be careful that you didn't previously access Facebook because in that case browser will know it must use https and it won't try to use http, even if you typed it in the URL bar. If this happens than just clean the complete browser history and try again.

VMWare Workstation 10/11 on Fedora 23 Beta

UPDATE. Take a look at the comment by Vincent Cojot below for a solution on how to run VMWare Workstation/Player on Fedora 23. Take care only to go into /usr/lib/vmware/lib instead of /usr/lib/vmware as pointed out by wolf (comment after), but other than that, everything works! Note that you must not change and/or set LD_LIBRARY_PATH as first suggested by this post!

I decided to upgrade one of my laptops from Fedora 22 to the newest Fedora 23 Beta. The upgrade process went smoothly but when I tried to run VMWare, it just silently failed. Googling around for Fedora 23 and VMWare Workstation didn't help. Then, I stumbled on this thread, about the problem that VMWare installation has with GCC 5.1 used on SuSE. Turns out that the same version is used on Fedora 23, so I tried the solution from a given thread which suggest the modification of LD_LIBRARY_PATH due to the missing symbol in system provided library gtkmm:

$ export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libglibmm-2.4.so.1/:/usr/lib64/gtk-2.0/modules/:$LD_LIBRARY_PATH
$ vmware
(vmware-modconfig:11918): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",
(vmware-modconfig:11918): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",
/usr/share/themes/Adwaita/gtk-2.0/gtkrc:1163: error: unexpected identifier `direction', expected character `}'
/usr/share/themes/Adwaita/gtk-2.0/gtkrc:1163: error: unexpected identifier `direction', expected character `}'
(vmware-tray:12017): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",
(vmware-tray:12017): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",
/usr/share/themes/Adwaita/gtk-2.0/gtkrc:1163: error: unexpected identifier `direction', expected character `}'

Basically, this helped a bit because now vmware doesn't fail silently. Yet, it still doesn't start. In the end, it seems that the problem is a bit more complex, as explained in this thread, so in case you need VMWare Workstation, don't upgrade yet. :)

Installing Zimbra 7.2.7 on CentOS 7 and upgrading to 8.6

I had a problem that my old mail system is Zimbra 7.2.7 running on CentOS 5. I can not upgrade to the newest version of Zimbra since it isn't supported on CentOS 5 any more. In addition, there is a problem that Zimbra 7.2.7 isn't available on CentOS 7. So, I'm in some kind of a deadlock situation here. To break from this deadlock I decided to first install Zimbra 7.2.7 on CentOS 7 and then to upgrade to Zimbra 8.6. Since Zimbra 7.2.7 isn't supported on CentOS 7, it is a bit of a hacky process. This post describes in detail what I had to do in order to install Zimbra 7 on CentOS 7.

Few things to note before I start with description of the installation. First, my new server has a different IP address from the old one, but I have given both servers the same name (the same FQDN). Also, I installed DNS server on the new server with almost the same configuration as the old one. The difference is in IP addresses pointing to the server itself (MX records primarily). Finally, both servers are 64-bit.

The whole post is structured into two parts. The first part is more in a diary form. The second part is a condensed, cookbook style text on how to upgrade without any text or discussions about problems.

Installation process

Start by downloading Zimbra 7.2.7 for RHEL6 and Zimbra 8.6.0 for RHEL7. Unpack both archives.

Go to the Zimbra 7.2.7 directory and before starting installation process open the file util/utilfunc.sh in the editor of you choice. In line 2287 you'll see the following statement:

PACKAGEINST='rpm -iv'

You should change it to:

PACKAGEINST='rpm --force -iv'

Now, start the installation process with:

# ./install.sh -s --platform-override

Note that I used the option -s so that configuration process isn't started before necessary changes are made. The problem is caused by newer version of Perl on CentOS 7 while installation version of Zimbra 7 relies on an older version from CentOS 5. Also be careful that you select packages for installation that are also installed on the old server!

The idea is to copy Perl from Zimbra 8.6. In order to do so, go to the directory /opt/zimbra/zimbramon and rename directory lib/ to lib.orig/, or something similar:

# cd /opt/zimbra/zimbramon
# mv lib lib.orig

Then, go to the directory where you unpacked Zimbra 8.6.0. You should now unpack zimbra-core RPM package using the following commands:

# mkdir tmp
# cd tmp
# rpm2cpio ../packages/zimbra-core-8.6.0_GA_1153.RHEL7_64-20141215151110.x86_64.rpm | cpio -id

and then go to directory opt/zimbra/zimbramon (note missing leading slash!) and copy lib directory to Zimbra7 tree:

# rsync --delete -av lib /opt/zimbra/zimbramon/

Additionally, make the following two symlinks in directory  /usr/lib64:

# ln -s libssl.so.1.0.1e libssl.so.1.0.0
# ln -s libcrypto.so.1.0.1e libcrypto.so.1.0.0

Finally, run the setup command:

# /opt/zimbra/libexec/zmsetup.pl

It can happen that zmsetup.pl complains about missing Perl modules. You should install any module missing. In my case the missing modules were:

perl-LDAP
perl-Net-DNS

Each time zmsetup.pl stops because some module is missing, just install the missing module and start zmsetup.pl again.

Finally, when the installation was finished, I saved freshly installed Zimbra in case I need it later, e.g. when I want to start from this point on. I did this with the following command:

rsync -av /opt/zimbra /opt/zimbra.orig

Note that in the following text I'll reference from time to time zimbra.orig directory because we'll need some files from there.

Experimenting with a new installation

This section is some form of a diary on how I tried to migrate Zimbra from the old server to the new one, and then how I tried to upgrade Zimbra 7 to Zimbra 8. I think that it is good to write about this trial-and-error process for at least two reasons:

1. You can learn from my experience, as (almost) nothing goes perfectly without trying.
2. If you google for some error messages, you might find it helpful to see what is the solution. There are no error messages in the condensed version of the diary section.

In case you just want a cookbook on how to do it, skip to the Cookbook section in this blog post.

Migrating existing Zimbra installation to a new server

I suggest that before you do this, you do some preparation steps:

1. Block outbound connections to port 25, i.e. prevent new instance of Zimbra to send emails until you migration is complete. And then, before going into production, check mail queue for orphaned/duplicate mails.
2. Synchronizations are time consuming on any, even moderately used host. So, do first live migration, and then stop old server and repeat migration.
3. In any case, don't do migration when the old server is running because you might leave your new server in inconsistent state.

Now, go to the old installation of Zimbra and copy the whole Zimbra tree over the Zimbra installation on the new server. The command should look something like this:

$ rsync -av --delete --exclude zimbramon/lib /opt/zimbra root@newserver:/opt/

Probably this will take some time. Also, don't forget to exclude zimbramon/lib directory or otherwise you'll end up with an old version of Perl and when you try to start some tool from Zimbra you'll get the following error message:

$ zmcontrol stop
Perl lib version (v5.8.8) doesn't match executable version (v5.16.3) at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/Config.pm line 46.
Compilation failed in require at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/lib.pm line 6.
BEGIN failed--compilation aborted at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/lib.pm line 6.
Compilation failed in require at /opt/zimbra/bin/zmcontrol line 25.
BEGIN failed--compilation aborted at /opt/zimbra/bin/zmcontrol line 25.

Anyway, in case you got the previous error just copy again zimbramon/lib directory from Zimbra 8 as explained earlier in the text.

At this point zmcontrol stop/start wan't work almost certainly. So, I suggest that you try to start service by service and check each one if it works or not. Checks can be performed looking at the process list or using netstat/ss command to see listening ports. In case some service doesn't work, check log files in /opt/zimbra/log directory and the file /var/log/zimbra.log. Actually, it would be good to check log files no matter if you think a service works or not.

The first service you should start is ldap. That one is easy, just run the following command as a zimbra user:

$ ldap start

then, look if there is slapd process running:

$ ps -ef | grep sldapd
zimbra   30224     1  0 06:56 ?        00:00:00 /opt/zimbra/openldap/sbin/slapd -l LOCAL0 -u zimbra -h ldap://mail.zemris.fer.hr:389 ldapi:/// -F /opt/zimbra/data/ldap/config

and you can also check if ldap port (389) is opened:

$ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 3.1.2.1:389      0.0.0.0:*               LISTEN
tcp        0      0 3.1.2.1:53       0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:53     0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22       0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:953    0.0.0.0:*               LISTEN
tcp6       0      0 ::1:53           :::*                    LISTEN
tcp6       0      0 :::22            :::*                    LISTEN
tcp6       0      0 ::1:953          :::*                    LISTEN

Logging, by the way, goes to syslog/journalctl because of the option -l, and that means we should look into /var/log/zimbra.log for slapd messages:

Next, I tried to start mailbox service:

$ zmmailboxdctl start

and that didn't work. Namely, no java process:

$ ps -ef | grep java
zimbra    4717  1470  0 07:19 pts/0    00:00:00 grep --color=auto java

and no listening sockets for imap, pop, etc.:

$ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 3.1.2.1:389      0.0.0.0:*   LISTEN
tcp        0      0 3.1.2.1:53       0.0.0.0:*   LISTEN
tcp        0      0 127.0.0.1:53     0.0.0.0:*   LISTEN
tcp        0      0 0.0.0.0:22       0.0.0.0:*   LISTEN
tcp        0      0 127.0.0.1:953    0.0.0.0:*   LISTEN
tcp6       0      0 ::1:53           :::*        LISTEN
tcp6       0      0 :::22            :::*        LISTEN
tcp6       0      0 ::1:953          :::*        LISTEN

Looking into log file (/opt/zimbra/log/zmmailboxd.out) reveals the following error messages:

Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
        at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        ... 18 more
1227 WARN  [main] log - Nested in java.lang.reflect.InvocationTargetException:
java.net.BindException: Cannot assign requested address

After few more minutes of scratching my head of why (while constantly resisting temptation to use strace tool) I finally realized that the problem is related to my customization on the old server. Namely, I changed jetty configuration so that Zimbra binds to a specific IP address which isn't available on the new server. The solution for this is easy, I saved a copy of Zimbra 7 immediately after installation (zimbra.orig directory, remember) so the following two commands solved this particular problem:

$ cd /opt/zimbra.orig/zimbra/$ rsync -av --delete jetty-6.1.22.z6 /opt/zimbra/

So, again I tried to start mailbox service on Zimbra:

$ zmmailboxdctl startStarting mailboxd...done.

This time, it seems to be good. But checking ports and processes again showed that it doesn't work! Looking at log file showed the following cryptic error:

Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented
        at javax.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:80)
        at com.zimbra.common.util.NetUtil.newBoundServerSocket(NetUtil.java:95)
        at com.zimbra.common.util.NetUtil.bindServerSocket(NetUtil.java:163)
        at com.zimbra.common.util.NetUtil.bindSslTcpServerSocket(NetUtil.java:54)
        ...

That was very cryptic error message. Luckily, Googling for it immediately gave the following thread. Namely, the problem is with SSL/TLS, and the following two commands from the given thread fixed that part of the problem:

$ mv /opt/zimbra/mailboxd/etc/keystore /opt/zimbra/mailboxd/etc/keystore.borked
$ sudo /opt/zimbra/bin/zmcertmgr deploycrt self

This time, it almost worked as now I had two java processes and listening sockets. But, there were a log of messages in log file like the following one:

1445 WARN  [main] log - failed org.mortbay.jetty.NCSARequestLog@64d55986: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN  [main] log - failed RequestLogHandler@72e8a021: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN  [main] log - failed HandlerCollection@6691177: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN  [main] log - failed RewriteHandler@5bf99eea: java.io.IOException: Cannot write log directory /opt/zimbra/log
1446 WARN  [main] log - failed DebugHandler@613043d2: java.io.IOException: Cannot write log directory /opt/zimbra/log

At first sight, that shouldn't happened. Zimbra is owner of those files and SELinux isn't controlling Zimbra. After some scratching I found out that there is a configuration file /opt/zimbra/jetty/etc/jetty.properties in which UID and GID values for Zimbra user/group are not correct, i.e. they are same as on the old server while Zimbra on the new server has different UID/GID values:

$ id zimbra
uid=997(zimbra) gid=996(zimbra) groups=996(zimbra),4(adm),5(tty),89(postfix)

So, I edited it and changed their values to 997 and 996. Note that you should check the correct values for your specific case. Trying again the error messages didn't go away. This time I checked process list and I found out that the process ID of Java is 500. So, I missed some configuration setting. Turns out that jetty.properties file is automatically generated from jetty.properties.in and UID and GID values are taken from Zimbra configuration. So, I had to change them permanently using the following commands (as a zimbra user):

$ zmlocalconfig -e zimbra_uid=997
$ zmlocalconfig -e zimbra_gid=996

And this time it worked! But, I realized now that ldap isn't working any more:

$ ldap start
Failed to start slapd.  Attempting debug start to determine error.
TLS: error:0200100D:system library:fopen:Permission denied bss_file.c:398
TLS: error:20074002:BIO routines:FILE_CTRL:system lib bss_file.c:400
55d5688d main: TLS init def ctx failed: -1

After some time I realized  that when I redeployed certificate using the link given above, it turns out that UID and GID used to set ownership of different certificate files were the ones specified in Zimbra configuration and not the actual UID and GID, i.e.:

$ ls -l /opt/zimbra/conf/slapd.*
-rw-r-----. 1 500 500 1107 Aug 20 07:36 /opt/zimbra/conf/slapd.crt
-rw-r-----. 1 500 500 1704 Aug 20 07:36 /opt/zimbra/conf/slapd.key

I also had to find all the files whose owner was set to previous value (500) and change it to a new UID/GID values. I did this using find command as a root user:

find /opt/zimbra -uid 500 -exec chown zimbra.zimbra {} \;

Now, both LDAP and mailbox services worked! Time to try zmcontrol start but first I stopped running services. Still didn't work. This time I received two error messages from Web and mail processes:

Starting apache.../opt/zimbra/httpd-2.2.22/bin/httpd: error while loading shared libraries: libexpat.so.0: cannot open shared object file: No such file or directory
failed.
/opt/zimbra/postfix/sbin/postalias: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory
/opt/zimbra/postfix/sbin/postfix: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory

But these are easy to solve, just copy missing libraries from CentOS 5 installation. And note that there is a library and symlink to the library! I tried starting Zimbra again using zmcontrol. Again, no luck. The problem this time is a missing Perl subroutine while starting stat module:

Undefined subroutine &main::getZimbraHome called at /opt/zimbra/bin/zmstatctl line 156.

This subroutine was removed in Perl packages distributed with Zimbra 8.6, which we are using here. I found this subroutine to be defined in Zmstat.pl file and first I tried to fix it by just copying this old file, but it turns out that more than one file was changed so I tried, and succeeded, with by copying all non-binary files:

# cd /opt/zimbra.orig/zimbra/zimbramon/lib.orig
# rsync -av --exclude \*.so --exclude x86_64-linux-thread-multi * /opt/zimbra/zimbramon/lib/

Trying again zmcontrol we are receiving warnings about deprecated use of certain Perl features:

Use of qw(...) as parentheses is deprecated at /opt/zimbra/zimbramon/lib/Zimbra/Util/Common.pm line 25.
Use of qw(...) as parentheses is deprecated at /opt/zimbra/zimbramon/lib/Zimbra/Util/Common.pm line 26.

But these can be safely ignored. Finally, Zimbra 7 works! I did some more checking but not thorough one since my intention now was to upgrade Zimbra to version 8.6.

Upgrading Zimbra 7 to Zimbra 8.6

Finally, I got to the step where I tried to upgrade Zimbra to a newer version. Go to the unpacked Zimbra 8 directory and start upgrade process as usual:

./install.sh --platform-override

After starting upgrade process, I received the following error message:

Validating ldap configuration
Error: Unable to create a successful TLS connection to the ldap masters.
       Fix cert configuration prior to upgrading.

Quick GoogleFu gave the following thread with the solution to the problem.

$ zmlocalconfig -e ldap_master_url=ldaps://mymaster.somewhere.com:636
$ zmlocalconfig -e ldap_url=ldaps://myreplica.somewhere.com:636
$ zmlocalconfig -e ldap_starttls_supported=0
$ zmlocalconfig -e ldap_port=636 

Be careful to change host names to names that match your configuration. This wasn't enough because it turned out to be that my CA expired, i.e.:

# openssl x509 -in /opt/zimbra/conf/ca/ca.pem -noout -text | grep 'Not After'
            Not After : Apr 11 11:22:21 2015 GMT

Which meant that I had to recreate CA:

# /opt/zimbra/bin/zmcertmgr createca -new
# /opt/zimbra/bin/zmcertmgr createcrt -new -days 1825
# /opt/zimbra/bin/zmcertmgr deploycrt self
# /opt/zimbra/bin/zmcertmgr deployca

Be sure to restart slapd after this, i.e. in my case slapd was already running so it didn't pick up new CA/CERT! Actually, it is much better to stop Zimbra and then to start upgrade process. Anyway, I restarted upgrade process! There were two warnings, but it seems they can be ignored:

ERROR 1133 (28000) at line 2: Can't find any matching row in the user table
ERROR 1396 (HY000) at line 1: Operation DROP USER failed for ''@'mail.somedomain.com'

Also, at one point installation stopped because some errors with LDAP's password. The configuration menu was presented with lot of marks that things have to be fixed. I selected LDAP password, and when the configuration program assigned new password everything after that went well.

And that's it! Unfortunately, my old mail has some additional services I had to migrate too, but I won't write about them in this post.

I have to note several things. First, this process heavily depends on configurations on both new and old server being as close as possible. This means that if you did customizations that are outside of /opt/zimbra tree, then you'll have to transfer those too. In my case, I integrated mailman with Zimbra, and mailman lives in other directories, so I had problems. Second,

Quick how to migrate Zimbra7/CentOS5 to Zimbra8/CentOS7

So, here is the recipe on how to move old Zimbra installation to the new server. This is condensed version of the previous section.

Preparation

1. Download ZCS 7.2.7 (for RHEL6) and ZCS 8.6.0 (you can download newer version if you wish, but I didn't test it so expect there might be some issues). I'll assume you downloaded them into /tmp directory.
2. Unpack both archives.
3. Enter into ZCS 8.6 directory and then into packages/ subdirectory. Make there temporary directory (e.g. tmp/) and enter into it. Unpack core package (zimbra-core-8.6.0_GA_1153.RHEL7_64-20141215151110.x86_64.rpm) using rpm2cpio and cpio commands.
4. Enter into ZCS 7.2.7 directory and modify file util/utilfunc.sh file. Line 2287 should be PACKAGEINST='rpm --force -iv'.
5. From the old server copy the following libraries into /lib64 directory on a new server: libssl.so.*, libcrypto.so.*, libpcre.so.*, and libexpat.so.*.

Installation

1. Enter direcotry ZCS 7.2.7 and start installation with the following command:
   ./install.sh -s --platform-override
2. When script finishes, go to /opt/zimbra/zimbramon directory rename lib directory to lib.orig.
3.  Go to the directory where you unpacked zimbra-core package (should be something like /tmp/zcs-8.6.0-something/packages/tmp/) and execute the following command:
   rsync -av opt/zimbra/zimbramon/lib /opt/zimbra/zimbramon/.
4. Run the command /opt/zimbra/libexec/zmsetup.pl. In case the script complains about missing Perl module and stops, install the missing module and run the script again. 
5. Stop Zimbra and disable starting during the boot process:
   
   # /etc/init.d/zimbra stop
   # chkconfig zimbra off
6. Make a copy of vanilla Zimbra directory tree, i.e. execute the following commands:
   
   # cd /opt
   # rsync -av zimbra zimbra.orig

Syncing with old server

1. Go to the old server and run the following command:
   
   rsync -av --delete --exclude zimbramon/lib --exclude jetty\* /opt/zimbra newserver:/opt/
2. Again on the new server run the following commands (as a zimbra user):
   
   $ zmlocalconfig -e zimbra_uid=997
   $ zmlocalconfig -e zimbra_gid=996
   
   Change UID and GID to match your new installation!
3. Turn on LDAP server, i.e. switch to the Zimbra user and execute the following command:
   
   $ ldap start
4. Run the following commands to generate new CA.
   
   # /opt/zimbra/bin/zmcertmgr createca -new
   # /opt/zimbra/bin/zmcertmgr createcrt -new -days 1825
   # /opt/zimbra/bin/zmcertmgr deploycrt self
   # /opt/zimbra/bin/zmcertmgr deployca
   
   Might not be necessary, but it doesn't hurt. And yes, if you have commercial CERT, you'll have to import it and these steps are not necessary.
5. Turn off LDAP server, i.e. switch to the Zimbra user and execute the following command:
   
   $ ldap stop
6. Run the following commands to "fix" Perl version mismatch on new and old servers:
   
   # cd /opt/zimbra.orig/zimbra/zimbramon/lib.orig
   # rsync -av --exclude \*.so --exclude x86_64-linux-thread-multi * /opt/zimbra/zimbramon/lib/

Upgrade

1. Go to the directory where you unpacked ZCS 8.6 and run upgrade process as usual:
   
   ./install.sh --platform-override

Some additional notes

In this section I'll add some additional notes.

I noticed after some time that while starting Zimbra, zmconfigd service reports error during startup. After some debugging I found out that the service is working, but that the problem is in how check if it is working is implemented. Namely, one step in the process is to connect to localhost port 7171 using nc tool. But since localhost first resolves to IPv6 address, and the service doesn't listen on IPv6, so the connection is refused, nc doesn't try IPv4 then, and script wrongfully thinks that the service isn't running. The solution was to edit /etc/hosts file and change it so that localhost resolves exclusively to 127.0.0.1 while localhost6 resolves to ::1.

How to fix weak DH key in Zimbra 7

I just had to fix a problem of weak DH keys in Zimbra 7. Namely, Firefox and Chrome, after upgrade, don't want to connect to servers that use DH keys less than 1024 bits. This means that IMAP won't work either, as it uses SSL/TLS too (or at least it should ;)). Note that the solution is to upgrade to newest Zimbra version, but for me at the moment there is no way I can upgrade my server, i.e. the upgrade is planned but currently it isn't possible. Googling around gave nothing for Zimbra 7, but only for Zimbra 8. In the end, it also turned out that in order to fix the length of DH keys it is necessary to have Java 8, while in Zimbra 7 Java 6 is used.

After a lot of search, the solution turned out to be easy. The key is to disable cipher suites that use DH keys. I managed to do that using the following commands:

zmprov mcf +zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_128_CBC_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_256_CBC_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_DES_CBC_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_DES_CBC3_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites TLS_EDH_RSA_WITH_3DES_EDE_CBC_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites SSL_EDH_RSA_WITH_3DES_EDE_CBC_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites TLS_DHE_DSS_WITH_AES_128_CBC_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites TLS_DHE_DSS_WITH_AES_256_CBC_SHA
zmprov mcf +zimbraSSLExcludeCipherSuites SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
zmmailboxdctl restart

After that, Webmail worked again. You can check supported ciphersuites using sslscan command, i.e. in my case after the given change I got the following ciphersuites:

$ sslscan webmail:443 | grep Accepted
    Accepted  SSLv3  256 bits  AES256-SHA
    Accepted  SSLv3  168 bits  EDH-RSA-DES-CBC3-SHA
    Accepted  SSLv3  168 bits  DES-CBC3-SHA
    Accepted  SSLv3  128 bits  AES128-SHA
    Accepted  SSLv3  128 bits  RC4-SHA
    Accepted  SSLv3  128 bits  RC4-MD5
    Accepted  TLSv1  256 bits  AES256-SHA
    Accepted  TLSv1  168 bits  EDH-RSA-DES-CBC3-SHA
    Accepted  TLSv1  168 bits  DES-CBC3-SHA
    Accepted  TLSv1  128 bits  AES128-SHA
    Accepted  TLSv1  128 bits  RC4-SHA
    Accepted  TLSv1  128 bits  RC4-MD5

Even though Webmail worked, Thunderbird didn't connect. Using Wireshark I found out that Thunderbird, for IMAP connection, tries to use EDH-RSA-DES-CBC3-SHA. I tried to disable that ciphersuite on the server side, but no matter what I've tried, it didn't work. In the end I disabled that cipher on the client side. I opened Thunderbird's configuration editor and there I manually disabled given cipher by setting configuration setting to false.

CentOS 7, Zimbra 8.6 and FirewallD

I just installed Zimbra 8.6 on a fresh CentOS 7. It seems that CentOS 7 uses FirewallD service by default instead of the old iptables and iptables6 scripts in /etc/init.d directory. Nevertheless I don't like when I see that someone recommends some critical security services/protections to be just turned off. Those services are there for a reason, and turning them off sounds to me like the old bad recommendation of chmod'ing everything to 777 when something didn't work. Anyway, I didn't turn off SELinux and Zimbra works as expected. What I needed is to configure FirewallD to allow access to mail services from the Internet. Turns out it isn't so hard as everything is already provided. Basically, the following services have to be enabled in your zone:

• dns
• https
• imaps
• smtp

To permanently enable each of the aforementioned services, use the following command:

firewall-cmd --permanent --add-service <service>

Note that the given command doesn't activate access to the service until you restart FirewallD. Anyway, that's it.

As a final note, I didn't allow access to admin port 7171. The reason is that I'm not so comfortable with allowing Internet wide access to admin console. To access admin console, I'm going to use ssh tunneling. Basically, I'll forward local port 7171, over ssh, to port 7171 on loopback interface of mail server. In case you are unlike me, and don't have problems with allowing access to that port, use the following command:

firewall-cmd --permanent --add-port=7171/tcp

Again, don't forget to restart FirewallD after issuing the given command.

VMWare Workstation 11 and Linux kernel 3.19

Well, I thought that starting with kernel 3.18 there will be no need any more for manual patching in order to make VMWare Workstation 11.0 work again (11.1 didn't work either). But, I was wrong. After updating vmnet compilation ended with errors and I had to search for a solution. I found it, on ArchWiki pages. Now, it happened once before to me that I just pointed to a page with a solution, and that page was changed so that solution disappeared. To avoid this, here is step by step what you have to do. First, download a patch. You don't need to be a root to execute this command:

$ curl http://pastie.org/pastes/9934018/download -o /tmp/vmnet-3.19.patch

Now, switch to root and execute the following commands:

# cd /usr/lib/vmware/modules/source
# tar -xf vmnet.tar
# patch -p0 -i /tmp/vmnet-3.19.patch
# mv vmnet.tar vmnet.tar.SAVED
# tar -cf vmnet.tar vmnet-only
# rm -r vmnet-only
# vmware-modconfig --console --install-all

And that should be it.

Short Tip: Renaming log files to include date...

I had a bunch of a log files in the format logfilename.N.gz, but I wanted to rename them into logfilename.YYYYMMDD.gz where YYYYMMDD is a date when the file was last modified. I did it using the following for loop:

for i in logfilename.*.gz
do
    mv -i $i logfilename.`date -r $i +%Y%m%d`.gz
done

The argument -r to date(1) command tells it to use the last modification date (mtime) of a file given as the argument to the option. Note that it is also possible to use stat(1) command instead of date(1).

Anomaly detection in Snort

I just got thoroughly confused when I found a statement in one whitepaper by SANS that Snort can do anomaly based detection. For me, anomaly based detection means that the software is capable of detecting something that deviates from the normal behavior in a profound ways and additionally, it wasn't possible to algorithmically define this deviated behavior in advance. Obviously, I started immediately to google around to find out more information about this since, lately, I was reading some surveys about research on anomaly based detection. This is still relatively unexplored area which means not much used in real-world scenarios.

After a bit of googling I found in Snort manual the following section:

2.2.3.4 Anomaly Detection 

TCP protocol anomalies, such as data on SYN packets, data received outside the TCP window, etc are configured via the detect_anomalies option to the TCP configuration. Some of these anomalies are detected on a per-target basis. For example, a few operating systems allow data in TCP SYN packets, while others do not. 

Turns out that the anomaly detection in Snort are actually anomalies that can be algorithmically codified (e.g. in TCP segment SYN bit is set and there is data in the segment). So, in conclusion, there is no algorithm for learning in standard Snort code.

That said, I found now defunct research project that experimented with anomaly based detection in Snort. By looking into the implementation, it turns out that the authors created plugin for Snort that was logging different features into textual log files. Those log files were then processed using R. In essence, this is good approach for experimentation but not for a production use.

Getting free disk space in Linux

While working on a script to have full Zimbra backups as many days in the past as possible, I was trying to automatically remove old backups based on the free space value. Basically, the idea was to remove directory by directory until free space reached some threshold. To find out free space on a disk is easy, use df(1) command. Basically, it looks like this:

$ df -k /
Filesystem 1K-blocks     Used Available Use% Mounted on
/dev/sda1   56267084 39311864  16938836  70% /

The problem is that it is necessary to use some postprocessing in order to obtain desired value, i.e. 5th or 5th column. cut(1) command, in this case, is a bit problematic because in general you can not expect that the output is so nicely formatted, nor it is fixed. For example, based on the width of the widest device node in the first column, it is automatically resized. That in turn means number of whitespaces varies, and you end up being forced to use something else than cut(1). Probably, the most appropriate tool is awk(1), since awk(1) can properly parse fields separated with variable number of whitespaces. In addition, you need to get rid of first line. That can be done using head(1)/tail(1), but it is more efficient to use awk(1) itself. So, you end up with the following construct:

$ df -k / | awk 'NR==2 {print $4}'
16938836

But, for some reason, I wasn't satisfied with the given solution because I thought I'm using too complex tools for something that should be simpler than that. So, I started to search is there some other way to obtain free space of some partition. It turned out that stat(1) command is able to do that, but it's rarely used for that purpose. It is used to find out data about files, or directories, but not file systems. Yet, there is an option, -f, that tells stat(1) we are querying file system, and also there is an option --format which accepts format sequences in a style of date(1) command. So, to get the free space on root file system you can use it as follows:

$ stat -f --format "%f" /
4238805

stat(1) command without --format option prints all the data about file system it can find out:

$ stat -f /
  File: "/"
    ID: b8a4e1f0a2aefb22 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 14066771   Free: 4238805    Available: 4234709
Inodes: Total: 3588096    Free: 2151591

This makes it in some way analogous to df(1) command. But, we are getting values in blocks, instead of kilobytes! You can get block size using %S format sequence, but that's it. So, some additional trickery is needed. One solution is to output arithmetic expression and evaluate it using bc(1) command, like this:

$ stat -f --format "%f * %S" / | bc
17362145280

Alternatively, it is also possible to use shell's arithmetic evaluation like this:

$ echo $((`stat -f --format "%f * %S" /`))17362145280

But, in both cases we are starting two process. In a first case the processes are stat(1) and bc(1), and in the second case it is a new subshell (for backtick) and stat(1). Note that this is the same as the solution with awk(1). But in case of awk(1) we are starting two more complex tools of which one, df(1), is more targeted to display value to a user than to be used in scripts. One additional advantage of a method using awk(1) might be portability, i.e. I'm df(1)/awk(1) combination is probably more common than stat(1)/bc(1) combination.

Anyway, the difference probably isn't so big with respect to performance, but obviously there is another way to do it, and it was interesting to pursue an alternative.