Tuesday, December 29, 2015

NetworkManager and OpenVPN - How it works?

I spent a lot of time trying to figure out how NetworkManager works. Very early in the process I came to conclusion that NM is a very complex piece of a software while in the same time documentation is lacking. What adds to the complexity is GObject mechanism that tries to fit object oriented programming into C programming language. So, in the end, I decided to write everything I managed to learn. First, in that way I'm leaving notes for myself later. In addition, I hope I'll help someone and thus save someone's time.


As a first goal to understand NetworkManager I set to understand how NM manages VPN connections. The following sequence of steps is a result of a research to answer the given question. Note that the flow isn't complete, but is enough to understand mechanics of VPN establishment. In addition, note that all the given file pathnames are relative to NetworkManager git repository.

So, everything starts when user activates VPN. At that moment a message is sent via DBus by nm-applet, nmcli or some other mechanism the following happens:
  1. Message to activate VPN sent via the DBus ends up in the function src/nm-manager.c:impl_manager_activate_connection().
  2. The function src/nm-manager.c:impl_manager_activate_connection() calls function src/nm-manager.c:_new_active_connection().
  3. Function _new_active_connection() function creates a new object of a type NM_TYPE_ACTIVE_CONNECTION, i.e. new active connection. To create new active connection object a method src/vpn-manager/nm-vpn-connection.c:nm_vpn_connection_new() in this particular case is used. This triggers a chain of initialization events described later.
  4. After the object is created a control returns to impl_manager_activate_connection() where asynchronous authorization check is initiated. Callback function to call when authorization check finishes is src/nm-manager.c:_activation_auth_done().
  5. After authorization is done the callback function _activation_auth_done() is called which in turn, if everything is OK, calls function src/nm-manager.c:_internal_activate_generic() which in turn calls function src/nm-manager.c:_internal_activate_vpn().
  6. The function _internal_activate_vpn() calls the function src/vpn-manager/nm-vpn-manager.c:nm_vpn_manager_activate_connection() is called.
  7. The function src/vpn-manager/nm-vpn-manager.c:nm_vpn_manager_activate_connection() calls function src/vpn-manager/nm-vpn-connection.c:nm_vpn_connection_activate().
  8. The function nm_vpn_connection_activate() connects asynchronously to DBus and when the connection is made a callback function on_proxy_acquired() is called.
  9. The function on_proxy_acquired() connects to signal "notify::g-name-owner" and then calls src/vpn-manager/nm-vpn-connection.c:nm_vpn_service_daemon_exec(). The purpose of the function nm_vpn_service_daemon_exec() is to start plugin binary (nm-openvpn-service).

    The goal of connecting to the signal "notify::g-name-owner" is to receive notification when the service appears, i.e. when it is initialized and available over the DBus so that appropriate signals can be registered. The most important registered signals, in our case, are Ip4Config and Ip6Config. When the function on_proxy_acquired() connects callback functions to DBus signals the process of establishing VPN can be continued so it also calls the function src/vpn-manager/nm-vpn-connection.c:get_secrets().
  10. The function get_secrets() calls function nm_settings_connection_get_secrets() and registers a callback src/vpn-manager/nm-vpn-connection.c:get_secrets_cb().
  11. The function src/vpn-manager/nm-vpn-connection.c:get_secrets_cb() sends secrets to VPN plugin via DBus. The send process is asynchronous and when finished callback src/vpn-manager/nm-vpn-connection.c:plugin_need_secrets_cb() is called.
  12. The callback function src/vpn-manager/nm-vpn-connection.c:plugin_need_secrets_cb() calls function src/vpn-manager/nm-vpn-connection.c:really_activate().
  13. The function src/vpn-manager/nm-vpn-connection.c:plugin_need_secrets_cb() calls Connect method on VPN plugin via DBus interface. Since DBus call is asynchronous callback function is registered, src/vpn-manager/nm-vpn-connection.c:connect_cb(). Callback function just calls src/vpn-manager/nm-vpn-connection.c:connect_success() that clears all timers.
The call to Connect method mentioned in the last step above initiates the chain of events in the VPN plugin described in the next section that, in the end, results in real connection establishment. When VPN connection is established, the VPN plugin will send three signals: Config, Ip4Config and Ip6Config. Those signals are caught by callbacks config_cb(), ip4_config_cb() and ip6_config_cb() respectively. Those signals, when activated, will be caught in nm-vpn-connection.c (look at step 9 in the previous list).

OpenVPN plugin

VPN plugins for NetworkManager are written so that they inherit some base classes from Network Manager and they only have to implement code specific to a particular VPN, while generic parts are implemented by NetworkManager  and placed in the base class. For example, DBus interface that each plugin has to implement is common to all of them and thus implemented in the base class. This is classic OO programming paradigm. But, in this case OO paradigm is emulated in C so when you start to study the code of some specific plugin, at first sight nothing will make sense and it will be hard to grasp what happens, where and when. So, before I describe sequence of events, I'll describe a code structure first as it aids a lot in understanding the code.

Static code structure and plugin initialization

The main part of the NetworkManager OpenVPN plugin is in the file src/nm-openvpn-service.c. This file inherits a class defined in libnm/nm-vpn-service-plugin.c from NetworkManager. Additionally, VPN DBus interface is defined in NetworkManager source in file introspection/nm-vpn-plugin.xml.

So, when you build OpenVPN plugin, a new binary is created, nm-openvpn-service. When NetworkManager executes this plugin its main method is invoked. In main method, the most important line is the following one:
plugin = nm_openvpn_plugin_new (bus_name);
That line causes new object of type NM_TYPE_OPENVPN_PLUGIN to be instantiated. Looking at the code of the function nm_openvpn_plugin_new() nothing special can be seen at a first glance. Basically, there is only the following line:
plugin = (NMOpenvpnPlugin *)
    g_initable_new (NM_TYPE_OPENVPN_PLUGIN,
                    NULL, &error,
But, because GObject type system is in background, actually a lot of things happen. First, initialization methods/constructors of OpenVPN plugin class and objects are called (nm_openvpn_plugin_init() and nm_openvpn_plugin_class_init()). Also, initialization methods/constructors of base class are called (nm_vpn_service_plugin_class_init() and nm_vpn_service_plugin_init()).

In addition, base class nm-vpn-service-plugin defines an interface that is initialized separately from class and object. The mechanism used is described here.

Note that after the plugin is initialized, NetworkManager receives information about this via a signal notify::g-name-owner in src/vpn-manager/nm-vpn-connection.c. This causes src/vpn-manager/nm-vpn-connection.c to connect to DBus interface and starts the sequence of steps described in the following subsection.

VPN activation sequence of steps

In the following code, when I write base class I mean on the code in the file libnm/nm-vpn-plugin-service.c in NetworkManager. When I write OpenVPN class or OpenVPN service then I'm thinking on file src/nm-openvpn-service in network-manager-openvpn code.

The following sequence of steps is initiated by calling method Connect on VPN plugin via DBus:
  1. DBus signal initiates a function impl_vpn_service_plugin_connect() in base class. This function is registered as a handler for DBus Connect method serviced by the plugin in the function init_sync() in the base class. If you look at the code of the given function you'll notice that it only transfers control to the function _connect_generic() in the base class.
  2. The function _connect_generic() does some checks and transfers function to OpenVPN class, i.e. the method real_connect() is called in the file src/nm-openvpn-service.c.
  3. The method real_connect() just redirects control to the function _connect_common().
  4. The method _connect_common() does some sanity check, obtains some parameters, and calls method nm_openvpn_start_openvpn_binary().
  5. The method nm_openvpn_start_openvpn_binary() searches for openvpn binary on a local file system, constructs command line options based on preferences set for the VPN connection and from environment variables, and finally starts openvpn binary (call to function g_spawn_async()). One important part of the command line construction is that openvpn binary is told not to assign parameters itself, but to call helper scripts and pass it all the parameters. This helper script is nm-openvpn-service-openvpn-helper contained in the src directory of networkmanager-openvpn-plugin. So, when OpenVPN binary establishes VPN connection it calls helper script.

    It also registers two callbacks. The first one monitors process using a g-child-watch-source-new() function. A callback function is 
    openvpn_watch_cb() that, if called, assumes an error occurred or openvpn binary just finished. The second callback is timer that, when fires, calls function nm_openvpn_connect_timer_cb(). It only tries to connect to OpenVPN's management socket. So, in other words, notification about successful VPN establishment to OpenVPN plugin isn't done using GLib or DBus notification system, but just by waiting and checking. As a final note, timer isn't used in every case. Actually, it seems that it is used rarely, only when authentication type is static key.
  6. The nm-openvpn-service-openvpn-helper script, in its main function, collects all network related data from the environment (set by openvpn binary) and sends it via DBus to nm-openvpn-service using methods SetConfig, SetIp4Config and SetIp6Config.
  7.  The configuration is caught by functions impl_vpn_service_plugin_set_config()impl_vpn_service_plugin_set_ip4_config(), and impl_vpn_service_plugin_set_ip6_config() in VPN plugin base class. Those function, in turn, call functions nm_vpn_service_plugin_set_config()nm_vpn_service_plugin_set_ip4_config() and nm_vpn_service_plugin_set_ip6_config(). They also call finish DBus proxies so that helper script can continue executing after each call.
  8. Each of the functions nm_vpn_service_plugin_set_config()nm_vpn_service_plugin_set_ip4_config() and nm_vpn_service_plugin_set_ip6_config() does two things. It emits signal ("config", "ip4-config" and "ip6-config") from base class and OpenVPN class. [Note: I don't fully understand this mechanism yet]


It seems to me that it is very hard to come up with a good documenation that describes this topic. So, here are some better references I used:

  1. For signals the best reference I could find was from Maemo 4 documentation, available on the following link: http://maemo.org/maemo_training_material/maemo4.x/html/maemo_Platform_Development_Chinook/Chapter_04_Implementing_and_using_DBus_signals.html

Saturday, December 26, 2015

Fedora 23 and VMWare Workstation 12.1

In the post about VMWare 11 I wrote about the problem that newest Fedoras are compiled using gcc5 which breaks C++ ABI thus making VMWare unrunnable. Several commenters suggested workarounds of which the simplest and most elegant solution is by user Andy who suggested that before running vmware command the environment variable LD_PRELOAD should be set to /usr/lib/vmware/lib/libglibmm-2.4.so.1/libglibmm-2.4.so.1, i.e. you should execute the following command:
export LD_PRELOAD=/usr/lib/vmware/lib/libglibmm-2.4.so.1/libglibmm-2.4.so.1 /usr/bin/vmware
This works for VMWare 12 as it did for VMWare 11.

Now, at some point it will happen that VMWare notifies you about newer version and offers you to download it and install it. But, you won't be able to run update and you'll receive the following error message:
/tmp/vmis.lLCMq3/install/vmware-installer/vmware-installer: line 56: 14595 Aborted (core dumped) VMWARE_INSTALLER="$VMWARE_INSTALLER" VMISPYVERSION="$VMISPYVERSION" "$VMWARE_INSTALLER"/vmis-launcher "$VMWARE_INSTALLER"/vmware-installer.py "$@"
What you have to do in that case is to manually download update and start it without defining LD_PRELOAD variable. Note that if you changed /usr/bin/vmware after upgrade you'll have to change it again.

Sunday, December 13, 2015

Research paper: "Development of a Cyber Warfare Training Prototype for Current Simulations"

One of my research directions I'm taking is simulation of security incidents and cyber security conflicts.  So, I'm searching for research papers that present work about that particular topic and one of them is the paper "Development of a Cyber Warfare Training Prototype for Current Simulations". I found out for this paper via announcement made on SCADASEC mailing list. The interesting thing is that the given paper couldn't be found on Google Scholar at the time this post was written. Anyway, it was presented on Fall 2014 Simulation Interoperability Workshop organized by Simulation Interoperability Standards Organization (SISO). All papers presented on the Workshop are freely available on SISO Web pages. The given workshop is, according to papers presented, mainly oriented towards military applications of simulation. Note that cybersecurity simulations only started to appear but the use of simulations in military are old thing.

Reading the paper Development of a Cyber Warfare Training Prototype for Current Simulations was valuable experience because I met for the first time a number of new terms specific to military domain. Also, there are references worth taking a look at, what I'm going to do.

In the end, I had the following conclusions about the paper:
  1. The paper talks about integrating cyber domain into  existing combat simulation tools. So, they are not interested in having a cybersecurity domain specific/isolated simulation tool. It might be extrapolated that this is based on the US military requirements.
  2. When the authors talk about cyber warfare training what they are basically describing is a cyber attack on command and control (C&C) infrastructure used on a battlefield.
  3. The main contribution of the paper is a description of requirements gathering phase based on use cases (section 3) and proposed component that would allow implementation of proposed scenarios (section 4).

Thursday, December 10, 2015

SCADA/ICS security conferences in 2016

On SCADASEC mailing list there was a question about security conferences in 2016 worth attending. I find this question very interesting so I decided to list here all responses received in this thread. The result is shown in the following table:

List of ICS/SCADA conferences in 2016
Important dates Conference name Venue Comment
Conference date:
January 12-14, 2016
S4x16 Week Miami South Beach Probably the best in the US for a heavy research focus, Dale and team do an excellent job on trying to do a "what's next" approach usually as well as a lot of flash/flair (fun time)
Conference date:
February 7-11, 2016
Kaspersky Security Analyst Summit Tenerife, Spain
Conference date:
February 9-11, 2016
DistribuTech Orange County Convention Center, West Halls A3-4 & B, Orlando, FL Focused on power grid, very OT centric and not security focused gives a unique look at broader industry
Conference date:
February 16-23, 2016
ICS Security Summit Orlando, FL Great 2 day conference with opportunity to take training classes if you want, hands-on challenges, live demos, and ~200 strong IT/OT mixed audience
Conference date:
April 26-28, 2016
ICS Cyber Security London, United Kingdom
Conference date:
May 3-5, 2016
ICSJWG 2016 Spring Meeting Scottsdale, AZ Multiple times a year in different locations): Definitely one to go for anyone new to the ICS community, it's free and the ICSCERT folks are always very kind/professional/awesome.
Conference date:
May 30, 2016
ACM Cyber-Physical System Security Workshop (CPSS 2016) Xi’an, China
Conference date:
October 25-27, 2016
4SICS Stockholm, Sweden Great IT/OT mix (50% practitioners this past year) with a very similar vibe to S4 in terms of flair/research
Conference date:
November 10-11, 2016
11th Annual API Cybersecurity Conference & Expo Westin Houston Memorial City Houston, Texas Focused on Oil/Gas, very diverse group of speakers with a lot of vendor interaction.

Note: I'm extrapolating in this case as when and where the next conference will take place.

The column titled Comment is taken from this mail message, so I'm crediting the original author as I don't have any first-person experience with the mentioned conferences. Also, you can find additional list of conferences here and here.

About Me

scientist, consultant, security specialist, networking guy, system administrator, philosopher ;)

Blog Archive