uClibc versus eglibc

This is a post about differences between uClibc and eglibc libraries. Namely, OpenWRT can be built on either of those two libraries with the default being uClibc, so the question one might ask is what's the difference between those libraries and why uClibc. I have to say that I'm not affiliated with neither of those, and what I wrote here is purely my personal opinion based on the information I managed to find on the Internet. I suppose that you know what C standard library is, what is its purpose and that the default C library on desktop and server Linux is glibc.

First, both eglibc and uClibc were developed with the intention of having a small footprint and thus to be suitable for use in embedded devices. glibc is a huge library that wasn't well suited for that purpose. Yet, those two libraries differ in a way they try to achieve that purpose.

eglibc

eglibc, or embedded glibc, was developed with the intention of being source and binary compatible with glibc. That means that isn't necessary to recompile existing binary applications compiled for glibc in order to be possible to run them on eglibc.  On the other hand, since it is source compatible with glibc, that means it is possible to recompile the source without any modifications. Yet, according to FAQ on eglibc page, eglibc development is stopped, and all embedded development will be done directly in glibc tree. This was announced on July 20th, 2013. That also means that almost all patches from eglibc will be ported to glibc. You can find more information here about the patches that are not going to be ported back.

What might confuse is that the newest eglibc release is based on glibc 2.19 which was released in April, 2014, a year after announcement that eglibc development will stop. But, this is according to the plan of phasing out separate eglibc tree. Also according to the plan, this is a last branch. All the branches will be maintained as long as the base glibc versions are maintained.

In June 2014 Debian announced that it's going to switch back from eglibc to glibc due to the changes in the governing structure of GLibc project. The reason for the change is that Ulrich Drepper left RedHat.

uClibc

uClibc, on the other hand, was developed with the intention to be source compatible only, i.e. no binary compatibility and thus binary programs compiled for glibc (or eglibc) have to be recompiled. uClibc is actively maintained even though the latest release is from 2012.

Conclusion

So, what is the conclusion? The conclusion is that if you don't need to be binary compatible, you should use uClibc  on OpenWRT. After all, all the binary packages on OpenWRT's site were compiled against that library. If binary compatibility is important to you, then glibc is the way to go. Unfortunately, since in OpenWRT there is eglibc and not glibc, you have to go eglibc route. Note that this also means you'll have to recompile all the sources for OpenWRT since you'll not be able to use precompiled binary packages!

Two or more applications listening on the same (IP, port) socket

It might happen that you need two applications to listen to the same (IP, port) UDP socket with the idea that the applications know how to differentiate between packets that are intended for them. If this is the case, then you'll have to do something special because the kernel doesn't allow two, or more, applications to bind in such a way. As a side note, starting with kernel 3.9 there is a socket option SO_REUSEPORT that allows multiple applications to bind to a single port (under certain preconditions) but it doesn't work the way I described here.

One solution is to have some kind of a demultiplexer application, i.e. it binds to a given port, receives packets and then sends them to appropriate application. This will work, but it wasn't appropriate for my situation. So, the solution is that one application binds to the given port, and the other uses PF_PACKET socket with appropriate filter so that it also receives UDP packets of interest. I hope that you realize that this works only for UDP, and not for TCP or other connection oriented protocols!

So, what you have to do is:

1. Open appropriate socket with socket() system call.
2. Bind to interface using bind() system call.
3. Attach filter to socket using setsockopt().
4. Receive packets.

If you want an example of how this is done, take a look into busybox, more specifically its udhcpc client.

Now, there are two problems with this approach that you need to be aware of. The first is that if you try to send via this socket you are avoiding routing code in the kernel! In other words, it might happen that you try to send packets to wrong directions. How this can be solved, and if it really needs solution, depends on the specific scenario you are trying to achieve.

The second problem is that if there is no application listening on a given port, the kernel will sent ICMP port unreachable error messages on each received UDP message. I found a lot of questions on the Internet about this issue, but without any real answer. So, I took a look at where this error message is generated, and if there is anything that might prevent this from happening.

UDP packets are received in function __udp4_lib_rcv() that, in case there is no application listening on a given port, sends ICMP destination port unreachable message. As it turns out, the only case when this message will not be sent is if the destination is multicast or broadcast address. So, your options are, from the most to the least preferred:

1. Be certain that you always have application listening on a given port.
2. Use iptables to block ICMP error messages (be careful not to block too much!).
3. The application on the other end ignores those error messages.

Installing OSSIM community edition in QEMU

Since OSSIM is based on Debian and it is a nightmare to compile it for something else (ehm, CentOS) I decided to use it in a headless QEMU virtual machine. To test the whole process, I first decided to do a regular installation of OSSIM, with display. But, I had a lot of obstacles while trying install OSSIM community edition in QEMU. It is even more interesting that when you google for ossim and qemu, there are almost no posts.

In the end, everything worked flawlessly but when using text based installation. To access text based installation edit boot command line (pressing TAB at the initial boot screen) and at the end add the following:

DEBIAN_FRONTEND=text

And that gave me text based installation. Basically, AlienVault uses Debian's installer so anything that can be configured for Debian, can be for OSSIM too. Take a look into manual for further information.

Few things to be aware of when doing this:

1. Don't use too small disk because the installation will stuck without any notification what happened.
2. I had problems with GUI based installation, and its fallback ncurses. The installation would stuck somewhere (e.g. in GUI after entering IP address, something would go wrong in package installation process, MySQL wasn't properly installed and there were errors that starting failed, apache wasn't properly installed and Web console wasn't accessible, etc.)

CentOS 6

On CentOS there is no qemu-kvm like in Fedora. Instead, you have to use libvirtd. Be sure that libvirt is installed, before continuing. That means packages virt-install and libvirt are installed. Additionally, libvirtd daemon must bi started.

So, first create file for disk image. You can do this using dd, but even better is to use fallocate(1) command. Also, fetch OSSIM ISO image file. Now, to start installation process use the following command:

virt-install -r 2560 --accelerate -n OSSIM \
        --cdrom /tmp/AlienVault_OSSIM_64Bits_4.3.4.iso \
        --os-variant=debiansqueeze --disk path=./sda.img \
        -w bridge --graphics vnc,password=replaceme

In the previous command I'm giving to OSSIM 2.5G RAM (option -r), the name will be OSSIM, disk image is in the current directory (with respect to the command virt-install) and I'm using bridged networking. Finally, console will be available via VNC and the password for access is replaceme.

There are several error messages you might receive when trying to start installation process:

ERROR    Error with storage parameters: size is required for non-existent disk '/etc/sysconfig/network-scripts/sda.img'

Well, this error message occured because I was trying to start installation process in the wrong directory, i.e. the one that didn't contain file for hard disk image.

The following error:

ERROR    Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

means that libvirtd daemon isn't started. Start it using:

service libvirtd on

and don't forget to make it start every time you boot your machine:

chkconfig libvirtd on

The next error:

Starting install...
ERROR    internal error Process exited while reading console log output: char device redirected to /dev/pts/1
qemu-kvm: -drive file=/root/AlienVault_OSSIM_64Bits_4.3.4.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw: could not open disk image /root/AlienVault_OSSIM_64Bits_4.3.4.iso: Permission denied

means that I placed ISO image in a directory where libvirt can not access it. Move image to, e.g. /tmp directory and try again.

After you managed to start installation process, connect to it using vncviewer application. libvirt-install binds vnc to localhost so you won't be able to access it directly from some remote host. This is actually OK, so you shouldn't change it, unless you know very well what you are doing. So, to connect to console, open terminal window and execute the following command:

ssh -L 5900:127.0.0.1:5900 host_where_installation_is_started

Now, in another terminal on local machine (i.e. the one where you started previous ssh command) run the following command:

vnc localhost

And that should be it. What happened is that with ssh you created a tunnel between your local machine and the remote where virtual machine is being installed. So, don't stop ssh until you vnc session is running!