Viber and Fedora 26 SSL errors

I just downloaded and updated Viber on my Fedora. When I tied to start it, it seg faulted with the following errors:

QSqlDatabasePrivate::removeDatabase: connection 'ConfigureDBConnection' is still in use, all queries will cease to work.
Qt WebEngine ICU data not found at /opt/viber/resources. Trying parent directory...
Qt WebEngine resources not found at /opt/viber/resources. Trying parent directory...
Qt WebEngine ICU data not found at /opt/viber/resources. Trying parent directory...
Qt WebEngine resources not found at /opt/viber/resources. Trying parent directory...
qt.network.ssl: QSslSocket: cannot resolve CRYPTO_num_locks
qt.network.ssl: QSslSocket: cannot resolve CRYPTO_set_id_callback
qt.network.ssl: QSslSocket: cannot resolve CRYPTO_set_locking_callback
qt.network.ssl: QSslSocket: cannot resolve ERR_free_strings
qt.network.ssl: QSslSocket: cannot resolve EVP_CIPHER_CTX_cleanup
qt.network.ssl: QSslSocket: cannot resolve EVP_CIPHER_CTX_init
qt.network.ssl: QSslSocket: cannot resolve sk_new_null
qt.network.ssl: QSslSocket: cannot resolve sk_push
qt.network.ssl: QSslSocket: cannot resolve sk_free
qt.network.ssl: QSslSocket: cannot resolve sk_num
qt.network.ssl: QSslSocket: cannot resolve sk_pop_free
qt.network.ssl: QSslSocket: cannot resolve sk_value
qt.network.ssl: QSslSocket: cannot resolve SSL_library_init
qt.network.ssl: QSslSocket: cannot resolve SSL_load_error_strings
qt.network.ssl: QSslSocket: cannot resolve SSL_get_ex_new_index
qt.network.ssl: QSslSocket: cannot resolve SSLv23_client_method
qt.network.ssl: QSslSocket: cannot resolve SSLv23_server_method
qt.network.ssl: QSslSocket: cannot resolve X509_STORE_CTX_get_chain
qt.network.ssl: QSslSocket: cannot resolve OPENSSL_add_all_algorithms_noconf
qt.network.ssl: QSslSocket: cannot resolve OPENSSL_add_all_algorithms_conf
qt.network.ssl: QSslSocket: cannot resolve SSLeay
qt.network.ssl: QSslSocket: cannot resolve SSLeay_version
qt.network.ssl: QSslSocket: cannot call unresolved function CRYPTO_num_locks
qt.network.ssl: QSslSocket: cannot call unresolved function CRYPTO_set_id_callback
qt.network.ssl: QSslSocket: cannot call unresolved function CRYPTO_set_locking_callback
qt.network.ssl: QSslSocket: cannot call unresolved function SSL_library_init
qt.network.ssl: QSslSocket: cannot call unresolved function SSLv23_client_method
qt.network.ssl: QSslSocket: cannot call unresolved function sk_num
Segmentation fault (core dumped)

After some digging the solution was simple, just execute the following command and it should work afterwards:

sudo ln -s /usr/lib64/libssl.so.10 /opt/viber/lib/libssl.so

Installing Zimbra 7.2.7 on CentOS 7 and upgrading to 8.6

I had a problem that my old mail system is Zimbra 7.2.7 running on CentOS 5. I can not upgrade to the newest version of Zimbra since it isn't supported on CentOS 5 any more. In addition, there is a problem that Zimbra 7.2.7 isn't available on CentOS 7. So, I'm in some kind of a deadlock situation here. To break from this deadlock I decided to first install Zimbra 7.2.7 on CentOS 7 and then to upgrade to Zimbra 8.6. Since Zimbra 7.2.7 isn't supported on CentOS 7, it is a bit of a hacky process. This post describes in detail what I had to do in order to install Zimbra 7 on CentOS 7.

Few things to note before I start with description of the installation. First, my new server has a different IP address from the old one, but I have given both servers the same name (the same FQDN). Also, I installed DNS server on the new server with almost the same configuration as the old one. The difference is in IP addresses pointing to the server itself (MX records primarily). Finally, both servers are 64-bit.

The whole post is structured into two parts. The first part is more in a diary form. The second part is a condensed, cookbook style text on how to upgrade without any text or discussions about problems.

Installation process

Start by downloading Zimbra 7.2.7 for RHEL6 and Zimbra 8.6.0 for RHEL7. Unpack both archives.

Go to the Zimbra 7.2.7 directory and before starting installation process open the file util/utilfunc.sh in the editor of you choice. In line 2287 you'll see the following statement:

PACKAGEINST='rpm -iv'

You should change it to:

PACKAGEINST='rpm --force -iv'

Now, start the installation process with:

# ./install.sh -s --platform-override

Note that I used the option -s so that configuration process isn't started before necessary changes are made. The problem is caused by newer version of Perl on CentOS 7 while installation version of Zimbra 7 relies on an older version from CentOS 5. Also be careful that you select packages for installation that are also installed on the old server!

The idea is to copy Perl from Zimbra 8.6. In order to do so, go to the directory /opt/zimbra/zimbramon and rename directory lib/ to lib.orig/, or something similar:

# cd /opt/zimbra/zimbramon
# mv lib lib.orig

Then, go to the directory where you unpacked Zimbra 8.6.0. You should now unpack zimbra-core RPM package using the following commands:

# mkdir tmp
# cd tmp
# rpm2cpio ../packages/zimbra-core-8.6.0_GA_1153.RHEL7_64-20141215151110.x86_64.rpm | cpio -id

and then go to directory opt/zimbra/zimbramon (note missing leading slash!) and copy lib directory to Zimbra7 tree:

# rsync --delete -av lib /opt/zimbra/zimbramon/

Additionally, make the following two symlinks in directory  /usr/lib64:

# ln -s libssl.so.1.0.1e libssl.so.1.0.0
# ln -s libcrypto.so.1.0.1e libcrypto.so.1.0.0

Finally, run the setup command:

# /opt/zimbra/libexec/zmsetup.pl

It can happen that zmsetup.pl complains about missing Perl modules. You should install any module missing. In my case the missing modules were:

perl-LDAP
perl-Net-DNS

Each time zmsetup.pl stops because some module is missing, just install the missing module and start zmsetup.pl again.

Finally, when the installation was finished, I saved freshly installed Zimbra in case I need it later, e.g. when I want to start from this point on. I did this with the following command:

rsync -av /opt/zimbra /opt/zimbra.orig

Note that in the following text I'll reference from time to time zimbra.orig directory because we'll need some files from there.

Experimenting with a new installation

This section is some form of a diary on how I tried to migrate Zimbra from the old server to the new one, and then how I tried to upgrade Zimbra 7 to Zimbra 8. I think that it is good to write about this trial-and-error process for at least two reasons:

1. You can learn from my experience, as (almost) nothing goes perfectly without trying.
2. If you google for some error messages, you might find it helpful to see what is the solution. There are no error messages in the condensed version of the diary section.

In case you just want a cookbook on how to do it, skip to the Cookbook section in this blog post.

Migrating existing Zimbra installation to a new server

I suggest that before you do this, you do some preparation steps:

1. Block outbound connections to port 25, i.e. prevent new instance of Zimbra to send emails until you migration is complete. And then, before going into production, check mail queue for orphaned/duplicate mails.
2. Synchronizations are time consuming on any, even moderately used host. So, do first live migration, and then stop old server and repeat migration.
3. In any case, don't do migration when the old server is running because you might leave your new server in inconsistent state.

Now, go to the old installation of Zimbra and copy the whole Zimbra tree over the Zimbra installation on the new server. The command should look something like this:

$ rsync -av --delete --exclude zimbramon/lib /opt/zimbra root@newserver:/opt/

Probably this will take some time. Also, don't forget to exclude zimbramon/lib directory or otherwise you'll end up with an old version of Perl and when you try to start some tool from Zimbra you'll get the following error message:

$ zmcontrol stop
Perl lib version (v5.8.8) doesn't match executable version (v5.16.3) at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/Config.pm line 46.
Compilation failed in require at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/lib.pm line 6.
BEGIN failed--compilation aborted at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/lib.pm line 6.
Compilation failed in require at /opt/zimbra/bin/zmcontrol line 25.
BEGIN failed--compilation aborted at /opt/zimbra/bin/zmcontrol line 25.

Anyway, in case you got the previous error just copy again zimbramon/lib directory from Zimbra 8 as explained earlier in the text.

At this point zmcontrol stop/start wan't work almost certainly. So, I suggest that you try to start service by service and check each one if it works or not. Checks can be performed looking at the process list or using netstat/ss command to see listening ports. In case some service doesn't work, check log files in /opt/zimbra/log directory and the file /var/log/zimbra.log. Actually, it would be good to check log files no matter if you think a service works or not.

The first service you should start is ldap. That one is easy, just run the following command as a zimbra user:

$ ldap start

then, look if there is slapd process running:

$ ps -ef | grep sldapd
zimbra   30224     1  0 06:56 ?        00:00:00 /opt/zimbra/openldap/sbin/slapd -l LOCAL0 -u zimbra -h ldap://mail.zemris.fer.hr:389 ldapi:/// -F /opt/zimbra/data/ldap/config

and you can also check if ldap port (389) is opened:

$ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 3.1.2.1:389      0.0.0.0:*               LISTEN
tcp        0      0 3.1.2.1:53       0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:53     0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22       0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:953    0.0.0.0:*               LISTEN
tcp6       0      0 ::1:53           :::*                    LISTEN
tcp6       0      0 :::22            :::*                    LISTEN
tcp6       0      0 ::1:953          :::*                    LISTEN

Logging, by the way, goes to syslog/journalctl because of the option -l, and that means we should look into /var/log/zimbra.log for slapd messages:

Next, I tried to start mailbox service:

$ zmmailboxdctl start

and that didn't work. Namely, no java process:

$ ps -ef | grep java
zimbra    4717  1470  0 07:19 pts/0    00:00:00 grep --color=auto java

and no listening sockets for imap, pop, etc.:

$ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 3.1.2.1:389      0.0.0.0:*   LISTEN
tcp        0      0 3.1.2.1:53       0.0.0.0:*   LISTEN
tcp        0      0 127.0.0.1:53     0.0.0.0:*   LISTEN
tcp        0      0 0.0.0.0:22       0.0.0.0:*   LISTEN
tcp        0      0 127.0.0.1:953    0.0.0.0:*   LISTEN
tcp6       0      0 ::1:53           :::*        LISTEN
tcp6       0      0 :::22            :::*        LISTEN
tcp6       0      0 ::1:953          :::*        LISTEN

Looking into log file (/opt/zimbra/log/zmmailboxd.out) reveals the following error messages:

Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
        at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        ... 18 more
1227 WARN  [main] log - Nested in java.lang.reflect.InvocationTargetException:
java.net.BindException: Cannot assign requested address

After few more minutes of scratching my head of why (while constantly resisting temptation to use strace tool) I finally realized that the problem is related to my customization on the old server. Namely, I changed jetty configuration so that Zimbra binds to a specific IP address which isn't available on the new server. The solution for this is easy, I saved a copy of Zimbra 7 immediately after installation (zimbra.orig directory, remember) so the following two commands solved this particular problem:

$ cd /opt/zimbra.orig/zimbra/$ rsync -av --delete jetty-6.1.22.z6 /opt/zimbra/

So, again I tried to start mailbox service on Zimbra:

$ zmmailboxdctl startStarting mailboxd...done.

This time, it seems to be good. But checking ports and processes again showed that it doesn't work! Looking at log file showed the following cryptic error:

Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented
        at javax.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:80)
        at com.zimbra.common.util.NetUtil.newBoundServerSocket(NetUtil.java:95)
        at com.zimbra.common.util.NetUtil.bindServerSocket(NetUtil.java:163)
        at com.zimbra.common.util.NetUtil.bindSslTcpServerSocket(NetUtil.java:54)
        ...

That was very cryptic error message. Luckily, Googling for it immediately gave the following thread. Namely, the problem is with SSL/TLS, and the following two commands from the given thread fixed that part of the problem:

$ mv /opt/zimbra/mailboxd/etc/keystore /opt/zimbra/mailboxd/etc/keystore.borked
$ sudo /opt/zimbra/bin/zmcertmgr deploycrt self

This time, it almost worked as now I had two java processes and listening sockets. But, there were a log of messages in log file like the following one:

1445 WARN  [main] log - failed org.mortbay.jetty.NCSARequestLog@64d55986: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN  [main] log - failed RequestLogHandler@72e8a021: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN  [main] log - failed HandlerCollection@6691177: java.io.IOException: Cannot write log directory /opt/zimbra/log
1445 WARN  [main] log - failed RewriteHandler@5bf99eea: java.io.IOException: Cannot write log directory /opt/zimbra/log
1446 WARN  [main] log - failed DebugHandler@613043d2: java.io.IOException: Cannot write log directory /opt/zimbra/log

At first sight, that shouldn't happened. Zimbra is owner of those files and SELinux isn't controlling Zimbra. After some scratching I found out that there is a configuration file /opt/zimbra/jetty/etc/jetty.properties in which UID and GID values for Zimbra user/group are not correct, i.e. they are same as on the old server while Zimbra on the new server has different UID/GID values:

$ id zimbra
uid=997(zimbra) gid=996(zimbra) groups=996(zimbra),4(adm),5(tty),89(postfix)

So, I edited it and changed their values to 997 and 996. Note that you should check the correct values for your specific case. Trying again the error messages didn't go away. This time I checked process list and I found out that the process ID of Java is 500. So, I missed some configuration setting. Turns out that jetty.properties file is automatically generated from jetty.properties.in and UID and GID values are taken from Zimbra configuration. So, I had to change them permanently using the following commands (as a zimbra user):

$ zmlocalconfig -e zimbra_uid=997
$ zmlocalconfig -e zimbra_gid=996

And this time it worked! But, I realized now that ldap isn't working any more:

$ ldap start
Failed to start slapd.  Attempting debug start to determine error.
TLS: error:0200100D:system library:fopen:Permission denied bss_file.c:398
TLS: error:20074002:BIO routines:FILE_CTRL:system lib bss_file.c:400
55d5688d main: TLS init def ctx failed: -1

After some time I realized  that when I redeployed certificate using the link given above, it turns out that UID and GID used to set ownership of different certificate files were the ones specified in Zimbra configuration and not the actual UID and GID, i.e.:

$ ls -l /opt/zimbra/conf/slapd.*
-rw-r-----. 1 500 500 1107 Aug 20 07:36 /opt/zimbra/conf/slapd.crt
-rw-r-----. 1 500 500 1704 Aug 20 07:36 /opt/zimbra/conf/slapd.key

I also had to find all the files whose owner was set to previous value (500) and change it to a new UID/GID values. I did this using find command as a root user:

find /opt/zimbra -uid 500 -exec chown zimbra.zimbra {} \;

Now, both LDAP and mailbox services worked! Time to try zmcontrol start but first I stopped running services. Still didn't work. This time I received two error messages from Web and mail processes:

Starting apache.../opt/zimbra/httpd-2.2.22/bin/httpd: error while loading shared libraries: libexpat.so.0: cannot open shared object file: No such file or directory
failed.
/opt/zimbra/postfix/sbin/postalias: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory
/opt/zimbra/postfix/sbin/postfix: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory

But these are easy to solve, just copy missing libraries from CentOS 5 installation. And note that there is a library and symlink to the library! I tried starting Zimbra again using zmcontrol. Again, no luck. The problem this time is a missing Perl subroutine while starting stat module:

Undefined subroutine &main::getZimbraHome called at /opt/zimbra/bin/zmstatctl line 156.

This subroutine was removed in Perl packages distributed with Zimbra 8.6, which we are using here. I found this subroutine to be defined in Zmstat.pl file and first I tried to fix it by just copying this old file, but it turns out that more than one file was changed so I tried, and succeeded, with by copying all non-binary files:

# cd /opt/zimbra.orig/zimbra/zimbramon/lib.orig
# rsync -av --exclude \*.so --exclude x86_64-linux-thread-multi * /opt/zimbra/zimbramon/lib/

Trying again zmcontrol we are receiving warnings about deprecated use of certain Perl features:

Use of qw(...) as parentheses is deprecated at /opt/zimbra/zimbramon/lib/Zimbra/Util/Common.pm line 25.
Use of qw(...) as parentheses is deprecated at /opt/zimbra/zimbramon/lib/Zimbra/Util/Common.pm line 26.

But these can be safely ignored. Finally, Zimbra 7 works! I did some more checking but not thorough one since my intention now was to upgrade Zimbra to version 8.6.

Upgrading Zimbra 7 to Zimbra 8.6

Finally, I got to the step where I tried to upgrade Zimbra to a newer version. Go to the unpacked Zimbra 8 directory and start upgrade process as usual:

./install.sh --platform-override

After starting upgrade process, I received the following error message:

Validating ldap configuration
Error: Unable to create a successful TLS connection to the ldap masters.
       Fix cert configuration prior to upgrading.

Quick GoogleFu gave the following thread with the solution to the problem.

$ zmlocalconfig -e ldap_master_url=ldaps://mymaster.somewhere.com:636
$ zmlocalconfig -e ldap_url=ldaps://myreplica.somewhere.com:636
$ zmlocalconfig -e ldap_starttls_supported=0
$ zmlocalconfig -e ldap_port=636 

Be careful to change host names to names that match your configuration. This wasn't enough because it turned out to be that my CA expired, i.e.:

# openssl x509 -in /opt/zimbra/conf/ca/ca.pem -noout -text | grep 'Not After'
            Not After : Apr 11 11:22:21 2015 GMT

Which meant that I had to recreate CA:

# /opt/zimbra/bin/zmcertmgr createca -new
# /opt/zimbra/bin/zmcertmgr createcrt -new -days 1825
# /opt/zimbra/bin/zmcertmgr deploycrt self
# /opt/zimbra/bin/zmcertmgr deployca

Be sure to restart slapd after this, i.e. in my case slapd was already running so it didn't pick up new CA/CERT! Actually, it is much better to stop Zimbra and then to start upgrade process. Anyway, I restarted upgrade process! There were two warnings, but it seems they can be ignored:

ERROR 1133 (28000) at line 2: Can't find any matching row in the user table
ERROR 1396 (HY000) at line 1: Operation DROP USER failed for ''@'mail.somedomain.com'

Also, at one point installation stopped because some errors with LDAP's password. The configuration menu was presented with lot of marks that things have to be fixed. I selected LDAP password, and when the configuration program assigned new password everything after that went well.

And that's it! Unfortunately, my old mail has some additional services I had to migrate too, but I won't write about them in this post.

I have to note several things. First, this process heavily depends on configurations on both new and old server being as close as possible. This means that if you did customizations that are outside of /opt/zimbra tree, then you'll have to transfer those too. In my case, I integrated mailman with Zimbra, and mailman lives in other directories, so I had problems. Second,

Quick how to migrate Zimbra7/CentOS5 to Zimbra8/CentOS7

So, here is the recipe on how to move old Zimbra installation to the new server. This is condensed version of the previous section.

Preparation

1. Download ZCS 7.2.7 (for RHEL6) and ZCS 8.6.0 (you can download newer version if you wish, but I didn't test it so expect there might be some issues). I'll assume you downloaded them into /tmp directory.
2. Unpack both archives.
3. Enter into ZCS 8.6 directory and then into packages/ subdirectory. Make there temporary directory (e.g. tmp/) and enter into it. Unpack core package (zimbra-core-8.6.0_GA_1153.RHEL7_64-20141215151110.x86_64.rpm) using rpm2cpio and cpio commands.
4. Enter into ZCS 7.2.7 directory and modify file util/utilfunc.sh file. Line 2287 should be PACKAGEINST='rpm --force -iv'.
5. From the old server copy the following libraries into /lib64 directory on a new server: libssl.so.*, libcrypto.so.*, libpcre.so.*, and libexpat.so.*.

Installation

1. Enter direcotry ZCS 7.2.7 and start installation with the following command:
   ./install.sh -s --platform-override
2. When script finishes, go to /opt/zimbra/zimbramon directory rename lib directory to lib.orig.
3.  Go to the directory where you unpacked zimbra-core package (should be something like /tmp/zcs-8.6.0-something/packages/tmp/) and execute the following command:
   rsync -av opt/zimbra/zimbramon/lib /opt/zimbra/zimbramon/.
4. Run the command /opt/zimbra/libexec/zmsetup.pl. In case the script complains about missing Perl module and stops, install the missing module and run the script again. 
5. Stop Zimbra and disable starting during the boot process:
   
   # /etc/init.d/zimbra stop
   # chkconfig zimbra off
6. Make a copy of vanilla Zimbra directory tree, i.e. execute the following commands:
   
   # cd /opt
   # rsync -av zimbra zimbra.orig

Syncing with old server

1. Go to the old server and run the following command:
   
   rsync -av --delete --exclude zimbramon/lib --exclude jetty\* /opt/zimbra newserver:/opt/
2. Again on the new server run the following commands (as a zimbra user):
   
   $ zmlocalconfig -e zimbra_uid=997
   $ zmlocalconfig -e zimbra_gid=996
   
   Change UID and GID to match your new installation!
3. Turn on LDAP server, i.e. switch to the Zimbra user and execute the following command:
   
   $ ldap start
4. Run the following commands to generate new CA.
   
   # /opt/zimbra/bin/zmcertmgr createca -new
   # /opt/zimbra/bin/zmcertmgr createcrt -new -days 1825
   # /opt/zimbra/bin/zmcertmgr deploycrt self
   # /opt/zimbra/bin/zmcertmgr deployca
   
   Might not be necessary, but it doesn't hurt. And yes, if you have commercial CERT, you'll have to import it and these steps are not necessary.
5. Turn off LDAP server, i.e. switch to the Zimbra user and execute the following command:
   
   $ ldap stop
6. Run the following commands to "fix" Perl version mismatch on new and old servers:
   
   # cd /opt/zimbra.orig/zimbra/zimbramon/lib.orig
   # rsync -av --exclude \*.so --exclude x86_64-linux-thread-multi * /opt/zimbra/zimbramon/lib/

Upgrade

1. Go to the directory where you unpacked ZCS 8.6 and run upgrade process as usual:
   
   ./install.sh --platform-override

Some additional notes

In this section I'll add some additional notes.

I noticed after some time that while starting Zimbra, zmconfigd service reports error during startup. After some debugging I found out that the service is working, but that the problem is in how check if it is working is implemented. Namely, one step in the process is to connect to localhost port 7171 using nc tool. But since localhost first resolves to IPv6 address, and the service doesn't listen on IPv6, so the connection is refused, nc doesn't try IPv4 then, and script wrongfully thinks that the service isn't running. The solution was to edit /etc/hosts file and change it so that localhost resolves exclusively to 127.0.0.1 while localhost6 resolves to ::1.

Incremental backup with rsync and cp with low disk demand

I just created very simple incremental backup solution for my Zimbra installation which efficiently uses disk space. The idea is simple. First, using rsync, I'm making a copy of existing Zimbra installation to another disk:

rsync --delete --delete-excluded -a \
        --exclude zimbra/data/amavisd/ \
        --exclude zimbra/data/clamav/ \
        --exclude zimbra/data/tmp \
        --exclude zimbra/data/mailboxd/imap-inactive-session-cache.data \
        --exclude zimbra/log \
        --exclude zimbra/zmstat \
        /opt/zimbra ${DSTDIR}/

I excluded from synchronization some directories that are not necessary for restoring Zimbra. Then, using cp I'm creating copy of this directory but which only consists of hard links to original files, the content isn't copied:

cd ${DSTDIR}
cp -al zimbra zimbra.`date +%Y%m%d%H%M`

Note the option -l that tells cp to hard link files instead of making a new copy. Also, note that the copy created is named so that it contains timestamp when it was created. Here is the content of the directory:

$ ls -l ${DSTDIR}
total 16
drwx------ 7 root   root    4096 Pro  9 15:31 zimbra
drwx------ 7 root   root    4096 Pro  9 15:31 zimbra.201412131551
drwx------ 7 root   root    4096 Pro  9 15:31 zimbra.201412140326
drwx------ 7 root   root    4096 Pro  9 15:31 zimbra.201412150325

Next time rsync runs, it will delete files that don't exist any more, and when it copies changed files it will create a new copy, and then remove the old one. Removing the old one means unlinking which in essence leaves the old version saved in the directory made by cp. This way you'll allocate space only for new and changed files, while the old ones will share disk space.

This system uses only the space it needs. Now, it is interesting to note du's command behavior in case of hard links. Here is an example:

# du -sh zimbra*
132G      zimbra
3.4G      zimbra.201412131551
3.2G      zimbra.201412140326
114M      zimbra.201412150325
# du -sh zimbra.201412131551
132G      zimbra.201412131551
# du -sh zimbra.201412150325
132G      zimbra.201412150325

In the first case it tells us how much space is used by main directory, zimbra, and then it tells us the difference in usage of the other directories, e.g. zimbra is using 132G and zimbra.201412131551 uses 3.4G more/differently. But, when we give specific directory to du command, then it tells us how much this directory is by itself, so we see that all the files in zimbra.201412131551 indeed use 132G.

And that's basically it. These two commands (rsync and cp) are placed in a script with some additional boilerplate code and everything is run from cron.

Fedora 20 update to kernel 3.17.2-200 and VMWare Workstation

Since I updated to VMWare Workstation 10.0.5 at the end of January 2015, things were again broken. Returning to this post I found that the link in the post now points to something that has changed and there is no patch nor instructions of what to do. So, I had to google again and now I placed the complete instructions in this post so that the next time I don't have to google again.

It turns out that there is a single line that has to be changed in the vmnet module in order for VMWare to be runnable again. So, here are the steps you have to do in order to patch the file:

1. Create temporary directory, e.g. /tmp/vmware and position yourself in that directory.
2. Create a file named vmnet.patch and put into it the following content:
   
   

   diff -ur vmnet-only.a/netif.c vmnet-only/netif.c
   --- vmnet-only.a/netif.c    2014-10-10 03:23:08.585920012 +0300
   +++ vmnet-only/netif.c  2014-10-10 03:23:09.245920008 +0300
   @@ -149,7 +149,7 @@
       memcpy(deviceName, devName, sizeof deviceName);
       NULL_TERMINATE_STRING(deviceName);
   
   -   dev = alloc_netdev(sizeof *netIf, deviceName, VNetNetIfSetup);
   +   dev = alloc_netdev(sizeof *netIf, deviceName, NET_NAME_UNKNOWN, VNetNetIfSetup);
       if (!dev) {
          retval = -ENOMEM;
          goto out;
   

   
   
3. Unpack /usr/lib/vmware/modules/source/vmnet.tar in the current directory (/tmp/vmware):
   
   

   tar xf /usr/lib/vmware/modules/source/vmnet.tar
   

   
   
4. Patch the module:
   
   

   cd vmnet-only; patch -p1 < ../vmnet.patch; cd ..
   

   
   
5. Make a copy of old, unpatched, archive:
   
   

   mv /usr/lib/vmware/modules/source/vmnet.tar /usr/lib/vmware/modules/source/vmnet.tar.SAVED
   

   
   
6. Create a new archive:
   
   

   tar cf /usr/lib/vmware/modules/source/vmnet.tar vmnet-only
   

   
   
7. Start vmware configuration process:
   
   

   vmware-modconfig --console --install-all
   

Hopefully, that should be it.

Old instructions (not valid any more!)

Well, here we go again. After recent update which brought kernel 3.17 to Fedora 20, VMWare Workstation 10.0.4 had problems with kernel modules. Luckily, after some short googling I found a solution. That solution works. There are two things that might confuse you though:

1. After cd command and before for loop you have to switch to root account (that is indicated by prompt sign change from $ to #).
2. The substring kernel-version in patch command should be replaced with a string "3.17". That is actually the name you gave to a file while executing curl command at the beginning of the process.

Anyway, that's it.

Private addresses in IPv6 protocol

It is almost a common wisdom that 172.16.0.0/12, 192.168.0.0/16, and 10.0.0.0/8 are private network addresses that should be used when you don't have assigned address, or you don't intend to connect to the Internet (at least not directly). With IPv6 being ever more popular, and necessary, the question is which addresses are used for private networks in that protocol. In this post I'll try to answer that question.

The truth is that in IPv6 there are two types of private addresses, link local and unique local addresses. Link local IPv6 addresses, as the name suggests, are valid only on a single link. For example, on a single wireless network. You'll recognize those addresses by their prefix, which is fe80::/10, and they are automatically configured by appending interface's unique ID. IPv4 also has link local address, though it is not so frequently used. Still, maybe you noticed it when your DHCP didn't work and suddenly you had address that starts with 169.254.0.0./16. This was a link local IPv4 address configured. The problem with link local addresses is that they can not be used in case you try to connect two or more networks. They are only valid on a single network, and packets having those addresses are not routable! So, we need something else.

Unique local addresses (ULA), defined in RFC4193, are closer to IPv4 private addresses. That RFC defines ULA format and how to generate them. Basically, those are addresses with the prefix FC00::/7. These addresses are treated as normal, global, addresses, but are only valid inside some restricted area and can not be used on the global Internet. This is the same as saying that 10.0.0.0/8 addresses can be used within some private networks, but are not allowed on a global Internet. You choose how this conglomerate of networks will be connected, what prefixes used, etc.

There is  difference, though. Namely, it is expected that ULA will be unique in the world. You might ask why is that important, when those addresses are not allowed on the Internet anyway. But, that is important. Did it ever happened to you that you had to connect two private IPv4 networks (directly via router, via VPN, etc.), and coincidentally, both used, e.g. 192.168.1.0/24 prefix? Such situations are a pain to debug, and require renumbering or some nasty tricks to make them work. So, being unique is an important feature.

So, the mentioned RFC, actually specifies how to generate ULA with /48 prefix and a high probability of the prefix being unique. Let's first see the exact format of ULA:

| 7 bits |1|  40 bits   |  16 bits  |          64 bits           |
+--------+-+------------+-----------+----------------------------+
| Prefix |L| Global ID  | Subnet ID |        Interface ID        |
+--------+-+------------+-----------+----------------------------+

First 8 bits have a fixed value 0xFD. As you can see, prefix is 7 bit, but L bit must be set to 1 if the address is specified according to the RFC4193. So, first 8 bits are fixed to the value 0xFD. Note that L bit set to 0 isn't specified, it is something left for the future. Now, the main part is Global ID, whose length is 40 bits. That one must be generated in such a way to be unique with high probability. This is done in the following way:

1. Obtain current time in a 64-bit format as specified in the NTP specification.
2. Obtain identifier of a system running this algorithm (EUI-64, MAC, serial number).
3. Concatenate the previous two and hash the concatenated result using SHA-1.
4. Take the low order 40 bits as a Global ID.

The prefix obtained can be used now for a site. Subnet ID can be further used for multiple subnets within a site. There are Web based implementations of the algorithm you can use to either get a feeling of the generated addresses, or to generate prefix for your concrete situation.

Occasionally you'll stumble upon so called site local addresses. Those addresses were defined starting with the initial IPv6 addressing architecture in RFC1884 and were also defined in subsequent revisions of addressing architecture (RFC2373, RFC3513) but were finally deprecated in RFC3879. Since they were defined for so long (8 years) you might stumble upon them in some legacy applications. They are recognizable by their prefix FEC0::/10. You shouldn't use them any more, but use ULA instead.

Installing OSSIM community edition in QEMU

Since OSSIM is based on Debian and it is a nightmare to compile it for something else (ehm, CentOS) I decided to use it in a headless QEMU virtual machine. To test the whole process, I first decided to do a regular installation of OSSIM, with display. But, I had a lot of obstacles while trying install OSSIM community edition in QEMU. It is even more interesting that when you google for ossim and qemu, there are almost no posts.

In the end, everything worked flawlessly but when using text based installation. To access text based installation edit boot command line (pressing TAB at the initial boot screen) and at the end add the following:

DEBIAN_FRONTEND=text

And that gave me text based installation. Basically, AlienVault uses Debian's installer so anything that can be configured for Debian, can be for OSSIM too. Take a look into manual for further information.

Few things to be aware of when doing this:

1. Don't use too small disk because the installation will stuck without any notification what happened.
2. I had problems with GUI based installation, and its fallback ncurses. The installation would stuck somewhere (e.g. in GUI after entering IP address, something would go wrong in package installation process, MySQL wasn't properly installed and there were errors that starting failed, apache wasn't properly installed and Web console wasn't accessible, etc.)

CentOS 6

On CentOS there is no qemu-kvm like in Fedora. Instead, you have to use libvirtd. Be sure that libvirt is installed, before continuing. That means packages virt-install and libvirt are installed. Additionally, libvirtd daemon must bi started.

So, first create file for disk image. You can do this using dd, but even better is to use fallocate(1) command. Also, fetch OSSIM ISO image file. Now, to start installation process use the following command:

virt-install -r 2560 --accelerate -n OSSIM \
        --cdrom /tmp/AlienVault_OSSIM_64Bits_4.3.4.iso \
        --os-variant=debiansqueeze --disk path=./sda.img \
        -w bridge --graphics vnc,password=replaceme

In the previous command I'm giving to OSSIM 2.5G RAM (option -r), the name will be OSSIM, disk image is in the current directory (with respect to the command virt-install) and I'm using bridged networking. Finally, console will be available via VNC and the password for access is replaceme.

There are several error messages you might receive when trying to start installation process:

ERROR    Error with storage parameters: size is required for non-existent disk '/etc/sysconfig/network-scripts/sda.img'

Well, this error message occured because I was trying to start installation process in the wrong directory, i.e. the one that didn't contain file for hard disk image.

The following error:

ERROR    Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

means that libvirtd daemon isn't started. Start it using:

service libvirtd on

and don't forget to make it start every time you boot your machine:

chkconfig libvirtd on

The next error:

Starting install...
ERROR    internal error Process exited while reading console log output: char device redirected to /dev/pts/1
qemu-kvm: -drive file=/root/AlienVault_OSSIM_64Bits_4.3.4.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw: could not open disk image /root/AlienVault_OSSIM_64Bits_4.3.4.iso: Permission denied

means that I placed ISO image in a directory where libvirt can not access it. Move image to, e.g. /tmp directory and try again.

After you managed to start installation process, connect to it using vncviewer application. libvirt-install binds vnc to localhost so you won't be able to access it directly from some remote host. This is actually OK, so you shouldn't change it, unless you know very well what you are doing. So, to connect to console, open terminal window and execute the following command:

ssh -L 5900:127.0.0.1:5900 host_where_installation_is_started

Now, in another terminal on local machine (i.e. the one where you started previous ssh command) run the following command:

vnc localhost

And that should be it. What happened is that with ssh you created a tunnel between your local machine and the remote where virtual machine is being installed. So, don't stop ssh until you vnc session is running!

Installing Snort 2.9.6.0 on CentOS 6.5 64-bit

Some time ago I wrote a post about installing Snort 2.9.1 on CentOS 6. In the mean time I decided it's time to upgrade so the idea of this post is to document what changed with respect to that older post. In short, binary packages for CentOS 6 are now provided on the Snort's download page. So, you only need to download them and install (or install using URL). Yet, there is a problem with a libdnet dependency (I don't know which one was used during compilation, but it certainly wasn't the one in EPEL).

Compiling and installing

In case you want to rebuild them, the process is now almost without any problems. In the following text I'll assume that you started with a minimal CentOS installation with the following packages installed (and their dependencies, of course): gcc, make, bison, flex, autoconf, automake, rpmbuild.

First, download daq source rpm file. Before rebuilding it, you should install pcap-devel. This is actually something rpmbuild tool will warn you that you have to install. When you installed it, rebuild daq:

rpmbuild --rebuild daq

then, install it:

yum localinstall ~/rpmbuild/RPMS/x86_64/daq-2.0.2-1.x86_64.rpm

Next, for snort you'll need libdnet library which is in EPEL. So, first install EPEL:

yum install http://mirrors.neterra.net/epel/6/i386/epel-release-6-8.noarch.rpm

Then, install necessary packages:

yum install libdnet-devel zlib-devel

Those two aren't listed as dependencies in Snort's SRPM file, so you'll get some cryptic error message. Now, download Snort's srpm file and rebuild it using:

rpmbuild --rebuild snort-2.9.6.0-1.src.rpm

Now, install it using:

yum localinstall ~/rpmbuild/RPMS/x86_64/snort-2.9.6.0-1.x86_64.rpm

That's all there is for installation.

Configuring and running

I'll assume that you are installing a fresh instance, i.e. no previous configuration. In case there is previous installation be careful not to overwrite existing configuration. To configure snort you'll have to download snortrules archive. Then, unpack it:

mkdir ~/snort
tar xzf snortrules-snapshot-2960.tar.gz -C ~/snort
chown root.root ~/snort

Next you have to move files in their place. First, move basic configuration file:

mv -f snort/etc/* /etc/snort/

Note that I'm using force option of move command to overwrite existing files. Next, move rules to their place:

mv -i snort/rules snort/preproc_rules snort/so_rules /etc/snort/

Now, if you are using SELinux you should change context of the files you moved to /etc/snort directory. Do it using the following commands:

chcon -R system_u:object_r:snort_etc_t:s0 /etc/snort
chcon -R system_u:object_r:lib_t:s0 /etc/snort/so_rules/precompiled/RHEL-6-0/

You should now modify configuration file. Here is a diff of the changes I made:

--- snort.conf.orig 2014-03-13 11:25:53.889609831 +0100
+++ snort.conf 2014-03-13 11:37:32.419292894 +0100
@@ -42,16 +42,16 @@
 ###################################################

 # Setup the network addresses you are protecting
-ipvar HOME_NET any
+ipvar HOME_NET 192.168.1.0/24

 # Set up the external network addresses. Leave as "any" in most situations
 ipvar EXTERNAL_NET any

 # List of DNS servers on your network
-ipvar DNS_SERVERS $HOME_NET
+ipvar DNS_SERVERS 192.168.1.8,192.168.1.9

 # List of SMTP servers on your network
-ipvar SMTP_SERVERS $HOME_NET
+ipvar SMTP_SERVERS 192.168.1.20

 # List of web servers on your network
 ipvar HTTP_SERVERS $HOME_NET
@@ -101,13 +101,13 @@
 # Path to your rules files (this can be a relative path)
 # Note for Windows users:  You are advised to make this an absolute path,
 # such as:  c:\snort\rules
-var RULE_PATH ../rules
-var SO_RULE_PATH ../so_rules
-var PREPROC_RULE_PATH ../preproc_rules
+var RULE_PATH rules
+var SO_RULE_PATH so_rules
+var PREPROC_RULE_PATH preproc_rules

 # If you are using reputation preprocessor set these
-var WHITE_LIST_PATH ../rules
-var BLACK_LIST_PATH ../rules
+var WHITE_LIST_PATH rules
+var BLACK_LIST_PATH rules

 ###################################################
 # Step #2: Configure the decoder.  For more information, see README.decode
@@ -240,13 +240,13 @@
 ###################################################

 # path to dynamic preprocessor libraries
-dynamicpreprocessor directory /usr/local/lib/snort_dynamicpreprocessor/
+dynamicpreprocessor directory /usr/lib64/snort-2.9.6.0_dynamicpreprocessor/

 # path to base preprocessor engine
-dynamicengine /usr/local/lib/snort_dynamicengine/libsf_engine.so
+dynamicengine /usr/lib64/snort-2.9.6.0_dynamicengine/libsf_engine.so.0

 # path to dynamic rules libraries
-dynamicdetection directory /usr/local/lib/snort_dynamicrules
+dynamicdetection directory /etc/snort/so_rules/precompiled/RHEL-6-0/x86-64/2.9.6.0/

 ###################################################
 # Step #5: Configure preprocessors

And you can download the complete snort.conf file that worked for me. Be careful, you need to change IP addresses in the configuration file to match your environment.

Finally, create two empty files, /etc/snort/rules/white_list.rules and /etc/snort/rules/black_list.rules.

Now, you should be able to start Snort, i.e.

# /etc/init.d/snortd start
Starting snort: Spawning daemon child...
My daemon child 1904 lives...
Daemon parent exiting (0)                         [  OK  ]

Compiling OVALDI 5.10.1.6 on CentOS 6.5

Some time ago I wrote about compiling Ovaldi on CentOS 6. Now, I tried to compile it again, and I found out that some things changed. Most importantly, there is no need to compile old Xalan/Xerces libraries any more. But, there are still problems with RPM. To make the story short, I managed to compile it and create RPM. Here are the files:

• patch you need to be able to compile ovaldi
• SRPM file you can use to recompile ovaldi; it contains patch
• RPM file if you don't want to compile it yourself (and you trust me ;))

Note that I didn't do any testing at all! So, it might happen that rpm based stuff doesn't work. If that's the case leave a comment and I'll take a look when I find time.

Kernel upgrade to 3.13.3-201 and VMWare Workstation...

I just started VMWare for the first time after upgrade and restart into kernel 3.13.3-201, and it didn't work. Well, I already got used to it. Anyway, the fix was very quick. I found a patch in a post on VMWare forums, downloaded it and applied it. The application process is a bit more manually involved that with the previous patches. You need to:

1. Switch to root user.
2. Go to the /usr/lib/vmware/modules/source directory
3. Unpack vmnet.tar using tar command.
4. Enter into newly created vmnet-only directory
5. Apply patch (patch -p1 < path_and_name_of_unpacked_patch_file). Patch shouldn't make any noise, only patched file is displayed.
6. Go one level up (exit vmnet-only directory)
7. Rename existing vmnet.tar into something else, just in case.
8. Create new vmnet.tar (tar cf vmnet.tar vmnet-only)
9. Start vmware

And that's it...

Messing up Zimbra upgrade, a.k.a. dangerous --platform-override option...

You know that Zimbra's installer has --platform-override option that enables you to install Zimbra on unsupported platforms, like CentOS. Well, it also allows you to mess up things. Namely, I downloaded Zimbra for RHEL6, but was upgrading an existing installation on CentOS5. And I didn't noticed anything unusual until upgrade process failed! It failed very early with an error message from loader that it couldn't start perl binary. Namely, the error message I got, was similar to:

/usr/bin/perl: symbol lookup error: /opt/zimbra/zimbramon/lib/i386-linux-thread-multi/auto/IO/IO.so: undefined symbol: Perl_Tstack_sp_ptr

It wasn't exactly that one, but similar! Well, I was surprised because Zimbra's upgrade process is good and robust, and I had never problems with it, but since I had some local customizations on my installation I suspected those to be the cause. So, I ended up by copying CentOS' Perl installation into Zimbra tree. This process wasn't actually so easy to do and that should've warned me that something is terribly wrong, but I continued nevertheless. But, when I managed to pass that point, the next one finally made me realize the mistake. Namely, mysql binary required Glibc 2.7, while in CentOS 5 is some earlier version. Now, I knew what mistake I've made.

Here is also a point I realised that I'm facing a long night trying to recover Zimbra installation!

So, I've downloaded the correct version of Zimbra, the one I intended to upgrade to, manually installed it using rpm tool, and then restarted upgrade process with:

/opt/zimbra/libexec/zmsetup.pl

And here my problems continued. Namely, the upgrade script correctly figured that I'm trying to upgrade, it also correctly figured the old and the new version, but it stuck on starting mysql server. After some time I managed to figure out where the problem is, namely, existing mysql database has password while the installation process assumes there isn't one. So, it couldn't determine if mysql was started or not, and everything stalled. I lost some time trying to make database not to ask for the password, and even though I managed to do something, luckily, in the mean time I stumbled on the following Wiki page:

Recovering from wrong platform upgrade

I made it large because it really got me out of the problem. It is written for version Zimbra version 5, but it works on 7, too. In a nutshell, you install the old Zimbra versions you had and then start from there with the idea of making a rollback. BUT, BIG WARNING, this is possible only if your localconfig.xml file is intact. In my case, I erased it, but the upgrade process saved a copy which I was able to recover! I had some additional problems while testing installation as suggested byt the Wiki page, but all of them were related to the tweaks I did to my installation of Zimbra, i.e. binding Zimbra to a single IP address.

What have I learned

Well, first and foremost, do a backup before doing an upgrade. Yes, I know it was stupid, and beginner's, mistake, but as I've said, Zimbra's upgrade process is really good, at least for me. The real reason I didn't do it was that it takes several hours to make it. If I had copy, I would revert to that copy, and I would only need to fix RPM database. So, to avoid the same thing happening again, now I did make a backup process which rsync'es all of Zimbra tree to an alternative location every day.

Next thing is, localconfig.xml file is extremely important, so I did additional backup of that file, too. Note that in the file are all passwords so, it's also critical from the security perspective!

Also, I would suggest that Zimbra adds checks into installation script to guard against mistakes like the one I did. It turns out, after some googling, that there are a lot more other posts with the similar problems. This can be easily done for CentOS/RHEL with appropriate requirements in RPM packages.

And for the end, here is one other story about failed Zimbra upgrade which I find interesting.

Modifying mail passing through Zimbra

I had a request to attach to (almost) each mail message that passes through the Zimbra mail server an image. Basically, what the owner wanted is that there is an image with advertisement in the mail that is sent by internal users. Additional requirements were:

1. Image should be added only once!
2. In case there is new image, and there is already on in the mail, the old one should be replaced!
3. Image has to be at exact spot within a message.

There were some additional requirements from my POV:

1. Not every mail should have image attached, e.g. automatically generated internal messages!
2. I should be careful about impact on the performance.
3. Mali messages that are not modified should not have any noticeable marks about image that isn't added (this one will be more clear later).
4. The solution should allow to define only certain senders to have mails modified.
5. It has to have a DRY RUN mode so that it is easily disabled.

I immediately knew that there is no way to place image somewhere on the Internet and put link into the mail message. Although the most elegant solution, the problem is that all mail clients don't show images by default, and that's it. So, image has to be within mail message itself. A bit of research showed up a potential solution. Namely, to embed image data within IMG tag itself, and mail messages are already altered by adding disclaimer which is HTML and a perfect place to add that IMG tag, so why not reuse disclaimer for the same purpose? In favor of that solution was also my intention not to add new scripts into the mail processing chain because of fear that it might impact performance. 

Unfortunately, that solution didn't work. The most important shortcoming is that Outlook and GMail don't handle IMG tag with embedded image data in it, in other words, the image isn't shown. To solve that, image has to be embedded as a MIME part within mail. Additionally, the other requirements aren't easy to achieve, especially, the one to replace old image with a new one. So, in the end I had to resort to writing scripts.

I already wrote about how I managed to solve more complex requirements for a disclaimer than the functionality of Zimbra allows. So, it was natural place for me to add that additional processing there. I called the script to add image altermail.py and now altermime script has the following form:

#!/bin/bash
grep "DISCLAIMER:" ${1#--input=} > /dev/null 2>&1
if [ ! "$?" = 0 ]; then
/opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
fi
#echo "`date +%Y%m%d%H%M%S` $@" >> /tmp/altermime-args
/opt/zimbra/altermime-0.3.10/bin/altermail.py "$@" >> /tmp/altermail.log 2>&1

I'm calling altermial.py after altermime because the image placeholder is within disclaimer! Also, I removed exec keyword before altermime-bin call so that altermail.py is finished.

Additionally, note that altermail.py accepts the same arguments as altermime! This is in order to simplify things a bit.

Obviously, I choose Python as a programming language of my choice. I could write all that in Perl too, but since I'm lately working a lot more with Python, Python was the way to go. Both languages have very good support for mail processing (MIME messages in particular).

The script is on the GitHub and you can fetch it there.

How the script works

First of, script doesn't work for mail messages that aren't MIME. So, after loading a message the first check if it is a multipart message. If not, then script just exits.

Next, white and black lists are checked.

There are two passess over the mail message. In the first pass, it searches through the mail message to see if there is already image attached. If so, then it additionally checks if it is an older version of the image. If it is, it replaces the image, but in both cases it doesn't do anything more and finishes execution.

The second pass is done when there is no image in the mail message and it has to be added. Now, when adding the image it has to be added with HTML as html/related so that they are both shown. If you add it as html/alternative, then only one of them will be shown!

All the configuration options are embedded within script itself. I chose not to have configuration file to reduce number of disk accesses, which is already very high (a lot of modules are necessary).

Systemd waiting for external crypted disks...

I have an external, encrypted, disk that I periodically connect to the laptop. In order to have some easy to remember name, instead of UUID, I placed an entry in /etc/crypttab:

EXTDISK UUID=a809c218-f828-4149-bd9e-1c352a5f94df none

That way, when I connect disk, it will be automatically named EXTDISK and it will have an entry in /dev/mapper directory. Eventually, it will be mounted under /run/media/<userid>/EXTDISK. Note that there are only three fields in the line, each field separated from the other using space. The last field is passphrase placeholder, but I didn't want to have it written on the disk, so I used keyword none to signal that I want to type it each time the disk is opened.

The problem with this setup is that during  boot procedure, systemd waits for this disk to appear and, since there is no disk, it has to timeout. In system logs there will be messages like the following ones:

Dec 29 13:32:15 w530 systemd: Dependency failed for Cryptography Setup for EXTDISK.
Dec 29 13:32:15 w530 systemd: Dependency failed for dev-mapper-EXTDISK.device.
Dec 29 13:37:27 w530 systemd: Expecting device dev-mapper-EXTDISK.device...

The solution is simple. In newer versions of /etc/crypttab there is nofail option that should be added as a part of the fourth field. Note that, if there are multiple options in the fourth field, they all should be separated using commas and no spaces are allowed there. This option isn't listed in the manual page I linked in the introductory section of the post, so check your local manual page about crypttab.

As a side note, while searching for the solution of this timeout problem, I needed at one point to know which physical devices are beneath luks devices, i.e. my /dev/mapper directory looks like this:

# ls -l /dev/mapper/
total 0
crw-------. 1 root root 10, 236 Dec 29 13:27 control
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 fedora-home -> ../dm-3
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 fedora-root -> ../dm-2
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 fedora-swap -> ../dm-1
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 luks-a2c17ceb-222e-4cd2-3330-24a0a1111b43 -> ../dm-0
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 luks-c7e8d2f7-1114-45c0-333b-fb8444222884 -> ../dm-4

What I was curious about are those two luks symlinks for which I didn't know their physical devices. It turned out its easy to find out, using cryptsetup tool:

# cryptsetup status luks-a2c17ceb-222e-4cd2-3330-24a0a1111b43
/dev/mapper/luks-a2c17ceb-222e-4cd2-3330-24a0a1111b43  is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  device:  /dev/sda2
  offset:  4096 sectors
  size:    999184384 sectors
  mode:    read/write

So, there is an answer, /dev/sda2.

Sudden crash during uprgrades on Fedora 20

It happened several times already that when I started upgrade via 'yum update' gnome-shell, or X, would crash and leave package database in an inconsistent state. One of those inconsistent states is the following one:

# yum check
Loaded plugins: langpacks, refresh-packagekit
apr-1.5.0-2.fc20.x86_64 is a duplicate with apr-1.4.8-2.fc20.x86_64
apr-util-1.5.3-1.fc20.x86_64 is a duplicate with apr-util-1.5.2-4.fc20.x86_64
apr-util-ldap-1.5.3-1.fc20.x86_64 is a duplicate with apr-util-ldap-1.5.2-4.fc20.x86_64
bijiben-3.10.2-1.fc20.x86_64 is a duplicate with bijiben-3.10.1-1.fc20.x86_64
cifs-utils-6.2-5.fc20.x86_64 is a duplicate with cifs-utils-6.2-4.fc20.x86_64
duplicity-0.6.22-1.fc20.x86_64 is a duplicate with duplicity-0.6.21-1.fc20.x86_64
ghc-numbers-3000.2.0.0-1.fc20.x86_64 is a duplicate with ghc-numbers-3000.1.0.3-3.fc20.x86_64

...

Error: check all

Now, you could try with yum-complete-transaction command, which should complete transaction, as suggested by its name:

# yum-complete-transaction
Loaded plugins: langpacks, refresh-packagekit
There are 1 outstanding transactions to complete. Finishing the most recent one
The remaining transaction had 46 elements left to run
--> Running transaction check
---> Package apr.x86_64 0:1.4.8-2.fc20 will be erased
---> Package apr-util.x86_64 0:1.5.2-4.fc20 will be erased
---> Package apr-util-ldap.x86_64 0:1.5.2-4.fc20 will be erased
---> Package bijiben.x86_64 0:3.10.1-1.fc20 will be erased
---> Package cifs-utils.x86_64 0:6.2-4.fc20 will be erased
---> Package duplicity.x86_64 0:0.6.21-1.fc20 will be erased
---> Package ghc-numbers.x86_64 0:3000.1.0.3-3.fc20 will be erased
--> Processing Dependency: ghc-numbers = 3000.1.0.3-3.fc20 for package: ghc-numbers-devel-3000.1.0.3-3.fc20.x86_64
...
--> Running transaction check
---> Package ghc-numbers-devel.x86_64 0:3000.1.0.3-3.fc20 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
Transaction size changed - this means we are not doing the
same transaction as we were before. Aborting and disabling
this transaction.
You could try running: package-cleanup --problems
                       package-cleanup --dupes
                       rpm -Va --nofiles --nodigest
Transaction files renamed to:
  /var/lib/yum/transaction-all.2013-12-29.10:53.49.disabled
  /var/lib/yum/transaction-done.2013-12-29.10:53.49.disabled

This didn't help, as evidenced by running 'yum check' command. Note that running suggested commands, package-cleanup and rpm, with given options will not help either.

I solved this using 'yum reinstall' command. Namely, for each line that 'yum check' showed, e.g.

apr-1.5.0-2.fc20.x86_64 is a duplicate with apr-1.4.8-2.fc20.x86_64

I run 'yum reinstall' with a first name, i..e.

yum reinstall apr-1.5.0-2.fc20.x86_64

The problem with this approach is that it is too demanding, you have to do it package by package. Now, if you try something like this:

yum reinstall `yum check | cut -f1 -d" "`

You'll receive a message like the following one:

Error:  Multilib version problems found. This often means that the root
       cause is something else and multilib version checking is just
       pointing out that there is a problem. Eg.:
     
         1. You have an upgrade for gnutls which is missing some
            dependency that another package requires. Yum is trying to
            solve this by installing an older version of gnutls of the
            different architecture. If you exclude the bad architecture
            yum will tell you what the root cause is (which package
            requires what). You can try redoing the upgrade with
            --exclude gnutls.otherarch ... this should give you an error
            message showing the root cause of the problem.
     
         2. You have multiple architectures of gnutls installed, but
            yum can only see an upgrade for one of those architectures.
            If you don't want/need both architectures anymore then you
            can remove the one with the missing update and everything
            will work.
     
         3. You have duplicate versions of gnutls installed already.
            You can use "yum check" to get yum show these errors.
     
       ...you can also use --setopt=protected_multilib=false to remove
       this checking, however this is almost never the correct thing to
       do as something else is very likely to go wrong (often causing
       much more problems).
     
       Protected multilib versions: gnutls-3.1.18-1.fc20.i686 != gnutls-3.1.17-3.fc20.x86_64

Looking a bit what is happening here didn't reveal a course. This is the problem with multilib programs/libraries and it seems that yum isn't processing them correctly due to either a bug or, more probably, some configuration setting. Luckily, in this case there is a way around this problem, and that is to first deinstall the old versions, and then to reinstall new ones, just in case. Because we are doing two operations that depend on state of the RPM database _before_ we begin, we'll save list of problematic packages into a file:

yum check > yum_check

Now, we first deinstall old versions of packages:

yum remove `cut -f6 -d" " yum_check`

That started all right, but again the GUI crashed!? Worse, I didn't paying attention on which package! Going through the whole process again, there were no problems!? Obviously a bug, but where is a whole different story. Anyway, after removing old packages, the new ones can be reinstalled using the following command:

yum reinstall `cut -f1 -d" " yum_check`

And that's it, the package database should be good again.

One cautionary note! When you issue 'yum remove' command very carefully check what is yum going to remove. It also removes dependencies and it might remove some critical components that can make your system unusable! As a rule of thumb, in this particular case, no dependencies should be listed for removal!

DHCPNAK messages in log file

When I was checking log files I spotted the following log entries that were strange:

Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 00:40:5a:18:83:56 via eth0
Sep  7 11:32:20 srv dhcpd: DHCPACK on 1.1.1.151 to 0:4:5:1:8:5 via eth0
Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 0:4:5:1:8:5 via 1.1.1.10
Sep  7 11:32:20 srv dhcpd: DHCPACK on 1.1.1.151 to 0:4:5:1:8:5 via 1.1.1.10
Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 0:4:5:1:8:5 via 1.1.0.10: wrong network.
Sep  7 11:32:20 srv dhcpd: DHCPNAK on 1.1.1.151 to 0:4:5:1:8:5 via 1.1.0.10

The problem is that DHCP request is received three times, on two of which the answer is positive (DHCPACK) while one received negative response (DHCPNAK) and dhcpd logged the error message 'wrong network'.

The important thing is the network configuration in this specific scenario, which looks something like follows:

  +----+            +-----+              +----+
  |    |------------|     |--------------|    |
  +----+            +-----+              +----+
  Client      Firewall/DHCP relay      DHCP server
1.1.1.151    1.1.1.10     1.1.0.10       1.1.0.4

Looking into log entries, not much can be inferred. The only thing that can be seen is that third DHCPREQUEST came from 1.1.0.10 which isn't on the same network with a client requesting IP address. Sniffing the network gave a bit more information on what's happening. Analyzing the network trace the following were conclusions:

1. There are three DHCPREQUEST messages with the same transaction ID, the same destination (1.1.0.4, i.e. DHCP server) and also client IP address field within DHCP request is set to 1.1.1.151.
2. The first DHCPREQUEST comes directly from the client. It has source IP 1.1.1.151, and there is no relay field (i.e. the value is 0.0.0.0). Also, client MAC address field within DHCP request has MAC address of a given client. 
3. The second DHCP request comes from DHCP relay on the firewall. It has source set to 1.1.0.10, and relay field is properly set to 1.1.1.10, i.e. the IP address from the client's network,.
4. The third DHCP request also comes from DHCP relay on the firewall, but this time relay field is set to 1.1.0.10. This contradicts client's IP address and DHCP server rejects this request.

So, the conclusion is that client sends request to 1.1.0.4. This request is forwarded by the firewall to the server, but also intercepted by DHCP relay on the firewall that creates two proxy requests and sends them to DHCP server too, one of which is rejected.

The interesting thing, not visible in logs, is that DHCP relay upon receiving NAK from the DCHP server, generates new NAK that is broadcasted on the network where DHCP server lives. 

So, the conclusion is that firewall is wrongly configured. It should not forward DHCP requests if there is a relay agent running. Furthermore, those NAKs aren't seen by the client, only by DHCP relay that reflects them back to DHCP servers.

Adding Zimbra disclaimer using shell scripts...

While Zimbra 8 (and 7, too) have domain wide disclaimer support built in, there are two shortcomings that forced me to fall back to the old way of doing it:

1. There is no support for not adding disclaimer if it already exists, and
2. No support to exclude some addresses from adding disclaimer.

The second problem I managed to solve by patching Amavis script. That approach adds extra effort for maintainability (primarily during the upgrades), but it works. To solve the first problem the same way was too much work that I wasn't prepared to invest so I had to abandon domain wide disclaimer provided by Zimbra. There was also a third problem. Namely, for all mail messages sent from Outlook, Zimbra added two extra characters at the end of a HTML disclaimer, namely characters "= A". Why is this, I don't have slightest clue. I suspect it has something to do with encoding and decoding messages while going through the mail system, but exact reasons are unknown to me.

So, I went to solve all those problems and first I tried the old way, namely modifying postfix subsystem. It turned out that it didn't work. Just for a reference, at the end of this post, I described what I did. Next, option was modifying amavis. But that turned out to be too complicated and error prone - as I said in the introduction paragraph. Finally, I decided to put a proxy script in front of altermime that will be called by amavis and that will check if there is already disclaimer. If it isn't, then it calls altermime. Note that in this way there was no need to change amavis, and that means a lot from the maintenance perspective. So, here is what I did.

First, I created the following simple script in /opt/zimbra/altermime directory:

#!/bin/bash
echo "`date +%Y%m%d%H%M%S` $@" >> /tmp/altermime-args
exec /opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"

What it does is it just logs how it was called and then it calls altermime. Note one more important thing here. In order to be able to put this script before altermime, I had to call it altermime, and altermime binary I renamed to altermime-bin. If you are doing this on a live system be very careful how you do this switch. I suggest that you first create script called altermime.sh, check that it works, and then use the following command to make a switch:

mv altermime altermime-bin && mv altermime.sh altermime

Ok, in this way I was able to find out how altermime is actually called. This is what I saw in /tmp/altermime-args file:

20130912100915 --input=/opt/zimbra/data/amavisd/tmp/amavis-20130912T100229-30384-pc8afS_K/email-repl.txt --verbose --disclaimer=/opt/zimbra/data/altermime/global-default.txt --disclaimer-html=/opt/zimbra/data/altermime/global-default.html

That's just one line of the output. As it can be seen, the first argument specifies file with mail message, and the rest specify disclaimer to be added. So, in order not to add disclaimer, if there is already one, I modified the altermime.sh script to have the following content:

#!/bin/bash
grep "DISCLAIMER:" ${1#--input=} > /dev/null 2>&1
if [ ! "$?" = 0 ]; then
    exec /opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
fi

Again, be careful if you are modifying this script on a live system.

Now, in order to control where disclaimer is added, you can modify this simple shell script. One more thing you should be aware of, this approach impacts performance as, instead of running one process, it now runs at least 3 per mail message, and there are few extra file accesses. 

Finally, as a side note, I managed to get rid of those strange characters added to Outlook's email messages. I just edited a little bit html file that contains disclaimer, and those characters were gone. That's definitely a bug somewhere, but who knows where...

The old way that didn't work

As I said, the first approach I tried is to use the procedure from Wiki. But it didn't work. Anyway, for a reference, here is what I tried to do. Note that, as Zimbra already ships with altermime, there is no need to install it. The altermime is in /opt/zimbra/altermime/bin directory and you can safely use it. Ok, now to changes:

First, change a line in master.cf.in that reads

smtp    inet  n       -       n       -       -       smtpd

into

smtp    inet  n       -       n       -       -       smtpd        -o content_filter=dfilt:

and also add the following two lines:

dfilt   unix  -       n       n       -       -       pipe
        flags=Rq user=filter argv=/opt/zimbra/postfix/conf/disclaimer.sh -f ${sender} -- ${recipient}

Note that by this last line you specified that your script is called disclaimer.sh and that it is placed in /opt/zimbra/postfix/conf directory. This script, when run, should be run with a user privileges filter. Also, be careful where you put those lines. Namely, put them after the following three lines:

%%uncomment SERVICE:opendkim%%  -o content_filter=scan:[%%zimbraLocalBindAddress%%]:10030
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_filter=[%%zimbraLocalBindAddress%%]:10027
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_options=speed_adjust

The reason is that those line logically belong to the first smtp line, and if you add dfilt in front of it, you'll mess things, probably very badly, depending on your luck!

If you had Zimbra's domain wide disclaimer enabled, then disable it using:

zmprov mcf zimbraDomainMandatoryMailSignatureEnabled FALSE

as a zimbra user, and then restart amavis:

zmamavisdctl restart

still as a zimbra user.

Finally, to active custom script to add disclaimer run the following command as zimbra user:

zmmtactl restart

After I did all that, it didn't work. :D But, then I realized that there are two content_filter options to smtp which might not work, and so I resorted to proxying altermime.