Messing up Zimbra upgrade, a.k.a. dangerous --platform-override option...

You know that Zimbra's installer has --platform-override option that enables you to install Zimbra on unsupported platforms, like CentOS. Well, it also allows you to mess up things. Namely, I downloaded Zimbra for RHEL6, but was upgrading an existing installation on CentOS5. And I didn't noticed anything unusual until upgrade process failed! It failed very early with an error message from loader that it couldn't start perl binary. Namely, the error message I got, was similar to:

/usr/bin/perl: symbol lookup error: /opt/zimbra/zimbramon/lib/i386-linux-thread-multi/auto/IO/IO.so: undefined symbol: Perl_Tstack_sp_ptr

It wasn't exactly that one, but similar! Well, I was surprised because Zimbra's upgrade process is good and robust, and I had never problems with it, but since I had some local customizations on my installation I suspected those to be the cause. So, I ended up by copying CentOS' Perl installation into Zimbra tree. This process wasn't actually so easy to do and that should've warned me that something is terribly wrong, but I continued nevertheless. But, when I managed to pass that point, the next one finally made me realize the mistake. Namely, mysql binary required Glibc 2.7, while in CentOS 5 is some earlier version. Now, I knew what mistake I've made.

Here is also a point I realised that I'm facing a long night trying to recover Zimbra installation!

So, I've downloaded the correct version of Zimbra, the one I intended to upgrade to, manually installed it using rpm tool, and then restarted upgrade process with:

/opt/zimbra/libexec/zmsetup.pl

And here my problems continued. Namely, the upgrade script correctly figured that I'm trying to upgrade, it also correctly figured the old and the new version, but it stuck on starting mysql server. After some time I managed to figure out where the problem is, namely, existing mysql database has password while the installation process assumes there isn't one. So, it couldn't determine if mysql was started or not, and everything stalled. I lost some time trying to make database not to ask for the password, and even though I managed to do something, luckily, in the mean time I stumbled on the following Wiki page:

Recovering from wrong platform upgrade

I made it large because it really got me out of the problem. It is written for version Zimbra version 5, but it works on 7, too. In a nutshell, you install the old Zimbra versions you had and then start from there with the idea of making a rollback. BUT, BIG WARNING, this is possible only if your localconfig.xml file is intact. In my case, I erased it, but the upgrade process saved a copy which I was able to recover! I had some additional problems while testing installation as suggested byt the Wiki page, but all of them were related to the tweaks I did to my installation of Zimbra, i.e. binding Zimbra to a single IP address.

What have I learned

Well, first and foremost, do a backup before doing an upgrade. Yes, I know it was stupid, and beginner's, mistake, but as I've said, Zimbra's upgrade process is really good, at least for me. The real reason I didn't do it was that it takes several hours to make it. If I had copy, I would revert to that copy, and I would only need to fix RPM database. So, to avoid the same thing happening again, now I did make a backup process which rsync'es all of Zimbra tree to an alternative location every day.

Next thing is, localconfig.xml file is extremely important, so I did additional backup of that file, too. Note that in the file are all passwords so, it's also critical from the security perspective!

Also, I would suggest that Zimbra adds checks into installation script to guard against mistakes like the one I did. It turns out, after some googling, that there are a lot more other posts with the similar problems. This can be easily done for CentOS/RHEL with appropriate requirements in RPM packages.

And for the end, here is one other story about failed Zimbra upgrade which I find interesting.