Messing up Zimbra upgrade, a.k.a. dangerous --platform-override option...

You know that Zimbra's installer has --platform-override option that enables you to install Zimbra on unsupported platforms, like CentOS. Well, it also allows you to mess up things. Namely, I downloaded Zimbra for RHEL6, but was upgrading an existing installation on CentOS5. And I didn't noticed anything unusual until upgrade process failed! It failed very early with an error message from loader that it couldn't start perl binary. Namely, the error message I got, was similar to:

/usr/bin/perl: symbol lookup error: /opt/zimbra/zimbramon/lib/i386-linux-thread-multi/auto/IO/IO.so: undefined symbol: Perl_Tstack_sp_ptr

It wasn't exactly that one, but similar! Well, I was surprised because Zimbra's upgrade process is good and robust, and I had never problems with it, but since I had some local customizations on my installation I suspected those to be the cause. So, I ended up by copying CentOS' Perl installation into Zimbra tree. This process wasn't actually so easy to do and that should've warned me that something is terribly wrong, but I continued nevertheless. But, when I managed to pass that point, the next one finally made me realize the mistake. Namely, mysql binary required Glibc 2.7, while in CentOS 5 is some earlier version. Now, I knew what mistake I've made.

Here is also a point I realised that I'm facing a long night trying to recover Zimbra installation!

So, I've downloaded the correct version of Zimbra, the one I intended to upgrade to, manually installed it using rpm tool, and then restarted upgrade process with:

/opt/zimbra/libexec/zmsetup.pl

And here my problems continued. Namely, the upgrade script correctly figured that I'm trying to upgrade, it also correctly figured the old and the new version, but it stuck on starting mysql server. After some time I managed to figure out where the problem is, namely, existing mysql database has password while the installation process assumes there isn't one. So, it couldn't determine if mysql was started or not, and everything stalled. I lost some time trying to make database not to ask for the password, and even though I managed to do something, luckily, in the mean time I stumbled on the following Wiki page:

Recovering from wrong platform upgrade

I made it large because it really got me out of the problem. It is written for version Zimbra version 5, but it works on 7, too. In a nutshell, you install the old Zimbra versions you had and then start from there with the idea of making a rollback. BUT, BIG WARNING, this is possible only if your localconfig.xml file is intact. In my case, I erased it, but the upgrade process saved a copy which I was able to recover! I had some additional problems while testing installation as suggested byt the Wiki page, but all of them were related to the tweaks I did to my installation of Zimbra, i.e. binding Zimbra to a single IP address.

What have I learned

Well, first and foremost, do a backup before doing an upgrade. Yes, I know it was stupid, and beginner's, mistake, but as I've said, Zimbra's upgrade process is really good, at least for me. The real reason I didn't do it was that it takes several hours to make it. If I had copy, I would revert to that copy, and I would only need to fix RPM database. So, to avoid the same thing happening again, now I did make a backup process which rsync'es all of Zimbra tree to an alternative location every day.

Next thing is, localconfig.xml file is extremely important, so I did additional backup of that file, too. Note that in the file are all passwords so, it's also critical from the security perspective!

Also, I would suggest that Zimbra adds checks into installation script to guard against mistakes like the one I did. It turns out, after some googling, that there are a lot more other posts with the similar problems. This can be easily done for CentOS/RHEL with appropriate requirements in RPM packages.

And for the end, here is one other story about failed Zimbra upgrade which I find interesting.

GnuCash - Troškovi na kreditnoj kartici

Vođenje troškova koje se ostvaruju putem kreditne kartice se ispostavilo kao najzahtjevnije. Postoji više načina na koji se to može obaviti, ali ja sam se u konačnici odlučio za detaljniji način, tj. onaj koji omogućava detaljnije vođenje troškova. Komplikaciju predstavlja činjenica kako nije moguće voditi troškove u različitim valutama unutar jedno računa (odnosno stavke kako sam ponekad pisao), bar ja ne znam kako bi to učinio. U tom smislu, odlučio sam se za zasebno vođenje evidencije kunskih te troškova u eurima.

Stvaranje računa

Dakle, kao prvi korak je kreirati račune, odnosno stavke, za kreditne kartice. To je relativno jednostavno, treba kliknuti na desnu tipku miša negdje unutar prozora s računima/stavkama te odabrati opciju New Account... i u prozoru koji se pojavi ispuniti polja na način prikazan sljedećom slikom:

Pod naziv je upisano Kreditne kartice, označena je opcija Placeholder što znači da se radi samo o računu koji sadrži druge račune ali neće u njemu moći biti upisivane stavke. Pod tip računa (Account Type) označeno je Credit Card i konačno, pod Parent Account, je označeno da se radi o novom vršnom računu. Sada je potrebno kreirati pojedine račune za Kunske troškove te troškove u Eurima. Opet je potrebno kliknuti na desnu tipku miša i odabrati opciju New Account... te potom ispuniti podatke kako je to prikazano na sljedećoj slici:

Ovim se stvara račun za MasterCard koji dolazi na naplatu u kunama. Primjetite kako je odabran tip računa Credit Card te je postavljen unutar stavke Kreditne kartice. Slično je i za Eure:

S tim da je u ovom slučaju odabrana valuta EUR (Euro).

Konačno stanje je prikazano sljedećom slikom:

Upisivanje troškova

Pretpostavimo kako smo s kreditnom karticom platili gorivo koje nas je koštalo 400Kn te je kupljeno na dan 29.11.2013. Dodatno, dana 4.12.2013. bili smo u nabavci te smo kupili hrane za 200kn. Te dvije stavke bi upisali pod račun MasterCard - Kunski troškovi na sljedeći način:

Primjetite kako je u koloni Transfer upisano na što se odnosi trošak. Dakle, kada sam kupio gorivo i platio ga kreditnom karticom, stavio sam da se radi o trošku za gorivo. Takav način upisivanja vezan je uz moju odluku kako želim voditi detaljne troškove kreditne kartice, tj. na ŠTO su novci potrošeni, a ne samo koliko je potrošeno. Kao zadnju stvar, primjetite da se trošak upisuje u kolonu Charge.

Upisivanje deviznih troškova je gotovo identično, jedino što se troškovi ne upisuju u kunama već u Eurima.

Naplata troškova

Kada vam banka konačno obračuna troškove i izvrši plaćanje onda je potrebno provesti postupak "rekonsilijacije" (ili kako god se to već zove u bankarskim krugovima). Naime, novci se povlače s vašeg tekućeg računa (ili ih uplaćujete u banci) te je potrebno obaviti tu transakciju. To se obavlja na sljedeći način

Prvo, kliknite desnom tipkom na račun kreditne kartice za koju je došla naplata. Pretpostavimo da je to kunski MasterCard. Na desni klik, otvorit će se izbornik unutar kojega je potrebno odabrati opciju Reconcile:

Kada odaberete tu opciju otvara se novi prozor. Unutar njega potrebno je odrediti datum kada se vrši naplata (ili obračun troškova) te suma koja je do trenutka obračuna učinjena:

U mom slučaju, odabrao sam datum 20.12.2013 jer na taj dan je stigla naplata te mi je sam GnuCash odredio da sam do tog trenutka napravio trošak od 600Kn. Ovom prilikom treba reći kako postoji razlika između datuma kada se vrši obračun (recimo, meni je to 10. u mjesecu) i datuma kada se vrši plaćanje (to je 20. u mjesecu). Ako ste upisali pod "Statement Date" datum plaćanja, a između datuma plaćanja i formiranja obavijesti je bilo troškova, oni će greškom također biti uključeni u naplatu. Dakle, pripazite na to. Možda je ipak najbolje upisivati datum formiranja troška, a ne kako sam ja napravio datum naplate.

Nakon što potvrdite informacije o datumu obračuna te konačne sume pritiskom na OK, otvara se novi prozor unutar kojega trebate odabreti koje sve stavke dolaze na plaćanje:

Ono što trebate učiniti je u desnom dijelu odabrati stavke koje se naplaćuju. Primjetite da svaka stavka, na kraju, ima kolonu R. To su "checkboxovi" koje treba selektirati. To morate napraviti tako da na kraju pokrijete sav trošak koji dolazi na naplatu, dakle svih 600kn. Razlika, koliko je još preostalo, navedena je u donjem desnom kutu, u retku "Difference". U našem slučaju, treba odabrati sve stavke, međutim, ako je nakon formiranja obavijesti bilo novih troškova, onda naravno nećete te nove troškove odabirati. Kada ste napravili odabri, stanje je sljedeće:

Primjetite dvije stvari. Prvo, razlika je 0 jer smo pokrili svu sumu koja dolazi na naplatu, i drugo, zelena tipka u traci s alatima je sada omogućena. Naime, tu zelenu tipku treba odabrati kako bi se nastavilo u sljedeći, i zadnji, korak:

U zadnjem koraku treba odabrati odakle se pokrivaju troškovi koji su načinjeni na kreditnoj kartici. U mom slučaju, troškovi se pokrivaju s tekućeg računa pa sam ja odabrao u polju "Transfer From" tekući račun, dok sam u polju "Transfer To" odabrao kunski kreditni račun. Klikom na OK, obavlja se transakcija, i cijeli postupak je završen. Sada situacija izgleda ovako:

Drugim riječima, na kunskom MasterCardu nemam nepodmirenih troškova, dok mi je tekući račun "tanji" za 600kn.

Modifying mail passing through Zimbra

I had a request to attach to (almost) each mail message that passes through the Zimbra mail server an image. Basically, what the owner wanted is that there is an image with advertisement in the mail that is sent by internal users. Additional requirements were:

1. Image should be added only once!
2. In case there is new image, and there is already on in the mail, the old one should be replaced!
3. Image has to be at exact spot within a message.

There were some additional requirements from my POV:

1. Not every mail should have image attached, e.g. automatically generated internal messages!
2. I should be careful about impact on the performance.
3. Mali messages that are not modified should not have any noticeable marks about image that isn't added (this one will be more clear later).
4. The solution should allow to define only certain senders to have mails modified.
5. It has to have a DRY RUN mode so that it is easily disabled.

I immediately knew that there is no way to place image somewhere on the Internet and put link into the mail message. Although the most elegant solution, the problem is that all mail clients don't show images by default, and that's it. So, image has to be within mail message itself. A bit of research showed up a potential solution. Namely, to embed image data within IMG tag itself, and mail messages are already altered by adding disclaimer which is HTML and a perfect place to add that IMG tag, so why not reuse disclaimer for the same purpose? In favor of that solution was also my intention not to add new scripts into the mail processing chain because of fear that it might impact performance. 

Unfortunately, that solution didn't work. The most important shortcoming is that Outlook and GMail don't handle IMG tag with embedded image data in it, in other words, the image isn't shown. To solve that, image has to be embedded as a MIME part within mail. Additionally, the other requirements aren't easy to achieve, especially, the one to replace old image with a new one. So, in the end I had to resort to writing scripts.

I already wrote about how I managed to solve more complex requirements for a disclaimer than the functionality of Zimbra allows. So, it was natural place for me to add that additional processing there. I called the script to add image altermail.py and now altermime script has the following form:

#!/bin/bash
grep "DISCLAIMER:" ${1#--input=} > /dev/null 2>&1
if [ ! "$?" = 0 ]; then
/opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
fi
#echo "`date +%Y%m%d%H%M%S` $@" >> /tmp/altermime-args
/opt/zimbra/altermime-0.3.10/bin/altermail.py "$@" >> /tmp/altermail.log 2>&1

I'm calling altermial.py after altermime because the image placeholder is within disclaimer! Also, I removed exec keyword before altermime-bin call so that altermail.py is finished.

Additionally, note that altermail.py accepts the same arguments as altermime! This is in order to simplify things a bit.

Obviously, I choose Python as a programming language of my choice. I could write all that in Perl too, but since I'm lately working a lot more with Python, Python was the way to go. Both languages have very good support for mail processing (MIME messages in particular).

The script is on the GitHub and you can fetch it there.

How the script works

First of, script doesn't work for mail messages that aren't MIME. So, after loading a message the first check if it is a multipart message. If not, then script just exits.

Next, white and black lists are checked.

There are two passess over the mail message. In the first pass, it searches through the mail message to see if there is already image attached. If so, then it additionally checks if it is an older version of the image. If it is, it replaces the image, but in both cases it doesn't do anything more and finishes execution.

The second pass is done when there is no image in the mail message and it has to be added. Now, when adding the image it has to be added with HTML as html/related so that they are both shown. If you add it as html/alternative, then only one of them will be shown!

All the configuration options are embedded within script itself. I chose not to have configuration file to reduce number of disk accesses, which is already very high (a lot of modules are necessary).