Modifying mail passing through Zimbra

I had a request to attach to (almost) each mail message that passes through the Zimbra mail server an image. Basically, what the owner wanted is that there is an image with advertisement in the mail that is sent by internal users. Additional requirements were:

1. Image should be added only once!
2. In case there is new image, and there is already on in the mail, the old one should be replaced!
3. Image has to be at exact spot within a message.

There were some additional requirements from my POV:

1. Not every mail should have image attached, e.g. automatically generated internal messages!
2. I should be careful about impact on the performance.
3. Mali messages that are not modified should not have any noticeable marks about image that isn't added (this one will be more clear later).
4. The solution should allow to define only certain senders to have mails modified.
5. It has to have a DRY RUN mode so that it is easily disabled.

I immediately knew that there is no way to place image somewhere on the Internet and put link into the mail message. Although the most elegant solution, the problem is that all mail clients don't show images by default, and that's it. So, image has to be within mail message itself. A bit of research showed up a potential solution. Namely, to embed image data within IMG tag itself, and mail messages are already altered by adding disclaimer which is HTML and a perfect place to add that IMG tag, so why not reuse disclaimer for the same purpose? In favor of that solution was also my intention not to add new scripts into the mail processing chain because of fear that it might impact performance. 

Unfortunately, that solution didn't work. The most important shortcoming is that Outlook and GMail don't handle IMG tag with embedded image data in it, in other words, the image isn't shown. To solve that, image has to be embedded as a MIME part within mail. Additionally, the other requirements aren't easy to achieve, especially, the one to replace old image with a new one. So, in the end I had to resort to writing scripts.

I already wrote about how I managed to solve more complex requirements for a disclaimer than the functionality of Zimbra allows. So, it was natural place for me to add that additional processing there. I called the script to add image altermail.py and now altermime script has the following form:

#!/bin/bash
grep "DISCLAIMER:" ${1#--input=} > /dev/null 2>&1
if [ ! "$?" = 0 ]; then
/opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
fi
#echo "`date +%Y%m%d%H%M%S` $@" >> /tmp/altermime-args
/opt/zimbra/altermime-0.3.10/bin/altermail.py "$@" >> /tmp/altermail.log 2>&1

I'm calling altermial.py after altermime because the image placeholder is within disclaimer! Also, I removed exec keyword before altermime-bin call so that altermail.py is finished.

Additionally, note that altermail.py accepts the same arguments as altermime! This is in order to simplify things a bit.

Obviously, I choose Python as a programming language of my choice. I could write all that in Perl too, but since I'm lately working a lot more with Python, Python was the way to go. Both languages have very good support for mail processing (MIME messages in particular).

The script is on the GitHub and you can fetch it there.

How the script works

First of, script doesn't work for mail messages that aren't MIME. So, after loading a message the first check if it is a multipart message. If not, then script just exits.

Next, white and black lists are checked.

There are two passess over the mail message. In the first pass, it searches through the mail message to see if there is already image attached. If so, then it additionally checks if it is an older version of the image. If it is, it replaces the image, but in both cases it doesn't do anything more and finishes execution.

The second pass is done when there is no image in the mail message and it has to be added. Now, when adding the image it has to be added with HTML as html/related so that they are both shown. If you add it as html/alternative, then only one of them will be shown!

All the configuration options are embedded within script itself. I chose not to have configuration file to reduce number of disk accesses, which is already very high (a lot of modules are necessary).

Adding Zimbra disclaimer using shell scripts...

While Zimbra 8 (and 7, too) have domain wide disclaimer support built in, there are two shortcomings that forced me to fall back to the old way of doing it:

1. There is no support for not adding disclaimer if it already exists, and
2. No support to exclude some addresses from adding disclaimer.

The second problem I managed to solve by patching Amavis script. That approach adds extra effort for maintainability (primarily during the upgrades), but it works. To solve the first problem the same way was too much work that I wasn't prepared to invest so I had to abandon domain wide disclaimer provided by Zimbra. There was also a third problem. Namely, for all mail messages sent from Outlook, Zimbra added two extra characters at the end of a HTML disclaimer, namely characters "= A". Why is this, I don't have slightest clue. I suspect it has something to do with encoding and decoding messages while going through the mail system, but exact reasons are unknown to me.

So, I went to solve all those problems and first I tried the old way, namely modifying postfix subsystem. It turned out that it didn't work. Just for a reference, at the end of this post, I described what I did. Next, option was modifying amavis. But that turned out to be too complicated and error prone - as I said in the introduction paragraph. Finally, I decided to put a proxy script in front of altermime that will be called by amavis and that will check if there is already disclaimer. If it isn't, then it calls altermime. Note that in this way there was no need to change amavis, and that means a lot from the maintenance perspective. So, here is what I did.

First, I created the following simple script in /opt/zimbra/altermime directory:

#!/bin/bash
echo "`date +%Y%m%d%H%M%S` $@" >> /tmp/altermime-args
exec /opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"

What it does is it just logs how it was called and then it calls altermime. Note one more important thing here. In order to be able to put this script before altermime, I had to call it altermime, and altermime binary I renamed to altermime-bin. If you are doing this on a live system be very careful how you do this switch. I suggest that you first create script called altermime.sh, check that it works, and then use the following command to make a switch:

mv altermime altermime-bin && mv altermime.sh altermime

Ok, in this way I was able to find out how altermime is actually called. This is what I saw in /tmp/altermime-args file:

20130912100915 --input=/opt/zimbra/data/amavisd/tmp/amavis-20130912T100229-30384-pc8afS_K/email-repl.txt --verbose --disclaimer=/opt/zimbra/data/altermime/global-default.txt --disclaimer-html=/opt/zimbra/data/altermime/global-default.html

That's just one line of the output. As it can be seen, the first argument specifies file with mail message, and the rest specify disclaimer to be added. So, in order not to add disclaimer, if there is already one, I modified the altermime.sh script to have the following content:

#!/bin/bash
grep "DISCLAIMER:" ${1#--input=} > /dev/null 2>&1
if [ ! "$?" = 0 ]; then
    exec /opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
fi

Again, be careful if you are modifying this script on a live system.

Now, in order to control where disclaimer is added, you can modify this simple shell script. One more thing you should be aware of, this approach impacts performance as, instead of running one process, it now runs at least 3 per mail message, and there are few extra file accesses. 

Finally, as a side note, I managed to get rid of those strange characters added to Outlook's email messages. I just edited a little bit html file that contains disclaimer, and those characters were gone. That's definitely a bug somewhere, but who knows where...

The old way that didn't work

As I said, the first approach I tried is to use the procedure from Wiki. But it didn't work. Anyway, for a reference, here is what I tried to do. Note that, as Zimbra already ships with altermime, there is no need to install it. The altermime is in /opt/zimbra/altermime/bin directory and you can safely use it. Ok, now to changes:

First, change a line in master.cf.in that reads

smtp    inet  n       -       n       -       -       smtpd

into

smtp    inet  n       -       n       -       -       smtpd        -o content_filter=dfilt:

and also add the following two lines:

dfilt   unix  -       n       n       -       -       pipe
        flags=Rq user=filter argv=/opt/zimbra/postfix/conf/disclaimer.sh -f ${sender} -- ${recipient}

Note that by this last line you specified that your script is called disclaimer.sh and that it is placed in /opt/zimbra/postfix/conf directory. This script, when run, should be run with a user privileges filter. Also, be careful where you put those lines. Namely, put them after the following three lines:

%%uncomment SERVICE:opendkim%%  -o content_filter=scan:[%%zimbraLocalBindAddress%%]:10030
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_filter=[%%zimbraLocalBindAddress%%]:10027
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_options=speed_adjust

The reason is that those line logically belong to the first smtp line, and if you add dfilt in front of it, you'll mess things, probably very badly, depending on your luck!

If you had Zimbra's domain wide disclaimer enabled, then disable it using:

zmprov mcf zimbraDomainMandatoryMailSignatureEnabled FALSE

as a zimbra user, and then restart amavis:

zmamavisdctl restart

still as a zimbra user.

Finally, to active custom script to add disclaimer run the following command as zimbra user:

zmmtactl restart

After I did all that, it didn't work. :D But, then I realized that there are two content_filter options to smtp which might not work, and so I resorted to proxying altermime.

Zimbra 7 and domain wide disclaimer...

In Zimbra 7 it is not necessary any more to hack scripts in order to get domain wide disclaimer. At least not if you want to add the same disclaimer to every mail that passes through your mail server. In case you want more control, like not attaching it to some mails, then you'll have to resort again to scripts and hacking.

So, to add disclaimer, the following would do:

zmprov mcf zimbraDomainMandatoryMailSignatureText <text disclaimer>
zmprov mcf zimbraDomainMandatoryMailSignatureHTML <html disclaimer>
zmprov mcf zimbraDomainMandatoryMailSignatureEnabled TRUE

and finally, reload amavis configuration using

zmamavisdctl reload

Note that whenever you change any of the previous three attributes you have to restart amavis. Those attributes are accessed only when amavis is started and they are used as an input to configuration writing routines.

You can query current values using gcf instead of mcf (don't forget to leave out last arguments because you are not setting values!).

Well, now, there were some problems.

The first one is that you disclaimer has probably more that one line, anyway more that you are prepared to type. In that case you can put it into a file and then use backtick shell notation with cat command to place the content of a file into appropriate place. Don't forget to place double quotes around everything so that spaces are not interpreted by the shell (like this "`cat somefile`")! There are other methods too, but this hack worked for me. Well, almost, becuse when text version of a disclaimer was attached to the text only mail, everything after the first line was cut off. HTML on the other hand worked, but, it also suffered from the second problem. And the second problem is...

The second problem is that I have Croatian in disclaimer which means it's encoded using UTF-8, and that, for some reason, doesn't play well with the LDAP, or some command inbetween, that transfers text/html disclaimer into LDAP. But I noticed that other tools, e.g. vi/vim, even bash, behaved strangely upon encountering some Croatian character. Trying to set environment variable LANG to en_US.utf8 before executing zmprov command didn't work either! And the command locale showed that C was still used. I managed to fix this by commenting out the following two lines in ~zimbra/.bash_profile file:

#export LANG=C
#export LC_ALL=C

After that (don't forget to login/logout) bash and vi started to work as expected but Zimbra still was messing up the disclaimer. That is, until I restarted amavis again. Then, it worked. In case it still doesn't work, set HTML attribute using zmprov command and restart amavis again!

This locale setting is nothing new in Zimbra. I checked previous version of Zimbra and it was also there. But until now I never had to put UTF8 into LDAP and so I never had hit this particular error.

This has left me with the last problem, text attachment isn't working as it should. So, I opened amavis configuration file /opt/zimbra/conf/amavisd.conf to see what's exactly going on. In there, I found that the file with the disclaimer should be in /opt/zimbra/data/altermime directory and it should be named _OPTION_.txt. Well, it is there but not called _OPTION_.txt. It could be that some postprocessing is performed on the values in amavis configuration file, but I ignored that as it is not important. What is important is that the file with disclaimer is called global-default.txt and it had wrong content, i.e. only the first line of the disclaimer. For a quick test I replaced it with the disclaimer I wanted to have, and voila, it worked!!!

So, why is there an error? I'll investigate later since now I have to do other things. :)