Tuesday, January 14, 2014

Modifying mail passing through Zimbra

I had a request to attach to (almost) each mail message that passes through the Zimbra mail server an image. Basically, what the owner wanted is that there is an image with advertisement in the mail that is sent by internal users. Additional requirements were:
  1. Image should be added only once!
  2. In case there is new image, and there is already on in the mail, the old one should be replaced!
  3. Image has to be at exact spot within a message.
There were some additional requirements from my POV:
  1. Not every mail should have image attached, e.g. automatically generated internal messages!
  2. I should be careful about impact on the performance.
  3. Mali messages that are not modified should not have any noticeable marks about image that isn't added (this one will be more clear later).
  4. The solution should allow to define only certain senders to have mails modified.
  5. It has to have a DRY RUN mode so that it is easily disabled.
I immediately knew that there is no way to place image somewhere on the Internet and put link into the mail message. Although the most elegant solution, the problem is that all mail clients don't show images by default, and that's it. So, image has to be within mail message itself. A bit of research showed up a potential solution. Namely, to embed image data within IMG tag itself, and mail messages are already altered by adding disclaimer which is HTML and a perfect place to add that IMG tag, so why not reuse disclaimer for the same purpose? In favor of that solution was also my intention not to add new scripts into the mail processing chain because of fear that it might impact performance. 

Unfortunately, that solution didn't work. The most important shortcoming is that Outlook and GMail don't handle IMG tag with embedded image data in it, in other words, the image isn't shown. To solve that, image has to be embedded as a MIME part within mail. Additionally, the other requirements aren't easy to achieve, especially, the one to replace old image with a new one. So, in the end I had to resort to writing scripts.

I already wrote about how I managed to solve more complex requirements for a disclaimer than the functionality of Zimbra allows. So, it was natural place for me to add that additional processing there. I called the script to add image altermail.py and now altermime script has the following form:
#!/bin/bash
grep "DISCLAIMER:" ${1#--input=} > /dev/null 2>&1
if [ ! "$?" = 0 ]; then
/opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
fi
#echo "`date +%Y%m%d%H%M%S` $@" >> /tmp/altermime-args
/opt/zimbra/altermime-0.3.10/bin/altermail.py "$@" >> /tmp/altermail.log 2>&1
I'm calling altermial.py after altermime because the image placeholder is within disclaimer! Also, I removed exec keyword before altermime-bin call so that altermail.py is finished.

Additionally, note that altermail.py accepts the same arguments as altermime! This is in order to simplify things a bit.

Obviously, I choose Python as a programming language of my choice. I could write all that in Perl too, but since I'm lately working a lot more with Python, Python was the way to go. Both languages have very good support for mail processing (MIME messages in particular).

The script is on the GitHub and you can fetch it there.

How the script works

First of, script doesn't work for mail messages that aren't MIME. So, after loading a message the first check if it is a multipart message. If not, then script just exits.

Next, white and black lists are checked.

There are two passess over the mail message. In the first pass, it searches through the mail message to see if there is already image attached. If so, then it additionally checks if it is an older version of the image. If it is, it replaces the image, but in both cases it doesn't do anything more and finishes execution.

The second pass is done when there is no image in the mail message and it has to be added. Now, when adding the image it has to be added with HTML as html/related so that they are both shown. If you add it as html/alternative, then only one of them will be shown!

All the configuration options are embedded within script itself. I chose not to have configuration file to reduce number of disk accesses, which is already very high (a lot of modules are necessary).


Sunday, December 29, 2013

Gnome Shell's calendar and Thunderbird

After I installed Fedora 20, I noticed that the calendar in Gnome Shell's clock doesn't work, i.e. it doesn't show scheduled entries. Actually, this didn't work in the older version of Fedora, neither. But now, I decided to make it work. Before describing what I've tried, and what I did, I have to describe my setup. First of, I'm using Thunderbird. Evolution proved too unstable for me so I ditched it. Next, all my calendars are stored on Google, so, no local calendar in Thunderbird. The reason I'm using Google calendars is to be able to sync with mobile phone. The reason for local client, instead of Web client, is pure habit and comotion, I like more desktop clients than the Web based. Finally, I don't mind having some additional software being installed no matter if I use it directly or not. So, usually, I have Evolution installed alongside Thunderbird.

Gnome Shell's calendar allows integration only with Evolution, Thunderbird isn't allowed. This is actually expected, as Thunderbird is primarily mail application. But using Lightning plugin, it is a very good calendar solution, too. So, no easy way to define Thunderbird as default calendar application. After some quick googling, I found the following plugin for Thunderbird. Basically, it synces Thunderbird's calendar with Evolution's, one way, and that's it. When I wanted to install it, I had a small, and unrelated, problem. Namely, I could not find how to install it in Thunderburd?! There was no option in Thunderbird for managing Addons!? After some short Googling I finally realized that the menu option on the upper right hand side (icon with three parallel lines) isn't actually the full menu! I managed to open full Tools menu item by pressing Alt+F.

After I installed this plugin, and restarting Gnome and then Thunderbird, it didn't work So, the next I tried this. But first, I checked what was the current setting:
$ gsettings get org.gnome.desktop.default-applications.office.calendar exec
'evolution -c calendar'
Ok, now I changed the value using the following command:
$ gsettings set org.gnome.desktop.default-applications.office.calendar exec thunderbird
That didn't work either. It occured to me that the problem might be that I'm using Thunderbird profiles, so I also tried to define a profile in the command:
gsettings set org.gnome.desktop.default-applications.office.calendar exec 'thunderbird -P MyProfile'
Still, no luck! Then I went back to the plugin page and saw there that I have to create initial Evolution profile. I tried that too, but again, no luck. Maybe the problem was that I'm not using local calendars but remote ones. But then I realized that there is very easy solution. Namely, I created Evolution profile that is connected to Google calendars and synces with them. Google calendars are, in turn, connected to Thunderbird and everything works!

Yet, I didn't managed to get Thunderbird used when I click on Open Calendar option in the Gnome Shell's clock. Evolution is always used. Note that I tried two things, and both didn't help. First, I tried to define Thunderbird as default Calendar application using gsettings command as described above. Next, I tried to define Thunderbird default Calendar application in menu that is accessed by selecting All setting application, Details button, and there Default applications menu option. Note that Thunderbird isn't shown as a possible Calendar application. That is because its MIME type doesn't specify it as such. To change that I used procedure described here. In short, open file /usr/share/applications/mozilla-thunderbird.desktop in text editor and modify line MimeType so that it is
MimeType=message/rfc822;x-scheme-handler/mailto;text/calendar;text/x-vcard;:
Now, close text editor and update database file:
update-desktop-database -q
Finally, go to Details button and you'll see Thunderbird is now offered as a Calendar application, too.

In the end, I lost few hours investigating this and trying different solutions. Hopefully, someone will find this useful and will have to waste a lot less time.

Systemd waiting for external crypted disks...

I have an external, encrypted, disk that I periodically connect to the laptop. In order to have some easy to remember name, instead of UUID, I placed an entry in /etc/crypttab:
EXTDISK UUID=a809c218-f828-4149-bd9e-1c352a5f94df none
That way, when I connect disk, it will be automatically named EXTDISK and it will have an entry in /dev/mapper directory. Eventually, it will be mounted under /run/media/<userid>/EXTDISK. Note that there are only three fields in the line, each field separated from the other using space. The last field is passphrase placeholder, but I didn't want to have it written on the disk, so I used keyword none to signal that I want to type it each time the disk is opened.

The problem with this setup is that during  boot procedure, systemd waits for this disk to appear and, since there is no disk, it has to timeout. In system logs there will be messages like the following ones:
Dec 29 13:32:15 w530 systemd: Dependency failed for Cryptography Setup for EXTDISK.
Dec 29 13:32:15 w530 systemd: Dependency failed for dev-mapper-EXTDISK.device.
Dec 29 13:37:27 w530 systemd: Expecting device dev-mapper-EXTDISK.device...
The solution is simple. In newer versions of /etc/crypttab there is nofail option that should be added as a part of the fourth field. Note that, if there are multiple options in the fourth field, they all should be separated using commas and no spaces are allowed there. This option isn't listed in the manual page I linked in the introductory section of the post, so check your local manual page about crypttab.

As a side note, while searching for the solution of this timeout problem, I needed at one point to know which physical devices are beneath luks devices, i.e. my /dev/mapper directory looks like this:
# ls -l /dev/mapper/
total 0
crw-------. 1 root root 10, 236 Dec 29 13:27 control
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 fedora-home -> ../dm-3
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 fedora-root -> ../dm-2
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 fedora-swap -> ../dm-1
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 luks-a2c17ceb-222e-4cd2-3330-24a0a1111b43 -> ../dm-0
lrwxrwxrwx. 1 root root       7 Dec 29 13:27 luks-c7e8d2f7-1114-45c0-333b-fb8444222884 -> ../dm-4
What I was curious about are those two luks symlinks for which I didn't know their physical devices. It turned out its easy to find out, using cryptsetup tool:
# cryptsetup status luks-a2c17ceb-222e-4cd2-3330-24a0a1111b43
/dev/mapper/luks-a2c17ceb-222e-4cd2-3330-24a0a1111b43  is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  device:  /dev/sda2
  offset:  4096 sectors
  size:    999184384 sectors
  mode:    read/write
So, there is an answer, /dev/sda2.

About Me

scientist, consultant, security specialist, networking guy, system administrator, philosopher ;)

Blog Archive