Interesting problem with OSSEC, active response and mail delivery...

We had a problem that manifested itself in such a way that mail messages didn't come from certain domains, or more specifically from certain mail servers. Furthermore, no clue was given in the mail log to know what went wrong and to make things worse, logs from the remote mail server were inaccessible to see there what actually happened. Finally, the worse thing was that this happened sporadically. It turned out that this was consequence of a circumstances and a bug with ossec active response. This post explains what happened.

We changed DNS domain several months ago, let me call the new domain newdomain.hr, and the old one olddomain.hr. DNS was reconfigured so that it correctly handled requests for a new domain, but we had to leave old domain because of some Web server. The the old domain was changed so that when someone asked which is a mail exchanger for olddomain.hr it would receive response: mail.newdomain.hr. Finally, domain olddomain.hr was removed from the mail server. This was the first error, and now I think that it is better either to leave old domain on mail server or to not return any response! Actually, if you want to get rid of the old domain, it is the best to remove it from the mail server and that DNS server doesn't return any response for a mail exchanger of a given domain. If you know how mail works, you'll know that by changing MX record for old domain from mail.olddomain.hr to mail.newdomain.hr didn't change anything!

Anyway, that's the part concerning mail. Now, about OSSEC. It has a possibility of active response, i.e. to block offending IP addresses for a certain amount of time, 10 minutes by default. One class of offending IP addresses are those that try to deliver mail messages which require mail server to be open relay. Since mail server is properly configured it rejects those messages with a message 'Relay denied'. After mail server rejects  such delivery attempt OSSEC kicks in and blocks offending IP address for 10 minutes.

This, by itself didn't have to be a problem because blocking rules are automatically removed after 10 minutes. But, there is a bug in the removal script that manifested itself in the logs like follows (found on agent in /var/ossec/logs/active-responses.log):

Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - 203.83.62.99 1326738019.2370422 3301

For some reason removal of IP address from block list wasn't successful and that basically meant that the source mail host is blocked indefinitely!

Majority of mail servers that to generate such 'Relay denied' messages are truly spammers and if some of them were indefinitely blocked that was actually good. But, this particular source mail server that was blocked is very popular one with many users serving many different domains, so now when some other user tried to send an email that was legal and had correct address, IPtables blocked access and the mail couldn't be delivered. There was nothing in the logs of destination mail server. Also, sending user didn't receive any response message since mail was being temporary put on hold on the source server.

This particular problem was solved by completely removing the old domain. Now, source mail servers won't even try to deliver mails for the old domain and thus OSSEC won't block legitimate servers. Furthermore, the sending users will get notification immediately about non-existent mail address.