Showing posts with label centos. Show all posts
Showing posts with label centos. Show all posts

Saturday, April 11, 2015

CentOS 7, Zimbra 8.6 and FirewallD

I just installed Zimbra 8.6 on a fresh CentOS 7. It seems that CentOS 7 uses FirewallD service by default instead of the old iptables and iptables6 scripts in /etc/init.d directory. Nevertheless I don't like when I see that someone recommends some critical security services/protections to be just turned off. Those services are there for a reason, and turning them off sounds to me like the old bad recommendation of chmod'ing everything to 777 when something didn't work. Anyway, I didn't turn off SELinux and Zimbra works as expected. What I needed is to configure FirewallD to allow access to mail services from the Internet. Turns out it isn't so hard as everything is already provided. Basically, the following services have to be enabled in your zone:
  • dns
  • https
  • imaps
  • smtp
To permanently enable each of the aforementioned services, use the following command:
firewall-cmd --permanent --add-service <service>
Note that the given command doesn't activate access to the service until you restart FirewallD. Anyway, that's it.

As a final note, I didn't allow access to admin port 7171. The reason is that I'm not so comfortable with allowing Internet wide access to admin console. To access admin console, I'm going to use ssh tunneling. Basically, I'll forward local port 7171, over ssh, to port 7171 on loopback interface of mail server. In case you are unlike me, and don't have problems with allowing access to that port, use the following command:
firewall-cmd --permanent --add-port=7171/tcp
Again, don't forget to restart FirewallD after issuing the given command.

Thursday, March 13, 2014

Installing Snort 2.9.6.0 on CentOS 6.5 64-bit

Some time ago I wrote a post about installing Snort 2.9.1 on CentOS 6. In the mean time I decided it's time to upgrade so the idea of this post is to document what changed with respect to that older post. In short, binary packages for CentOS 6 are now provided on the Snort's download page. So, you only need to download them and install (or install using URL). Yet, there is a problem with a libdnet dependency (I don't know which one was used during compilation, but it certainly wasn't the one in EPEL).

Compiling and installing

In case you want to rebuild them, the process is now almost without any problems. In the following text I'll assume that you started with a minimal CentOS installation with the following packages installed (and their dependencies, of course): gcc, make, bison, flex, autoconf, automake, rpmbuild.

First, download daq source rpm file. Before rebuilding it, you should install pcap-devel. This is actually something rpmbuild tool will warn you that you have to install. When you installed it, rebuild daq:
rpmbuild --rebuild daq
then, install it:
yum localinstall ~/rpmbuild/RPMS/x86_64/daq-2.0.2-1.x86_64.rpm
Next, for snort you'll need libdnet library which is in EPEL. So, first install EPEL:
yum install http://mirrors.neterra.net/epel/6/i386/epel-release-6-8.noarch.rpm
Then, install necessary packages:
yum install libdnet-devel zlib-devel
Those two aren't listed as dependencies in Snort's SRPM file, so you'll get some cryptic error message. Now, download Snort's srpm file and rebuild it using:
rpmbuild --rebuild snort-2.9.6.0-1.src.rpm
Now, install it using:
yum localinstall ~/rpmbuild/RPMS/x86_64/snort-2.9.6.0-1.x86_64.rpm
That's all there is for installation.

Configuring and running

I'll assume that you are installing a fresh instance, i.e. no previous configuration. In case there is previous installation be careful not to overwrite existing configuration. To configure snort you'll have to download snortrules archive. Then, unpack it:
mkdir ~/snort
tar xzf snortrules-snapshot-2960.tar.gz -C ~/snort
chown root.root ~/snort
Next you have to move files in their place. First, move basic configuration file:
mv -f snort/etc/* /etc/snort/
Note that I'm using force option of move command to overwrite existing files. Next, move rules to their place:
mv -i snort/rules snort/preproc_rules snort/so_rules /etc/snort/
Now, if you are using SELinux you should change context of the files you moved to /etc/snort directory. Do it using the following commands:
chcon -R system_u:object_r:snort_etc_t:s0 /etc/snort
chcon -R system_u:object_r:lib_t:s0 /etc/snort/so_rules/precompiled/RHEL-6-0/
You should now modify configuration file. Here is a diff of the changes I made:
--- snort.conf.orig 2014-03-13 11:25:53.889609831 +0100
+++ snort.conf 2014-03-13 11:37:32.419292894 +0100
@@ -42,16 +42,16 @@
 ###################################################

 # Setup the network addresses you are protecting
-ipvar HOME_NET any
+ipvar HOME_NET 192.168.1.0/24

 # Set up the external network addresses. Leave as "any" in most situations
 ipvar EXTERNAL_NET any

 # List of DNS servers on your network
-ipvar DNS_SERVERS $HOME_NET
+ipvar DNS_SERVERS 192.168.1.8,192.168.1.9

 # List of SMTP servers on your network
-ipvar SMTP_SERVERS $HOME_NET
+ipvar SMTP_SERVERS 192.168.1.20

 # List of web servers on your network
 ipvar HTTP_SERVERS $HOME_NET
@@ -101,13 +101,13 @@
 # Path to your rules files (this can be a relative path)
 # Note for Windows users:  You are advised to make this an absolute path,
 # such as:  c:\snort\rules
-var RULE_PATH ../rules
-var SO_RULE_PATH ../so_rules
-var PREPROC_RULE_PATH ../preproc_rules
+var RULE_PATH rules
+var SO_RULE_PATH so_rules
+var PREPROC_RULE_PATH preproc_rules

 # If you are using reputation preprocessor set these
-var WHITE_LIST_PATH ../rules
-var BLACK_LIST_PATH ../rules
+var WHITE_LIST_PATH rules
+var BLACK_LIST_PATH rules

 ###################################################
 # Step #2: Configure the decoder.  For more information, see README.decode
@@ -240,13 +240,13 @@
 ###################################################

 # path to dynamic preprocessor libraries
-dynamicpreprocessor directory /usr/local/lib/snort_dynamicpreprocessor/
+dynamicpreprocessor directory /usr/lib64/snort-2.9.6.0_dynamicpreprocessor/

 # path to base preprocessor engine
-dynamicengine /usr/local/lib/snort_dynamicengine/libsf_engine.so
+dynamicengine /usr/lib64/snort-2.9.6.0_dynamicengine/libsf_engine.so.0

 # path to dynamic rules libraries
-dynamicdetection directory /usr/local/lib/snort_dynamicrules
+dynamicdetection directory /etc/snort/so_rules/precompiled/RHEL-6-0/x86-64/2.9.6.0/

 ###################################################
 # Step #5: Configure preprocessors
And you can download the complete snort.conf file that worked for me. Be careful, you need to change IP addresses in the configuration file to match your environment.

Finally, create two empty files, /etc/snort/rules/white_list.rules and /etc/snort/rules/black_list.rules.

Now, you should be able to start Snort, i.e.
# /etc/init.d/snortd start
Starting snort: Spawning daemon child...
My daemon child 1904 lives...
Daemon parent exiting (0)                         [  OK  ]

Tuesday, March 11, 2014

Compiling OVALDI 5.10.1.6 on CentOS 6.5

Some time ago I wrote about compiling Ovaldi on CentOS 6. Now, I tried to compile it again, and I found out that some things changed. Most importantly, there is no need to compile old Xalan/Xerces libraries any more. But, there are still problems with RPM. To make the story short, I managed to compile it and create RPM. Here are the files:
  • patch you need to be able to compile ovaldi
  • SRPM file you can use to recompile ovaldi; it contains patch
  • RPM file if you don't want to compile it yourself (and you trust me ;))
Note that I didn't do any testing at all! So, it might happen that rpm based stuff doesn't work. If that's the case leave a comment and I'll take a look when I find time.

Tuesday, January 14, 2014

Modifying mail passing through Zimbra

I had a request to attach to (almost) each mail message that passes through the Zimbra mail server an image. Basically, what the owner wanted is that there is an image with advertisement in the mail that is sent by internal users. Additional requirements were:
  1. Image should be added only once!
  2. In case there is new image, and there is already on in the mail, the old one should be replaced!
  3. Image has to be at exact spot within a message.
There were some additional requirements from my POV:
  1. Not every mail should have image attached, e.g. automatically generated internal messages!
  2. I should be careful about impact on the performance.
  3. Mali messages that are not modified should not have any noticeable marks about image that isn't added (this one will be more clear later).
  4. The solution should allow to define only certain senders to have mails modified.
  5. It has to have a DRY RUN mode so that it is easily disabled.
I immediately knew that there is no way to place image somewhere on the Internet and put link into the mail message. Although the most elegant solution, the problem is that all mail clients don't show images by default, and that's it. So, image has to be within mail message itself. A bit of research showed up a potential solution. Namely, to embed image data within IMG tag itself, and mail messages are already altered by adding disclaimer which is HTML and a perfect place to add that IMG tag, so why not reuse disclaimer for the same purpose? In favor of that solution was also my intention not to add new scripts into the mail processing chain because of fear that it might impact performance. 

Unfortunately, that solution didn't work. The most important shortcoming is that Outlook and GMail don't handle IMG tag with embedded image data in it, in other words, the image isn't shown. To solve that, image has to be embedded as a MIME part within mail. Additionally, the other requirements aren't easy to achieve, especially, the one to replace old image with a new one. So, in the end I had to resort to writing scripts.

I already wrote about how I managed to solve more complex requirements for a disclaimer than the functionality of Zimbra allows. So, it was natural place for me to add that additional processing there. I called the script to add image altermail.py and now altermime script has the following form:
#!/bin/bash
grep "DISCLAIMER:" ${1#--input=} > /dev/null 2>&1
if [ ! "$?" = 0 ]; then
/opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
fi
#echo "`date +%Y%m%d%H%M%S` $@" >> /tmp/altermime-args
/opt/zimbra/altermime-0.3.10/bin/altermail.py "$@" >> /tmp/altermail.log 2>&1
I'm calling altermial.py after altermime because the image placeholder is within disclaimer! Also, I removed exec keyword before altermime-bin call so that altermail.py is finished.

Additionally, note that altermail.py accepts the same arguments as altermime! This is in order to simplify things a bit.

Obviously, I choose Python as a programming language of my choice. I could write all that in Perl too, but since I'm lately working a lot more with Python, Python was the way to go. Both languages have very good support for mail processing (MIME messages in particular).

The script is on the GitHub and you can fetch it there.

How the script works

First of, script doesn't work for mail messages that aren't MIME. So, after loading a message the first check if it is a multipart message. If not, then script just exits.

Next, white and black lists are checked.

There are two passess over the mail message. In the first pass, it searches through the mail message to see if there is already image attached. If so, then it additionally checks if it is an older version of the image. If it is, it replaces the image, but in both cases it doesn't do anything more and finishes execution.

The second pass is done when there is no image in the mail message and it has to be added. Now, when adding the image it has to be added with HTML as html/related so that they are both shown. If you add it as html/alternative, then only one of them will be shown!

All the configuration options are embedded within script itself. I chose not to have configuration file to reduce number of disk accesses, which is already very high (a lot of modules are necessary).


Tuesday, September 17, 2013

DHCPNAK messages in log file

When I was checking log files I spotted the following log entries that were strange:
Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 00:40:5a:18:83:56 via eth0
Sep  7 11:32:20 srv dhcpd: DHCPACK on 1.1.1.151 to 0:4:5:1:8:5 via eth0
Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 0:4:5:1:8:5 via 1.1.1.10
Sep  7 11:32:20 srv dhcpd: DHCPACK on 1.1.1.151 to 0:4:5:1:8:5 via 1.1.1.10
Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 0:4:5:1:8:5 via 1.1.0.10: wrong network.
Sep  7 11:32:20 srv dhcpd: DHCPNAK on 1.1.1.151 to 0:4:5:1:8:5 via 1.1.0.10
The problem is that DHCP request is received three times, on two of which the answer is positive (DHCPACK) while one received negative response (DHCPNAK) and dhcpd logged the error message 'wrong network'.

The important thing is the network configuration in this specific scenario, which looks something like follows:
  +----+            +-----+              +----+
  |    |------------|     |--------------|    |
  +----+            +-----+              +----+
  Client      Firewall/DHCP relay      DHCP server
1.1.1.151    1.1.1.10     1.1.0.10       1.1.0.4
Looking into log entries, not much can be inferred. The only thing that can be seen is that third DHCPREQUEST came from 1.1.0.10 which isn't on the same network with a client requesting IP address. Sniffing the network gave a bit more information on what's happening. Analyzing the network trace the following were conclusions:

  1. There are three DHCPREQUEST messages with the same transaction ID, the same destination (1.1.0.4, i.e. DHCP server) and also client IP address field within DHCP request is set to 1.1.1.151.
  2. The first DHCPREQUEST comes directly from the client. It has source IP 1.1.1.151, and there is no relay field (i.e. the value is 0.0.0.0). Also, client MAC address field within DHCP request has MAC address of a given client. 
  3. The second DHCP request comes from DHCP relay on the firewall. It has source set to 1.1.0.10, and relay field is properly set to 1.1.1.10, i.e. the IP address from the client's network,.
  4. The third DHCP request also comes from DHCP relay on the firewall, but this time relay field is set to 1.1.0.10. This contradicts client's IP address and DHCP server rejects this request.
So, the conclusion is that client sends request to 1.1.0.4. This request is forwarded by the firewall to the server, but also intercepted by DHCP relay on the firewall that creates two proxy requests and sends them to DHCP server too, one of which is rejected.

The interesting thing, not visible in logs, is that DHCP relay upon receiving NAK from the DCHP server, generates new NAK that is broadcasted on the network where DHCP server lives. 

So, the conclusion is that firewall is wrongly configured. It should not forward DHCP requests if there is a relay agent running. Furthermore, those NAKs aren't seen by the client, only by DHCP relay that reflects them back to DHCP servers.

Thursday, September 12, 2013

Adding Zimbra disclaimer using shell scripts...

While Zimbra 8 (and 7, too) have domain wide disclaimer support built in, there are two shortcomings that forced me to fall back to the old way of doing it:
  1. There is no support for not adding disclaimer if it already exists, and
  2. No support to exclude some addresses from adding disclaimer.
The second problem I managed to solve by patching Amavis script. That approach adds extra effort for maintainability (primarily during the upgrades), but it works. To solve the first problem the same way was too much work that I wasn't prepared to invest so I had to abandon domain wide disclaimer provided by Zimbra. There was also a third problem. Namely, for all mail messages sent from Outlook, Zimbra added two extra characters at the end of a HTML disclaimer, namely characters "= A". Why is this, I don't have slightest clue. I suspect it has something to do with encoding and decoding messages while going through the mail system, but exact reasons are unknown to me.

So, I went to solve all those problems and first I tried the old way, namely modifying postfix subsystem. It turned out that it didn't work. Just for a reference, at the end of this post, I described what I did. Next, option was modifying amavis. But that turned out to be too complicated and error prone - as I said in the introduction paragraph. Finally, I decided to put a proxy script in front of altermime that will be called by amavis and that will check if there is already disclaimer. If it isn't, then it calls altermime. Note that in this way there was no need to change amavis, and that means a lot from the maintenance perspective. So, here is what I did.

First, I created the following simple script in /opt/zimbra/altermime directory:
#!/bin/bash
echo "`date +%Y%m%d%H%M%S` $@" >> /tmp/altermime-args
exec /opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
What it does is it just logs how it was called and then it calls altermime. Note one more important thing here. In order to be able to put this script before altermime, I had to call it altermime, and altermime binary I renamed to altermime-bin. If you are doing this on a live system be very careful how you do this switch. I suggest that you first create script called altermime.sh, check that it works, and then use the following command to make a switch:
mv altermime altermime-bin && mv altermime.sh altermime
Ok, in this way I was able to find out how altermime is actually called. This is what I saw in /tmp/altermime-args file:
20130912100915 --input=/opt/zimbra/data/amavisd/tmp/amavis-20130912T100229-30384-pc8afS_K/email-repl.txt --verbose --disclaimer=/opt/zimbra/data/altermime/global-default.txt --disclaimer-html=/opt/zimbra/data/altermime/global-default.html
That's just one line of the output. As it can be seen, the first argument specifies file with mail message, and the rest specify disclaimer to be added. So, in order not to add disclaimer, if there is already one, I modified the altermime.sh script to have the following content:
#!/bin/bash
grep "DISCLAIMER:" ${1#--input=} > /dev/null 2>&1
if [ ! "$?" = 0 ]; then
    exec /opt/zimbra/altermime-0.3.10/bin/altermime-bin "$@"
fi
Again, be careful if you are modifying this script on a live system.

Now, in order to control where disclaimer is added, you can modify this simple shell script. One more thing you should be aware of, this approach impacts performance as, instead of running one process, it now runs at least 3 per mail message, and there are few extra file accesses. 

Finally, as a side note, I managed to get rid of those strange characters added to Outlook's email messages. I just edited a little bit html file that contains disclaimer, and those characters were gone. That's definitely a bug somewhere, but who knows where...

The old way that didn't work

As I said, the first approach I tried is to use the procedure from Wiki. But it didn't work. Anyway, for a reference, here is what I tried to do. Note that, as Zimbra already ships with altermime, there is no need to install it. The altermime is in /opt/zimbra/altermime/bin directory and you can safely use it. Ok, now to changes:

First, change a line in master.cf.in that reads
smtp    inet  n       -       n       -       -       smtpd
into
smtp    inet  n       -       n       -       -       smtpd        -o content_filter=dfilt:
and also add the following two lines:
dfilt   unix  -       n       n       -       -       pipe
        flags=Rq user=filter argv=/opt/zimbra/postfix/conf/disclaimer.sh -f ${sender} -- ${recipient}
Note that by this last line you specified that your script is called disclaimer.sh and that it is placed in /opt/zimbra/postfix/conf directory. This script, when run, should be run with a user privileges filter. Also, be careful where you put those lines. Namely, put them after the following three lines:
%%uncomment SERVICE:opendkim%%  -o content_filter=scan:[%%zimbraLocalBindAddress%%]:10030
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_filter=[%%zimbraLocalBindAddress%%]:10027
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_options=speed_adjust
The reason is that those line logically belong to the first smtp line, and if you add dfilt in front of it, you'll mess things, probably very badly, depending on your luck!

If you had Zimbra's domain wide disclaimer enabled, then disable it using:
zmprov mcf zimbraDomainMandatoryMailSignatureEnabled FALSE
as a zimbra user, and then restart amavis:
zmamavisdctl restart
still as a zimbra user.

Finally, to active custom script to add disclaimer run the following command as zimbra user:
zmmtactl restart
After I did all that, it didn't work. :D But, then I realized that there are two content_filter options to smtp which might not work, and so I resorted to proxying altermime.

Tuesday, June 18, 2013

Upgrading Alfresco

This is a short note on how I upgraded Alfresco. The basic idea was to just replace WAR files while keeping configuration files with local modifications intact. To be able to do that, I'll unpack WAR archive, integrate changes I made to running instance of Alfresco, create new WAR archives and place them in tomcat webapps folder so that he unpacks and deploys them.

Preparation

So, I downloaded Alfresco 4.2.c. More specifically, I downloaded file alfresco-community-4.2.c.zip. The version I had was 4.0.d.

To find out what configuration files are changed in the running instance I unpacked alfresco.war archive (that file is in the downloaded archive) into a separate directory using unzip command. I suggest that you create directory alfresco, enter that directory and then run unzip command. There, I run the following script:
#!/bin/bash
OLDPATH=/var/lib/tomcat6/webapps/alfresco/
for i in `find . -path ./WEB-INF/classes/alfresco/messages -prune -o -name \*properties -print`
do
        # If file doesn't exist we don't need to check it, go to next one
        [ -f $OLDPATH/$i ] || continue
        # If the old and new files are the same, then skip it also
        cmp -s $OLDPATH/$i $i && continue
        # diff -uN $OLDPATH/$i $i | less
        echo $i
done
The script showed me what files have changed:
./WEB-INF/classes/test/alfresco/test-hibernate-cfg.properties./WEB-INF/classes/alfresco/model/dataTypeAnalyzers_en.properties./WEB-INF/classes/alfresco/workflow/invitation-nominated-workflow-messages_ja.properties./WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/slingshot/wiki/move.post_it.properties./WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/slingshot/wiki/move.post_de.properties./WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/slingshot/wiki/move.post_nl.properties./WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/slingshot/wiki/move.post_es.properties./WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/slingshot/wiki/move.post_fr.properties./WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/slingshot/wiki/move.post_ja.properties./WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/slingshot/wiki/move.post.properties./WEB-INF/classes/alfresco/domain/hibernate-cfg.properties./WEB-INF/classes/alfresco/repository.properties./WEB-INF/classes/alfresco/subsystems/email/OutboundSMTP/outboundSMTP.properties./WEB-INF/classes/alfresco/subsystems/thirdparty/default/swf-transform.properties./WEB-INF/classes/alfresco/subsystems/thirdparty/default/imagemagick-transform.properties./WEB-INF/classes/alfresco/subsystems/fileServers/default/file-servers.properties./WEB-INF/classes/alfresco/subsystems/Synchronization/default/default-synchronization.properties./WEB-INF/classes/alfresco/subsystems/Search/solr/solr-search.properties./WEB-INF/classes/alfresco/subsystems/Search/solr/solr-backup.properties./WEB-INF/classes/alfresco/subsystems/Authentication/ldap-ad/ldap-ad-authentication.properties./WEB-INF/classes/alfresco/subsystems/Authentication/ldap/ldap-authentication.properties./WEB-INF/classes/alfresco/subsystems/Authentication/kerberos/kerberos-authentication.properties./WEB-INF/classes/alfresco/subsystems/Authentication/alfrescoNtlm/alfresco-authentication.properties./WEB-INF/classes/alfresco/subsystems/Authentication/passthru/passthru-authentication-context.properties./WEB-INF/classes/alfresco/subsystems/OOoDirect/default/openoffice-transform.properties./WEB-INF/classes/alfresco/version.properties./WEB-INF/classes/log4j.properties
Of those, localization files are not important (the ones ending in _[a-z][a-z].properties). If you uncomment a line containing less then the script will compare each file and show you difference in less. Based on that, I factored out the following configuration files that were changed:
./WEB-INF/classes/alfresco/repository.properties
./WEB-INF/classes/alfresco/subsystems/Authentication/ldap/ldap-authentication.properties
./WEB-INF/classes/alfresco/subsystems/Authentication/kerberos/kerberos-authentication.properties
./WEB-INF/classes/log4j.properties
I also found that I didn't change mail configuration data in the file:
./WEB-INF/classes/alfresco/subsystems/email/OutboundSMTP/outboundSMTP.properties
The next step was to find what changes are due to local configuration, and which ones are due to the changes in upstream. Namely, I'll take old configuration files but the changes in the new version have to be propagated. It turned out that repository.properties doesn't have any changes, while the other three have. So, I started to change files in the new version of Alfresco, that I unpacked. Finally, when all the changes are done, create new archive:
cd ..
mv alfresco.war alfresco.war.OLD
cd alfresco
zip -9rv ../alfresco .
cd ..
mv alfresco.zip alfresco.war
The first two commands rename old archive, next two create a new archive, and the final command changes name to have extension WAR. I assumed here that you unpacked original WAR archive into directory called alfresco.

All this has to be done with share.war archive too. In my case, the script showed that only log4j.properties has been changed so I incorporated changes and created a modified share.war archive.

Update

Finally, I stopped tomcat, and alfresco using:
service tomcat6 stop
created a copy of existing alfresco and share directories in tomcat's webapp directory. I also renamed old alfresco.war and share.war, and moved the ones I prepared in their place. Take care about permissions, tomcat has to be owner of everything! Then, I started tomcat with:
service tomcat6 start
and also started to pray that it works. Well, almost, I watched logs (/var/log/tomcat6/catalina.out) because I doubt that praying would help. ;)

Everything was OK, i.e. errors that I received (openoffice, pdf2swf) were expected because I didn't inistalled them. But, two errors were not expected:
java.io.FileNotFoundException: alfresco.log (Permission denied)
well, that one was cause by share.war not being able to reopen log file already being used by alfresco.war. So, I changed in appropriate log4j.properties that it uses its own separate log file. Except that it turned that I forgot to change log4j.properties. Anyway, I gave separate log to share.war just to be on a safe side and to finish this finally. The second error was:
java.net.BindException: Permission denied
That one was caused by FTP server not being able to bind to a low numbered port. This is OK, because I'm not running tomcat as a root. So, I'm safe to ignore it.

Tuesday, March 12, 2013

Storing arpwatch output into database

arpwatch is very useful tool which logs its output via syslog and also sends mail alerts. Unfortunately, this isn't configurable, i.e. arpwatch, out-of-the-box, doesn't support any other way of logging.  One approach is to modify arpwatch to be able to log into some SQL database, but this isn't straightforward way, i.e. not an easy one. Namely, arpwatch is written in C, and besides, it's hard to know if this would be accepted by upstream (who ever that migh be).

So, I decided to go with a different approach. I configured arpwatch to log its output into log file and wrote a Python script that executes via cron and transfers all the data into the database. Here is how I did it along with all the scripts.

Configuring logging

The first step is to configure arpwatch to log its output into a separate file. This isn't possible to do in arpwatch itself, but it is possible to achieve it by configuring syslog, or rsyslog to be more precise. On CentOS 6 rsyslog is used that allows just that. All you have to do is to place a file named (for example) arpwatch.conf in directory /etc/rsyslog.d with the following content:
if $programname == 'arpwatch' then /var/log/arpwatch.log
&~
Don't forget to restart rsyslog after that. This will write anything logged by arpwatch binary into /var/log/arpwatch.log file. All the different log lines that can appear are documented in arpwatch's manual page so I won't replicate them here.

Configuring database

In my case I created a single table using the following SQL statement:
CREATE TABLE arpwatch (
  macaddr char(17) NOT NULL,
  ip_addr int(10) unsigned NOT NULL,
  state varchar(8) NOT NULL,
  timestamp datetime NOT NULL,
  oldmac char(17) DEFAULT NULL
)
I think it's pretty obvious what goes where. Only thing that might be strange is that I'm using INT(10) for IP address. But that is because SNORT also stores IP addresses in such a way so in order to be compatible with it I used it also. Also, what is missing is primary key, but for the time being I'm not using it.

Script

Here is the script that should be started from the cron. For example, store it in /usr/local/sbin directory and to start it every 20 minutes add the following line (as root user) to cron using 'crontab -e' command:
*/20 * * * * /usr/local/sbin/arpwatchlog2sql.py
Note that the script expects configuration file. Here is a sample configuration file you'll have to modify. The script expects configuration file to be in its current directory, but you can place it into /usr/local/etc and modify the line CONFIGFILE in script accordingly.

Log rotation

Finally, you should be certain that logs are properly handled, i.e. rotated along with other logs. Since arpwatch is logging via syslog, that means that you have to modify rsyslog's log configuration file, i.e. /etc/logrotate.d/syslog. In there you'll see that logfiles maintained by rsyslog are enumerated, one per line. Just add arpwatch.log to that list and that should be it.

Tuesday, February 5, 2013

Fun when mail server receives SERVFAIL instead of NXDOMAIN...

Ok, I got log files overflowed with error messages like this one:
Feb 5 11:01:35 mail named[994]: error (host unreachable) resolving 'sbs-music.com/NS/IN': 50.56.243.69#53
In essence, name server for this domain (50.56.243.69) is unreachable from the DNS server used by mail server. Trying to manually query the server, I get:
$ host -t ns sbs-music.com
Host sbs-music.com not found: 2(SERVFAIL)
Note the status, it's SERVFAIL. The result is that mail server thinks it is a temporary error and retries later, with the same results. Trying this on another host (that uses another DNS server) I get:
$ host -t ns sbs-music.com
Host sbs-music.com not found: 3(NXDOMAIN)
Well, this time it tells me that there is no such domain. An error message like this would tell mail server to give up and return error response.

So, why is there discrepancy between the two? Using tcpdump in the first case (i.e. when we get SERVFAIL) the following requests/responses are exchanged (slightly edited for readability):
192.168.x.y.51892 > 206.72.97.238.53: 39504 [1au] NS? sbs-music.com. (42)
206.72.97.238.53 > 192.168.x.y.51892: 39504 4/0/5 NS ns4.shepherdhosting.com., NS ns1.shepherdhosting.com., NS ns2.shepherdhosting.com., NS ns3.shepherdhosting.com. (194)
192.168.x.y.10749 > 50.56.243.69.53: 40104 [1au] NS? sbs-music.com. (42)
timeout
192.168.x.y.63081 > 206.72.97.238.53: 56636 [1au] NS? sbs-music.com. (42)
206.72.97.238.53 > 192.168.x.y.63081: 56636 4/0/5 NS ns1.shepherdhosting.com., NS ns2.shepherdhosting.com., NS ns3.shepherdhosting.com., NS ns4.shepherdhosting.com. (194)
192.168.x.y.31948 > 50.56.243.69.53: 27220 [1au] NS? sbs-music.com. (42)
timeout
So, let me interpret this trace. The first query is to IP address 206.72.97.238 and it asks for the name server of a domain sbs-music.com. Doing reverse DNS query, we get:
# host 206.72.97.238
238.97.72.206.in-addr.arpa domain name pointer sh214.shepherdhosting.com.
So, it's some hosting provider. Now, what we get in response is that name servers for that domain are ns1.shepherdhosting.com through ns4.shepherdhosting.com. Ok, our DNS server choose ns4.shepherdhosting.com with IP address 50.56.243.69. Then, it queried it for sbs-music.com domain. This time the query timed out. So, our DNS server decided to query again 206.72.97.238 for name servers. It again received the same list and then it again queried ns4 which didn't answer the query.

Let us manually try some other server. So, querying ns1.shepherdhosting.com we get:
# host -t ns sbs-music.com. 206.72.97.238
Using domain server:
Name: 206.72.97.238
Address: 206.72.97.238#53
Aliases:

sbs-music.com name server ns2.shepherdhosting.com.
sbs-music.com name server ns3.shepherdhosting.com.
sbs-music.com name server ns4.shepherdhosting.com.
sbs-music.com name server ns1.shepherdhosting.com.
This is the same response we saw in the first part of the trace. Trying ns2:
$ host -t ns sbs-music.com. 206.72.100.134
;; connection timed out; trying next origin
;; connection timed out; no servers could be reached
Well, ns2 doesn't respond. Neither do ns3, nor as was always obvious, ns4. What does the trace looks like (again, slightly edited):
12:20:22.532877 IP 192.168.x.y.34364 > 206.72.100.134.53: 31539+ NS? sbs-music.com. (31)
12:20:27.532864 IP 192.168.x.y.34364 > 206.72.100.134.53: 31539+ NS? sbs-music.com. (31)
12:20:32.533445 IP 192.168.x.y.45322 > 206.72.100.134.53: 5827+ NS? sbs-music.com. (31)
12:20:52.733389 IP 206.72.100.134.53 > 192.168.x.y.34364: 31539 ServFail 0/0/0 (31)
12:20:52.733410 IP 192.168.x.y > 206.72.100.134: ICMP 192.168.x.y udp port 34364 unreachable, length 67
12:20:52.734042 IP 206.72.100.134.53 > 192.168.x.y.45322: 5827 ServFail 0/0/0 (31)
12:20:52.734053 IP 192.168.x.y > 206.72.100.134: ICMP 192.168.x.y udp port 45322 unreachable, length 67
Now, we have interesting situation here. First, DNS server for sbs-music.com takes significant time to answer, and when it answers our local DNS isn't listening any more (thus those ICMP error messages). But, in the end local DNS concludes correctly that something's wrong with the name servers for that domain.

The final piece of puzzle comes from querying com name server for sbs-music.com domain:
$ host -t ns sbs-music.com l.gtld-servers.net.
Using domain server:
Name: l.gtld-servers.net.
Address: 192.41.162.30#53
Aliases:

sbs-music.com has no NS record
Obviously, this domain was removed. It is clear now that this sbs-music.com domain existed for some time, and DNS server that produces error cached IP address of its domain name. If it were to query com domain name server again, it would receive NXDOMAIN error and properly notify mail server.

To see currently cached entries of BIND name server use the following command:
rndc dumpdb
Then look for file cache_dump.db in /var/named/data (or in /var/named/chroot/var/named/data if you are running BIND in chroot). It is a textual file that you can inspect with text editor, less or something similar. In my case there were the following lines there:
; glue
sbs-music.com.          172669  NS      ns1.shepherdhosting.com.
                        172669  NS      ns2.shepherdhosting.com.
                        172669  NS      ns3.shepherdhosting.com.
                        172669  NS      ns4.shepherdhosting.com.

To flush a single entry use the following command
rndc flushname sbs-music.com internal
This removes sbs-music.com from caches in internal view (I configured viewes so that server behaves differently depending on who asks it). Yet, this didn't help. Then I tried to flush everything in internal view using:
rndc flush internal
But this, while helped, didn't actually solve the problem. Namely, looking into packet trace it turns out that  BIND server receives from b.gtld-servers.net. that given domain doesn't exist and from somewhere it pulls the old IP address 206.72.97.238?!

So, I finally decided to look into log and there are a lot of the following messages:
error (FORMERR) resolving 'sbs-music.com/NS/IN': 206.72.97.238#53
along with the one that triggered all this. Ok, I also tried to upgrade bind, but no luck, still SERVFAIL errors.

Now, its time for heavy artillery, or Wireshark. So I saved packet trace and loaded it into Wireshark. Guess what! Wireshark crashed on requests sent by local DNS server!?

Ok, after a bit more fiddling I realised that some com domain name servers do know for this domain. But note the difference between the output of host command (that I've used previously) and nslookup command:
$ nslookup -type=ns sbs-music.com 192.5.6.30
Server: 192.5.6.30
Address: 192.5.6.30#53

Non-authoritative answer:
*** Can't find sbs-music.com: No answer

Authoritative answers can be found from:
sbs-music.com nameserver = ns1.shepherdhosting.com.
sbs-music.com nameserver = ns2.shepherdhosting.com.
sbs-music.com nameserver = ns3.shepherdhosting.com.
sbs-music.com nameserver = ns4.shepherdhosting.com.
ns1.shepherdhosting.com internet address = 206.72.97.238
ns2.shepherdhosting.com internet address = 206.72.100.134
ns3.shepherdhosting.com internet address = 206.72.97.237
ns4.shepherdhosting.com internet address = 50.56.243.69
Ok, if this com server tells me that there is no this domain, why is then it pointing me to those nameservers? dig command is a bit more informative:
$ dig @192.5.6.30 sbs-music.com

; <<>> DiG 9.9.2-P1-RedHat-9.9.2-6.P1.fc18 <<>> @192.5.6.30 sbs-music.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64382
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 5
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;sbs-music.com. IN A

;; AUTHORITY SECTION:
sbs-music.com. 172800 IN NS ns1.shepherdhosting.com.
sbs-music.com. 172800 IN NS ns2.shepherdhosting.com.
sbs-music.com. 172800 IN NS ns3.shepherdhosting.com.
sbs-music.com. 172800 IN NS ns4.shepherdhosting.com.

;; ADDITIONAL SECTION:
ns1.shepherdhosting.com. 172800 IN A 206.72.97.238
ns2.shepherdhosting.com. 172800 IN A 206.72.100.134
ns3.shepherdhosting.com. 172800 IN A 206.72.97.237
ns4.shepherdhosting.com. 172800 IN A 50.56.243.69
xxx
;; Query time: 149 msec
;; SERVER: 192.5.6.30#53(192.5.6.30)
;; WHEN: Tue Feb  5 13:49:22 2013
;; MSG SIZE  rcvd: 194
What a mess?!

Then, I googled a bit to find why BIND is returning SERVFAIL instead of NXDOMAIN. This is something interesting that I found:
  • BIND could have returned SERVFAIL instead of NXDOMAIN responses for nonexistent resource records from the unsigned child zone if the parent zone was signed. (BZ#643012)
Trying to lookup that bug in RedHat's Bugzilla gives be big red square which tells me that I'm not allowed to see it (despite being logged in) so it's some security issue obviously!?

Looking at how different BIND versions behave I get the following results:
  • bind-9.3.6-20.P1.el5_8.6 and bind-9.9.2-6.P1.fc18.x86_64 return NXDOMAIN.
  • bind-9.8.2-0.10.rc1.el6_3.6.x86_64 and bind-9.8.2-0.10.rc1.el6_3.5.x86_64 return SERVFAIL.
Could it be somehow related to DNSSEC?

Ok, let me conclude. The problem is that name servers for domain sbs-music.com aren't correctly configured, while the domain itself is registered with com domain servers. This triggers different behavior from BIND. So, there are two possibilities from here:
  1. Persuade somehow BIND to return NXDOMAIN instead of SERVFAIL.
  2. Find what is causing queries for this domain in the first place.
Stay tuned... :)

Tuesday, January 29, 2013

How to change Volume Group's name...

In default installation of CentOS LVM is used and all volume groups are named VolGroup00. This can create problems when multiple machines' disks have to be accessed from a single machine. So, one of the options is to rename volume groups. This is actually very easy to do in the following four steps that can be done on a live machine:
  1. Rename volume group.
  2. # vgrename VolGroup00 <newname>
  3. Change /etc/fstab
  4. Open it in some text editor and do a search and replace through the file, i.e. any occurrence of VolGroup00 change to <newname>.
  5. Change /etc/boot/grub.conf
  6. Open it in some text editor and do a search and replace through the file, i.e. any occurrence of VolGroup00 change to <newname>.
  7. Recreate initrd image.
  8. First, rename old initrd image. initrd images are in /boot directory and their name contains the version of currently running kernel (use uname -r but without architecture part).
    # initrd <initrdname> <kernel version>
    Be careful that you don't have newer kernel installed which will be started during the next boot process. In that case you'll have problems! Maybe it's best to restart machine before doing this whole procedure.
Restart machine and that should be it. :) Of course, just to be safe try this first on some test machine.

Friday, November 23, 2012

Zimlets for managing posix & samba attributes...

Well, this isn't actually new news, but nevertheless I managed to avoid it for some time now. Namely, Zimbra, with upgrade to 7.2, removed plugins that are used to manage Samba and Posix accounts in its LDAP directory. Now, whenever someone asked what about this Zimlets, the answer was "This was never supported by Zimbra and thus someone from community has to step in." If you Google a bit, you'll easily find it, e.g. here or here. Now, this probably is a perfectly reasonable answer from the Zimbra's standpoint, but I believe that Zimbra should know that this plugin was more frequently used one (let me guess it: because it's useful?) and they have to listen their users.

But whatever is/was with those plugins, I had to have them because one of my setups is such that it contains all account databases in Zimbra. When I disabled Samba and Posix Zimlets, everything worked as usual, apart that I was unable to add new users via Web interface. After I managed to get with it for some time now, the time has come that I had to add another user account and I had to see what I'm going to do with non-working Web interface.

After some googling I discovered that someone managed to fix those two plugins, and also at the time this post was written, there are no news if those plugins work with version 8. So, in short, don't upgrade yet if you are using those plugins. To see what was changed in the plugins to make them work, take a look at this post. In any case, go to the Zimbra's gallery and download Posix and Samba Zimlets. The versions I used are 28.5.12 - v6.1 for both Zimlets. Now, before installing open the archives and in each one you'll find config_template.xml files. Open those files in text editor and fill in the correct values. The most important one is LDAP suffix which is by default set to dc=domain,dc=tld and which you should change to reflect your domain. For example, if your domain is example.com then the suffix will be dc=example,dc=com. After you've made changes, save files and put them back in the archive. If you don't do that you'll receive error reports when logging in admin console, and also there will be no existing samba and possix groups. Not to mention that you'll be unable to create new accounts.

Ok, the last step is to undeploy the old versions - in the case you didn't already, and deploy new ones. After deploying, you should log out and then back in and you should see their options under the Configuration section (in the left pane). If you click on, e.g. Manage Samba Groups, you'll see your existing Samba groups. Similarly has to be with the option Posix Groups. If there are no groups (and you know there should be) than you probably messed with LDAP suffixes I was talking about.

And that's it. For the end, if someone from Zimbra is reading this post, then I have a message for you. Namely, don't answer to so many people that you don't support something because it was always unsupported. I don't think it's relevant. If people are using it, and they find it useful, then you should support it. Or at least devote one engineer day to fix the problem. It isn't so expensive and people will have much better opinion about Zimbra.

Thursday, November 8, 2012

Installing certificate for Alfresco...

This post is continuation of the post about installing Alfresco using native Tomcat6 installation (on CentOS6). If you followed steps given in that post, you have running Alfresco installation but Tomcat uses self-signed certificate.

To install your own certificate first obtain it (you can use your own, self managed, CA or you can buy commercial one), then install it on your Tomcat instance. You'll find a lot of information about this in SSL Howto on Tomcat's Web pages, but that page assumes that everything you do, you are doing using keytool.

Here is a quick Howto with an assumption that you have files newcert.pem (containing certificate), newkey.pem (containing private key) and cacert.pem (your CA certificate). By default, tomcat's keystore is in its home (/usr/share/tomcat6) and it is named .keystore. Keystore file is password protected and default password for it is changeit. Note that the period isn't part of password! I suggest that you copy this file to root's home under the name keystore (note no leading dot!) or whatever else you wish so that you can restore old copy in case something goes wrong with the following steps.

The installation is two step process. First, you create keystore containing you certificate, private key and CA's certificate. In second step, you import that information to Tomcat's keystore.

First step is to pack certificate for Alfresco, its private key and CA's certificate into PKCS12 store using openssl tool as follows:
$ openssl pkcs12 -export \
        -in newcert.pem -inkey newkey.pem \
        -out mycert.p12 -name tomcat \
        -CAfile cacert.pem -caname root -chain
Enter Export Password:
Verifying - Enter Export Password:
This command assumes that all necessary files (newcert.pem, newkey.pem and cacert.pem) are in you current directory. Output of the command is also stored into current directory. Note that you are asked for password that will protect all the data. Enter something or later you'll see the following warning:
*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in the srckeystore*
* has NOT been verified!  In order to verify its integrity, *
* you must provide the srckeystore password.                *
*****************  WARNING WARNING WARNING  *****************
And then you'll receive the following error:
keytool error: java.security.UnrecoverableKeyException: Get Key failed: / by zero
Second step is to import this pkcs12 file to tomcat's keystore using keytool as follows:
$ keytool -importkeystore -srckeystore mycert.p12 \
        -srcstoretype pkcs12 -destkeystore /usr/share/tomcat6/.keystore
Enter destination keystore password:
Enter source keystore password:
Existing entry alias tomcat exists, overwrite? [no]:  yes
Entry for alias tomcat successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
Again, input file is in the current directory and you are importing directly into tomcat's keystore. Note that the existing certificate with the alias tomcat will be removed and you are asked to confirm that! The default alias Tomcat searches when it start is called tomcat.

Third step is to change private key's password that has to be the same as for the keystore. Do that using the following command:
keytool -keypasswd -alias tomcat -new <keypassword> -keystore /usr/share/tomcat6/.keystore
You'll be asked for the keystore's password and the password for the key will be set to keypassword.

And that's it. Restart tomcat and check if it is using new certificate.

Tuesday, October 30, 2012

Installing ossec client on CentOS 6...

Ok, I did this already, but I managed to forget it. Still, it isn't strange, after all, it's not that you are adding new machines every day. Anyway, here are the steps that are need in order to install OSSEC client on a CentOS machine, more specifically CentOS 6. I decided to write this post if someone also needs these instructions, but certainly for me so that next time I have to do it I don't have to think a lot. Note that I like to install RPM packages because it is easier to update them instead compiling from source, and also someone else is worrying about new releases. Additionally, it's not so good to install development environment on production machines that don't need it, for security reasons. Ok, here we go.

First, make sure that you have EPEL repository added. The easiest way to do this is using the following command (note, bold is what you type, the rest is what you get from the machine):
# rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Retrieving http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
warning: /var/tmp/rpm-tmp.7IMdWB: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing...                ##################################### [100%]
1:epel-release   ##################################### [100%]
Second, fetch necessary packages. I didn't want to install Atomicorp's repository, so I only fetched ossec packages using wgetossec-hids and ossec-hids-client are what you need. Select the newest versions you can find. Next, install them using yum command:
# yum localinstall ossec-hids-client-2.6-15.el6.art.x86_64.rpm ossec-hids-2.6-15.el6.art.x86_64.rpm
I assumed that yum is executed in the same directory where you placed downloaded packages. Also, if you downloaded some other versions, change names appropriately.

Open ossec's configuration file, /var/ossec/etc/ossec-agent.conf, and change the line that has <server-ip></server-ip> element. It has to point to your server's IP address. You can also add files to be monitored in addition to the existing ones, or remove some of the existing ones if they are not used on the machine you are installing ossec client.

Now, go to the OSSEC server and run there agent management tool. It is probably in /var/ossec/bin:
# ./manage_agents


****************************************
* OSSEC HIDS v2.5-SNP-100907 Agent manager.     *
* The following options are available: *
****************************************
   (I)mport key from the server (I).
   (Q)uit.
Choose your action: I or Q: A

- Adding a new agent (use '\q' to return to the main menu).
  Please provide the following:
   * A name for the new agent: centos6.domain.local
   * The IP Address of the new agent: 192.168.10.41
   * An ID for the new agent[030]: <just press ENTER>
Agent information:
   ID:030
   Name:centos6.domain.local
   IP Address:192.168.10.41

Confirm adding it?(y/n): y
Agent added.
Note that the tool doesn't display all the options you have on your disposal. Next what you need to do is to extract a key that you'll import into the client. This is also done using manage_clients tool, so either start it again, or in case you didn't exit after you added an agent just continue:
 ****************************************
* OSSEC HIDS v2.5-SNP-100907 Agent manager.     *
* The following options are available: *
****************************************
   (I)mport key from the server (I).
   (Q)uit.
Choose your action: I or Q: e

Available agents:
   ID: 002, Name: somehost, IP: 10.0.10.1
   ID: 030, Name: centos6.domain.local, IP: 192.168.10.41
Provide the ID of the agent to extract the key (or '\q' to quit): 030
Agent key information for '030' is:
<here a very long string will be printed>
** Press ENTER to return to the main menu.
Again, option to export the key isn't listed in the help message! Anyway, copy the very long string that is printed (agent's key) and you can quit from the tool and logout from the OSSEC server.

Go now to ossec client, change directory to /var/ossec/bin and run manage_client tool:
# ./manage_client


****************************************
* OSSEC HIDS v2.6 Agent manager.     *
* The following options are available: *
****************************************
   (I)mport key from the server (I).
   (Q)uit.
Choose your action: I or Q: I

* Provide the Key generated by the server.
* The best approach is to cut and paste it.
*** OBS: Do not include spaces or new lines.

Paste it here (or '\q' to quit):
<very long string copied here!>

Agent information:
   ID:030
   Name:centos6.domain.local
   IP Address:192.168.10.41

Confirm adding it?(y/n): y
Added.
Finally, restart ossec client:
# /etc/init.d/ossec-hids restart
Shutting down ossec-hids:                      [  OK  ]
Starting ossec-hids:                           [  OK  ]
You should see you new client in OSSEC's Web interface which should confirm that it is running OK.

Monday, October 29, 2012

yum and fastestmirror plugin...

Few hours ago I lost my nerves because when I started yum to update my system, download was painfully slow, somewhere around 20kB/s. It is outrageous because I was using 100 Mbps link and that is probably the slowest link in the chain that ends up somewhere in GEANT. Thus, things have to be much faster than that! The best speed that can be achieved is somewhere around 50Mbps and what I was getting wasn't even remotely close to it! This wasn't something I was prepared to accept as is, so I decided to see what's happening.

Yum has a plugin, fastestmirror. The purpose of that plugin is to determine the fastest available mirror and makes yum download from it, not some random one. Usually, this plugin works very well, but this time it didn't. I tried to reset everything with
yum clean all
and than again
yum update
But it didn't help. Googling around I quickly determined that the first command didn't remove fastestmirror's data. What is necessary is to remove cache file stored in /var/cache/yum/x86_64/17/timedhosts.txt (this is location on 64-bit Fedora 17). Well, guess what, this didn't help either. Namely, fastestmirror plugin determines which mirror is the best one based on measuring how much time is necessary to establish connection with a mirror, and then it immediately disconnects. This is all OK, until mirror starts to apply some throttling effectively capping maximum speed. And this was exactly what happened to me.

It used to be possible to send SIGINT signal to yum (pressing Ctrl+C) on which yum would switch to another mirror. But this doesn't work any more. When you press Ctrl+C yum exits. Now, this is expected behavior, but the previous one was actually useful! So, there should be some way to tell yum to switch to next mirror.

In the end I solved this by looking which mirror(s) yum was using. This is printed when yum starts, e.g.:

Loading mirror speeds from cached hostfile
 * fedora: gd.tuwien.ac.at
 * fedora-debuginfo: fedora.inode.at
 * rpmfusion-free: mirrors.coreix.net
 * rpmfusion-free-debuginfo: mirrors.coreix.net
 * rpmfusion-free-updates: mirrors.coreix.net
 * rpmfusion-free-updates-debuginfo: mirrors.coreix.net
 * rpmfusion-nonfree: mirrors.coreix.net
 * rpmfusion-nonfree-debuginfo: mirrors.coreix.net
 * rpmfusion-nonfree-updates: mirrors.coreix.net
 * rpmfusion-nonfree-updates-debuginfo: mirrors.coreix.net
 * rpmfusion-nonfree-updates-testing: mirrors.coreix.net
 * rpmfusion-nonfree-updates-testing-debuginfo: rpmfusion.blizoo.mk
 * updates: gd.tuwien.ac.at
 * updates-debuginfo: fedora.intergenia.de
The problem was Fedora's main repository, which was downloaded from gd.tuwien.ac.at. So, I edited fastestmirror's configuration file /etc/yum/pluginconf.d/fastestmirror.conf and added the following line:
exclude=.at
That excluded a bit more mirrors than I intended, but it definitely solved my problem.

Thursday, July 26, 2012

Searching for packet catpuring and interface manipulation library for Python...

I needed a script that would monitor network traffic and capture and process only DHCP traffic. It turned out I couldn't find such script so I decided to write one (more about that script in another post). For a language I decided to use Python. That was the easy part. Now, I had to decide which libraries I will use that will allow me to capture network traffic, decode DHCP request and responses, and manipulate IP addresses on interfaces.

I started with the network traffic capturing. pcap library is the library for network capture, so it was natural for me to search for a Python interface to this library. I found several such interfaces, i.e. pcap, pylibpcap, pypcap, and pcapy. There is also library interface specifically for Python 3, i.e. py3kcap. While searching for pcap interface, three other Python libraries poped out: libdnet (here is the old project page), dpkt and scapy.

But, not all libraries are equal, nor they serve the same purpose. libdnet allows sending packets, manipulation with kernel's routing tables, firewall and arp cache. So, besides Ethernet and IP, it doesn't offer much more in term of supported protocols. dpkt, on the other hand, is made just for this purpose! It supports easy creation and parsing of different TCP/IP protocols. Finally, Scapy is a swiss army knife of network manipulation. It offers shell in which one can manipulate packets, but also can be used within other scripts. Unfortunately, while browsing the source of Scapy I realized that it uses os.popen interface and calls external programs. So, this actually was enough for me to eliminate scapy from further consideration.

The next elimination criteria is availability of the packages within CentOS and Fedora. I try to hold on prepackaged software as much as possible, so quick search (yum search) showed that on both, CentOS 6 and Fedora 17, there are packages for pcapy and dpkt (named python-dpkt). For some reason, there is dnet, but python interface isn't packaged. I found this bugzilla entry, but without any answer!

So, I settled on pcapy and dpkt. The only piece of puzzle that was missing now is how to manipulate interface addresses. I stumbled on netifaces, which allows me to obtain information about interfaces and also on this post for Windows. But all the results I got were on how to obtain IP address. In the end, I gave up and decided that I'll try to use libdnet even though I'll have to compile it from the source. Either that, or I'll use raw sockets and ioctls which are accessible from Python using standard libraries.

And for the end, as a curiosity, I'll mention that there is Python interface to IPTables, python-iptables, which is also packaged for Fedora.

Tuesday, July 24, 2012

ntop 5 on CentOS 6...

Last week I decided to install ntop on one of my CentOS 6 machines and, to much of my surprise, it turned out that there is no ntop package in the standard CentOS6 repositories (i.e. Base, EPEL, RPMFusion). Then, I looked into Fedora repository and it turned out that there is package, but for the older version, i.e. 4.0 (the newest version of ntop at the time this post was written was 5.0). So, I downloaded that older version, placed new version of ntop, modified a bit SPEC file and tried to build it. It didn't work intermediately, but after few more tweaks it worked. I filled a bug report on RedHat's bugzilla so that maintainter can upgrade a package, if he wishes so.

In the mean time, I decided to build package for CentOS 6. The main problem is that Fedora introduced Systemd instead of traditional SysV init used by CentOS. To cut the story short, I managed to do that, too. The resulting SPEC file can be used in both Fedora and EPEL6. I uploaded new SPEC file (and init file) to bugzilla, so you can fetch them there if you wish.

Until maintainers decide what to do, and if anything to do with it, here are the SRPM file and resulting binary RPM file for 64-bit CentOS6.

Friday, July 20, 2012

Integrating FreeIPA and Alfresco...

After describing how to install CentOS, DNS and reverse DNS, FreeIPA and Alfresco, in this post I'm going to describe how to integrate Alfresco with FreeIPA. I want to achieve the following goals with the integration:
  • Users and groups are kept within FreeIPA and authentication is done by FreeIPA.
  • Alfresco Web interface honors Kerberos tickets. Upon opening Web interface users are immediately presented with their pages withoug necessity for authentication (if, of course, they have valid Kerberos tickets).
  • Authentication when mounting DAV share is also done via Kerberos tickets.
In short, I want to achieve SSO (Single Sign-On) as much as possible. Users sign in when they start to use their workstations once, that's the only time they have to enter password.

Tuesday, July 17, 2012

Zimbra log cluttering...

When you run Zimbra, logs generated by it are duplicated in system log files (i.e. /var/log/messages, or /var/log/secure) but also in Zimbra specific log files (i.e. /var/log/zimbra.log). The problem with this is that it clutters system logfiles, i.e. takes unnecessary space and makes them hard to analyze. So, it would be good to make Zimbra log only in its own specific log files.

Googling for solution I found this post but with no satisfactory solution. Since there was no ready solution, I turned to Googling on how to configure rsyslog to do that. Namely, CentOS (on which I'm running Zimbra) uses rsyslog as a replacement for a more traditional syslog. It turns out it is possible to filter according to application doing logging.

So, two offending applications are zimbramon and zmmailboxdmgr. In order to prevent them from logging into /var/log/messages add the following lines before section RULES (that resemble classical syslog rules):
if $programname == 'zimbramon' then /var/log/zimbra-stats.log
& ~

if $programname == 'zmmailboxdmgr' then /var/log/zimbra-stats.log
& ~
The first two lines will redirect zimbramon messages, while the second two will do the same for zmmailboxdmgr.

Friday, June 29, 2012

BIND and network unreachable messages...

Sometimes you'll see messages like the following ones in your log file (messages are slightly obfuscated to protect innocent :)):
Jun 29 14:32:11 someserver named[1459]: error (network unreachable) resolving 'www.eolprocess.com/A/IN': 2001:503:a83e::2:30#53
Jun 29 14:32:11 someserver named[1459]: error (network unreachable) resolving 'www.eolprocess.com/A/IN': 2001:503:231d::2:30#53
What these messages say is that network that contains address 2001:503:231d::2:30 is unreachable. So, what's happening?

The problem is that all modern operating systems support IPv6 out of the box. The same is for growing number of software packages, among them is BIND too. So, operating system configures IPv6 address on interface and application thinks that IPv6 works and configures it. But, IPv6 doesn't work outside of the local network (there is no IPv6 capable router) so, IPv6 addresses, unless in local networks, are unreachable.

So, you might ask now: but everything otherwise works, why is this case special! Well, the problem is that some DNS servers, anywhere in hierarchy, support IPv6, but not all. And when our resolver gets IPv6 address in response, it defaults to it and ignores IPv4. It obviously can not reach it so it logs a message and then tries IPv4. Once again, note that this IPv6 address can pop up anywhere in hierarchy, it isn't necessary to be on the last DNS server. In this concrete case name server for eolprocess.com doesn't support IPv6, but some name server for the top level com domain do support it!

To prevent those messages from appearing add option -4 to bind during startup. On CentOS (Fedora/RHEL) add or modify the line OPTIONS in /etc/sysconfig/named so that it includes option -4, i.e.
OPTIONS="-4"

Thursday, June 28, 2012

Another internal error trying to access IPA Web UI

I just tried to access IPA's Web UI and I got 'Internal Server Error' dialog box:


Looking into log file (/var/log/httpd/error_log) I found the following entry that obviously was the reason dialog box appeared:
[Thu Jun 28 21:10:28 2012] [error] [client 192.168.178.1] gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information (, No key table entry found for HTTP/ipa.example-domain.local.localdomain@EXAMPLE-DOMAIN.HR), referer: https://ipa.example-domain.local/ipa/ui/
It's immediately obvious that something is wrong with the name of IPA server and that somehow .localdomain was appended!? At first, I thought that the problem is in the Firefox and that the value of keys network.negotiate-auth.trusted-uris and network.negotiate-auth.delegation-uris have to end with a dot so that no domain is appended. But quick test showed that I was wrong, when I added dots there nothing worked any more. :)

So, I thought that there must be something on a server that causes that behavior. And then, I looked into /etc/resolv.conf and there it was:
search localdomain example-domain.local
So, this search statement cause localdomain to be appended to the IPA's FQDN. So, I removed that statement and tried again, but the error was still there. Then, it occured to me that Apache probably memorized the statement so I restarted it. And, lo and behold, everyting worked.

You might wonder from where came this search statement. Well, I play tricks with my network setup, and in this case DHCP was used to obtain list of DNS servers which later I manually changed into 127.0.0.1. But, I forgot to remove search statement and so the error occurred. Playing games with network setup obviously bites sometimes... ;)

About Me

scientist, consultant, security specialist, networking guy, system administrator, philosopher ;)

Blog Archive