Installing ossec client on CentOS 6...

Ok, I did this already, but I managed to forget it. Still, it isn't strange, after all, it's not that you are adding new machines every day. Anyway, here are the steps that are need in order to install OSSEC client on a CentOS machine, more specifically CentOS 6. I decided to write this post if someone also needs these instructions, but certainly for me so that next time I have to do it I don't have to think a lot. Note that I like to install RPM packages because it is easier to update them instead compiling from source, and also someone else is worrying about new releases. Additionally, it's not so good to install development environment on production machines that don't need it, for security reasons. Ok, here we go.

First, make sure that you have EPEL repository added. The easiest way to do this is using the following command (note, bold is what you type, the rest is what you get from the machine):

# rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Retrieving http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
warning: /var/tmp/rpm-tmp.7IMdWB: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing...                ##################################### [100%]
1:epel-release   ##################################### [100%]

Second, fetch necessary packages. I didn't want to install Atomicorp's repository, so I only fetched ossec packages using wget. ossec-hids and ossec-hids-client are what you need. Select the newest versions you can find. Next, install them using yum command:

# yum localinstall ossec-hids-client-2.6-15.el6.art.x86_64.rpm ossec-hids-2.6-15.el6.art.x86_64.rpm

I assumed that yum is executed in the same directory where you placed downloaded packages. Also, if you downloaded some other versions, change names appropriately.

Open ossec's configuration file, /var/ossec/etc/ossec-agent.conf, and change the line that has <server-ip></server-ip> element. It has to point to your server's IP address. You can also add files to be monitored in addition to the existing ones, or remove some of the existing ones if they are not used on the machine you are installing ossec client.

Now, go to the OSSEC server and run there agent management tool. It is probably in /var/ossec/bin:

# ./manage_agents


****************************************
* OSSEC HIDS v2.5-SNP-100907 Agent manager.     *
* The following options are available: *
****************************************
   (I)mport key from the server (I).
   (Q)uit.
Choose your action: I or Q: A

- Adding a new agent (use '\q' to return to the main menu).
  Please provide the following:
   * A name for the new agent: centos6.domain.local
   * The IP Address of the new agent: 192.168.10.41
   * An ID for the new agent[030]: <just press ENTER>
Agent information:
   ID:030
   Name:centos6.domain.local
   IP Address:192.168.10.41

Confirm adding it?(y/n): y
Agent added.

Note that the tool doesn't display all the options you have on your disposal. Next what you need to do is to extract a key that you'll import into the client. This is also done using manage_clients tool, so either start it again, or in case you didn't exit after you added an agent just continue:

 ****************************************
* OSSEC HIDS v2.5-SNP-100907 Agent manager.     *
* The following options are available: *
****************************************
   (I)mport key from the server (I).
   (Q)uit.
Choose your action: I or Q: e

Available agents:
   ID: 002, Name: somehost, IP: 10.0.10.1
   ID: 030, Name: centos6.domain.local, IP: 192.168.10.41
Provide the ID of the agent to extract the key (or '\q' to quit): 030
Agent key information for '030' is:
<here a very long string will be printed>
** Press ENTER to return to the main menu.

Again, option to export the key isn't listed in the help message! Anyway, copy the very long string that is printed (agent's key) and you can quit from the tool and logout from the OSSEC server.

Go now to ossec client, change directory to /var/ossec/bin and run manage_client tool:

# ./manage_client


****************************************
* OSSEC HIDS v2.6 Agent manager.     *
* The following options are available: *
****************************************
   (I)mport key from the server (I).
   (Q)uit.
Choose your action: I or Q: I

* Provide the Key generated by the server.
* The best approach is to cut and paste it.
*** OBS: Do not include spaces or new lines.

Paste it here (or '\q' to quit):
<very long string copied here!>

Agent information:
   ID:030
   Name:centos6.domain.local
   IP Address:192.168.10.41

Confirm adding it?(y/n): y
Added.

Finally, restart ossec client:

# /etc/init.d/ossec-hids restart
Shutting down ossec-hids:                      [  OK  ]
Starting ossec-hids:                           [  OK  ]

You should see you new client in OSSEC's Web interface which should confirm that it is running OK.

OSSEC active response...

This is still work in progress (I need to add more about configuration part). But since OSSEC is so badly documented and I don't know when this will be finished I'm publishing it now.

Prompted by the problem caused by the OSSEC active response, I decided to try to debug why an error is occurring in logs and fine tune it. But in order to do that I had first to understand how it works. There is a section in OSSEC manual about active response, but what wasn't immediately clear to me is who runs active response and where this is configured. But soon I found out that the active response is run on the client via agent (actually this part wasn't problematic) but that active response is configured on the server and the server is the one that instructs agents to run active response.

On server you'll find active response configuration in $OSSEC/etc/ossec.conf. In that file you have several parts of the configuration:

1. Set of <command> blocks. Each one defines a command for active response and the arguments it expects. Note that those commands have to exist on agents.
2. Set of <active-response> blocks that define circumstances under which there will be active response and which command will be executed.

Scripts for active response

Scripts for active response receive five arguments. The first argument is either add or delete. If add is specified, given IP address should be added to a ban list, in case delete is specified address should be removed from the ban list. The second argument is a user name. The third argument is offending IP address. Fourth argument is Unix timestamp (microsecond resolution) when the script was called. Final argument is rule number that triggered active response.

firewall-drop.sh script

This script ships with OSSEC and it adds or removes IP addresses from firewall. It is a relatively simple shell script that accepts command line arguments as specified in the introduction of this section and installs IP address in the ban list (or removes it from there, depending on the command line arguments). The script could be run on Linux, FreeBSD, NetBSD, Solaris and AIX. The following description is specific to Linux behavior even though some things will be common across different platforms.

This script logs its activity into active-responses.log file (in my case in /var/ossec/logs directory). For each invocation one line is emitted into that file. Here is an example of one log entry:

Thu Jan 19 13:52:29 CET 2012 /var/ossec/active-response/bin/firewall-drop.sh add - 193.41.36.141 1326977549.1358625 3301

First group of fields is timestamp when the log entry was generated. Next is a full path and name of the script itself. Finally, all five arguments given to script are also recorded.

Ocasionally, you'll also see error messages like the following one:

Thu Jan 19 06:17:19 CET 2012 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - 208.115.236.82 1326949601.551727 3301

This log entry is a bit misleading. What happened is that iptables command returned exit code 1 (judging from the log entry it could be interpreted as if it returned something else and 1 was expected, but that's not true). What is important to note is that you'll usually see multiple log entries like the previous one grouped together and the only thing that will differ between them is the number 1 (shown in bold above). What basically happens is that in case of an error returned by iptables command the script tries to run it five times, so, you'll usually see five records and each of those records is numbered.

There are two places where this error might occur. The first one is when the IP address is removed from INPUT chain, while the other is when it is removed from FORWARD chain.

The only reason this error can occur is because someone or something already removed IP address (or added it). But, this should not happen. Still, it happened to em but I don't know the reason for that.

Looking into this script it was clear that it could be improved from the logging perspective. Currently, if you manually run this command from the command line it will write part of error messages to stdout and some other to log file.

Manual control of scripts

Scripts for active response can be started from the server using agent_control tool. Note that the help message of this tool isn't updated to reflect real arguments (bug?) so I had to look into source to infer how to call it. Let me give you several examples of its use.

To list all available agent use this command in the following way:

# ./agent_control -l
   ID: 000, Name: agent0.somedomain.local (server), IP: 127.0.0.1, Active/Local
   ID: 001, Name: agent1.somedomain.local, IP: 192.168.1.2, Active
   ID: 002, Name: agent2.somedomain.local, IP: 192.168.1.3, Active
   ID: 003, Name: agent3.somedomain.local, IP: 192.168.1.4, Disconnected

The output of the command is a list of known agents, either active or non-active. In case you want only active agents use -lc option instead.

Next, if you want to find out some information about a certain agent, you can query it in the following way:

# ./agent_control -i 002

OSSEC HIDS agent_control. Agent information:
   Agent ID:   002
   Agent Name: agent2.somedomain.local
   IP address: 192.168.1.3
   Status:     Active

   Operating system:    Linux agent2.somedomain.local 2.6.32-131.17.1.el6.x86_64..
   Client version:      OSSEC HIDS v2.5-SNP-100907
   Last keep alive:     Thu Jan 19 13:26:01 2012

   Syscheck last started  at: Thu Jan 19 12:10:16 2012
   Rootcheck last started at: Thu Jan 19 06:33:06 2012

To activate active response on a certain agent use the following form of the agent_control command:

./agent_control -b 1.1.1.1 -f firewall-drop600 -u 002

Here, the IP address to be blocked is argument of the -b option (in this case 1.1.1.1). There could be more responses available (defined in ossec.conf on server) and the option -f selects which one to run. To see which responses are available use the option -L, like this:

# ./agent_control -L

OSSEC HIDS agent_control. Available active responses:

   Response name: host-deny600, command: host-deny.sh
   Response name: firewall-drop600, command: firewall-drop.sh

Agent on which this command should initiate active response is specified via ID given as the parameter to option -u. Note that, if you look into help output of the agent_control, this option is not listed, at least not in the version 2.6.0. There is a bit of inconsistency here, as some commands use agent ID as a parameter, while this one requires separate parameter. It would be more uniform if all command would instead use -u option.

Note that when you manually initiate active response then fourth argument to the script will be '(from_the_server)' and the fifth argument will be '(no_rule_id)'.

Also, the rule that was added can not be removed with agent_control, you have to wait for it to timeout when it will be automatically removed.

Problem with inactive agent in OSSEC Web Interface

I was just debugging OSSEC Web interface. Namely, it incorrectly showed that one host was not responding event though there were log entries that showed otherwise. The problem was that this particular host was transferred to another network, and thus, its address was changed.

I figured out that the list of available agents within Web interface is generated from a files found in /var/ossec/queue/agent-info directory. There, you'll find one file per agent. The file name itself consists of agent name and IP address separated by a single dash. In order to display if an agent is connected or not the PHP code from Web interface (which itself is placed in /usr/share/ossec-wui directory) obtains time stamp of a file belonging to a particular client and if this time stamp is younger that 20 minutes, it proclaims agent OK, otherwise, it shows it as inaccessible.

In this case it turned out that the old agent wasn't removed using manage_client tool (selecting option R, for remove). So, the old information remained, which wasn't updated and thus the Web interface reported inactive agent.