I'm using OSSEC for log monitoring, and while it is a great tool there are some drawbacks. One is that (not so) occasionally Java stack traces are dumped into syslog and they are not properly handled/parsed by OSSEC. The other drawback is that I want OSSEC to monitor mod_security logs, which are multi line logs with variable number of lines belonging to each HTTP request/response processed by mod_security. So, I wanted to modify OSSEC in order to allow it to handle those cases, too. To be able to modify OSSEC I started with the analysis of its source (version 2.6) and based on the analysis, I wrote this text. The goal of the analysis was to get acquainted on how OSSEC handles log files, document this knowledge, and to propose solution that would handle Java stack traces and mod_security logs.
Configuring log files monitoring
First, ehere are some global parameters that control global log monitoring behavior of each OSSEC agent, or server. More specifically,
logcollector.loop_timeout in
internaloptions.conf defines how much time (in seconds) OSSEC will wait before checking if there are additions to all log files that it monitors. Default value is 2s, with minimal allowed value of 1 second and maximum 120s.
This is actually the most important parameter. There are two additional ones,
logcollector.open_attempts with allowed values between 2 and 998, and
logcollector.debug with allowed values 0, 1 and 2.
Supported log types in OSSEC
If I correctly understand it, OSSEC treats log files as consisting of records. Each record being independent from the previous, or any later ones. Rules, then, act on records. In the majority of cases the record is identical to a single line of a log file. This is, I suppose, legacy from syslog in which each line was (and still is) separate log entry. In the mean time OSSEC was extended so that it supports more different log files:
- multiline log (read_multiline). The problem with this log format is that it expects that each record consists of a fixed number of lines.
- snort full log (read_snortfull). This is the closest one to what I need with respect to mod_security, i.e. this one reads several lines, combines them, and sends them to log analyzer.
- some others I won't analyze now.
The length of the record is restricted to 6144 characters (constant
OS_MAXSTR defined in
headers/defs.h). Everything above that length will be cut off and discarded, with an error message logged.
Configuring log files
So, there are different log types and to tell ossec for some file which type it is, you use
<logformat> element associated with each monitored file. Each monitored file is defined in element
<localfile> directly beneath
<ossec_conf> element in the ossec configuration file, for example:
<ossec_config>
...
<localfile>
<log_format>syslog</log_format>
<location>path_to_log_file</location>
</localfile>
...
</ossec_config>
The configuration file, along with all of its elements, is read by configuration loader placed in
src/config subdirectory. The
localfile element is processed in function
Read_Localfile (file
localfile-config.c). For each
<localfile> one
logreader structure (defined in
localfile-config.h) is allocated and initialized. This structure has the following content:
typedef struct _logreader
{
unsigned int size;
int ign;
#ifdef WIN32
HANDLE h;
int fd;
#else
ino_t fd;
#endif
/* ffile - format file is only used when
* the file has format string to retrieve
* the date,
*/
char *ffile;
char *file;
char *logformat;
char *djb_program_name;
char *command;
char *alias;
void (*read)(int i, int *rc, int drop_it);
FILE *fp;
}logreader;
Analyzing
localfile-config.c file I came to the following conclusions:
- Underneath <localfile> element the following elements are accepted/valid: <location>, <command>, <log_format>, <frequency>, and <alias>.
- <location> defines the file, with full path, that is being monitored.
- <log_format> tells OSSEC in which format are records stored. The following are hardcoded values in the Read_Local.c file: syslog, generic, snort-full, snort-fast, apache, iis, squid, nmapq, mysql_log, mssql_log, postgresql_log, djb_multilog, syslog-pipe, command, full_command, multi-line, and eventlog. What's interesting is that some of those (marked in bold) don't appear later in log collector!
- the syntax of accepted multi-line log_format is:
<log_format>multiline[ ]*:[ ]*[0-9]*[ ]*</log_format>
note that it is not an error, i.e. you can avoid writing any number, but that number defines how many lines should be concatenated before being sent to analyzer module, so it is important you don't skip it. The number part is made available in logformat field of logreader structure. Furthermore, when check is made to determine which type of logformat is some logff structure, to detemine it is a multiline the first character is checked if it is a digit. If it is, then it is a multiline format. This is actually a hack because in the original design it wasn't planned to have parameters to certain log types!
- <command> is used to run some external process, collect its output, and send it to central log. Each line outputed by the command will be treated as a separate log record. Empty lines are removed, i.e. ignored. Note that command is passed to shell using popen library call. So, you can place shell commands there too.
- in the code there is a reference to a full_command log type format, but it is not supported in the configuration file. The difference is that this form reads command's output as a single log record.
- the value of element <frequency> is stored into logreader.ign field. But the use of this field is strange because it is overwritten with number of failed attempts to open log file. I would assume that this parameter would, somehow, allow rate limiting.
- When defining the value of <location> element certain variables can be used. In case of Windows operating systems, that means % variables (e.g. %SYSTEM% and similar). In case of Unix/Linux glob patterns are allowed (these are not allowed on Windows). Also, strftime time format specifiers can be used and in that case strftime function will be called with time format specifier and current local time to produce final log's file name.
- <alias> defines an alias for a log file. It is used in read_command.c and read_fullcommand.c files, only. Probably to be substituted instead of command in the output sent to log collector (for readability purposes, instead of the whole complex command line you see just its short version).
- The function pointer should point to a function returning nothing (void). But, all the read functions return void * pointer and this return value is forced during assignment of a function to this field. Finally, this return value is never used.
- The return code (*rc parameter) of read functions is also used in a strange way and only once it is different than 0.
Reading and processing log files
Actual monitoring of log files is done by logcollector module (contained in the
src/logcollector subdirectory). If you look at process list you'll see it under the name
ossec-logcollector.
C's
main() function of Logcollector module obtains array of logreader structures from config module and calls main function
LogCollectorStart().
LogCollectorStart() references logreader structures through
logff array pointer.
logreader structures are then initialized.
Functions to read different type of files are placed in separate files prefixed with string
read_. Each file has one globally visible entry function that has the following prototype:
void *read_something(int pos, int *rc, int drop_it);
The function return value isn't used, and most (but not all!) functions just return NULL.
pos is index into structure that defines which particular file should be checked by the function. Basically, it indexes array
logff. So,
logff[pos] is the log file that has to be processed.
rc is output parameter that contains return code. Finally,
drop_it is a flag that tells reader function to drop record (
drop_it != 0) or to process it as usual (
drop_it == 0).
So, the conclusion is that I should/would create the following log readers:
- regex reader that uses regex to define start of the block, so everything between one line that matches given regex is treated as a log record until the first line that matches the regex again. The variation of the theme is to have separate regex for the end of the block.
- modsec_audit reader. A separate log reader that would combine the multiline output of modsec into a single line/record understood by ossec. In particular, I'll read only audit log of mod_security, there is also debug log which I'll ignore for the time being.
Plan
So, as I said, the goal was to solve Java and mod_security log problems. While studying the source I decided to implement two log readers. I'll leave Java for now, since I think it is better solved with modifying syslog (concatenating next line if it doesn't start at column 1). So, one reader that will process mod_security's audit log files and another one that will use regex to search for start and end of each record. To do so, obviously those two have to be implemented, but also appropriate configurations has to be defined.
Since I definitely decided to go with attributes instead of hacks (as multiline is) I also decided to convert multiline to use attribute to define number of lines that has to be concatenated.
So, new multiline definition in the configuration file will be:
<log_format lines="N">multiline</log_format>
And the attribute
lines is mandatory, it defines how many lines in the log file makes one log record! On the other hand,
mod_security for now will use very simple configuration style:
<log_format>modsec_audit</log_format>
Finally,
regex based log type will use the following type:
<log_format start_regex="" end_regex="">regex</log_format>
And if
end_regex isn't specified, then the value of
start_regex will be assumed to be also
end_regex.
start_regex attribute is mandatory.
Well, basically, that's for the plan.
Implementation
I first implemented code that reads and parses configuration. In order to do that I had to change files
src/config/localfile-config.[ch] and
src/error_messages/error_messages.h files. Note that I introduced a new field
void *data into
logreader structure who's purpose is to keep private data of each log reader. In that way I'm not cluttering structure with lots of attributes used only sporadically. Alternatively, I could use union and place everything there, but for the time being this will do.
Then I modified
src/logcollector/read_multiline.c to take its parameter from a new place in logreader structure, i.e. from private
*data pointer. The small problem with how I did it is that it is dependent on 32/64-bit architecture as I'm directly manipulating with pointers, which is actually one big no-no. :) But, for the prototype it will do.
Next, I copied
read_multiline.c into
read_modsec_audit.c and modified it to work with mod_security audit logs. Note that mod_security, for each request and response (that are treated as a single record) creates a series of lines and blocks. Block are separated by blank lines, but everything is kept between the following lines:
-- 62a78a12-A--
...
-- 62a78a12-Z--
Blank lines before and after those that mark beginning and an end are ignored. Random looking number (
62a78a12) is an identifier that binds multiple blocks together in log files. In my implementation I assume there is no interleaving of those!
Finally, I implemented
read_regex.c. This one is the same as the
read_modsec_audit.c but instead of fixed delimiters for start and the end or a record, delimiters are provided via configuration file in a form of regular expressions. Note that it would be possible to make read_modsec_audit a special case of read_regex via appropriate use of regular expressions in the configuration file.
When modules were finished I just had to integrate them into
logcollector.[ch] (and modify how multiline is detected). And the implementation was finished.
All the changes are provided in
new_log_types-v00.patch.
Some conclusions and ideas for a future work
There are few shortcomings of this implementation. First, it is implemented on Linux and only slightly tested even on that platform. So, it is very likely buggy and non-portable across different platforms. Next, there is certainly a non-portable part with respect to 32-bit vs. 64-bit pointers. I marked that part in the code. Finally,
security review has to be done, after all, this is security sensitive application!
It seems to me that multiline module doesn't seem to work right, i.e. there are some corner cases when it misbehaves. Namely, there is a check that a single line doesn't exceed maximum line size, but there is no check if more than one line exceeds that threshold. And, probably, if it exceeds, then the reading will get out of sync, i.e. wrong lines will be grouped together.
For
regexp module written as a part of this treatise probably additional attributes should be included, like
flags so that you can say, e.g., if the match should be case sensitive or not. Or, that you can remember matches from
start_regex (using () operator) and reuse them in
end_regex regular expression.
For the end, I don't want to criticize too much, but the build system of OSSEC isn't what you would call: flexible. This isn't a problem for production environment, but for development it is since, to test a single change, you have to rebuild all the source. Also, OSSEC assumes you run it from the installation directory, which is owned by a root. Again, this is a problem for a development, more specifically testing. I think there is a lot of room for the improvement.
