arpwatch on multiple interfaces

I'm regularly using arpwatch on all servers I install in order to track MAC changes and to notice potential MAC spoofings. But the problem is that on CentOS 6.2 the startup script shipped with arpwatch (package arpwatch-2.1a15-14.el6.x86_64) doesn't support multiple interfaces. More specifically, I can tell arpwatch on which interface to listen by modifying OPTIONS variable in /etc/sysconfig/arpwatch file and inserting -i <interface> option. But, I'm still restricted to a single interface. That is, it is possible to specify multiple -i options, but arpwatch still listens only on a single interface. I checked that in the source (version 2.1a15), and the last -i command is in effect, the previous one's are ignored.

So, I modified startup script so that it now accepts INTERFACES variable within /etc/syconfig/arpwatch configuration file and starts arpwatch on each specified interface. If this variable isn't defined then it behaves as before. For example, to start it on interfaces eth0 and eth1 you should add the following line in /etc/syconfig/arpwatch:

INTERFACES="eth0 eth1" 

The basic idea behind this change is to start arpwatch tool multiple times, once per each specified interface. Also, to each instance I give different database (arp.dat) so that multiple instances don't overwrite each other data.

Note that the script is a bit rough on edges, i.e. it properly behaves during startup phase, but not on shudown. Also, I embedded fixed path to data files. I'll improve this script in a due course when I find more time, or when it turns out that it's necessary to do so. :)

[20120203] Update: I had a an error in script because of which database files were placed in wrong directory and, as a consequence, arpwatch couldn't write database when it was exiting. Now, the script is updated and it works, furthermore, I tested stoping arpwatch using that script and it also works

Why we should study history...

Oh, it happened so many times! I stumbled upon it already while reading John Day's book titled Patterns in network architecture: A return to fundamentals. In short, at one point something is controversial and in next, it's regarded as a some kind of a rule that people passionately protect! One good example is 7-layer ISO/OSI Reference Model. When it was created it was problematic how many layers there should be, now, it is taken as something set in stone that there are 7-layers, while in reality it is dubious if this is a correct number. I'm certain that there are a large number of similar examples in every area you can think of. What this implies is that we have to always question the correctness of our current knowledge knowing that something might happen by chance, or politics of a certain time, and that ultimately hampers us from making further progress, maybe even clean start.

And today, I found this post. written by Rob Landley. It's ubeliviable! I'm using Unix/Linux for over 20 years now, always knowing there is a split between /bin, /sbin, /usr/sbin and /usr/bin and knowing why it is done so. But I realise now that, till today, I didn't actually know and, what's more, this is again an example of something that by accident becomes a law. What's more interesting is that not once I stumbled upon some heated discussion about file system layout (an example) in which there were proponents of this split with a simple argument that it is a Unix way of things! Boy, how wrong they are! :)

I'm copying this post here for a refence:

Usefull Unix commands, tips and tricks...

Have you seen this question on Hacker News? It's great and actually there are really some useful commands and tips for use of the command line more efficiently. Here I'll list some of them in case you want a quick glance, but be aware that something mentioned there might already be known to me and/or I decided that it's not so important and excluded it.

So, without further ado, here's my list of favorites:

Biseri naših neukih novinara 2...

Eto, nastavak serije bisera potaknut člankom o napadu na Index u Novom listu. Ok, moram priznati da ovaj puta biser nije toliko velik kao prethodni, ali svejedno ga smatram biserom. Međutim, ovom prilikom neću kritizirati samo novinare Novog lista već i Index (iako, praktično gledajući radi se opet o novinarima :)).

Citirani članak piše kako je "najpoznatiji" domaći "haker" napao Index i zamijenio mu naslovnu stranicu. Stranica je nakon napada, a prije obnove, imala sljedeći oblik:

Ok, ajmo sada s kritikama.

How to detect your script is started using su...

I wrote a script that had a problem when started via su command. Actually, this is a script within /etc/profile.d so it is executed when new login shell is executed. I'll write about that problem in another post, but here I'll concetrate on how to detect su command.

But before continuing let me clarify that this is a bit of a misnomer. Namely, the goal is to detect whether current environment is a consequence of user ID switching after login, but since this is almost exclusively done using su command, then I think I can put this title. There is also one more "problem". Namely, all user IDs currently having running processes descended from user id 0. But, we are not going so far with philosophy. :)

I started by thinking/hoping that id command could identify originating user, i.e. real user. But that was not possible since the distinction between real and effective user ids is preserved only via setuid flag on files. So, another approach has to be used. There are three possibilities, each one with its own advantages and shortcomings.

Tvrtki Symantec ukraden izvorni kod...

Symantecu je ukraden izvorni kod nekih njegovih proizvoda! O tome sada bruji cijeli Internet jer se radi o velikom problemu.

Ironija je da proizvođač čija primarna djelatnost je vezana uz sigurnost računala i mreža, i koji bi trebao druge štiti, je sam uspješno napadnut 2006. godine te mu je tom prilikom ukraden izvorni kod nekih njegovih proizvoda. Kradljivci koda prijete da će ga objaviti, no, bez obzira na to da li je javno objavljen ili ne već sama činjenica da napadači imaju kod znači da mogu (i vjerojatno hoće) pronaći sve ranjivosti. U najgoroj situaciji su korisnici pcAnyware proizvoda za koji Symantec preporuča da ga se isključi! Još gore, na jednom tweet-u sam primjetio kako se tvrdi da je Symantec nekoliko godina znao za postojanje tri stražnja vrata (backdoor) u tom proizvodu te da ih nije ispravio.

Ovdje ima još jedna zanimljivost. Naime, stalno se vode rasprave da li je sa sigurnosnog stanovišta bolje objaviti kod ili ne. Pobornici zatvorenog koda tvrde kako na taj način napadači imaju manje informacija te su manje efikasni prilikom napada. Teoretski, takav pristup bi se moglo podvesti pod sigurnost korištenjem prikrivanja (engl. security through obscurity) za koji se općenito tvrdi da nije dobar (plus/minus, pogledajte ovaj post). S druge strane, zagovornici otvorenog koda tvrde kako se objavljivanjem koda postiže bolja kvaliteta dotičnoga. Ja sam osobno negdje na sredini, iako, bliže ovom drugom pristupu. Ovaj incident sa Symantecom pokazuje da u slučaju krađe zatvorenog koda nastaju veliki problemi jer je gotovo sigurno da je kod nesiguran, tj. da ima dosta propusta. Dodatno, slučaj Symanteca (a i Microsofta prije toga) pokazuje da je jako teško zaštititi kod od ciljane krađe. I ne samo to. Prva vijest je bila da je kod ukraden Indijskoj vladi. Ispostavilo se da je ta vijest lažna, tj. da je baš Symantecu ukraden kod. Međutim, ne smije se podcijeniti činjenica da se kod nalazi i u posjedu nekih vlada - a koje bi ga mogle zloupotrijebiti za svoje potrebe!

Sve u svemu, škakljiva tema.

Kritpiranje diskova - razlozi za i neki pravni problemi...

Prilično je jasno kako je trend zadnjih nekoliko godina da prijenosna računala zamjenjuju stolna. Više je prednosti prijenosnih računala, ali imaju i jednu veliku manu. Puno lakše ih je izgubiti ili ukrasti nego što je to slučaj sa stolnim računalima. To samo po sebi ne bi predstavljalo veliki problem da se na diskovima ne nalaze tajni podaci koje, iz raznih razloga, ne bi željeli da vide drugi. To mogu biti osobni podaci, podaci tvrtke za koju radite i slično.

Sada će netko reći da je stavio/la lozinku te da je nemoguće pristupiti računalu bez da se prijavi na sustav. Dodatno, moguće je i da ste administratorski račun dodatno osigurali, primjerice, prilično dugačkom i jakom lozinkom. Problem s tom zaštitom je što ništa ne sprečava napadača da otvori računalo, izvadi disk, premjesti ga u drugo računalo kao dodatni disk te na tom drugom računalu pogleda sadržaj diska! To je jednostavno za napraviti i mora se uzeti u obzir kao vrlo vjerojatni slijed događaja.

Međutim, problem je, sam po sebi, relativno jednostavno riješiti. Treba uključiti enkripciju diskova. To drugim riječima znači da se cjelokupni sadržaj diska šifrira te se prilikom uključivanja računala traži da upišete tu šifru. Ako ne upišete šifru, ne možete pristupiti disku. Ne moram ni govoriti koliko je bitno da tu šifru ne zaboravite! Uglavnom, napadač sada - u slučaju da izvadi disk i stavi ga u drugo računalo - ne može do njegovog sadržaja, nema šifru! Ono što može je pokušati ju pogađati. Zbog toga je vrlo bitno da šifra bude dugačka i dobra. Najbolje da je to cijela rečenica zajedno s interpunkcijskim i numeričkim znakovima. U tom slučaju pogađanje i s najbržim računalnim klasterima može trajati godinama.

I da, ako koristite kriptirane diskove ne smijete hibernirati računalo jer se u tom slučaju hibernira otključani disk! Ako ste postavili lozinku na korisničko ime, tj. potrebno je pisati lozinku prije nego što se mora nastaviti s radom, sigurni ste koliko i ta lozinka. Postoje i dodatne metode uz pomoć kojih bi se moglo na hiberniranom računalu otkriti koja je lozinka koja otključava disk, ali to nije toliko trivijalno.

Što se tiče programske podrške za kritpiranje diskova, imaju ih skoro svi operacijski sustavi, a postoje i slobodno dostupni alati za kritpiranje diskova, primjerice, jedan od poznatijih je TrueCrypt. Prema tome, radi se o prilično pristupačnoj tehnologiji.

Sada dolazimo i do jednog velikog problema. Naime, što je s pitanjem kada kriminalac, ili jednostavno neka osoba koja je potencijalno prekršila zakon, koristi takvu zaštitu te policija ne može do podataka na računalu? Na žalost, mislim da je to kod nas siva zona, odnosno, da policija ne može ništa u tom slučaju. Volio bi kada bi me netko ispravio, ali mislim da je tako. Prije nekih godinu dana bio sam na jednom predavanju na kojemu je pričao jedan naš forenzičar, sudski vještak. Uglavnom, postavio sam pitanje što se dešava u takvim situacijama kada osumnjičeni ne želi otkriptirati disk te se pravi da je, primjerice, zaboravio lozinku. Na žalost, mislim da me uopće nije shvatio što pitam(!) tako da odgovor nisam dobio.

Danas sam naletio na vijest vezanu uz takvu situaciju u SAD-u. U tom slučaju, sudac je "naredio" optuženoj da omogući pristup svome prijenosnom računalu te je rekao da se (famozni) peti amandman ne odnosi na taj slučaj.

Istraga bande iza Koobeface zloćudnog koda...

This is Croatian translation of text published here. I tried to preserve original content and meaning as much as possible. I recommend that if you know English read the original version.

Ovo je prijevod teksta objavljenog na ovoj stranici. Preporučam da prije čitanja ovog teksta pogledate informacije o samom prijevodu.

Uvod: ne postoji savršen kiberzločin

Koobface botnet - proizvod samoproglašene grupe "Ali Baba & 4", odnosno, "Koobface banda" - neprestano terorizira milijune Internet korisnika još od sredine 2008. godine, bez obzira na višestruke pokušaje njegova gašenja.

Istraživanje opisano u nastavku, koje su proveli nezavisni istražitelji Jan Drömeri i Dirk Kollberg, pripadnici SophosLabsa, usredotočeno je na osumnjičene koji se kriju iza jedne od najvećih kiberkriminalnih prijetnji zadnjih godina te na proces njihove identifikacije.

Istraživanje je većim dijelom provedeno od početka listopada 2009. godine do veljače 2010. godine i od tada je dano na raspolaganje različitim međunarodnim agencijama za provedbu zakona (engl. law enforcement agency).

Baš kao i u stvarnom svijetu, savršeni kiberkriminal je mit. Činjenica je da se u današnjem kiberkrimalnom miljeu želi postići maksimalna dobit s minimalnim investicijama, i to za sobom povlači svjesno prihvaćanje određene doze nesavršenosti.

Dakle, ta nesavršenost uz aroganciju kriminalaca (and an uncontrollable threat environment such as the internet) je u konačnici dovela do identifikacije više osumnjičenih koji čine Koobface bandu.

Prijevod teksta o istrazi Koobeface zloćudnog koda...

Danas sam pročitao tekst u kojemu se opisuje istraga koju su obavili Jan Drömeri i Dirk Kollberg i čiji rezultat je bilo otkrivanje pojedinaca koji se nalaze iza Koobeface zloćudnog koda. Kao rezultat te istrage, nakon njenog objavljivanja, kriminalci su se povukli što je popraćeno i kod nas. S obzirom da je taj tekst izuzetno zanimljiv, iz nekoliko razloga, odlučio sam ga prevesti na Hrvatski kako bi ga približio širem krugu ljudi. Tekst je dosta lagan na tehničkim detaljima, međutim, određen broj njih nije moguće izbjeći. Ako želite, ostavite komentar i mogu dodati opis stvari koje vas bune u nekom od idućih postova ili staviti poveznice na objašnjenja na Hrvatskom jeziku. Bez obzira na trud, siguran sam da bi se na terminologiji Hrvatskog prijevoda, isto kao i na pravopisu, moglo dosta poraditi. (Posebno sam nesretan s prijevodom pojma Cyber - Kiber, ali ne znam bolje). Ako imate kakvih komentara u tom smislu svakako mi pošaljite poruku putem elektroničke pošte.

Inače, ako znate Engleski jezik svakako preporučam da pročitate originalni tekst. Ja sam se, koliko god je to moguće, trudio zadržati originalno značenje teksta. Moguće je da sam pri tome načinio i pokoju grešku pa ako primjetite nešto što ne valja, molim vas da me obavijestite. Inače, nadam se da ne kršim nikakva autorska prava s ovim prijevodom i objavom. Pokušavao sam naći na originalnoj stranici uvjete korištenja, ali nisam mogao. Također sam tražio nekakav link ili adresu elektroničke pošte za kontakt, međutim ni tu nisam bio uspješan. U konačnici, nadam se da će biti dovoljno to što jasno navodim koji je izvor teksta, također da je to pokušaj direktnog prijevoda te sam dao poveznicu na original.

Kao dodatne razloge zašto je taj tekst zanimljiv, osim što opisuje jednu istragu, je da pokazuje kako djeluje kriminalni milje, a također pokazuje kako informacije dostupne na Internetu omogućavaju da se puno sazna o svakome od nas. To je problem anonimnosti i privatnosti koji, sa sve većim korištenjem Interneta, postaje sve značajniji. Međutim, odmah moram istaknuti kako nije niti vrijeme niti mjesto, a nema ni potrebe, za paničarenjem, a posebno ne za izmišljanjem nekakvih teorija urote!

Cybercriminal underground - results of an investigation...

I just finished reading blog post The Koobface malware gang - exposed! This is a fantastic read for a several reasons. First, it shows how one investigation is performed and what information sources were used for that purpose. Next, it shows inner workings of cybercriminal underground which by itself is very very interesting.  Finally, it shows also what can be inferred about every individual that uses Internet, and it emphasizes how important privacy and anonymity are. I strongly recommend that you read it too.

I'm really thinking about translating this into Croatia, but I first have to see if I'm allowed to publish it and if I need to obtain a permission for that.

Anti-SOPA protest... results...

This week there was anti-SOPA protest in which several high profile Web sites took part, e.g. Wikipedia, Google, YCombinator, to name a few. I already blogged about SOPA. PIPA appeared in the mean time and I didn't covered it, but in the end I think it's no better than SOPA.

The blackout event was also covered by all the media in the world and now, even my parents know there was Wikipedia blackout, even though they vaguely knew for what purpose. :)

What is interesting are the effects, and it seems that there was some kind of DDoS attack on Congress. You can see some stats here, that is the only information I found, but nevertheless, numbers are impressive. If I correctly understand, there were over 200,000 calls to Congress that day, quite a huge amount.

Cisco's bug in ARP behavior - A story of firewall configuration oddisey...

Well, a very interesting situation happened to me. I was changing an old firewall with a new one and after switching I added secondary IP address to a firewall's public interface with the intention that all the traffic that was comming to that secondary IP address is redirected to one internal server. All nice except that it didn't work! I created such configuration numerous times and it simply had to work, but it didn't! The similar functionality for the primary IP address worked as expected, but secondary didn't! This was driving me nuts, until I finally figured out what was happening. This is a story of how I finally resolved this problem.

In cases when something isn't working as expected I use Wireshark, or better yet, tcpdump, and start to watch what's happening with packets on interfaces. I use tcpdump as it is more ubiquitous than Wireshark, i.e. available with default install on many operating systems, Unix like I mean. Now using this tool has one drawback, at least on Linux. The point where it catches packets is before PREROUTING chain and there is no way (at least I don't now how) to see packet flows between different chains. This is actually a restriction of Linux kernel so any other tool (Wireshark included) will behave the same.

Additional complication in this particular case was that to debug firewall I had to run it in production. This makes things quite complicated because when you run something in production that doesn't (fully) work as expected there will be many unhappy people and in the end you don't have much time to experiment, you have to revert old firewall so that people can continue to work. In the end this translates into longer debug period as you have relatively short time windows in which you can debug. Oh yeah, and it didn't helped that this was done at 8pm, outside of usual working hours, for various reasons I won't go into now.

So, using tcpdump I saw that packets with secondary address were reaching the firewall interface and they mysteriously disappeared within a box! Naturally, based on that I concluded that something strange is happening within Linux itself.

I have to admit that usually this would be a far reached hypothesis as it would mean that there is a bug in a relatively simple NAT configuration and it had to be due to the bug which would certainly be known. Quick googling revealed nothing at all and added a further confirmation that this hypothesis isn't good. But what kept me going in that direction was that I decided to use Firewall Builder as a tool to manage firewall and firewall rules. This was my first use of this tool ever (very good tool by the way!). The reason I selected that tool was that this firewall is used by one customer for which I intended to allow him to change rules by himself (so that I have less work to do :)). I wasn't particularly happy with the way rules are generated by that tool, and so I suspected that maybe it messed something, or I didn't define rules as it expects me to. To see if this is true, I flushed all the rules on the firewall and quickly generated a test case by manually entering iptables rules. Well, it turned out that it doesn't work either.

The next hypothesis was that FWBuilder somehow messed something within /proc filesystem. For example, it could be that I told him to be overlay restrictive. But trying out different combinations and poking throughout /proc/sys/net didn't help either, the things were not working and that was it!

Finally, at a moment of despair I started again tcpdump but this time I requested it to show me MAC addresses too. And then I noticed that destination MAC address doesn't belong to firewall! I rarely use this mode of tcpdump operation as L2 simply works, but now I realized what the problem was. The core of the problem was that the router (which is some Cisco) didn't change MAC address assigned to secondary IP address when I switched firewall. What I would usually do in such situation is to restart Cisco. But, since this router was within cabinet that I didn't have access to, and also I didn't have access to its power supply, it was not an option. Yet, it turned out that it is easy to "persuade" some device to change MAC address, just send it a gratuitous ARP response:

arping -s myaddr -I eth0 ciscoaddr

Both addresses are IP addresses, with myaddr being the address for which I want to change MAC address and ciscoaddr is device where I want this mapping to be changed. And that was it! Cisco now had correct MAC address and thing worked as expected. The primary address worked correctly because firewall itself sent a packet to Cisco and in that way changed MAC address belonging to primary IP address.

To conclude, this is a short version of everything that happened as I also used iptables' logging functionality (that obviously didn't help, as there was nothing to log for a start :)). Finally, there's left only one question to answer, i.e. How did I saw packets with "wrong" MAC address it tcpdump output? First, switch shouldn't send it to me, and second, interface should drop them before OS sees them (and by extension tcpdump). Well, switch was sending those frames to me because it's entry for MAC address expired and it didn't know where the old MAC address is, so it sent every frame to all the outputs/ports (apart from the receiving one, of course). The reason input interface didn't drop the packet was that sniffing tool places interface into promiscuous mode, and so it sees every frame that reaches it. Easy, and interesting how things combine to create problem and false clues. :)

Biseri naših neukih novinara 1...

S vremena na vrijeme naletim na naslov, ili članak, naših vrlih novinara koji jednostavno zrači neizmjernom glupošću i besmislenošću. I to nije baš tako rijetka pojava. Uglavnom, kada naletim na takve stvari napisat ću pokoji post i ovo je prvi u toj seriji.

Naslov koji me je inspirirao je Poznati Hrvati na UDARU: Ljubi Jurčiću HAKIRALI karticu s dnevnim limitom od 7000 eura objavljen u Jutarnjem listu. Pa da malo seciramo i analiziramo taj naslov i njegove implikacije.

Prvi komentar je sitnica,kaže Hrvati (množina), a onda samo navode Ljubu Jurčića. Hm, neobično, nisam baš neki jezičar, ali to mi zvuči čudno. No dobro, iz teksta je jasno da je više ljudi oštećeno u tom napadu, a da su iz nekog samo njima znanog razloga istaknuli Ljubu Jurčića.

Drugo, i sada idemo na bitne stvari, navedeni naslov implicira da su napadači, na neki način, direktno napali Ljubu Jurčića! To je debilana (u nedostatku bolje riječi)!!! Naime, navedeni korisnik je samo kolateralna žrtva napada koji je zadesio neku tvrtku. U ovom konkretnom slučaju radilo se o napadu na tvrtku Stratfor.

Treće, kažu "HAKIRALI karticu"?!?! Kako se točno hakira kartica!? Broj kartice je ukraden, zajedno s dodatnim podacima koji omogućavaju da se obave neovlaštene transakcije na računu!

Osim toga, naslov implicira još kako odjednom napadači direktno ciljaju "poznate" Hrvate, kao da ti napadači znaju tko je Ljubo Jurčić, ili općenito tko su "poznati" Hrvati - što god značio taj izraz! :)

I za kraj, lako je kritizirati, treba biti i konstruktivan. :) Pa evo prijedloga "normalnog" i nepretencioznog naslova: Poznati Hrvati žrtve hakerskog napada  na tvrtku Stratfor. Inače, jedne Australijske novine su napisale baš tako: The Australian: Leading Aussies victims of Stratfor hacking.

Ostale postove iz ove "serije" možete pronaći ovdje.

Problem with avformat_open_input()

I lost too much time because of a misleading error message reported by avformat_open_input function! I used the following code that exhibit the error:

AVFormatContext *pFormatCtx = NULL;
...
ret = avformat_open_input(&pFormatCtx, argv[1], NULL, NULL);
if (ret < 0)
   print_error_and_exit(ret, "avformat_open_input()");

print_error_and_exit() is a helper function that uses av_strerror() to give textual representation of the error code stored in ret. Running this code produced the following output:

$ ./a.out infile.wav outfile.wav
avformat_open_input(): No such file or directory

But the infile.wav was there! Using strace I found that open system call wasn't called and so it was certainly an internal error. This was frustrating! The reason I started to write this code was to find out why I'm getting another error in VoIP tool simulator, but I was stuck on something even more basic: Not being able to open a wav file. Googling around I finally found the following link which explained me a real cause of the error, i.e. I forgot to call av_register_all() initialization function.

This again shows how important good error reporting is, in both libraries and application programs!

I had also another problem with the previous code. It was segfaulting within avformat_open_input(). At first, I thought that I found a bug within a library but then I realized I forgot to initialize pFormatCtx to NULL. Namely, allocating a dynamic variable on stack didn't zero it so it was non-NULL and avformat_open_input() misbehaved. Mea culpa! :)

OSSEC active response...

This is still work in progress (I need to add more about configuration part). But since OSSEC is so badly documented and I don't know when this will be finished I'm publishing it now.

Prompted by the problem caused by the OSSEC active response, I decided to try to debug why an error is occurring in logs and fine tune it. But in order to do that I had first to understand how it works. There is a section in OSSEC manual about active response, but what wasn't immediately clear to me is who runs active response and where this is configured. But soon I found out that the active response is run on the client via agent (actually this part wasn't problematic) but that active response is configured on the server and the server is the one that instructs agents to run active response.

On server you'll find active response configuration in $OSSEC/etc/ossec.conf. In that file you have several parts of the configuration:

1. Set of <command> blocks. Each one defines a command for active response and the arguments it expects. Note that those commands have to exist on agents.
2. Set of <active-response> blocks that define circumstances under which there will be active response and which command will be executed.

Scripts for active response

Scripts for active response receive five arguments. The first argument is either add or delete. If add is specified, given IP address should be added to a ban list, in case delete is specified address should be removed from the ban list. The second argument is a user name. The third argument is offending IP address. Fourth argument is Unix timestamp (microsecond resolution) when the script was called. Final argument is rule number that triggered active response.

firewall-drop.sh script

This script ships with OSSEC and it adds or removes IP addresses from firewall. It is a relatively simple shell script that accepts command line arguments as specified in the introduction of this section and installs IP address in the ban list (or removes it from there, depending on the command line arguments). The script could be run on Linux, FreeBSD, NetBSD, Solaris and AIX. The following description is specific to Linux behavior even though some things will be common across different platforms.

This script logs its activity into active-responses.log file (in my case in /var/ossec/logs directory). For each invocation one line is emitted into that file. Here is an example of one log entry:

Thu Jan 19 13:52:29 CET 2012 /var/ossec/active-response/bin/firewall-drop.sh add - 193.41.36.141 1326977549.1358625 3301

First group of fields is timestamp when the log entry was generated. Next is a full path and name of the script itself. Finally, all five arguments given to script are also recorded.

Ocasionally, you'll also see error messages like the following one:

Thu Jan 19 06:17:19 CET 2012 Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - 208.115.236.82 1326949601.551727 3301

This log entry is a bit misleading. What happened is that iptables command returned exit code 1 (judging from the log entry it could be interpreted as if it returned something else and 1 was expected, but that's not true). What is important to note is that you'll usually see multiple log entries like the previous one grouped together and the only thing that will differ between them is the number 1 (shown in bold above). What basically happens is that in case of an error returned by iptables command the script tries to run it five times, so, you'll usually see five records and each of those records is numbered.

There are two places where this error might occur. The first one is when the IP address is removed from INPUT chain, while the other is when it is removed from FORWARD chain.

The only reason this error can occur is because someone or something already removed IP address (or added it). But, this should not happen. Still, it happened to em but I don't know the reason for that.

Looking into this script it was clear that it could be improved from the logging perspective. Currently, if you manually run this command from the command line it will write part of error messages to stdout and some other to log file.

Manual control of scripts

Scripts for active response can be started from the server using agent_control tool. Note that the help message of this tool isn't updated to reflect real arguments (bug?) so I had to look into source to infer how to call it. Let me give you several examples of its use.

To list all available agent use this command in the following way:

# ./agent_control -l
   ID: 000, Name: agent0.somedomain.local (server), IP: 127.0.0.1, Active/Local
   ID: 001, Name: agent1.somedomain.local, IP: 192.168.1.2, Active
   ID: 002, Name: agent2.somedomain.local, IP: 192.168.1.3, Active
   ID: 003, Name: agent3.somedomain.local, IP: 192.168.1.4, Disconnected

The output of the command is a list of known agents, either active or non-active. In case you want only active agents use -lc option instead.

Next, if you want to find out some information about a certain agent, you can query it in the following way:

# ./agent_control -i 002

OSSEC HIDS agent_control. Agent information:
   Agent ID:   002
   Agent Name: agent2.somedomain.local
   IP address: 192.168.1.3
   Status:     Active

   Operating system:    Linux agent2.somedomain.local 2.6.32-131.17.1.el6.x86_64..
   Client version:      OSSEC HIDS v2.5-SNP-100907
   Last keep alive:     Thu Jan 19 13:26:01 2012

   Syscheck last started  at: Thu Jan 19 12:10:16 2012
   Rootcheck last started at: Thu Jan 19 06:33:06 2012

To activate active response on a certain agent use the following form of the agent_control command:

./agent_control -b 1.1.1.1 -f firewall-drop600 -u 002

Here, the IP address to be blocked is argument of the -b option (in this case 1.1.1.1). There could be more responses available (defined in ossec.conf on server) and the option -f selects which one to run. To see which responses are available use the option -L, like this:

# ./agent_control -L

OSSEC HIDS agent_control. Available active responses:

   Response name: host-deny600, command: host-deny.sh
   Response name: firewall-drop600, command: firewall-drop.sh

Agent on which this command should initiate active response is specified via ID given as the parameter to option -u. Note that, if you look into help output of the agent_control, this option is not listed, at least not in the version 2.6.0. There is a bit of inconsistency here, as some commands use agent ID as a parameter, while this one requires separate parameter. It would be more uniform if all command would instead use -u option.

Note that when you manually initiate active response then fourth argument to the script will be '(from_the_server)' and the fifth argument will be '(no_rule_id)'.

Also, the rule that was added can not be removed with agent_control, you have to wait for it to timeout when it will be automatically removed.

Interesting problem with OSSEC, active response and mail delivery...

We had a problem that manifested itself in such a way that mail messages didn't come from certain domains, or more specifically from certain mail servers. Furthermore, no clue was given in the mail log to know what went wrong and to make things worse, logs from the remote mail server were inaccessible to see there what actually happened. Finally, the worse thing was that this happened sporadically. It turned out that this was consequence of a circumstances and a bug with ossec active response. This post explains what happened.

We changed DNS domain several months ago, let me call the new domain newdomain.hr, and the old one olddomain.hr. DNS was reconfigured so that it correctly handled requests for a new domain, but we had to leave old domain because of some Web server. The the old domain was changed so that when someone asked which is a mail exchanger for olddomain.hr it would receive response: mail.newdomain.hr. Finally, domain olddomain.hr was removed from the mail server. This was the first error, and now I think that it is better either to leave old domain on mail server or to not return any response! Actually, if you want to get rid of the old domain, it is the best to remove it from the mail server and that DNS server doesn't return any response for a mail exchanger of a given domain. If you know how mail works, you'll know that by changing MX record for old domain from mail.olddomain.hr to mail.newdomain.hr didn't change anything!

Anyway, that's the part concerning mail. Now, about OSSEC. It has a possibility of active response, i.e. to block offending IP addresses for a certain amount of time, 10 minutes by default. One class of offending IP addresses are those that try to deliver mail messages which require mail server to be open relay. Since mail server is properly configured it rejects those messages with a message 'Relay denied'. After mail server rejects  such delivery attempt OSSEC kicks in and blocks offending IP address for 10 minutes.

This, by itself didn't have to be a problem because blocking rules are automatically removed after 10 minutes. But, there is a bug in the removal script that manifested itself in the logs like follows (found on agent in /var/ossec/logs/active-responses.log):

Unable to run (iptables returning != 1): 1 - /var/ossec/active-response/bin/firewall-drop.sh delete - 203.83.62.99 1326738019.2370422 3301

For some reason removal of IP address from block list wasn't successful and that basically meant that the source mail host is blocked indefinitely!

Majority of mail servers that to generate such 'Relay denied' messages are truly spammers and if some of them were indefinitely blocked that was actually good. But, this particular source mail server that was blocked is very popular one with many users serving many different domains, so now when some other user tried to send an email that was legal and had correct address, IPtables blocked access and the mail couldn't be delivered. There was nothing in the logs of destination mail server. Also, sending user didn't receive any response message since mail was being temporary put on hold on the source server.

This particular problem was solved by completely removing the old domain. Now, source mail servers won't even try to deliver mails for the old domain and thus OSSEC won't block legitimate servers. Furthermore, the sending users will get notification immediately about non-existent mail address.

About Git and Perforce..

All this started with a reading of a simple post why Java is slower that C. This is actually interesting, and controversial, question which I'm going to address in another post, but this time I ended up in reading what different developers are saying about git and perforce. It's also interesting to find out about different repositories that exists, and I mean on those that are proprietary. The source for the material in this post, apart from the mail that started it, was this blog post, and this presentation from Google. I also looked at Perforce's user conferences materials.

Perforce is code versioning system popular in many companies, i.e. Google, FreeBSD, RIM, and many others that is a proprietary product with a hefty price tag (somewhere between $740 and $900 per user per year). There are free and trial versions available, and as far as I know, clients are free. Also, I think that Perforce offers free versions of software to open source projects. In essence, Perforce is not a distributed version control system. It requires a user to have a centralized repository/machine that will host all the necessary data. Furthermore, it seems as there is no way to have some kind of a cluster to host that repository, at least not at the application level (i.e. Perforce). That was true several years ago and could be true today. Because of all this if you look at what's the main talk on Perforce User Conferece you'll quickly conclude that majority of talks are about performance and how to achieve it. It is also interesting to note that I stumbled on one presentation that claims it is better to host Perforce repository on Linux server than on Windows.

Git on the other hand is a completely free and extremely popular distributed version control system. The most known application of Git is as a repository for the Linux kernel, but it's used all over the map, both for open source projects and for closed source projects. Git is specifically aimed for revision control of source files and not binary files.

What I concluded from reading all the comments about git vs. perforce is that Perforce behaves better in case binary objects are in repository too. This is a primary use case for game development that requires all the artwork (video, audio, picture) to be attached to specific source version. Git wasn't designed to handle that use case even though there were discussions about that in 2009 (see the thread that started my inquiry in all this). But git is praised for its branching capabilities as well as some other capabilities not present in Perforce. In essence, those capabilities are the consequence of its distributed nature. For example, you can send your local changes to other developer without committing the changes to some central server. Also, Perforce allows (and forces) developers to mark the files as being edited. This makes it possible to find out who is editing which file. But it creates some problems, e.g. scalability.

One more interesting thing I learned is that Linux is actually a small project. :) This was a surprise to me, but then, after second thought it seems natural, if nothing else than for games that have to put a lot of non-programming stuff into the repository. Reported sizes of the repositories are from 6GB up to 100GB as Google claims in their presentation from 2006 (the one linked above). This is impressive, and what occurred to me is that it would be interesting to do some kind of survey of the sizes of repositories and their use cases. Anyway, it is particuarly interesting to know that Google runs largest Perforce server with over 12000 users.

While looking for information about Perforce I also found out that there is competitor called ClearCase. But, according to what I've found ClearCase is more expensive than Perforce.

For the end, I have to say this is very interesting subject that woule be interesting to do some kind of a survey. Wikipedia, as usual, has very good comparison chart of major version control systems that could be a good starting point for anyone's research into version control systems.

Configuration management...

Anybody that ever administered more than few machines knows how hard it is to keep them maintained. Especially when you have a mix of different operating systems, and I mean different like Windows, Linux, AIX, HP-UX, etc. For that purpose many different configuration management applications were written, but there is no perfect solution. In case you are running Windows environment, then I have to admit that Active directory, along with several supporting tools is actually very good tool. On the Unix scene, there is much more diversity and, apparently, not such a good solution.

I stubled upon this blog post that compares different CM applications, most notably puppet, cfengine and bcfg2. It turns out that in comments section you'll find info and links to a lot more, e.g. cdist, chef, and many others.

Finally, I looked on Wikipedia while writing this blog post and found that there is a page devoted to the comparison of different CM application. I have to admit that Wikipedia is really becoming an ubiquitous source of information, despite occasional inaccuracies.

Keyboard autorepeat...

It happens from time to time that the keyboard autorepeat suddenly stops working. It just happened to me after I used VMware with Windows 7 as a guest operating system. I don't believe it's Windows' fault. Actually, I'm quite certain that it isn't! :) It's probably fault of VMware's mouse and keyboard driver that allows seamless integration of desktops.

Anyway, it is very annoying when autorepeat doesn't work, so I Googled a bit and found out that using 'xset q' should show me wether the autorepeat is on or off. What I got was:

$ xset q
Keyboard Control:
  auto repeat:  off    key click percent:  0    LED mask:  00000000
  XKB indicators:
...

So, it was obvious that autorepeat is turned off. It was relatively straightforward to find how to turn it back on. It's just necessary to run 'xset r', and it was on again:

$ xset r
$ xset q
Keyboard Control:
  auto repeat:  on    key click percent:  0    LED mask:  00000000
  XKB indicators:
...

That made me very happy as I didn't need to restart X session. It's a nightmare to restart X (or just logout) since I have over hundred opened tabs in Firefox...