Logging: Do they think?

Last week I had to discover the reason for some obscure log entry in ESXi vmkernel log. It looks something like this:

Nov  2 11:34:36 vmkernel: 16:21:06:29.145 cpu2:2446675)WARNING: UserObj: 569: Failed to crossdup fd 6, fs: def5 oid: 19000000030000003 type CHAR: Busy

Do I need to say that I found nothing?

This is very nice entry, on which, if you try to use your best google-fu, you'll end up with nothing. And this is only a single example. There are multitude of such log entries, not only from vmware but from almost every possible software vendor.

So, what is exactly the problem, you may ask? Well, the problem is that there are many variable parts that depend on a particular installation!  If you try to paste string into google search it will find nothing! This is a problem, not only for novice users, but also for experienced ones. If I have an error message that includes a single word, PID, pathname, maybe IP address or something like that, this is completely useless to google for. Because when you remove all the variable parts you end up with nothing.

Now, administrators/users could agree to always use some standard address (e.g. 1.1.1.1) but this is also not a solution. First, there has to be many predefined parameters, and second, no one could keep with them! And yes, in addition people are very good in making exceptions (i.e. being ignorant).

So, what is the conclusion?

Well, the conclusion is that those that produce applications have to think about logging and that today each long entry will be googled for! Furthermore, this is in their best interest. Beacuse, if users/administrator quickly find the solution for some problem they will be happier (and regard your software better) and in the same time pressing on helpdesk will be lower. Both cases translate into money gain!

And what is the recommendation?

Well, I recommend software vendors to carefully design log messages that contain some fixed and unique string that will be easy to find in google. Variable parts should be separated.

Finally, I have to say that I particularly hate log4j logs. It's not that they are bad, they are wonderful, but only if you are a developer! Again, if you try to google for them, or to process them in OSSEC, you'll have big problems.