VMWare Workstation DNS server...

I just figured out very interesting thing about DNS server used by VMWare virtual machines. It uses /etc/hosts file on host machine to resolve names. The trick is that those names have to have .localdomain suffix. So, for example, if you have the following entry in /etc/hosts file of host machine:

1.1.1.1        test

and then you ask with nslookup within guest machine for that name, you'll receive:

# nslookup test
Server: 192.168.178.2
Address: 192.168.178.2#53

Name: test.localdomain
Address: 1.1.1.1

Note that localdomain is automatically appended. This is specified with search directive in /etc/resolv.conf of guest machine (there is a line search localdomain or something similar). Otherwise, without localdomain, the name wouldn't be found:

# nslookup test.
Server: 192.168.178.2
Address: 192.168.178.2#53

** server can't find test.: NXDOMAIN

In this example, the dot I appended after the name prevents anything from being appended and only name test is looked for.

To conclude, the rule is simple. If the name to be looked up ends with localdomain, then it is searched within /etc/hosts file of the host operating system.

I discovered this by accident. Namely, I did nslookup within guest machine expecting to receive error message about non-existing host name, but I got good response!? At first, I was confused, and it took me several minutes to figure out what's happening. Well, so the things go when you don't read manuals. ;)

BIND and network unreachable messages...

Sometimes you'll see messages like the following ones in your log file (messages are slightly obfuscated to protect innocent :)):

Jun 29 14:32:11 someserver named[1459]: error (network unreachable) resolving 'www.eolprocess.com/A/IN': 2001:503:a83e::2:30#53
Jun 29 14:32:11 someserver named[1459]: error (network unreachable) resolving 'www.eolprocess.com/A/IN': 2001:503:231d::2:30#53

What these messages say is that network that contains address 2001:503:231d::2:30 is unreachable. So, what's happening?

The problem is that all modern operating systems support IPv6 out of the box. The same is for growing number of software packages, among them is BIND too. So, operating system configures IPv6 address on interface and application thinks that IPv6 works and configures it. But, IPv6 doesn't work outside of the local network (there is no IPv6 capable router) so, IPv6 addresses, unless in local networks, are unreachable.

So, you might ask now: but everything otherwise works, why is this case special! Well, the problem is that some DNS servers, anywhere in hierarchy, support IPv6, but not all. And when our resolver gets IPv6 address in response, it defaults to it and ignores IPv4. It obviously can not reach it so it logs a message and then tries IPv4. Once again, note that this IPv6 address can pop up anywhere in hierarchy, it isn't necessary to be on the last DNS server. In this concrete case name server for eolprocess.com doesn't support IPv6, but some name server for the top level com domain do support it!

To prevent those messages from appearing add option -4 to bind during startup. On CentOS (Fedora/RHEL) add or modify the line OPTIONS in /etc/sysconfig/named so that it includes option -4, i.e.

OPTIONS="-4"

Problems with resolver library...

I just had a problem that manifested itself in a very strange way. I couldn't open Web page hosted on a local network, while everything else seemingly worked. The behavior was same for Chrome and Firefox. In due course I realized that every application had this problem. On the other hand, resolving with nslookup worked flawlesly. This was very confusing. To add more to the confusion, while running tcpdump it was obvious that there were no DNS requests sent to the network! So, it was obvious that the problem was somewhere in the local resolver. At first, I suspected on nscd that was used as a caching daemon on Fedora, but in Fedora 16 this daemon is not installed. So, how to debug this situation? Quick google query didn't yield anything useful.

Reading manual page of resolv.conf there is section that says that you can use directive option debug. But trying to do that yielded no output! Neither there were any results using the same option but via RES_OPTIONS environment variable. This is strange, and needs additional investigation as why it is so, and more importantly to know how to debug local resolver.

In the mean time I figured out that the ping command behaves the same as browser and since ping command is much smaller it is easier to debug it using strace command. So, while running ping via strace I noticed the following line in the output:

open("/lib64/libnss_mdns4_minimal.so.2", O_RDONLY|O_CLOEXEC) = 3

which immediately rung a bell that the problem could be nsswitch! And indeed, opening it I saw the following line:

hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname

which basically said that, if mdns4 returns not found dns is not tried. It seems that mdns4 is used whenever the domain name ends in .local, which was true in my case. So, I changed that line into:

hosts:      files dns

and everything works as expected.

Since I didn't install explicitly mdns, I decided to remove it. But then it became clear that wine (Windows Emulator) depends on it. So, I left it.