End2end design principle, middleboxes and a bit about TCP...

I was just watching guest lecture given by Jana Iyengar to the students of CS144 course on Stanford. In his lecture he talks about e2e design principle, and the rise of middleboxes. He then goes on to conclude that middleboxes (especially NATs) are a problem of today's Internet. And I couldn't agree more! It's known fact that Internet was built in such a way that the network is dumb, while end nodes are smart. When I say smart, it means that functionality is placed within smart part of the whole system, while when I say dumb, it means it only performs one simple function. In the case of the Internet, network only moves data from one end to the other, and almost nothing else. This design principle was a key feature that allowed Internet to evolve to today's form and become ubiquitous network. To better illustrate this point, contrast Internet with telephone network. Telephone network was built so that the network itself is smart, while the end nodes (telephones!) are dumb. There is a nice illustration of this difference I saw once I liked very much. Here is the reconstruction of the illustration I saw:

Now, when you wanted something new from your telephone, which included telephone network, you had to wait your telephone company to introduce the service in the network, and only after that you could use it. Contrast that to the Internet, you just had to install a server/service on you computer and whoever wanted to access it had to install client on its machine. And that's it, no changes necessary in the network. Actually, the network doesn't know, neither care, what you are doing and everything works. There is a great example of this: the Web. The Web wouldn't succeed if Tim Berners-Lee had to wait for telephone companies to do something. And since the Internet is popular thanks to the Web, the Internet itself wouldn't succeed. Note that the telephone network functions in such a way because of a historical reasons. But, the telephone provides don't have incentive to change that. When/if they control network, they have revenues. The moment they only transfer data, the revenues are with someone else. And that's the case today with content providers (Google, Yahoo, Microsoft, Youtube, ...) and ISPs.

There is also one additional reason to design network by pushing  functionality to the edge and that's for scalability reasons. I think that it's quite obvious that the simpler something is, the bigger it can be, and it's easier to increase size, so I won't argue this any more.

What is happening now, and for some time, is that NATs are proliferating throughout the network. And since NATs heavily inspect the packets that pass through them and they depend on knowing higher layer protocols, it means they have to have built-in knowledge of higher layer protocols. What that means in turn is that if you are introducing a new service, you have to have support built into NATs. And there are two problems there. First, abundance of installed NATs that can not be changed, and second, bugs within those devices. So, in essence, we are approaching the way telephone networks work. Of course, there are other problems with NATs, but this one is a huge one!

Jana Iyengar then talks about SCTP, and the fact that this protocol exists from 2000 and it still didn't manage to take some ground. And middleboxes, more specifically NATs, are to blame. They pass TCP, fiddling with it, but nothing else. So, one of the things he was doing is using TCP as a communication substrate. In other words, he relied on TCP being passed through middleboxes, and then he went to built protocol on top of it. This protocol then could be used to build another protocols that will work across middleboxes. The modification that they did to TCP allowed it to deliver out-of-order data which they termed as uTCP, unreliable TCP. And, it seems no-one thought of that before.

But, I have to say that in 2011. I worked with a student and we were trying to introduce certain QoS parameters into TCP. The motivation were streaming services. As a part of this we allowed TCP to be unreliable, i.e. it could drop data in order to meet other QoS parameters. The work is described in diploma thesis available online, unfortunately, only in Croatian. But, I intend now to rewrite it into a paper as it was an interesting experiment...

Setting up reverse DNS server...

In the post about DNS configuration I skipped reverse DNS configuration. But, it is necessary to have it in some cases, like FreeIPA installation or for mail servers. So, I'm going to explain how to configure reverse DNS server.

While "normal" DNS resolution works by names, from root server down to the authoritative one for the name we are looking for, reverse DNS resolution works within special top-level domain (in-addr.arpa). Within this domain, sub-domains are comprised from octets within IP address in reverse order. Now, if your block of IP addresses ends on byte boundary (e.g. /8, /16, /24) the setup is relatively simple. Otherwise, you upstream provider (the one that holds larger IP address block) has to point to your domain on a per address base.

Let us bring this to more concrete values. Suppose that our public IP address space is 192.0.2.0/24. Also, suppose that your mail server has public IP address 192.0.2.2. In that case, reverse query is sent for name 2.2.0.192.in-addr.arpa and query type is set to PTR, i.e. we are looking for a name 2 within 2.0.192.in-addr.arpa zone.

So, it's relatively easy to setup reverse DNS. You need to define appropriate zones that include only network part of your IP addresses. In our case we have two zones, but IP addresses used for one of them depends on who's asking (client from the local network or client on the Internet). So, we have three zones in effect:

1. DMZ, when asked by local clients, is in the network 10.0.0.0/24. This means we have reverse zone 0.0.10.in-addr.arpa for local clients.
2. DMZ, when asked by internet clients, is in the network 192.0.2.0/24. This means that for them reverse zone is 2.0.192.in-addr.arpa.
3. Finally, clients in local network (non-DMZ one) have IP addresses from a block 172.16.1.0/24 and so they are placed within reverse zone 1.16.172.in-addr.arpa.

So, within internal view you should add the following two zone statements:

zone "0.0.10.in-addr.arpa" {
    type master;
    file "example-domain.com.local.rev";
};

zone "1.16.172.in-addr.arpa" {
    type master;
    file "example-domain.local.rev";
};

And within internet view you should add the following zone statement:

zone "2.0.192.in-addr.arpa" {
   type master;
    file "example-domain.com.rev";
};

Then, you should create the three zone files (example-domain.com.local.rev, example-domain.local.rev, and example-domain.com.rev) with the following content:

# cat example-domain.com.local.rev
 $TTL 1D
@    IN    SOA    @ root.example-domain.com. (
            2012062601 ; serial
            1D         ; refresh
            1H         ; retry
            1W         ; expire
            3H )       ; minimum
           NS    ns1.example-domain.com.

1          PTR    ns1.example-domain.com.
# cat example-domain.local.rev
 $TTL 1D
@    IN    SOA    @ root.example-domain.com. (
            2012062601 ; serial
            1D         ; refresh
            1H         ; retry
            1W         ; expire
            3H )       ; minimum
           NS    ns1.example-domain.com.

1          PTR    test.example-domain.local.
# cat example-domain.com.rev
$TTL 1D
@    IN    SOA    @ root.example-domain.com. (
            2012062601 ; serial
            1D         ; refresh
            1H         ; retry
            1W         ; expire
            3H )    ; minimum
           NS    ns1.example-domain.com.

1          PTR    ns1.example-domain.com.

Don't forget to change permissions on those files as explained in the previous post. Now, restart BIND and test server:

# nslookup ns1.example-domain.com 127.0.0.1
Server:  127.0.0.1
Address: 127.0.0.1#53

Name:    ns1.example-domain.com
Address: 10.0.0.1
[root@ipa ~]# nslookup 10.0.0.1 127.0.0.1
Server:  127.0.0.1
Address: 127.0.0.1#53

1.0.0.10.in-addr.arpa    name = ns1.example-domain.com.

As it can be seen, DNS server correctly handles request for IP addres 10.0.0.1 and returns ns1.sistemnet.hr. Let's try with a name from LAN:

# nslookup test.example-domain.local 127.0.0.1
Server:  127.0.0.1
Address: 127.0.0.1#53

Name:    ipa.example-domain.local
Address: 192.0.2.1

[root@ipa named]# nslookup 192.0.2.1 127.0.0.1
Server:  127.0.0.1
Address: 127.0.0.1#53

1.2.0.192.in-addr.arpa    name = test.example-domain.local

That one is correct too. So, that's it, you have reverse DNS correctly configured. Testing from the outside I'm leaving to you as an exercise. ;)

Setting up DNS server...

In the previous post I described how to install minimal CentOS distribution with some additional useful tools. In this part I'm going to describe how to install DNS server. This DNS server will exhibit certain behavior depending on where the client is, in other words we are going to setup split DNS. Note that since DNS server is accessible from the Internet, it is placed in DMZ network, i.e. 10.0.0.0/24.

Environment and configuration parameters

In the following text we assume network topology from the previous post, but we need some additional assumptions and requirements before going further. First, our public domain is example-domain.com and all hosts within DMZ will belong to that zone, i.e. FQDN of mail server will be mail.example-domain.com. But, when some client from the Internet asks for certain host, e.g. mail.example-domain.com DNS server will return public IP address. On the other hand, when client from a local network, or even DMZ, asks for the same host private IP address will be returned, i.e. 10.0.0.2. The reason for such behavior is more efficient routing. In other words, if client from a local network would receive public IP address, then all the traffic would have to be NAT-ed. We'll also assume that we are assigned a public block of IP addresses 192.0.2.0/24.

There will be  also additional domain example-domain.local in which all the hosts from the local domain will be placed. Additionally, this domain will be visible only from the local network and DMZ, it will not exist for clients on the Internet.

To simplify things we are not going to install secondary DNS server nor we are going to install reverse zones. For a test purposes this is OK, but if you are doing anything serious, you should definitely install slave DNS servers (or use someone's service) and configure reverse zones. Also, I completely skipped IPv6 configuration and DNSSEC configuration.

Installation of necessary packages

There are a number of DNS server implementations to choose from. The one shipped with CentOS is BIND. It is the most popular DNS server, with most features, but, in the past it had a lot of vulnerabilities. Nevertheless, we'll use that one. Maybe sometime in future, I write something about alternatives, too.

So, the first step, is to install necessary packages:

yum install bind.x86_64 bind-chroot.x86_64 bind-utils.x86_64

Server itself is in bind package, bind-chroot is used to confine server to alternative root filesystem in case it is compromised. Finally, bind-utils contains utilities we need to test server. This is 4.1M of download data, and it takes around 7M after installation. Not much.

Note that it is a good security practice to confine server into alternative root. As I already said, in the past bind had many vulnerabilities, and in case new one is discovered and exploited, we want to minimize potential damage.

That's all about installation.

Configuration

Next, we need to configure server before starting it. Main configuration file for bind is /etc/named.conf . So, open it with your favorite editor as we are going to make some changes in it. Remember that our domain is called example-domain.com and that we want private addresses to be returned for internal clients, and public ones for external clients. Additionally, we have a local domain example-domain.local. Here is the content of the /etc/named.conf file:

acl lnets {
    10.0.0.0/24;
    172.16.1.0/24;
    127.0.0.0/8;
};

options {
  listen-on port 53 { any; };
  directory     "/var/named";
  dump-file     "/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  memstatistics-file "/var/named/data/named_mem_stats.txt";
  allow-update     { none; };
  recursion no;

  dnssec-enable no;
};

view "internal" {
    match-clients { lnets; };
    recursion yes;

    zone "." IN {
        type hint;
        file "named.ca";
    };

    zone "example-domain.local" {
        type master;
        file "example-domain.local";
    };

    zone "example-domain.com" {
        type master;
        file "example.domain.local";
    };

    include "/etc/named.rfc1912.zones";
};

view "internet" {
    match-clients { any; };
    recursion no;

    zone "example-domain.com" {
        type master;
        file "example-domain.com";
    };
};

Let me now explain what this configuration file actually specifies. In global, there are four blocks, i.e. acl, options, and two view blocks.

acl block (or blocks, because there may be more such statements) specifies some symbolic name that will refer to group of addresses.  This is good from maintenance perspective because there is only one place where something is defined. In our case, we define our local networks, i.e. DMZ and local LAN. There is also loopback address because when some application on DNS server itself asks something, DNS server should treat it as if it is coming from a local network.

Next is an option block. In our case we specify that DNS server should listen (listen-on) port 53 on any IP address (any). In case you want to restrict on which addresses DNS server listens, you can enumerate them here instead of any keyword. Better yet, create ACL and use symbolic name. In option block we also disabled DNSSEC (dnssec-enable no), disabled updates to server (allow-update no), and disabled recursion (recursion no). Apart from that we defined directories, e.g. for zone files.

Finally, there are two view statements. They define zones depending on where the client asking is. If it is on the local networks, then first view (i.e. internal) is consulted, and if the client is anywhere else, then the second view is consulted (i.e. internet). match-clients statement classifies clients, and it is here that we use our ACL lnets. When query arrives, the source address from the packet is taken and compared to match-clients. The first one that matches is used. So, it is obvious that the second view matches anything but since it is the last one, then it is a catch-all view. Anyway, for our local clients we allow recursive queries (recursion yes), while we disallow it for a global clients (recursion no). This is a good security practice! Also, note that for internal clients we define example-domain.local zone, which isn't defined for global clients. And, for the same example-domain.com zone, two different files (i.e. databases with names) are used. This reflects the fact that we want local clients to get local IP addresses, while global clients should get globally valid IP addresses.

Zone configurations

There are three zones in our case. First one is a global one, used to answer queries to clients on the Internet. Looking into the main configuration file the name of the file containing this zone has to be example-domain.com and it has to be placed in /var/named directory! The content of this file is:

$TTL 1D
@    IN    SOA    @ root.example-domain.com. (
            2012062501    ; serial
            1D            ; refresh
            1H            ; retry
            1W            ; expire
            3H )          ; minimum
           NS    ns1.example-domain.com.

ns1        A    192.0.2.1

There are three records in the file. The first one defines zone itself, it is called Start of Authority, or SOA. The @ sign is shorthand for zone name and it is taken from the /etc/named.conf file. Then, there is a continuation that defines name server (NS) for the zone. It is continuation because the first column (where @ sign is) isn't defined so it is taken from the previous record. Name server is ns1.example-domain.com. Finally, there is IP address of the name server (A record). You should keep in mind that any name that doesn't end with dot is appended with a domain name. A lot of errors are caused by that omission. For example, if you wrote ns1.example-domain.com (without ending dot) this would translate into ns1.example-comain.com.example-domain.com. which is obviously wrong. But note that we are counting on this behavior in A records since we only wrote a name, without a domain!

The other zone files are similar to the previous one and also have to be within /var/named directory. I.e. for local view of example-domain.com. the content of zone file is:

$TTL 1D
@    IN    SOA    @ root.example-domain.com. (
            2012062501 ; serial
            1D         ; refresh
            1H         ; retry
            1W         ; expire
            3H )       ; minimum
        NS    ns1.example-domain.com.

ns1        A    10.0.0.2

Note that it is almost identical, except that IP address for name server is now from a private range! Finally, here is the content of zone that contains names from a local network:

$TTL 1D
@    IN    SOA    @ root.example-domain.local. (
            2012062501 ; serial
            1D         ; refresh
            1H         ; retry
            1W         ; expire
            3H )       ; minimum
        NS    ns1.example-domain.com.

test        A    172.16.1.1

Again, almost the same. Note that I added one test record with phony IP address. This is so that I can test DNS server if it is correctly resolving IP addresses.

One final thing before starting DNS server! You should change file ownership and SELinux context. To do so, use the following two commands (run them in /var/named directory):

chcon system_u:object_r:named_zone_t:s0 example-domain.*
chown root.named example-domain.*

The first command changes SELinux context, and the second one owner and group of files. So, it is time to start BIND, i.e. DNS, server:

/etc/init.d/named start

If there were some error messages, like the following one:

zone example-domain.com/IN/internal: loading from master file example-domain.com.local failed: permission denied
zone example-domain.com/IN/internal: not loaded due to errors.

Then something is wrong. Look carefully what it says to you as the error messages are quite self explanatory! In this particular case the problem is that DNS server can not read zone file. If you get that particular error message, check that SELinux context and ownership were changed properly.

Testing

Now, if you get just OK, it doesn't yet mean that everything is OK. You have to query server in order to see if it responds correctly. But, if there were any errors, they will be logged in /var/log/messages.

To test DNS server, use nslookup like this:

nslookup <name> 127.0.0.1

Here, I assume you are testing on a DNS server itself. If you are testing on some other host then you should change address 127.0.0.1 with proper IP address of DNS server (public one in case you ask from the internet, local one if you are asking from some computer in DMZ or local network). In any case you should receive correct answers. Here are few examples:

# nslookup ns1.example-domain.com 127.0.0.1
Server:        127.0.0.1
Address:    127.0.0.1#53

Name:    ns1.example-domain.com
Address: 10.0.0.2

# nslookup test.example-domain.local 127.0.0.1
Server:        127.0.0.1
Address:    127.0.0.1#53

Name:    test.example-domain.local
Address: 172.16.1.1

As expected, when internal host sends request internal IP address are returned. You should also check that any valid name is properly resolved (recursive queries). For example, you could ask for www.google.hr:

# nslookup www.google.hr 127.0.0.1
Server:        127.0.0.1
Address:    127.0.0.1#53

Non-authoritative answer:
www.google.hr    canonical name = www-cctld.l.google.com.
Name:    www-cctld.l.google.com
Address: 173.194.70.94

Which, in this case, resolves properly.

Now, there is a question how to test if names are properly resolved for clients on the Internet. The obvious solution is to try from some host on the Internet and see what's happening. In case you don't have host on the Internet, you can cheat using NAT. ;)

Do the following. First, add some random IP address to your loopback interface, i.e.:

ip addr add 3.3.3.3/32 dev lo

Then, configure NAT so that any query sent via loopback gets source IP address changed:

iptables -A POSTROUTING -o lo -t nat -p udp --dport 53 -j SNAT --to-source 3.3.3.3

And now, try to send few queries:

# nslookup www.slashdot.org. 127.0.0.1
Server:        127.0.0.1
Address:    127.0.0.1#53

** server can't find www.slashdot.org: REFUSED

That one is OK. Namely, we don't want to resolve third party names for outside clients (remember, recursive is off!). Next test:

# nslookup ns1.example-domain.local. 127.0.0.1
Server:        127.0.0.1
Address:    127.0.0.1#53

** server can't find ns1.example-domain.local: REFUSED

This one doesn't work either. Which is what we wanted, i.e. our local domain isn't visible to the outside world. Finally, this one:

# nslookup ns1.example-domain.com. 127.0.0.1
Server:        127.0.0.1
Address:    127.0.0.1#53

Name:    ns1.example-domain.com
Address: 192.0.2.1

Which sends us correct outside address.

So, all set and done. Don't forget to remove temporary records (i.e. test), iptables rule, lo address, and also don't forget to modify /etc/resolv.conf so that your new server is used by default by local applications.