TCP client self connect...

This is so cool and unexpected, but then nothing out of spec, that I had to reblog it. Namely, if you run the following snippet of the Bourne shell code:

while true
do
   telnet 127.0.0.1 50000
done

You'll constantly receive message 'Connection refused', but at one point the connection will be established and whatever you type, will be echoed back:

Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
test1
test1
test2
test2

Note that you didn't start any server and there is no process listening on port 50000 on localhost, but yet, it connected! Looking at the output of netstat command we see that there is really established connection:

$ netstat -tn | grep 50000
tcp    0   0 127.0.0.1:50000  127.0.0.1:50000  ESTABLISHED

and, if we monitor traffic using tcpdump we observe a three way handshake:

21:31:02.327307 IP 127.0.0.1.50000 > 127.0.0.1.50000: Flags [S], seq 2707282816, win 43690, options [mss 65495,sackOK,TS val 41197287 ecr 0,nop,wscale 7], length 0
21:31:02.327318 IP 127.0.0.1.50000 > 127.0.0.1.50000: Flags [S.], seq 2707282816, ack 2707282817, win 43690, options [mss 65495,sackOK,TS val 41197287 ecr 41197287,nop,wscale 7], length 0
21:31:02.327324 IP 127.0.0.1.50000 > 127.0.0.1.50000: Flags [.], ack 1, win 342, options [nop,nop,TS val 41197287 ecr 41197287], length 0

What happened? In short, client connected to itself. :) A bit longer explanation follows...

Let's start with a fact that when a client (in this case it is a telnet application) creates socket and tries to connect to server, kernel assigns it a random source port number. This is because each TCP connection is uniquely identified with 4-tuple:

(source IP, source port, destination IP, destination port)

Of those, three parameters are predetermined, i.e. source IP, destination IP and destination port, what's left is source port that has to be somehow arbitrarily assigned, and usually applications leave that to the kernel which takes it from the range of ephemeral ports. Applications can choose source port using bind(2) system call, but it is very rarely done. Now, in what range do these ephemeral ports live? They are high ports, and you can take a look into /proc file system to see specific values for your Linux machine, e.g.:

$ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000

In this case, ephemeral ports are taken between 32768 and 61000.

Now, back to our example with telnet application. When telnet is started, kernel selects some free port from the given range of ephemeral ports and tries to connect to localhost (destination IP 127.0.0.1), port 50000. Since no process usually listens on ephemeral ports RST response is sent back and telnet client gives error message Connection refused. This exchange on the network can be seen by using tcpdump tool:

# tcpdump -nni lo port 50000
21:31:02.326447 IP 127.0.0.1.49999 > 127.0.0.1.50000: Flags [S], seq 1951433652, win 43690, options [mss 65495,sackOK,TS val 41197286 ecr 0,nop,wscale 7], length 0
21:31:02.326455 IP 127.0.0.1.50000 > 127.0.0.1.49999: Flags [R.], seq 0, ack 387395547, win 0, length 0

It is interesting to note that Linux chooses ephemeral ports sequentially, not randomly. This allows easy guessing of the ports, and might be a security problem, but further investigation is necessary to confirm this.

Anyway, during many unsuccessful connections, at one iteration, telnet client is assigned source port 50000 and SYN request is sent to port 50000, i.e. to itself. So, it establishes connection with itself! This is actually fully according to the TCP specification which supports a so called feature simultaneous open, illustrated in Figure 8 in RFC793 (note, there is errata to this example in RFC1122).

Yet, the example from RFC793 assumes that there are two independent endpoints trying to connect at the same time, but in our case it is only one side so there is a small deviation from prescribed behavior. Let's take a look. Here is a TCP state machine taken from Wikipedia page about TCP:

When telnet client starts, source port 50000 is assigned and state machine is instantiated which is immediately initialized into CLOSED state. Then, telnet tries to connect to a server which means SYN is sent and TCP state machine of the source port goes to SYN SENT state. Now, this same source port, i.e. state machine, receives this SYN and because of this goes into SYN RECEIVED state (arrow from right to left marked with SYN/SYN+ACK). While transiting to a new state, SYN+ACK is emitted that is again received by the state machine. Now, we come to a bit of a mystery, namely how the state machine transitions to ESTABLISHED state and when an ACK is emitted to finish three way handshake?

To answer that, we'll have to dig a bit into the kernel's source code. First, note that there is an explicit case for self connect which is also commented. This case is triggered in TCP_SYN_SENT state. Then, socket is placed into TCP_SYN_RECV state and SYN+ACK is sent back. This SYN+ACK is immediately looped back and processed in function tcp_rcv_state_process(). In that function, the function tcp_validate_incoming() is called. That function, finally, after few checks calls function tcp_send_challenge_ack() that sends ACK. The state of the TCP connection (i.e. socket) is changed to ESTABLISHED in function tcp_rcv_state_process() within a part that processes ACK flag. And that concludes the description what happens actually happens, and what is seen on a network.

The scenario of self connect described in this post is quite specific and requires specific preconditions. First, obviously, you need to (ab)use ephemeral ports for listening servers so that you clients try to connect to ephemeral ports. Next, client and server have to run on the same IP address, otherwise client will not be able to self connect. Finally, this can only happen during initial handshake phase. If you find some client using some ephemeral port and try to connect to it, you'll be refused. So, the conclusion is: Don't use ephemeral ports for servers! Or otherwise, you risk very interesting behavior that is nondeterministic and hard to debug.