Network troubleshooting...

Yesterday, I was giving a lecture to a third year students of computer science within a course Communication networks. Of course, that is not the only computing module on the Faculty, i.e. there are also modules computer engineering and software engineering, but other lecturers are giving lectures to them. Anyway, the topic of the lecture was Internet's networking layer and, among other things I was talking about autonomus systems, BGP routing protocol, and forwarding process. As a part of ICMP protocol, ping and traceroute commands were mentioned as an important addition to troubleshooting tool. I also mentioned several times to the students that ping and traceroute are the main troubleshooting tools of any network technician. I also told them that after they finish with this course they should be able to do basic troubleshooting and never ever again say something like "Internet isn't working"! Finally, I mentioned that there is a Routeviews project on the Internet that provides (read only) access to BGP routers that can be used to see routes exchanged on certain parts of the Internet.

So, the reason I'm writing this post is that I stumbled on a post Why Google Went Offline Today and a Bit about How the Internet Works which is highly recommended read for them, but also for anyone else learning about networking, with an emphasis on the Internet. They (students) should be able to follow and understand this post now after they learned basic terminology and mechanisms of the Internet layer. To fully understand it, they'll have to wait until we explain how DNS works.

Routeviews

Let me (mis)use this post to say more about Routeviews. Actually, there is also Looking Glass which is very similar, i.e. it allows to peek at certain points on the Internet, but it doesn't offer direct access to BGP routers so, its a bit less interesting, at least to me.

Before I continue with description of Routeviews let me state that Internet is highly irregular network, connecting autonomous systems, with the main irregularity coming from the peering relations between autonomous systems. This peering is largely kept confidential, and besides BGP routing is mainly driven by politics not by technology. What this means is that the real topology if the Internet is not known to anybody. And besides, it's a dynamic and constantly moving target. There are attempts of mapping Internet, but nevertheless they are only approximations.

So, how Internet looks depends on where are you looking from. The routeviews allow one to look on the Internet from different points and this is used for troubleshooting purposes, as well for research purposes. There are also historical data and that's very valuable information.

Anyway, if you go to Routeviews project you'll see a table with a list of DNS names. For each name there is additional information, how to access it (mainly telnet), what type of software/hardware it is running, and where it is. So, you can telnet to one of those routers, and using command show ip bgp determine BGP routing table of that router. Note that Cisco IOS (as well as Zebra/Quagga which are modelled after IOS) offer help command in the form of a question mark. At any point in the command you can type ? and the OS will show you what can you type at that point.

Here is a c/p from a session I made. It's obviously edited to be short and up to the point. First, I logged in to one of the routers:

$ telnet route-views.routeviews.org
Trying 128.223.51.103...
Connected to route-views.routeviews.org.
Escape character is '^]'.
 **********************************************************************
                    Oregon Exchange BGP Route Viewer
          route-views.oregon-ix.net / route-views.routeviews.org
 route views data is archived on http://archive.routeviews.org
 This hardware is part of a grant from Cisco Systems.
 Please contact help@routeviews.org if you have questions or
 comments about this service, its use, or if you might be able to
 contribute your view.
 This router has views of the full routing tables from several ASes.
 The list of ASes is documented under "Current Participants" on
 http://www.routeviews.org/.
                          **************
 route-views.routeviews.org is now using AAA for logins.  Login with
 username "rviews".  See http://routeviews.org/aaa.html
 **********************************************************************

User Access Verification
Username: rviews
route-views>

What I typed is in bold, and what I received is in ordinary text. Note that I used username rviews, as specified in a greeting message. Here is a bit of using help:

route-views>?
Exec commands:
  <1-99>           Session number to resume
  access-enable    Create a temporary Access-List entry
  access-profile   Apply user-profile to interface
  clear            Reset functions
  connect          Open a terminal connection
  crypto           Encryption related commands.
  disable          Turn off privileged commands
...
route-views>show ?
  aaa                   Show AAA values
  aal2                  Show commands for AAL2
  adjacency             Adjacent nodes
  alps                  Alps information
  appfw                 Application Firewall information
  aps                   APS information
  arp                   ARP table
  auto                  Show Automation Template
  backup                Backup status
  bfd                   BFD protocol info
  bgp                   BGP information
...
route-views>show i?
if-mgr  ima           inventory  ip
ipc     iphc-profile  ipv6    

Again, in bold is what I typed, and with three dots I marked output that was cutted not to clutter this post. Finally, here is part of the show bgp command output (note, show ip bgp is equivalent form):

route-views>show bgp
BGP table version is 3000708321, local router ID is 128.223.51.103
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network     Next Hop         Metric LocPrf Weight Path
r> 0.0.0.0  208.74.64.40                        0 19214 12989 2828 i
*  1.0.0.0/24  217.75.96.60          0             0 16150 15169 i
*              154.11.98.225         0             0 852 15169 i
*              129.250.0.11          6             0 2914 3356 15169 i
*              4.69.184.193          0             0 3356 15169 i
*              194.85.102.33                       0 3277 15169 i
*              194.85.40.15                        0 3267 15169 i
*              193.0.0.56                          0 3333 3356 15169 i
*              209.124.176.223                     0 101 101 15169 i
*              216.218.252.164                     0 6939 15169 i
*              114.31.199.1          0             0 4826 15169 i
*              207.172.6.20          0             0 6079 15169 i
*              208.51.134.254        1             0 3549 15169 i
...

Let me give you a short description of what you see here. First line is some general data (table version, and router's ID). Then there are status codes and origin code that are used later. Finally, the BGP table is dumped (and piped through more, use q to quit).

What you see in BGP table is:

1. Network destination, i.e. some network on the Internet that is reachable from that particular BGP router who's tables you are examining. Note that the multiple lines belong to a single destination network and in that case, network isn't repeated. This is the case for network 1.0.0.0/24 in the previous output.
2. Each line in the output is one possible path to reach given network. Each path consists of a next hop (second column), and exact path (last column with numbers). The path that was found to be the best is marked with greater-then (>) in the first column (also, note the legend in the beginning of the output).
3. Exact path is a sequence of autonomous systems through which destination network is reachable. Note that each path ends with a same number, i.e. same autonomous system number. That's because given network belongs to that autonomous system.
4. At the end of each path there is letter that informs us from where this route was obtained. In this case i means it came from the peering BGP router in the same autonomous system as the router we are examining.

Finally, to exit use exit command. :) And that's it. I suppose you can play and research for yourself from this point on...