GNS3 with ExaBGP Part 2 – Full Internet Routing Table

Following on the the initial concept for using ExaBGP in part 1 we will kick it up a notch and inject a replica BGP full routing table feed. Thankfully the people at RIPE NCC have something we can use available through their Routing Information Service Raw Data page.

For my example I’m going to pull data from rrc00 which is based in Amsterdam and has a good number of peers. The data archive for rrc00 is located at http://data.ris.ripe.net/rrc00/ and the most straight forward file to download is the latest-bview.gz file.

A note about these bview files – These archives contain BGP data encoded in Multi-Threaded Routing Toolkit (MRT) Routing Information Export Format, which ExaBGP on its own cannot digest. Thankfully this is addressed by a python based MRT Parsing tool which includes a script to generate ExaBGP compatible configs (mrt2exabgp.py).

I recommend installing via a clone the git repositorary as you get the freshest version of the tool.

adam@gns3:~$ sudo -H pip install https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip
Collecting https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip
  Downloading https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip (52kB)
    100% |████████████████████████████████| 61kB 122kB/s
Installing collected packages: mrtparse
  Running setup.py install for mrtparse ... done
Successfully installed mrtparse-1.4

adam@gns3:~$ wget http://data.ris.ripe.net/rrc00/latest-bview.gz

After downloading the 50MB+ file, we can use the mrt2exabgp.py script – details on its operation are here however for my purposes I will just use the -G and -P flags and redirect the output to exabgptable.py which will end becoming a python script that is called fullbgptable.py

My Linux box is a long in the tooth AMD Athlon II X4 630 based system with 16GB RAM, so not precisely a speed demon, and doing this conversion is going to take awhile, so I’m using the time command to see how long the process will actually take, while I grab a coffee.

adam@gns3:~$ time /usr/local/lib/python2.7/dist-packages/mrtparse/examples/mrt2exabgp.py -G -P latest-bview.gz  > fullbgptable.py

real    24m25.654s
user    24m19.396s
sys     0m0.612s
adam@gns3:~$ ls -la fullbgptable.py
-rw-rw-r-- 1 adam adam 35282115 Nov  8 23:54 fullbgptable.py

Okay… so that took nearly half an hour, lets have a quick look at this.

adam@gns3:~$ head fullbgptable.py
#!/usr/bin/env python

import sys
import time

msgs = [
'announce attributes origin IGP as-path [29608 3356 29396 29396 29396 39686 44953 ] med 13 community [3356:2 3356:22 3356:100 3356:123 3356:503 3356:2067 29608:30600] next-hop 79.143.241.12 nlri 93.95.248.0/21',
'announce attributes origin IGP as-path [29608 3356 6453 9498 58682 24389 ] med 13 community [3356:2 3356:22 3356:86 3356:502 3356:666 3356:2066 6453:2000 6453:2200 6453:2204 29608:30600] next-hop 79.143.241.12 nlri 202.56.5.0/24 202.56.6.0/23 202.56.7.0/24',
'announce attributes origin IGP as-path [29608 3356 25795 202773 ] med 13 community [3356:3 3356:22 3356:100 3356:123 3356:575 3356:2003 25795:100 25795:40000 29608:30600] next-hop 79.143.241.12 nlri 185.152.130.0/24 185.152.131.0/24',
'announce attributes origin IGP as-path [29608 6939 31027 44869 203646 ] med 11 community [29608:40090] next-hop 2a01:678::2 nlri 2a03:9aa0::/32',
adam@gns3:~$ tail fullbgptable.py
    msg = msgs.pop(0)
    if isinstance(msg, str):
        sys.stdout.write(msg + '\n')
        sys.stdout.flush()
    else:
        time.sleep(msg)

while True:
    time.sleep(1)

It appears good, so lets create the ini file for exabgp to use.

group SR4 {
    router-id 1.2.3.3;
    neighbor 1.2.3.4 {
        local-address 1.2.3.3;
        local-as 64512;
        peer-as 1234;
    }
    process fullbgp {
        run /usr/bin/python /home/adam/fullbgptable.py;
    }
}

Time to fire up SR4 and start pushing a whole lot of IPv4 routes to it (you may notice in the fullbgptable.py extract above an IPv6 prefix, however since we are only using the IPv4 address family, this will be ignored)

While ExaBGP is still running, lets see how many bgp routes we have received.

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512   21732    0 00h00m32s 21726/0/0 (IPv4)
                            7    0
-------------------------------------------------------------------------------

In a little over 30 seconds we have recieved 21726 routes but none of them are active, lets have a look at what we have been offered.

A:SR4# show router bgp routes
===============================================================================
 BGP Router ID:10.10.10.4       AS:1234        Local AS:1234
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path
-------------------------------------------------------------------------------
i     1.0.4.0/24                                         None        13
      79.143.241.12                                      None        -
      29608 3356 4637 1221 38803 56203
i     1.0.5.0/24                                         None        13
      79.143.241.12                                      None        -
      29608 3356 4637 1221 38803 56203
i     1.0.6.0/24                                         None        13
A:SR4#  show router bgp routes 1.0.7.0/24 detail | match expression "Flags|Next"
Nexthop        : 79.143.241.12
Res. Nexthop   : Unresolved
Flags          : Invalid  IGP  Nexthop-Unresolved
Nexthop        : 79.143.241.12
Res. Nexthop   : Unresolved
Flags          : Invalid  IGP  Nexthop-Unresolved

The IP Nexthop is 79.143.241.12 but since SR4 doesn’t know how to get there, the route is invald.

Lets have a look at how many IP Nexthops are in fullbgptable.py

adam@gns3:~$ grep -Eo 'next-hop [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' fulltable.py | cut -f 2 | sort | uniq -c
   1177 12.0.1.63
     79 146.228.1.3
   1330 176.12.110.8
     58 178.255.145.243
    681 193.0.0.56
    849 193.150.22.1
    153 193.160.39.1
    847 202.12.28.1
   1876 203.119.76.5
    112 203.123.48.6
    543 208.51.134.248
    290 212.25.27.44
    110 213.200.87.254
    157 45.61.0.85
 102315 79.143.241.12

So we can see that there are a number of nexthops with the vast majority associated with 79.143.241.12 (the counts are not precisely the number of individual routes due to the way the fullbgptable.py has consolidated similar prefixes for performance purposes). To support installing these routes into the routing table on SR4 rather than modifying fullbgptable.py to use an alternate nexthop (say 1.2.3.3 which is used by ExaBGP), a quick and dirty workaround is to use a static blackhole route for 79.143.241.12/32 (on IOS this would be the equivalent of a null route)

A:SR4# configure router static-route 79.143.241.12/32 black-hole

And restart ExaBGP (If we really wanted to we could add blackhole routes for the other nexthops, however 79.143.241.12 covers the vast majority of routes).

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512   17966    0 00h00m32s 17950/16879/4204 (IPv4)
                          849    0
-------------------------------------------------------------------------------
A:SR4# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.0.7.0/24                                    Blackh* BGP       00h01m12s  170
       Black Hole                                                   0
1.0.128.0/17                                  Blackh* BGP       00h00m14s  170
       Black Hole                                                   0
1.1.20.0/24                                   Blackh* BGP       00h00m58s  170
       Black Hole                                                   0
1.1.128.0/17                                  Blackh* BGP       00h00m14s  170
       Black Hole                                                   0
1.2.3.0/24                                    Local   Local     00h22m49s  0
       ExaBGP                                                       0
1.2.4.0/24                                    Blackh* BGP       00h00m35s  170
       Black Hole                                                   0
1.2.11.0/24                                   Blackh* BGP       00h00m50s  170
       Black Hole                                                   0
1.2.128.0/17                                  Blackh* BGP       00h00m14s  170

This looks pretty good now (however it looks like picking 1.2.3.0/24 for my link addressing wasn’t the best choice and I should have used something from RFC1918)

While ExaBGP is pushing routes to SR4, lets check the status:

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  347185    0 00h03m00s 347165/318512/269061 (IPv4)
                        44567    0
-------------------------------------------------------------------------------
A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  405376    0 00h03m31s 405354/371121/325132 (IPv4)
                        54221    0
-------------------------------------------------------------------------------
A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  483575    0 00h04m13s 483552/442325/431845 (IPv4)
                        72792    1
-------------------------------------------------------------------------------
A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512      23    0 00h00m10s Disabled
                           10    0
-------------------------------------------------------------------------------

Okay, this isn’t a good thing, the state is showing up as disabled and there are no longer any prefix counts – something serious must have happened. On SROS routers, log 99 is a good first place to investigate potential problems.

A:SR4# show log log-id 99

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
warning: 1 events dropped from log
Memory Log contents  [size=500   next event=43  (not wrapped)]

42 2016/11/08 14:29:56.40 UTC WARNING: BGP #2012 Base Peer 1: 1.2.3.3
"Peer 1: 1.2.3.3: Closing connection: VR 1: Group ExaBGP: Peer 1.2.3.3 not enabled or not in configuration"

41 2016/11/08 14:29:53.09 UTC CRITICAL: BGP #2015 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: out of memory - disabled the peer"

40 2016/11/08 14:29:53.08 UTC WARNING: BGP #2005 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: sending notification: code CEASE subcode OUT_OF_RESR"

39 2016/11/08 14:29:53.08 UTC WARNING: BGP #2002 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: moved from higher state ESTABLISHED to lower state IDLE due to event OUT_OF_MEMORY"

38 2016/11/08 14:25:10.64 UTC MINOR: BGP #2001 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: moved into established state"

We can see that Event 38 was when the BGP session came up, however event 39 brought the session down because we ran out of memory.

In GNS3 SR4 had been allocated 2GB of memory, obviously this is not enough if we expect to take a full BGP feed – lets bump the memory to 3GB and see if that fixes it.

Up SR4's Memory from 2GB to 3GB
Up SR4’s Memory from 2GB to 3GB

Restart SR4 and kick off ExaBGP again…

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  647537    0 00h06m03s 647522/595050/595050 (IPv4)
                       102359    0
-------------------------------------------------------------------------------

Well, the session is alive for longer than previous, and the ExaBGP session console output has gone quiet, so it looks like it has transmitted everything. Lets wait a bit more just to be certain.

A:SR4# show router bgp summary | match Summary post-lines 100
A:SR4# A:SR4>show>router>bgp# show router bgp summary
===============================================================================
 BGP Router ID:10.10.10.4       AS:1234        Local AS:1234
===============================================================================
BGP Admin State         : Up          BGP Oper State              : Up
Total Peer Groups       : 1           Total Peers                 : 1
Total BGP Paths         : 110581      Total Path Memory           : 26928008
Total IPv4 Remote Rts   : 647522      Total IPv4 Rem. Active Rts  : 595050
Total McIPv4 Remote Rts : 0           Total McIPv4 Rem. Active Rts: 0
Total McIPv6 Remote Rts : 0           Total McIPv6 Rem. Active Rts: 0
Total IPv6 Remote Rts   : 0           Total IPv6 Rem. Active Rts  : 0
Total IPv4 Backup Rts   : 0           Total IPv6 Backup Rts       : 0

Total Supressed Rts     : 0           Total Hist. Rts             : 0
Total Decay Rts         : 0

Total VPN Peer Groups   : 0           Total VPN Peers             : 0
Total VPN Local Rts     : 0
Total VPN-IPv4 Rem. Rts : 0           Total VPN-IPv4 Rem. Act. Rts: 0
Total VPN-IPv6 Rem. Rts : 0           Total VPN-IPv6 Rem. Act. Rts: 0
Total VPN-IPv4 Bkup Rts : 0           Total VPN-IPv6 Bkup Rts     : 0

Total VPN Supp. Rts     : 0           Total VPN Hist. Rts         : 0
Total VPN Decay Rts     : 0

Total L2-VPN Rem. Rts   : 0           Total L2VPN Rem. Act. Rts   : 0
Total MVPN-IPv4 Rem Rts : 0           Total MVPN-IPv4 Rem Act Rts : 0
Total MDT-SAFI Rem Rts  : 0           Total MDT-SAFI Rem Act Rts  : 0
Total MSPW Rem Rts      : 0           Total MSPW Rem Act Rts      : 0
Total RouteTgt Rem Rts  : 0           Total RouteTgt Rem Act Rts  : 0
Total McVpnIPv4 Rem Rts : 0           Total McVpnIPv4 Rem Act Rts : 0
Total MVPN-IPv6 Rem Rts : 0           Total MVPN-IPv6 Rem Act Rts : 0
Total EVPN Rem Rts      : 0           Total EVPN Rem Act Rts      : 0
Total FlowIpv4 Rem Rts  : 0           Total FlowIpv4 Rem Act Rts  : 0
Total FlowIpv6 Rem Rts  : 0           Total FlowIpv6 Rem Act Rts  : 0

===============================================================================
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  647547    0 00h10m58s 647522/595050/595050 (IPv4)
                       102369    0
-------------------------------------------------------------------------------

The session is still up and no new routes have been received.

While this post has been using SROS, this example can be applied to any BGP system, just make sure you have sufficient memory in your system as the IPv4 routing table is not small! ExaBGP is a pretty interesting tool and in this case can help make a simulated environment seem more like the real world, the only thing is that this is a static snapshot and there is ongoing churn within the internet.