Using BGP Flowspec with Nokia SROS to protect the network edge

This post is about how you can use BGP to augment your network edge protection. Specifically BGP Flow spec can be used to dynamically add entries to an access control list (or ip-filter in Nokia SROS speak). Previous posts have been using ExaBGP as the method to inject traffic, and while it’s good at what it does, I’ve moved on to use GoBGP not because it seems to be quite a high performance BGP implementation (I’m just doing proof of concept work) but because GoBGP’s command line interface is more friendly and better documented than ExaBGP, making it the more versatile tool to generate route advertisements in a lab on the fly (at least for me).

BGP Flowspec Test Topology

The lab topology is designed to demonstrate filtering with flowspec – two routers (SR1 is under my control, while the External router is a gateway to unscrupulous entities that I cannot control) and GoBGP running on a computer in my Network Operations centre.

Before getting into the flowspec piece, I’ll go through the installation process for GoBGP and the Go Language for Ubuntu 16.04 (as the routers I’m using as running on eve-ng) and the computer in the example is bridged to SR1) and then how to develop the initial GoBGP configuration.

The Go Programming language is available via the standard Ubuntu repository
adam@m4600:~$ sudo apt-get install golang-go

GoBGP is pulled down as source code and compiled as part of the download process. An environmental variable needs to be set so Go knows where to pull down code, and where to build the binaries.
adam@m4600:~$ export GOPATH=$HOME/go
Tell go to pull down and build gobgpd (this takes awhile at least on my machine and you don’t get an indication on what it’s doing)
adam@m4600:~$ go get github.com/osrg/gobgp/gobgpd
The gobgp cli is a separate program that talks to gobgpd using the GoBGP API – downloading and building it is pretty quick in comparison to gobgpd:
adam@m4600:~$ go get github.com/osrg/gobgp/gobgp
Once built, we shall copy the generated binaries to the an executable path
adam@m4600:~$ sudo cp $GOPATH/bin/* /usr/local/sbin/
For those of us using the default bash shell, we can enable tab completion for the cli

adam@m4600:~$ sudo cp $GOPATH/src/github.com/osrg/gobgp/tools/completion/*.bash /etc/bash_completion.d/

Once GoBGP is installed, we can develop a configuration file that will be used to set up the GoBGP instance – in this case we’re defining the router itself and SR1 as a neighbor for the IPv4 FlowSpec address family:
A configuration file that will be used by the GoBGP Daemon enabling FlowSpec for IPv4 where we peer against SR1:

adam@m4600:~$ cat gobgp-flowspec.conf
[global.config]
  as = 64512
  router-id = "1.2.3.4"

[[neighbors]]
[neighbors.config]
  neighbor-address = "192.168.1.123"
  peer-as = 64512
[[neighbors.afi-safis]]
  [neighbors.afi-safis.config]
  afi-safi-name = "ipv4-flowspec"

The GoBGP documentation is also a bit nicer than ExaBGP too – additional information on setting up a configuration (including different formats is described in the GoBGP getting started document)

Start the GoBGP Daemon and read our new config:

adam@m4600:~$ sudo gobgpd -f gobgp-flowspec.conf
{"level":"info","msg":"gobgpd started","time":"2017-05-18T22:57:22+10:00"}
{"Topic":"Config","level":"info","msg":"Finished reading the config file","time":"2017-05-18T22:57:22+10:00"}
{"level":"info","msg":"Peer 192.168.1.123 is added","time":"2017-05-18T22:57:22+10:00"}
{"Topic":"Peer","level":"info","msg":"Add a peer configuration for:192.168.1.123","time":"2017-05-18T22:57:22+10:00"}
{"Key":"192.168.1.123","State":"BGP_FSM_OPENCONFIRM","Topic":"Peer","level":"info","msg":"Peer Up","time":"2017-05-18T22:57:40+10:00"}
We can see shortly after gobgpd started, it has already established a peering session with SR1 (192.168.1.123)

Opening up a new shell, we can use the gobgp client to verify that:

adam@m4600:~$ gobgp neighbor
Peer             AS  Up/Down State       |#Received  Accepted
192.168.1.123 64512 00:00:22 Establ      |        0         0
adam@m4600:~$ gobgp neighbor 192.168.1.123
BGP neighbor is 192.168.1.123, remote AS 64512
  BGP version 4, remote router ID 123.123.123.123
  BGP state = established, up for 00:01:53
  BGP OutQ = 0, Flops = 0
  Hold time is 90, keepalive interval is 30 seconds
  Configured hold time is 90, keepalive interval is 30 seconds
  %s
  Neighbor capabilities:
    multiprotocol:
        ipv4-flowspec:  advertised and received
    route-refresh:      advertised and received
    4-octet-as: advertised and received
  Message statistics:
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             4          5
    Route Refresh:          0          0
    Discarded:              0          0
    Total:                  5          6
  Route statistics:
    Advertised:             0
    Received:               0
    Accepted:               0

We can also get SR1 to confirm it too:
*A:SR1# show router bgp summary | match Summ post-lines 100
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
192.168.1.46
               64512        9    0 00h00m54s 0/0/0 (FlowIPv4)
                           13    0
-------------------------------------------------------------------------------

Now that the flowspec session is up and running but no one has exchanged any routes yet, I’ll park this for a moment to create a simple ip-filter (or ACL)

*A:SR1# /configure filter
*A:SR1>config>filter# info
----------------------------------------------
        match-list
            ip-prefix-list "FPL_RFC1918" create
                prefix 10.0.0.0/8
                prefix 172.16.0.0/12
                prefix 192.168.0.0/16
            exit
        exit
        ip-filter 100 create
            default-action forward
            embed-filter flowspec router "Base" offset 1000
            entry 10 create
                match
                    src-ip ip-prefix-list "FPL_RFC1918"
                exit
                action
                    drop
                exit
            exit
            entry 20 create
                match
                    dst-ip ip-prefix-list "FPL_RFC1918"
                exit
                action
                    drop
                exit
            exit
        exit
----------------------------------------------

The match-list ip-prefix-list “FPL_RFC1918” is a way to consolidate a number of network prefixes into a single named list enabling filter configurations to be smaller and more manageable (and easier to debug)

ip-filter 100 is the thing of interest. The default-action is what happens if none of the previous entries are matched – in this case it is a black-list ip-filter, since anything we don’t specifically drop will get forwarded
Each filter entry has a number which is used for the ascending evaluation order (entry 10 before entry 20 here) and usually has a match entry and an action to perform when the match was successful. This filter is relatively simple since all it is doing is blocking IP packets that either have source or destination IP addresses from RFC1918 space.

The interesting thing is the embed-filter flowspec entry. Router “Base” is the base routing instance (as opposed to a VPRN) and offset 1000 means that any flowspec entries will be dynamically added to ip-filter 100 starting at entry 1000 (and with an additional offset determined by flowspec)

This filter will be applied to the IP interface facing the external router – This interface is associated with an Internet Enhanced Service (IES) associated with the SR1’s Global Routing Table (Router “Base”)

*A:SR1>config>service>ies# /configure service ies 1
*A:SR1>config>service>ies# info
----------------------------------------------
            interface "External" create
                address 200.200.200.1/24
                sap 1/1/1 create
                exit
            exit
            no shutdown
----------------------------------------------
*A:SR1>config>service>ies# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     00h20m17s  0
       Loop_1                                                       0
0.0.0.0/0                                     Remote  Static    00h20m02s  5
       200.200.200.2                                                1
123.123.123.123/32                            Local   Local     00h20m17s  0
       system                                                       0
192.168.1.0/24                                Local   Local     00h20m02s  0
       NetOps                                                       0
200.200.200.0/24                              Local   Local     00h20m02s  0
       External                                                     0
-------------------------------------------------------------------------------
No. of Routes: 5
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available

In this contrived example SR1 just has a default route point to the External interface in IES 1.
Before we apply the IP filter – check to see we can reach an RFC1918 IP Address:
A:External# ping 1.1.1.1 source 172.16.0.1 count 2
PING 1.1.1.1 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=1.40ms.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=1.91ms.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min = 1.40ms, avg = 1.66ms, max = 1.91ms, stddev = 0.253ms

Apply the IP Filter in the ingress direction of interface External on SR1:
*A:SR1>config>service>ies# interface "External" sap 1/1/1 ingress filter ip 100
*A:SR1>config>service>ies# info
----------------------------------------------
            interface "External" create
                address 200.200.200.1/24
                sap 1/1/1 create
                    ingress
                        filter ip 100
                    exit
                exit
            exit
            no shutdown

And repeat the test:
*A:External# ping 1.1.1.1 source 172.16.0.1 count 2
PING 1.1.1.1 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 0 packets received, 100% packet loss

We can see the ping failed and if we look at the counters
*A:SR1>config>service>ies# show filter ip 100 counters

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2
Sub-Entries         : 6
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
We can see the ping was unsuccessful because of entry 10 stopping traffic coming back to SR1 (because we weren’t filtering traffic leaving SR1)

Now all of a sudden we find a ping flood (well okay, calling two packets a flood is a rather long bow to draw but anyway) is coming from 100.100.100.100 and hitting 1.1.1.1

A:External# ping 1.1.1.1 source 100.100.100.100 count 2
PING 1.1.1.1 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=1.16ms.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=1.26ms.

These ICMP packets are causing havok to 1.1.1.1, so it would be nice to add some precision filtering to stop that from occuring, while allowing other traffic through – this is where our GoBGP instance and flowspec comes back into the story. We will ask gobgp to announce a flowspec “route” that specifies a discard action (actually a rate-limit of 0 kbps) for traffic from 100.100.100.100 to 1.1.1.1 using ICMP:
adam@m4600:~$ gobgp global rib add -a ipv4-flowspec match source 100.100.100.100/32 destination 1.1.1.1/32 protocol icmp then discard
SR1 can confirm it recieved this route:
*A:SR1>config>service>ies# show router bgp routes flow-ipv4
===============================================================================
 BGP Router ID:123.123.123.123  AS:64512       Local AS:64512
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP FLOW IPV4 Routes
===============================================================================
Flag  Network             Nexthop                 LocalPref       MED
      As-Path
-------------------------------------------------------------------------------
u*>?  --                  0.0.0.0                 100             None
      No As-Path

      Community Action:  rate-limit: 0 kbps
      NLRI Subcomponents:
      Dest Pref : 1.1.1.1/32
      Src Pref  : 100.100.100.100/32
      Ip Proto  : [ == 1 ]
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Flowspec can do quite interesting things, besides rate-limiting traffic to 0kbps (dropping it), it’s possible to use flowspec to tell the router to rate limit flows rather than completely blocking, remark traffic to a different level of priority or even redirect traffic to a VRF by setting the appropriate route targets (this could for example allow the forwarding of traffic into a VPRN or VRF to reach a scrubbing network prior to re-injection of clean traffic back into the network)

Lets see if ip filter has taken this flowspec entry:

*A:SR1>config>service>ies# show filter ip 100

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2/0/0/1 (Fixed/Radius/Cc/Embedded)
Sub-Entries         : 6/0/0/1
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : ip-prefix-list "FPL_RFC1918"
Src. Port           : n/a
Dest. IP            : 0.0.0.0/0
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 0.0.0.0/0
Src. Port           : n/a
Dest. IP            : ip-prefix-list "FPL_RFC1918"
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

Entry               : 1256
Origin              : Inserted by embedded filter fSpec-0 entry 256
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 100.100.100.100/32
Src. Port           : n/a
Dest. IP            : 1.1.1.1/32
Dest. Port          : n/a
Protocol            : 1                            Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
We had specified that the offset for flowspec was 1000, this flowspec entry (number 256) became ip filter 100 entry 256. Time to verify that the ping now should fail:
*A:External# ping 1.1.1.1 source 100.100.100.100 count 2
PING 1.1.1.1 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 0 packets received, 100% packet loss

Yes, and we can verify that it’s only blocking icmp between those endpoints by sending IP packets from another protocol say UDP by doing a traceroute:
*A:External# traceroute 1.1.1.1 source 100.100.100.100
traceroute to 1.1.1.1 from 100.100.100.100, 30 hops max, 40 byte packets
  1  1.1.1.1 (1.1.1.1)    1.74 ms  1.51 ms  1.52 ms
Yes, this is pretty good as far as network based filters (that aren’t performing payload based inspection) are doing, there’s quite a few match conditions that can be used to narrow things down even further. In this contrived example it may be similar effort to configure a filter on the fly but in a production environment, you could hook GoBGP to your route reflector and through a single procedure push updates to all your edge routers in one go. Another good thing about this method, is if the filtering is only temporary, it’s just as easy to remove as it was to put in.
To see what’s currently in the rib for ipv4-flowspec:
adam@m4600:~$ gobgp global rib  -a ipv4-flowspec
    Network                                                              Next Hop             AS_PATH              Age        Attrs
*>  [destination:1.1.1.1/32][source:100.100.100.100/32][protocol:==icmp ]fictitious                                00:12:06   [{Origin: ?} {Extcomms: [discard]}]
Withdraw the announcement:
adam@m4600:~$ gobgp global rib del -a ipv4-flowspec match source 100.100.100.100/32 destination 1.1.1.1/32 protocol icmp then discard
adam@m4600:~$ gobgp global rib  -a ipv4-flowspec                                Network not in table

Verify SR1 ip filter 100 no longer has the entry:
*A:SR1>config>service>ies# show filter ip 100

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2
Sub-Entries         : 6
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : ip-prefix-list "FPL_RFC1918"
Src. Port           : n/a
Dest. IP            : 0.0.0.0/0
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 0.0.0.0/0
Src. Port           : n/a
Dest. IP            : ip-prefix-list "FPL_RFC1918"
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
Entry 1256 has left the building.

Hopefully that was a reasonable introduction to using BGP Flowspec with filtering and GoBGP.

Flowspec is quite often a piece of the puzzle in DDoS mitigation tools, usually there is some kind of automation attached which makes decisions using inputs such a Netflow/IPFIX data, SNMP polling of interfaces to track anomalous loading and in some cases threat intelligence feeds from security vendors.