Simplified Provisioning of RSVP-TE Signaled LSPs with Nokia SROS

This post will demonstrate some RSVP-TE LSP concepts using Nokia SROS. It will start off with the basic method of establishing a RSVP-TE Signaled LSP and a cursory glance of some of the TE options and then will jump into a more useful method to cope with larger deployments, depending on your impatience, it may could be worthwhile to jump directly to the “LSP Templates” section, however if you would like a bit of background, grab a cup of coffee and read on.

MPLS Label Switched Paths (LSPs) established using Label Distribution Protocol (LDP) signaling is fine for many applications and a primary advantage is that it is very quick to deploy and is widely supported across a range of device, however the Interior Gateway Routing Protocol (Typically OSPF or IS-IS) dictates the path taken by the LSP which may not always be desired particularly if you wish to steer particular flows one way or another. LSPs signaled using Resource Reservation Protocol with Traffic Engineering Extensions (RSVP-TE) introduces a lot more fine grained control (however it is equally valid to let the LSP follow the path of the IGP as well) however a disadvantage using such LSPs is that it can become quite cumbersome to build each one particularly if you have many to manage, or there is a lot of ongoing change.

The following 4 router partial mesh will be used to support this the configurations in this post.

4 Router Partial Mesh RSVP-TE

We’ll start with the initial configurations on the routers (simulated 7750 SR12s) which will including everything except for the MPLS related configuration.

configure
    system
        name "R1"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        ethernet
        exit
        no shutdown
    exit
    router
        interface "R2"
            address 10.1.2.1/24
            port 1/1/2
            no shutdown
        exit
        interface "R3"
            address 10.1.3.1/24
            port 1/1/3
            no shutdown
        exit
        interface "system"
            address 10.10.10.1/32
            no shutdown
        exit
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "R2"
                    no shutdown
                exit
                interface "R3"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R2"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        ethernet
        exit
        no shutdown
    exit
    port 1/1/4
        ethernet
        exit
        no shutdown
    exit
    router
        interface "R1"
            address 10.1.2.2/24
            port 1/1/2
            no shutdown
        exit
        interface "R3"
            address 10.2.3.2/24
            port 1/1/4
            no shutdown
        exit
        interface "R4"
            address 10.2.4.2/24
            port 1/1/3
            no shutdown
        exit
        interface "system"
            address 10.10.10.2/32
            no shutdown
        exit
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "R1"
                    no shutdown
                exit
                interface "R3"
                    no shutdown
                exit
                interface "R4"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R3"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        ethernet
        exit
        no shutdown
    exit
    port 1/1/4
        ethernet
        exit
        no shutdown
    exit
    router
        interface "R1"
            address 10.1.3.3/24
            port 1/1/3
            no shutdown
        exit
        interface "R2"
            address 10.2.3.3/24
            port 1/1/4
            no shutdown
        exit
        interface "R4"
            address 10.3.4.3/24
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.3/32
            no shutdown
        exit
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "R1"
                    no shutdown
                exit
                interface "R2"
                    no shutdown
                exit
                interface "R4"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R4"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        ethernet
        exit
        no shutdown
    exit
    router
        interface "R2"
            address 10.2.4.4/24
            port 1/1/3
            no shutdown
        exit
        interface "R3"
            address 10.3.4.4/24
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.4/32
            no shutdown
        exit
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "R3"
                    no shutdown
                exit
                interface "R2"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
exit all
There are 3 initial steps to enable RSVP-TE on Nokia SROS platforms:
1: Add the router interfaces into the MPLS Process (including the system interface) and enable the process. The follow will be applied on R1
configure
    router
        mpls
            interface "system"
                no shutdown
            exit
            interface "R2"
                no shutdown
            exit
            interface "R3"
                no shutdown
            exit
            no shutdown
        exit

2: RSVP automatically adds the interfaces inserted into MPLS into its process, however RSVP is shutdown by default and required enabling
        rsvp
            no shutdown
        exit

3: As we are intending to use constrained paths which are not limited to that of the shortest path, the IGP traffic engineering extensions need enabling
        ospf
            traffic-engineering
        exit
    exit
exit all

Once done we are ready to build MPLS paths and assign them to LSPs but before we do that, here is the equivalent configuration applied to R2, R3 and R4

configure
    router
        mpls
            interface "system"
                no shutdown
            exit
            interface "R1"
                no shutdown
            exit
            interface "R3"
                no shutdown
            exit
            interface "R4"
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            no shutdown
        exit
        ospf
            traffic-engineering
        exit
    exit
exit all
configure
    router
        mpls
            interface "system"
                no shutdown
            exit
            interface "R1"
                no shutdown
            exit
            interface "R2"
                no shutdown
            exit
            interface "R4"
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            no shutdown
        exit
        ospf
            traffic-engineering
        exit
    exit
exit all
configure
    router
        mpls
            interface "system"
                no shutdown
            exit
            interface "R2"
                no shutdown
            exit
            interface "R3"
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            no shutdown
        exit
        ospf
            traffic-engineering
        exit
    exit
exit all

The first LSP to be created will be configured on R1 and destined to R3, however rather than use the directly connected path, the path will go via the outer Square (R1 -> R2 -> R4 -> R3). On R1 we will first define the path between R1 and R3 using strict hops where each hop is the system address of the next router:

/configure
    router
        mpls
            path "R1-R3"
                hop 1 10.10.10.2 strict
                hop 2 10.10.10.4 strict
                hop 3 10.10.10.3 strict
                no shutdown
            exit
Once the path is defined, the LSP “LSP_R1-R3” will be created ensuring that a constraint based shortest based path can be used and using “R1-R3” as the primary path:
            lsp "LSP_R1-R3"
                to 10.10.10.3
                cspf
                primary "R1-R3"
                exit
                no shutdown
            exit
        exit
    exit
exit all
Once configured and enable, we can verify if the LSP is up and running:
*A:R1# show router mpls lsp

===============================================================================
MPLS LSPs (Originating)
===============================================================================
LSP Name                           To               Tun     Fastfail  Adm  Opr
                                                    Id      Config
-------------------------------------------------------------------------------
LSP_R1-R3                          10.10.10.3       3       No        Up   Up
-------------------------------------------------------------------------------
LSPs : 1
===============================================================================
Yes it is, however that summary doesn’t provide details on the path, so a more detailed command will be called
*A:R1# show router mpls lsp "LSP_R1-R3" path detail | match Explicit post-lines 100
Explicit Hops    :
    10.10.10.2(S)      -> 10.10.10.4(S)      -> 10.10.10.3(S)
Actual Hops      :
    10.1.2.1 (10.10.10.1)                        Record Label        : N/A
 -> 10.1.2.2 (10.10.10.2)                        Record Label        : 262143
 -> 10.2.4.4 (10.10.10.4)                        Record Label        : 262139
 -> 10.3.4.3 (10.10.10.3)                        Record Label        : 262143
Computed Hops    :
    10.1.2.1(S)
 -> 10.1.2.2(S)
 -> 10.2.4.4(S)
 -> 10.3.4.3(S)
Resignal Eligible: False
Last Resignal    : n/a                  CSPF Metric          : 300
===============================================================================
The path is taken as specified, and we can also confirm this on the transit (MPLS Label Switch Routers) and terminating (Egress MPLS Label Edge Router)
*A:R2# show router mpls lsp transit
===============================================================================
MPLS LSPs (Transit)
===============================================================================
Legend :  @ - Active Detour
===============================================================================
From            To              In I/F   Out I/F  State LSP Name
-------------------------------------------------------------------------------
10.10.10.1      10.10.10.3      1/1/2    1/1/3    Up    LSP_R1-R3::R1-R3
-------------------------------------------------------------------------------
LSPs : 1
-------------------------------------------------------------------------------
*A:R4# show router mpls lsp transit
===============================================================================
MPLS LSPs (Transit)
===============================================================================
Legend :  @ - Active Detour
===============================================================================
From            To              In I/F   Out I/F  State LSP Name
-------------------------------------------------------------------------------
10.10.10.1      10.10.10.3      1/1/3    1/1/2    Up    LSP_R1-R3::R1-R3
-------------------------------------------------------------------------------
LSPs : 1
-------------------------------------------------------------------------------
*A:R3# show router mpls lsp terminate
===============================================================================
MPLS LSPs (Terminate)
===============================================================================
Legend :  @ - Active Detour
===============================================================================
From            To              In I/F   Out I/F  State LSP Name
-------------------------------------------------------------------------------
10.10.10.1      10.10.10.3      1/1/2    n/a      Up    LSP_R1-R3::R1-R3
-------------------------------------------------------------------------------
LSPs : 1
-------------------------------------------------------------------------------
As MPLS LSPs are unidirectional, we would need to provision a corresponding LSP on R3 to terminate on R1, to have something more than just a demonstration – however I’ll skip that step and shutdown the LSP on R1 and do something a little different.*A:R1# /configure router mpls lsp "LSP_R1-R3" shutdownFirstly we are going to define an admin-group or link color which can be used to give input into the specific hops the path of an LSP will take (rules can be made based on the presence or non-presence of certain admin-groups however with 32 values available, care needs to be given on how these are allocated)
/configure
    router
        if-attribute
            admin-group "R1-R2" value 5
        exit
Then this admin-group will be associated to the R2 facing interface within the MPLS context
        mpls
            interface "R2"
                admin-group "R1-R2"
            exit
Now to create an unconstrained path (by simply creating and administratively enabling the path, providing nothing else limits the selection criteria, the IGP path will be used for the LSP)
            path "PATH_LOOSE"
                no shutdown
            exit
First we will create a LSP to R1 that is not constrained at all:
            lsp "LSP_R1-R2-U"
                to 10.10.10.2
                cspf
                primary "PATH_LOOSE"
                exit
                no shutdown
            exit
and now a constrained path that is configured in the same manner, except that it requires the path to exclude links with the R1-R2 admin-group
            lsp "LSP_R1-R2-C"
                to 10.10.10.2
                cspf
                exclude "R1-R2"
                primary "PATH_LOOSE"
                exit
                no shutdown
            exit
To compare the two different paths taken for each LSP:
*A:R1>config>router>mpls# show router mpls lsp "LSP_R1-R2-U" path detail | match Actual post-lines 100
Actual Hops      :
    10.1.2.1 (10.10.10.1)                        Record Label        : N/A
 -> 10.1.2.2 (10.10.10.2)                        Record Label        : 262143
Computed Hops    :
    10.1.2.1(S)
 -> 10.1.2.2(S)
Resignal Eligible: False
Last Resignal    : n/a                  CSPF Metric          : 100
===============================================================================
*A:R1>config>router>mpls# show router mpls lsp "LSP_R1-R2-C" path detail | match Actual post-lines 100
Actual Hops      :
    10.1.3.1 (10.10.10.1)                        Record Label        : N/A
 -> 10.1.3.3 (10.10.10.3)                        Record Label        : 262143
 -> 10.2.3.2 (10.10.10.2)                        Record Label        : 262142
Computed Hops    :
    10.1.3.1(S)
 -> 10.1.3.3(S)
 -> 10.2.3.2(S)
Resignal Eligible: False
Last Resignal    : n/a                  CSPF Metric          : 200
===============================================================================
The output shows that the constrained path avoided the R1-R2 link due to the admin-group “R1-R2” attached to the interface and the directive to exclude links with that admin-group. R2 can be used to confirm the entry of each LSP as well with the terminate directive:
*A:R2# show router mpls lsp terminate
===============================================================================
MPLS LSPs (Terminate)
===============================================================================
Legend :  @ - Active Detour
===============================================================================
From            To              In I/F   Out I/F  State LSP Name
-------------------------------------------------------------------------------
10.10.10.1      10.10.10.2      1/1/2    n/a      Up    LSP_R1-R2-U::PATH_LOOSE
10.10.10.1      10.10.10.2      1/1/4    n/a      Up    LSP_R1-R2-C::PATH_LOOSE
-------------------------------------------------------------------------------
LSPs : 2
-------------------------------------------------------------------------------

This post has not touched upon secondary or standby paths, Fast Reroute, bandwidth reservation, set up or hold priority or a myriad of other features that may be considered within a production environment, however even when using admin-groups as a tool to steer traffic, the setup of LSPs can become quite tiresome in very short order, if you require a full-mesh of LSPs to be defined, each time you add a router to the network, you have to revist each router and build an LSP to the new one (similar to the IBGP problem before concepts like route reflectors reduced the configuration overhead) – larger configurations are more prone to errors and can make the time to deploy nodes more painful which may result in deciding that LDP signaled LSPs are a reasonable trade off.

LSP-Templates

LSP-Templates are one way to help take back the configuration overhead for initial builds, and to make it easier when adding nodes to an existing network. This example will be performed on R2. Define the a loose path with the name “PATH_LOOSE” as was done on R1 and create a Point-to-Point Mesh LSP-Template named AUTOLSP, that will use that path and be enabled for cspf:

configure
    router
        mpls
            path "PATH_LOOSE"
                no shutdown
            exit
            lsp-template "AUTOLSP" mesh-p2p
                default-path "PATH_LOOSE"
                cspf
                no shutdown
            exit
        exit all
To leverage the LSP-Template, we need to create a routing-policy which will be used to determine what nodes we will wish to establish a LSP with. In this case each nodes system address is in the 10.10.10.0/24 network, so we’ll create a match for host routes within that network using the prefix list and policy statement below.
configure
    router
        policy-options
            begin
            prefix-list "PL_MPLSNETWORK"
                prefix 10.10.10.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_AUTOMESH_RSVP"
                entry 10
                    from
                        prefix-list "PL_MPLSNETWORK"
                    exit
                    action accept
                    exit
                exit
            exit
            commit
        exit all
The OSPF router LSA will be associated with the system addresses (since we haven’t changed the router ids):
*A:R2# show router ospf database type router

===============================================================================
Rtr Base OSPFv2 Instance 0 Link State Database (type: Router)
===============================================================================
Type    Area Id         Link State Id   Adv Rtr Id      Age  Sequence   Cksum
-------------------------------------------------------------------------------
Router  0.0.0.0         10.10.10.1      10.10.10.1      1698 0x8000000a 0xaf19
Router  0.0.0.0         10.10.10.2      10.10.10.2      1114 0x8000000c 0xbb6a
Router  0.0.0.0         10.10.10.3      10.10.10.3      67   0x8000000d 0x1efb
Router  0.0.0.0         10.10.10.4      10.10.10.4      101  0x8000000b 0x4d5c
-------------------------------------------------------------------------------
No. of LSAs: 4
===============================================================================
Back within the MPLS configuration context, auto-created lsps using the lsp-template AUTOLSP and the routing policy PS_AUTOMESH_RSVP will be enabled:
configure
    router
        mpls
            auto-lsp lsp-template "AUTOLSP" policy "PS_AUTOMESH_RSVP"
        exit
    exit all
Now to determine what LSPs have been automatically created:
*A:R2# show router mpls lsp auto-lsp mesh-p2p

===============================================================================
MPLS Auto-LSP
===============================================================================
LSP Name                           Type             Fastfail    Admin   Oper
                                                    Config      State   State
-------------------------------------------------------------------------------
AUTOLSP-10.10.10.1-61453           MeshP2P          Yes         Up      Up
AUTOLSP-10.10.10.3-61454           MeshP2P          Yes         Up      Up
AUTOLSP-10.10.10.4-61455           MeshP2P          Yes         Up      Up
-------------------------------------------------------------------------------
Auto-LSPs : 3
===============================================================================
The LSP name is system derived from the lsp-template name, the destination router, and the tunnel id:
*A:R2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
10.10.10.1/32     rsvp      MPLS  61453     7        10.1.2.1       100
10.10.10.3/32     rsvp      MPLS  61454     7        10.2.3.3       100
10.10.10.4/32     rsvp      MPLS  61455     7        10.2.4.4       100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================
The same configuration can be applied to R1, R3 and R4, when consolidated it looks like this:
configure
    router
        policy-options
            begin
            prefix-list "PL_MPLSNETWORK"
                prefix 10.10.10.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_AUTOMESH_RSVP"
                entry 10
                    from
                        prefix-list "PL_MPLSNETWORK"
                    exit
                    action accept
                    exit
                exit
            exit
            commit
        exit
        mpls
            path "PATH_LOOSE"
                no shutdown
            exit
            lsp-template "AUTOLSP" mesh-p2p
                default-path "PATH_LOOSE"
                cspf
                no shutdown
            exit
            auto-lsp lsp-template "AUTOLSP" policy "PS_AUTOMESH_RSVP"
        exit
    exit
exit all

What I really like about this, is that it brings configuration consistency, relatively small configurations that can do some good things, particularly if you have a well thought out deployment strategy – e.g. if you deploy another router, e.g. R5 with the system address of 10.10.10.5/32, R1-R4 will automatically attempt to establish LSPs to it which resulting in lower touch provisioning when adding new network elements to your network. LSP-Templates have been around on the 7×50 platform for some time, however it is only recently that the software on the 7705 SAR platforms have had this feature introduced allowing simplified RSVP-TE provisioning on large, medium and small platforms.

Please note that the specific LSP-Template that was used in this example is very simple and only used to demonstrate the basic concept – in a live deployment additional attributes such as fast-reroute, adspec and admin-groups if appropriate should be considered.

Carrier Supporting Carrier with SROS

Carrier of Carriers or Carrier supporting Carrier is a hierarchical construct to allow a network provider to provide MPLS connectivity with relatively low complexity. While it is even simpler to build layer 2 circuits over a backhaul network, in some cases routed connectivity may be preferred as you have a common connection point to the far end regions without needing to concern yourself with the provisioning scale and bandwidth management of each path. While this is quite a large posting it is mainly to show the configuration and verification steps. The summary of actions to create a CsC configuration is at the end of the post if you run out of patience.

This Example will have a three router Carrier of Carriers Network Operator (SC-1, SC-2 and SC-3) providing connectivity between a customer (VPRN 200) with two regions – Region A (RA-1, RA-2 and RA-3) and Region B (RB-1, RB-2 and RB-3)

9 Router Carrier Supporting Carrier Topology

Prior to SROS version 14.0R4, a single RIB was used for both labeled and unlabeled prefixes. With 14.0R4, two RIBs were established (actually, two RIBs for IPv4 and another two for IPv6) – this demonstration will have half the routers using 12.0R6 and half running 14.0R8 to highlight the configuration differences. The routers that are running SROS 14.0R8 in this example are configured using the VSR-D model which has the Control Plane Module operating in a VM (VM-CP) independent of the Input/Output Module (VM-DP) each VM-CP and VM-DP is logically connected via Switch Fabric ports – this is transparent to the configuration and operation of the simulated routers.

The initial configurations for Region A without the CSC Uplink on RA-1 as as below. OSPF as the IGP, LDP signalled LSPs are used, RA-3 is the local VPN Route Reflector for AS65000, VPRN 123 is created on each router.

configure
    system
        name "RA-1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-2"
            address 1.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "RA-3"
            address 1.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 1.1.1.1/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-2"
                    no shutdown
                exit
                interface "RA-3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-2"
                exit
                interface "RA-3"
                exit
            exit
            no shutdown
        exit
        bgp
            split-horizon
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.1:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.1/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RA-2"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-1"
            address 1.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "RA-3"
            address 1.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 1.1.1.2/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-1"
                    no shutdown
                exit
                interface "RA-3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-1"
                exit
                interface "RA-3"
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.2:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.2/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RA-3"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-1"
            address 1.1.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "RA-2"
            address 1.2.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "system"
            address 1.1.1.3/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-1"
                    no shutdown
                exit
                interface "RA-2"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-1"
                exit
                interface "RA-2"
                exit
            exit
        exit
        bgp
            group "VPN-RRC"
                family vpn-ipv4
                cluster 1.1.1.3
                peer-as 65000
                neighbor 1.1.1.1
                    description "RA-1"
                exit
                neighbor 1.1.1.2
                    description "RA-2"
                exit
            exit
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.3:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.3/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

Similarly for Region B, the initial configurations without the CSC Uplink on RB-1 as as below. OSPF as the IGP, LDP signalled LSPs, RB-3 is the local VPN Route Reflector for AS65000, VPRN 123 is created on each router. Besides the card/mda differences on the emulated hardware, the primary difference here compared to Region A which is using an older SROS version is the LSP binding (auto-bind) syntax changes to give more control as to the LSP types that may be desired.

configure
    system
        name "RB-1"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-2"
            address 2.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "RB-3"
            address 2.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 2.2.2.1/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-2"
                    no shutdown
                exit
                interface "RB-3"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-2" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-3" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            split-horizon
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.1:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.1/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RB-2"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-1"
            address 2.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "RB-3"
            address 2.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 2.2.2.2/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-1"
                    no shutdown
                exit
                interface "RB-3"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-1" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-3" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.2:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.2/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RB-3"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-1"
            address 2.1.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "RB-2"
            address 2.2.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "system"
            address 2.2.2.3/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-1"
                    no shutdown
                exit
                interface "RB-2"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-1" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-2" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RRC"
                family vpn-ipv4
                cluster 2.2.2.3
                peer-as 65000
                neighbor 2.2.2.1
                    description "RB-1"
                exit
                neighbor 2.2.2.2
                    description "RB-2"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.3:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.3/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

The CSC Operators Network uses IS-IS as the IGP, RSVP-TE signalled LSPs, SC-R3 is the local VPN Route Reflector for AS64512.

configure
    system
        name "SC-R1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R2"
            address 10.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R3"
            address 10.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.1/32
            no shutdown
        exit
        autonomous-system 64512
        isis
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R2"
                to 10.10.10.2
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R3"
                to 10.10.10.3
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                peer-as 64512
                neighbor 10.10.10.3
                    description "SC-R3"
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "SC-R2"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R3"
            address 10.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.2/32
            no shutdown
        exit
        autonomous-system 64512
        isis 0
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R3"
                to 10.10.10.3
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R1"
                to 10.10.10.1
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                peer-as 64512
                neighbor 10.10.10.3
                    description "SC-R3"
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "SC-R3"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R2"
            address 10.2.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.3/32
            no shutdown
        exit
        autonomous-system 64512
        isis
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R1"
                to 10.10.10.1
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R2"
                to 10.10.10.2
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                cluster 10.10.10.3
                peer-as 64512
                neighbor 10.10.10.1
                    description "SC-R1"
                exit
                neighbor 10.10.10.2
                    description "SC-R2"
                exit
            exit
            no shutdown
        exit
    exit
exit all

At this stage, we have three Islands – Region A, Region B and the Carrier of Carriers Network (without the CSC configuration)

RA-1 is the Region A CSC-CE Router (which is also operating as a regular PE Router) which peers with the CSC Network SC-R1 to exchange labeled routes using BGP. Note: Both of these routers are running SROS 12.0R6 and are configured in the “old way”

Create VPRN 200 on SC-R1 and BGP Peering for RA-1:

configure
    router
        policy-options
            begin
            policy-statement "PS_MPBGP_VPN_TO_BGP"
                entry 10
                    from
                        protocol bgp-vpn
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
    exit
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    service
        vprn 200 customer 1 create
            carrier-carrier-vpn
            autonomous-system 64512
            route-distinguisher 10.10.10.1:200
            auto-bind rsvp-te
            vrf-target target:64512:200
            network-interface "RA-1" create
                address 10.1.1.11/24
                port 1/1/3
                no shutdown
            exit
            bgp
                split-horizon
                group "RegionA"
                    family ipv4
                    as-override
                    export "PS_MPBGP_VPN_TO_BGP"
                    peer-as 65000
                    neighbor 10.1.1.1
                        advertise-label ipv4
                    exit
                exit
                no shutdown
            exit
            no shutdown
        exit
    exit
exit all

Of note is the definition of the VPRN type: carrier-carrier-vpn, once enabled regular IP interfaces cannot be created, only “network-interfaces” which must be on Ethernet ports that are of mode network. As both Region A and Region B both use AS 65000, as-override has been used to stop AS-Path loops occuring when advertising prefixes from other regions with the same BGP AS. We are using the IPv4 Address family but are specifically advertising associated MPLS Labels (This particular configuration component is different after SROS Release 14.0R4) Routing policy is defined and used to export VPN routes to RA-1.

Create the BGP Peering for RA-1 with SC-R1:

configure
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.1.1/24
            port 1/1/3
            no shutdown
        exit
        policy-options
            begin
            prefix-list "RegionA"
                prefix 1.1.1.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_BGP_EXP_CSC"
                entry 10
                    from
                        prefix-list "RegionA"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        bgp
            group "CSC"
                export "PS_BGP_EXP_CSC"
                neighbor 10.1.1.11
                    description "SC-R1"
                    peer-as 64512
                    advertise-label ipv4 include-ldp-prefix
                exit
            exit
        exit
    exit all

Routing policy is used to advertise any system addresses from Region A (1.1.1.x/32) with MPLS labels to the CSC

RB-1 is the Region B CSC-CE Router (which is also operating as a regular PE Router) which peers with the CSC Network SC-R2 to exchange labeled routes using BGP. Note: Both of these routers are running SROS 14.0R8 and are configured in the “new way”

Create VPRN 200 on SC-R2 and BGP Peering for RB-1:

configure
    router
        policy-options
            begin
            policy-statement "PS_MPBGP_VPN_TO_BGP"
                entry 10
                    from
                        protocol bgp-vpn
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
    exit
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    service
        vprn 200 customer 1 create
            carrier-carrier-vpn
            autonomous-system 64512
            route-distinguisher 10.10.10.2:200
            auto-bind-tunnel
                resolution-filter
                    rsvp
                exit
                resolution filter
            exit
            vrf-target target:64512:200
            network-interface "RB-1" create
                address 10.2.2.11/24
                port 1/1/3
                no shutdown
            exit
            bgp
                split-horizon
                group "RB"
                    family label-ipv4
                    as-override
                    export "PS_MPBGP_VPN_TO_BGP"
                    peer-as 65000
                    neighbor 10.2.2.1
                    exit
                exit
                no shutdown
            exit
            no shutdown
        exit
    exit
exit all

The main difference compared with the configuration on SC-1 besides the auto-bind-tunnel changes, are that we are specifically peering using the label-ipv4 address family instead of the ipv4 address family and advertising labels.

Create the BGP Peering for RB-1 with SC-R2:

configure
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    router
        interface "SC-R2"
            address 10.2.2.1/24
            port 1/1/3
            no shutdown
        exit
        policy-options
            begin
            prefix-list "RegionB"
                prefix 2.2.2.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_BGP_EXP_CSC"
                entry 10
                    from
                        prefix-list "RegionB"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        bgp
            group "CSC"
                export "PS_BGP_EXP_CSC"
                neighbor 10.2.2.11
                    description "SC-R2"
                    family label-ipv4
                    peer-as 64512
                    advertise-ldp-prefix
                exit
            exit
        exit
    exit all

Routing policy is used to advertise any system addresses from Region B (2.2.2.x/32) with MPLS labels to the CSC

Once all peers are configured, examine the CSC VPRN 200 Routing Tables:

A:SC-R1#  show router 200 route-table

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
1.1.1.2/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
1.1.1.3/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
2.2.2.1/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
2.2.2.2/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
2.2.2.3/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
10.1.1.0/24                                   Local   Local     00h08m29s  0
       RA-1                                                         0
10.2.2.0/24                                   Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
-------------------------------------------------------------------------------
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

A:SC-R2# show router 200 route-table

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
1.1.1.2/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
1.1.1.3/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
2.2.2.1/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
2.2.2.2/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
2.2.2.3/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
10.1.1.0/24                                   Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
10.2.2.0/24                                   Local   Local     02h11m42s  0
       RB-1                                                         0
-------------------------------------------------------------------------------
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

With Region A and Region B connectivity via the CSC we should be able to see the initial list of tunnels on the CSC-CE routers (RA-1 and RA-2):

A:RA-1# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.2/32            ldp   MPLS   -        9        1.1.2.2        100
1.1.1.3/32            ldp   MPLS   -        9        1.1.3.3        100
2.2.2.1/32            bgp   MPLS   -        10       10.1.1.11      1000
2.2.2.2/32            bgp   MPLS   -        10       10.1.1.11      1000
2.2.2.3/32            bgp   MPLS   -        10       10.1.1.11      1000
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-1# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        bgp       MPLS  262301    12       10.2.2.11      1000
1.1.1.2/32        bgp       MPLS  262303    12       10.2.2.11      1000
1.1.1.3/32        bgp       MPLS  262304    12       10.2.2.11      1000
2.2.2.2/32        ldp       MPLS  65540     9        2.1.2.2        100
2.2.2.3/32        ldp       MPLS  65541     9        2.1.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-2# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.2.1        100
1.1.1.3/32            ldp   MPLS   -        9        1.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-3# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.3.1        100
1.1.1.2/32            ldp   MPLS   -        9        1.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
2.2.2.1/32        ldp       MPLS  65596     9        2.1.2.1        100
2.2.2.3/32        ldp       MPLS  65544     9        2.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-3# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
2.2.2.1/32        ldp       MPLS  65597     9        2.1.3.1        100
2.2.2.2/32        ldp       MPLS  65537     9        2.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

RA-2, RA-3, RB-2 and RB-3 need the CSC-CEs to to redistribute the prefixes we learnt from the CSC into OSPF and LDP.
configure
    router
        policy-options
            begin
            prefix-list "RegionB"
                prefix 2.2.2.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_OSPF_LDP_EXP_RB"
                entry 10
                    from
                        prefix-list "RegionB"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        ospf
            asbr
            export "PS_OSPF_LDP_EXP_RB"
        exit
        ldp
            export-tunnel-table "PS_OSPF_LDP_EXP_RB"
        exit
    exit
exit all

configure
    router
        policy-options
            begin
            prefix-list "RegionA"
                prefix 1.1.1.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_OSPF_LDP_EXP_RA"
                entry 10
                    from
                        prefix-list "RegionA"
                    exit
                    action accept
                    exit
                exit
                default-action drop
                exit
            exit
            commit
        exit
        ospf
            asbr
            export "PS_OSPF_LDP_EXP_RA"
        exit
        ldp
            export-tunnel-table "PS_OSPF_LDP_EXP_RA"
        exit
Remember: SROS specifically requires OSPF in the GRT to be defined as an ASBR if you intend to export routes.
A:RA-2# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.2.1        100
1.1.1.3/32            ldp   MPLS   -        9        1.2.3.3        100
2.2.2.1/32            ldp   MPLS   -        9        1.1.2.1        1
2.2.2.2/32            ldp   MPLS   -        9        1.1.2.1        1
2.2.2.3/32            ldp   MPLS   -        9        1.1.2.1        1
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-3# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.3.1        100
1.1.1.2/32            ldp   MPLS   -        9        1.2.3.2        100
2.2.2.1/32            ldp   MPLS   -        9        1.1.3.1        1
2.2.2.2/32            ldp   MPLS   -        9        1.1.3.1        1
2.2.2.3/32            ldp   MPLS   -        9        1.1.3.1        1
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        ldp       MPLS  65619     9        2.1.2.1        1
1.1.1.2/32        ldp       MPLS  65620     9        2.1.2.1        1
1.1.1.3/32        ldp       MPLS  65621     9        2.1.2.1        1
2.2.2.1/32        ldp       MPLS  65596     9        2.1.2.1        100
2.2.2.3/32        ldp       MPLS  65544     9        2.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-3# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        ldp       MPLS  65620     9        2.1.3.1        1
1.1.1.2/32        ldp       MPLS  65621     9        2.1.3.1        1
1.1.1.3/32        ldp       MPLS  65622     9        2.1.3.1        1
2.2.2.1/32        ldp       MPLS  65597     9        2.1.3.1        100
2.2.2.2/32        ldp       MPLS  65537     9        2.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================
Okay, we have tunnels between all routers in both Regions A and B.
The final step to get VPRN connectivity between Region A and Region B is to peer the VPN Route Reflectors (RA-3 and RB-3)
configure
        bgp
            group "VPN-RR-RegionB"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
        exit
    exit all

configure
        bgp
            group "VPN-RR-RegionA"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
        exit
    exit all

We should be able to now verify this with VPRN 123
A:RA-1# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Local   Local     00h50m13s  0
       Loop                                                         0
123.1.1.2/32                                  Remote  BGP VPN   00h44m50s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h45m49s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.1 (tunneled:BGP)                                       0
123.2.2.2/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.2 (tunneled:BGP)                                       0
123.2.2.3/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.3 (tunneled:BGP)                                       0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-1# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=0.245ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.245ms, avg = 0.245ms, max = 0.245ms, stddev = 0.000ms
A:RA-1# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=1.26ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.26ms, avg = 1.26ms, max = 1.26ms, stddev = 0.000ms
A:RA-1# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=1.37ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.37ms, avg = 1.37ms, max = 1.37ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=3.46ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.46ms, avg = 3.46ms, max = 3.46ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=3.92ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.92ms, avg = 3.92ms, max = 3.92ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=3.83ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.83ms, avg = 3.83ms, max = 3.83ms, stddev = 0.000ms

A:RA-2# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h46m12s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Local   Local     00h48m34s  0
       Loop                                                         0
123.1.1.3/32                                  Remote  BGP VPN   00h46m12s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-2# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=1.12ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.12ms, avg = 1.12ms, max = 1.12ms, stddev = 0.000ms
A:RA-2# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=0.171ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.171ms, avg = 0.171ms, max = 0.171ms, stddev = 0.000ms
A:RA-2# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=1.31ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.31ms, avg = 1.31ms, max = 1.31ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=4.12ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.12ms, avg = 4.12ms, max = 4.12ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=7.76ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 7.76ms, avg = 7.76ms, max = 7.76ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=4.53ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.53ms, avg = 4.53ms, max = 4.53ms, stddev = 0.000ms

A:RA-3# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h47m48s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h46m56s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Local   Local     00h49m05s  0
       Loop                                                         0
123.2.2.1/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-3>config>router# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=1.17ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.17ms, avg = 1.17ms, max = 1.17ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=1.07ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.07ms, avg = 1.07ms, max = 1.07ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=0.166ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.166ms, avg = 0.166ms, max = 0.166ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=6.13ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 6.13ms, avg = 6.13ms, max = 6.13ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=5.04ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.04ms, avg = 5.04ms, max = 5.04ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=4.37ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.37ms, avg = 4.37ms, max = 4.37ms, stddev = 0.000ms

A:RB-1# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.1 (tunneled:BGP)                                       0
123.1.1.2/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.2 (tunneled:BGP)                                       0
123.1.1.3/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.3 (tunneled:BGP)                                       0
123.2.2.1/32                                  Local   Local     02h35m54s  0
       Loop                                                         0
123.2.2.2/32                                  Remote  BGP VPN   02h11m21s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   02h11m21s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-1# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=4.42ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.42ms, avg = 4.42ms, max = 4.42ms, stddev = 0.000ms
A:RB-1# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=3.89ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.89ms, avg = 3.89ms, max = 3.89ms, stddev = 0.000ms
A:RB-1# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=5.97ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.97ms, avg = 5.97ms, max = 5.97ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=0.118ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.118ms, avg = 0.118ms, max = 0.118ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=1.96ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.96ms, avg = 1.96ms, max = 1.96ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=1.80ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.80ms, avg = 1.80ms, max = 1.80ms, stddev = 0.000ms

A:RB-2# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   02h11m55s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Local   Local     03h04m35s  0
       Loop                                                         0
123.2.2.3/32                                  Remote  BGP VPN   02h36m00s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-2# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=3.43ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.43ms, avg = 3.43ms, max = 3.43ms, stddev = 0.000ms
A:RB-2# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=5.63ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.63ms, avg = 5.63ms, max = 5.63ms, stddev = 0.000ms
A:RB-2# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=4.40ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.40ms, avg = 4.40ms, max = 4.40ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=1.85ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.85ms, avg = 1.85ms, max = 1.85ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=0.150ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.150ms, avg = 0.150ms, max = 0.150ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=1.90ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.90ms, avg = 1.90ms, max = 1.90ms, stddev = 0.000ms

A:RB-3# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   02h13m05s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   02h37m03s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Local   Local     02h37m07s  0
       Loop                                                         0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-3# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=3.83ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.83ms, avg = 3.83ms, max = 3.83ms, stddev = 0.000ms
A:RB-3# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=4.38ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.38ms, avg = 4.38ms, max = 4.38ms, stddev = 0.000ms
A:RB-3# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=4.47ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.47ms, avg = 4.47ms, max = 4.47ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=1.97ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.97ms, avg = 1.97ms, max = 1.97ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=1.88ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.88ms, avg = 1.88ms, max = 1.88ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=0.269ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.269ms, avg = 0.269ms, max = 0.269ms, stddev = 0.000ms
While this is quite a long post, the actual configuration itself for the CSC is straight forward:

  • The CSC VPRN must be set to carrier-carrier-vpn
  • The CSC VPRN uses Ethernet ports with mode network even though its for customers
  • The CSC-PE and CSC-CE exchange labeled routes
  • The CSC-CE redistributes between OSPF/LDP and BGP

Changes to the way the RIB works on SROS and how that impacts CsC configurations was also briefly discussed.

Resolving OSPF MTU problems with SROS

OSPF is a popular Interior Gateway Routing Protocol and in many instances it “just works” for a lot of situations, however care must be taken even in simple deployments. An issue that comes up from time to time is with regards to the maximum transmission unit (MTU). The network topology is a three router topology where I only have direct control a Nokia SROS based router.

OSPF MTU Test Topology

TL;DR – OSPF neighbor in ExchStart – you need to increase your MTU, OSPF neighbor in Exchange – you need to decrease your MTU. Keep reading to see how you can identify and resolve the MTU issues on Nokia Routers with SROS.

Below is the configuration of SR (the router under our administrative control):

configure
    system
        name "SR"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
            mode access
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
            mode access
        exit
        no shutdown
    exit
#--------------------------------------------------
echo "Router (Network Side) Configuration"
#--------------------------------------------------
    router Base
        interface "system"
            address 1.1.1.1/32
            no shutdown
        exit
#--------------------------------------------------
echo "OSPFv2 Configuration"
#--------------------------------------------------
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit

#--------------------------------------------------
echo "Service Configuration"
#--------------------------------------------------
    service
        customer 1 create
            description "Default customer"
        exit
        ies 100 customer 1 create
            description "PEER1"
            interface "PEER1" create
                address 10.1.2.1/27
                sap 1/1/1 create
                exit
            exit
            no shutdown
        exit
        ies 200 customer 1 create
            description "PEER2"
            interface "PEER2" create
                address 10.1.3.1/27
                sap 1/1/2 create
                exit
            exit
            no shutdown
        exit
    exit
#--------------------------------------------------
echo "Router (Service Side) Configuration"
#--------------------------------------------------
    router
        ospf 0
            area 0.0.0.1
                interface "PEER1"
                    no shutdown
                exit
                interface "PEER2"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
exit all

One thing to note is that the Peer routers are attached to an Internet Enhanced Service (IES) and not part of the OSPF Backbone Area – from a stored configuration perspective there is a distinction between core network and customer configurations but from a protocol pespective things are the same. IES Interfaces that are bound to Service Access Points (SAPs) which must be changed from the default mode of network – in this case we are using access, however hybrid is an option as well.

As this post is about resolving issues, obviously things are not working as straight forward as expected.

A:SR# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     00h40m51s  0
       system                                                       0
10.1.2.0/27                                   Local   Local     00h33m32s  0
       PEER1                                                        0
10.1.3.0/27                                   Local   Local     00h34m02s  0
       PEER2                                                        0
-------------------------------------------------------------------------------
No. of Routes: 3
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

So nothing from OSPF is in the routing table, while its possible (but unlikely) that our peers aren’t advertising anything, an alternate explanation is that it could be a connectivity issue, so we’ll ping each peer router first
A:SR# ping 10.1.2.2 count 3
PING 10.1.2.2 56 data bytes
64 bytes from 10.1.2.2: icmp_seq=1 ttl=64 time=1.28ms.
64 bytes from 10.1.2.2: icmp_seq=2 ttl=64 time=1.15ms.
64 bytes from 10.1.2.2: icmp_seq=3 ttl=64 time=1.08ms.

---- 10.1.2.2 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.08ms, avg = 1.17ms, max = 1.28ms, stddev = 0.081ms
A:SR# ping 10.1.3.3 count 3
PING 10.1.3.3 56 data bytes
64 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.51ms.
64 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.31ms.
64 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.24ms.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.24ms, avg = 1.35ms, max = 1.51ms, stddev = 0.116ms
Okay so IP connectivity is established, lets check the OSPF interface state
A:SR# show router ospf interface

===============================================================================
Rtr Base OSPFv2 Instance 0 Interfaces
===============================================================================
If Name               Area Id         Designated Rtr  Bkup Desig Rtr  Adm  Oper
-------------------------------------------------------------------------------
system                0.0.0.0         1.1.1.1         0.0.0.0         Up   DR
PEER1                 0.0.0.1         1.1.1.1         100.100.100.100 Up   DR
PEER2                 0.0.0.1         1.1.1.1         0.0.0.0         Up   DR
-------------------------------------------------------------------------------
No. of OSPF Interfaces: 3
===============================================================================

From first glance PEER1 seems okay but PEER2 doesn’t have a BDR and since we are using the default ospf interface type (broadcast) we would expect that to see both the DR and BDR – lets get some more details
A:SR# show router ospf interface "PEER2" detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Interface "PEER2" (detail)
===============================================================================
-------------------------------------------------------------------------------
Configuration
-------------------------------------------------------------------------------
IP Address       : 10.1.3.1
Area Id          : 0.0.0.1              Priority         : 1
Hello Intrvl     : 10 sec               Rtr Dead Intrvl  : 40 sec
Retrans Intrvl   : 5 sec                Poll Intrvl      : 120 sec
Cfg Metric       : 0                    Advert Subnet    : True
Transit Delay    : 1                    Cfg IF Type      : None
Passive          : False                Cfg MTU          : 0
LSA-filter-out   : None                 Adv Rtr Capab    : Yes
LFA              : Include              LFA NH Template  :
RIB-priority     : None
Auth Type        : None
-------------------------------------------------------------------------------
State
-------------------------------------------------------------------------------
Admin Status     : Enabled              Oper State       : Designated Rtr
Designated Rtr   : 1.1.1.1              Backup Desig Rtr : 0.0.0.0
IF Type          : Broadcast            Network Type     : Stub
Oper MTU         : 1500                 Last Enabled     : 06/02/2017 01:44:23
Oper Metric      : 100                  Bfd Enabled      : No
Te Metric        : 100                  Te State         : Down
Admin Groups     : None
Ldp Sync         : outOfService         Ldp Sync Wait    : Disabled
Ldp Timer State  : Disabled             Ldp Tm Left      : 0
-------------------------------------------------------------------------------
Statistics
-------------------------------------------------------------------------------
Nbr Count        : 0                    If Events        : 2
Tot Rx Packets   : 0                    Tot Tx Packets   : 76
Rx Hellos        : 0                    Tx Hellos        : 76
Rx DBDs          : 0                    Tx DBDs          : 0
Rx LSRs          : 0                    Tx LSRs          : 0
Rx LSUs          : 0                    Tx LSUs          : 0
Rx LS Acks       : 0                    Tx LS Acks       : 0
Retransmits      : 0                    Discards         : 78
Bad Networks     : 0                    Bad Virt Links   : 0
Bad Areas        : 78                   Bad Dest Addrs   : 0
Bad Auth Types   : 0                    Auth Failures    : 0
Bad Neighbors    : 0                    Bad Pkt Types    : 0
Bad Lengths      : 0                    Bad Hello Int.   : 0
Bad Dead Int.    : 0                    Bad Options      : 0
Bad Versions     : 0                    Bad Checksums    : 0
LSA Count        : 0                    LSA Checksum     : 0x0
===============================================================================
Okay, we can see that there are discards which align with the Bad Area Count – this means that PEER2 doesn’t believe it’s part of OSPF Area 1.

Log-id 99 is automatically configured on Nokia SROS devices to capture a number of event messages however it can get a bit overwhelming to find something specific. Fortunately there are ways to reduce the output by specifying the application (OSPF) and something that may be part of the log message itself we want to see (PEER2)

A:SR# show log log-id 99 application OSPF message PEER2

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=944  (wrapped)]

941 2017/06/02 02:10:15.55 UTC WARNING: OSPF #2043 Base VR:  1 OSPFv2 (0)
"LCL_RTR_ID 1.1.1.1: Conflicting configuration areaMismatch on interface PEER2 from 10.1.3.3 in hello"
So while we have identified a problem – OSPF Area MisMatch, we need to overcome it – remembering we cant configure PEER2 (the person that manages it is on a training course and cannot be contacted, while your project manager is wanting solutions, not problems..)

This is where using show and debug commands can help identify and resolve issues – SROS is quite powerful with its debugging tools and while they can be used in production, it is always best to attempt to narrow down what you are attempting to collect – firstly we need to create a debug log if one doesn’t already exist – for this example I’m just logging to a circular memory buffer but it could go to SNMP, syslog or a file if necessary.

A:SR# configure log log-id 10
*A:SR>config>log>log-id$ from debug-trace
*A:SR>config>log>log-id$ to memory
*A:SR>config>log>log-id$ no shutdown
*A:SR>config>log>log-id$ back
*A:SR>config>log# info
----------------------------------------------
        log-id 10
            from debug-trace
            to memory
            no shutdown
        exit
----------------------------------------------
Now to set up the debug – we know it’s from interface PEER2 and the log message kindly told us the packet type (in hello)..
*A:SR>config>log# /debug router ospf packet hello "PEER2"
*A:SR>config>log# show debug
debug
    router "Base"
        ospf
            packet hello "PEER2"
        exit
    exit
exit
Router Base is the global routing table of the router, the debug can reference other services e.g. a VPRN if necessary by changing the router – After a few seconds (OSPF hello packets will come every 10 seconds or so) we can look in log 10 to see what was received.
*A:SR>config>log# show log log-id 10

===============================================================================
Event Log 10
===============================================================================
Description : (Not Specified)
Memory Log contents  [size=100   next event=10  (not wrapped)]

9 2017/06/02 02:19:27.10 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT

>> Outgoing OSPF packet on I/F PEER2 area 0.0.0.1
OSPF Version      : 2
Router Id         : 1.1.1.1
Area Id           : 0.0.0.1
Checksum          : ecb9
Auth Type         : Null
Auth Key          : 00 00 00 00 00 00 00 00
Packet Type       : HELLO
Packet Length     : 44 "

8 2017/06/02 02:19:26.55 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT DROPPED
area mismatch"

7 2017/06/02 02:19:26.54 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT

>> Incoming OSPF packet on I/F PEER2 area 0.0.0.2
OSPF Version      : 2
Router Id         : 200.200.200.200
Area Id           : 0.0.0.2
Checksum          : 5d27
Auth Type         : Null
Auth Key          : 00 00 00 00 00 00 00 00
Packet Type       : HELLO
Packet Length     : 44 "
SR is configured with PEER2 in Area 1 but it should be in Area 2, lets fix that
*A:SR>config>log# /configure router ospf
*A:SR>config>router>ospf# info
----------------------------------------------
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
            exit
            area 0.0.0.1
                interface "PEER1"
                    no shutdown
                exit
                interface "PEER2"
                    no shutdown
                exit
            exit
            no shutdown
----------------------------------------------
*A:SR>config>router>ospf# area 1 interface "PEER2" shutdown
*A:SR>config>router>ospf# area 1 no interface "PEER2"
*A:SR>config>router>ospf# area 2 interface "PEER2" no shutdown
*A:SR>config>router>ospf# info
----------------------------------------------
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
            exit
            area 0.0.0.1
                interface "PEER1"
                    no shutdown
                exit
            exit
            area 0.0.0.2
                interface "PEER2"
                    no shutdown
                exit
            exit
            no shutdown
----------------------------------------------
Now see if that fixes that problem.
*A:SR>config>router>ospf# show router ospf interface

===============================================================================
Rtr Base OSPFv2 Instance 0 Interfaces
===============================================================================
If Name               Area Id         Designated Rtr  Bkup Desig Rtr  Adm  Oper
-------------------------------------------------------------------------------
system                0.0.0.0         1.1.1.1         0.0.0.0         Up   DR
PEER1                 0.0.0.1         1.1.1.1         100.100.100.100 Up   DR
PEER2                 0.0.0.2         200.200.200.200 1.1.1.1         Up   BDR
-------------------------------------------------------------------------------
No. of OSPF Interfaces: 3
===============================================================================
Yes we can see both the DR and BDR for our OSPF peers but before we move on, we should stop the debug activity
*A:SR>config>router>ospf# /debug router no ospf
*A:SR>config>router>ospf# show debug
debug
exit
Now lets see if OSPF routing exchange is occurring.
*A:SR>config>router>ospf# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     01h15m46s  0
       system                                                       0
10.1.2.0/27                                   Local   Local     01h08m27s  0
       PEER1                                                        0
10.1.3.0/27                                   Local   Local     01h08m57s  0
       PEER2                                                        0
-------------------------------------------------------------------------------
No. of Routes: 3
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
Well that isn’t fixed yet (which should be no surprise as this is about MTU issues) so lets move onto the next phase and examine the state of our OSPF neighbors
*A:SR>config>router>ospf# show router ospf neighbor

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors
===============================================================================
Interface-Name                   Rtr Id          State      Pri  RetxQ   TTL
   Area-Id
-------------------------------------------------------------------------------
PEER1                            100.100.100.100 ExchStart  1    0       34
   0.0.0.1
PEER2                            200.200.200.200 Exchange   1    0       32
   0.0.0.2
-------------------------------------------------------------------------------
No. of Neighbors: 2
===============================================================================
A router that is stuck in ExchStart or Exchange is a hallmark of OSPF MTU related problems.
Let’s start working on PEER1.
*A:SR>config>router>ospf# show router ospf neighbor "PEER1" detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface "PEER1" (detail)
===============================================================================
-------------------------------------------------------------------------------
Neighbor : 10.1.2.2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Neighbor Rtr Id : 100.100.100.100  Interface: PEER1
-------------------------------------------------------------------------------
Neighbor IP Addr : 10.1.2.2
Local IF IP Addr : 10.1.2.1
Area Id          : 0.0.0.1
Designated Rtr   : 1.1.1.1              Backup Desig Rtr : 100.100.100.100
Neighbor State   : ExchStart            Priority         : 1
Retrans Q Length : 0                    Options          : - E - -  -  - - --
Events           : 1068                 Last Event Time  : 06/02/2017 02:59:34
Up Time          : 0d 01:11:00          Time Before Dead : 38 sec
GR Helper        : Not Helping          GR Helper Age    : 0 sec
GR Exit Reason   : None                 GR Restart Reason: Unknown (0)
Bad Nbr States   : 0                    LSA Inst fails   : 0
Bad Seq Nums     : 0                    Bad MTUs         : 1066
Bad Packets      : 0                    LSA not in LSDB  : 0
Option Mismatches: 0                    Nbr Duplicates   : 0
Num Restarts     : 0                    Last Restart at  : Never
===============================================================================
There are quite a few Bad MTUs being reported – While some vendors have an option to ignore the OSPF MTU, there are quite a number of MTU implications that can occur within the core when you consider various tunnel options that this is not provided.

Before we start to change things lets see if our trusty log 99 to see says anything about this:

*A:SR>config>router>ospf# show log log-id 99 application OSPF message PEER1

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=1473  (wrapped)]

1472 2017/06/02 02:40:17.67 UTC WARNING: OSPF #2043 Base VR:  1 OSPFv2 (0)
"LCL_RTR_ID 1.1.1.1: Conflicting configuration mtuMismatch on interface PEER1 from 10.1.2.2 in dbDescript"
We can use another debug to determine what the actual MTU should be (as before with the area mismatch, log 99 gave us a hint as to the packet type we should be investigating):
*A:SR>config>router>ospf# /debug router ospf packet dbdescr ingress "PEER1"
Clear the log and see what we are recieving:
*A:SR>config>router>ospf# /clear log 10
*A:SR>config>router>ospf# /show log log-id 10

===============================================================================
Event Log 10
===============================================================================
Description : (Not Specified)
Memory Log contents  [size=100   next event=3  (not wrapped)]

2 2017/06/02 02:55:17.67 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT DROPPED
MTU mismatch"

1 2017/06/02 02:55:17.67 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT

>> Incoming OSPF packet on I/F PEER1 area 0.0.0.1
OSPF Version      : 2
Router Id         : 100.100.100.100
Area Id           : 0.0.0.1
Checksum          : e35a
Auth Type         : Null
Auth Key          : 00 00 00 00 00 00 00 00
Packet Type       : DB_DESC
Packet Length     : 32

Interface MTU     : 1504
Options           : 000042
Flags             : 7   INIT MORE MAST
Sequence Num      : 2514
"

Okay, so PEER1 requires an MTU of 1504, lets modify that within the OSPF configuration:
*A:SR>config>router>ospf# info
----------------------------------------------
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
            exit
            area 0.0.0.1
                interface "PEER1"
                    no shutdown
                exit
            exit
            area 0.0.0.2
                interface "PEER2"
                    no shutdown
                exit
            exit
            no shutdown
----------------------------------------------
*A:SR>config>router>ospf# area 1 interface "PEER1" mtu 1504
When applying a configuration it is good to verify things are working as expected:
*A:SR>config>router>ospf# show router ospf interface "PEER1" detail | match MTU
Passive          : False                Cfg MTU          : 1504
Oper MTU         : 1500                 Last Enabled     : 06/02/2017 01:44:23
Although we configured the MTU to be 1504, the Operational MTU is 1500 (This is because the IP MTU is 1500 so OSPF cant be given a larger MTU on this interface)
*A:SR>config>router>ospf# show router interface "PEER1" detail | match MTU
IP MTU           : (default)
IP Oper MTU      : 1500

When Ethernet Ports are configured as mode access and left at the default encapsulation (null) the Ethernet port MTU is 1514 bytes (to support a 1500 byte IP MTU and 14 bytes of Ethernet Header – FCS is not included in MTU calculations)
*A:SR>config>router>ospf# show port 1/1/1 | match MTU
Physical Link      : Yes                        MTU              : 1514
To get a 1504 byte IP MTU, we can just add 4 bytes to the Port Ethernet MTU
*A:SR>config>router>ospf# /configure port 1/1/1 ethernet mtu 1518
*A:SR>config>router>ospf# show router interface "PEER1" detail | match MTU
IP MTU           : (default)
IP Oper MTU      : 1504
*A:SR>config>router>ospf# show router ospf interface "PEER1" detail | match MTU
Passive          : False                Cfg MTU          : 1504
Oper MTU         : 1504                 Last Enabled     : 06/02/2017 01:44:23
This should mean that the OSPF neighbor will now perform the database exchange and enter the Full state.
*A:SR>config>router>ospf# show router ospf neighbor "PEER1"

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface "PEER1"
===============================================================================
Interface-Name                   Rtr Id          State      Pri  RetxQ   TTL
   Area-Id
-------------------------------------------------------------------------------
PEER1                            100.100.100.100 Full       1    0       34
   0.0.0.1
-------------------------------------------------------------------------------
No. of Neighbors: 1
===============================================================================
*A:SR>config>router>ospf# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     02h10m36s  0
       system                                                       0
10.1.2.0/27                                   Local   Local     02h03m16s  0
       PEER1                                                        0
10.1.3.0/27                                   Local   Local     02h03m46s  0
       PEER2                                                        0
100.100.100.100/32                            Remote  OSPF      00h03m18s  10
       10.1.2.2                                                     100
-------------------------------------------------------------------------------
No. of Routes: 4
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
The OSPF issue with PEER1 appears to have been resolved so back to PEER2.

*A:SR>config>router>ospf# show router ospf neighbor "PEER2" detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface "PEER2" (detail)
===============================================================================
-------------------------------------------------------------------------------
Neighbor : 10.1.3.3
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Neighbor Rtr Id : 200.200.200.200  Interface: PEER2
-------------------------------------------------------------------------------
Neighbor IP Addr : 10.1.3.3
Local IF IP Addr : 10.1.3.1
Area Id          : 0.0.0.2
Designated Rtr   : 200.200.200.200      Backup Desig Rtr : 1.1.1.1
Neighbor State   : Exchange             Priority         : 1
Retrans Q Length : 3                    Options          : - E - -  -  - O --
Events           : 3                    Last Event Time  : 06/02/2017 02:22:36
Up Time          : 0d 01:01:10          Time Before Dead : 37 sec
GR Helper        : Not Helping          GR Helper Age    : 0 sec
GR Exit Reason   : None                 GR Restart Reason: Unknown (0)
Bad Nbr States   : 0                    LSA Inst fails   : 0
Bad Seq Nums     : 0                    Bad MTUs         : 0
Bad Packets      : 0                    LSA not in LSDB  : 0
Option Mismatches: 0                    Nbr Duplicates   : 917
Num Restarts     : 0                    Last Restart at  : Never
===============================================================================
There are no Bad MTUs being reported here, all we can see is that we are forever in Exchange state – lets check log 99 to see if anything at all related to PEER2 is present
*A:SR>config>router>ospf# show log log-id 99 message PEER2

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=2033  (wrapped)]
There is nothing present (the older events have wrapped around since we are only keeping the last 500 events)
What I have found is when OSPF neighbors are stuck in ExchStart, your router is the one with the MTU too small but while the router that is stuck in Exchange is the one with the MTU that is too big for its peer.
To work out what the smaller MTU should be, we’ll send ping packets of various lengths to work out what is the biggest unfragmented packet that can be sent to PEER2. Note: when we send a ping and specify the size, we are actually calling out what the ICMP payload size should be, so we need to ensure for IPv4 we consider the 20 byte IP header and 8 byte ICMP header – so an IP interface with an IP-MTU of 1500 would work for a ping with a payload size of 1472 but would fail at 1473.
We can test this concept on a known quantity (PEER1 which has an IP MTU of 1504) we should be able to get a ping payload of 1476 through okay but 1477 should fail – make sure we set the DF bit!
*A:SR>config>router>ospf# ping 10.1.2.2 size 1476 do-not-fragment count 3
PING 10.1.2.2 1476 data bytes
1484 bytes from 10.1.2.2: icmp_seq=1 ttl=64 time=1.34ms.
1484 bytes from 10.1.2.2: icmp_seq=2 ttl=64 time=1.27ms.
1484 bytes from 10.1.2.2: icmp_seq=3 ttl=64 time=1.16ms.

---- 10.1.2.2 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.16ms, avg = 1.26ms, max = 1.34ms, stddev = 0.072ms
*A:SR>config>router>ospf# ping 10.1.2.2 size 1477 do-not-fragment count 3
PING 10.1.2.2 1477 data bytes

---- 10.1.2.2 PING Statistics ----
3 packets transmitted, 3 packets bounced, 0 packets received, 100% packet loss
This works as expected, so the concept appears sound.

*A:SR>config>router>ospf# show router interface "PEER2" detail | match MTU
IP MTU           : (default)
IP Oper MTU      : 1500
We know that we have a ceiling of 1500 and we know the MTU must be lower than this. But just to be certain, we’ll try based on a 1500 byte IP packet anyway
*A:SR>config>router>ospf# ping 10.1.3.3 size 1472 do-not-fragment count 3
PING 10.1.3.3 1472 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.
Request timed out. icmp_seq=3.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 0 packets received, 100% packet loss
Unsurprising, the Peer MTU is less than 1500 bytes, lets try a slightly smaller payload
*A:SR>config>router>ospf# ping 10.1.3.3 size 1462 do-not-fragment count 3
PING 10.1.3.3 1462 data bytes
1470 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.44ms.
1470 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.36ms.
1470 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.34ms.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.34ms, avg = 1.38ms, max = 1.44ms, stddev = 0.042ms
Okay, time to divide and conquer to determine the largest payload that gets through
*A:SR>config>router>ospf# ping 10.1.3.3 size 1467 do-not-fragment count 3
PING 10.1.3.3 1467 data bytes
1475 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.24ms.
1475 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.30ms.
1475 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=2.16ms.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.24ms, avg = 1.57ms, max = 2.16ms, stddev = 0.417ms
*A:SR>config>router>ospf# ping 10.1.3.3 size 1469 do-not-fragment count 3
PING 10.1.3.3 1469 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.
Request timed out. icmp_seq=3.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 0 packets received, 100% packet loss
*A:SR>config>router>ospf# ping 10.1.3.3 size 1468 do-not-fragment count 3
PING 10.1.3.3 1468 data bytes
1476 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.19ms.
1476 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.30ms.
1476 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.26ms.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.19ms, avg = 1.25ms, max = 1.30ms, stddev = 0.043ms
An ICMP payload of 1468 fits within an IP packet with a size of 1496 – adjust the OSPF MTU to 1496 and see if that results in getting a full adjacency.
*A:SR>config>router>ospf# area 2 interface "PEER2" mtu 1496
*A:SR>config>router>ospf# show router ospf interface "PEER2" detail | match MTU
Passive          : False                Cfg MTU          : 1496
Oper MTU         : 1496                 Last Enabled     : 06/02/2017 02:22:36
*A:SR>config>router>ospf# show router ospf neighbor "PEER2"

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface "PEER2"
===============================================================================
Interface-Name                   Rtr Id          State      Pri  RetxQ   TTL
   Area-Id
-------------------------------------------------------------------------------
PEER2                            200.200.200.200 Full       1    0       35
   0.0.0.2
-------------------------------------------------------------------------------
No. of Neighbors: 1
===============================================================================
The adjacency is up – lets see what routes we have learnt
*A:SR>config>router>ospf# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     02h40m46s  0
       system                                                       0
10.1.2.0/27                                   Local   Local     02h33m27s  0
       PEER1                                                        0
10.1.3.0/27                                   Local   Local     02h33m56s  0
       PEER2                                                        0
100.100.100.100/32                            Remote  OSPF      00h33m29s  10
       10.1.2.2                                                     100
200.200.200.200/32                            Remote  OSPF      00h01m27s  10
       10.1.3.3                                                     100
-------------------------------------------------------------------------------
No. of Routes: 5
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

We now have learnt routes from PEER1 and PEER2, time for a quick dataplane verification:
*A:SR>config>router>ospf# ping 100.100.100.100 source 1.1.1.1 count 1
PING 100.100.100.100 56 data bytes
64 bytes from 100.100.100.100: icmp_seq=1 ttl=64 time=1.38ms.

---- 100.100.100.100 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.38ms, avg = 1.38ms, max = 1.38ms, stddev = 0.000ms
*A:SR>config>router>ospf# ping 200.200.200.200 source 1.1.1.1 count 1
PING 200.200.200.200 56 data bytes
64 bytes from 200.200.200.200: icmp_seq=1 ttl=64 time=1.14ms.

---- 200.200.200.200 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.14ms, avg = 1.14ms, max = 1.14ms, stddev = 0.000ms
We now have successful routing exchange and data plane reachability.

Transactional (Candidate) configurations with Nokia SROS

While it’s not a new feature for SROS, the introduction of transactional configurations with SROS was not a day one feature (unlike JunOS), so it may be less known to existing users.

Firstly before getting into rollback configurations, I would like to point out that SROS has been saving multiple copies of configurations via rotation based mechanism for quite some time without needing specific activation.

A:SR1# show system information | match expression "Config|Backup"
Config Source          : primary
Last Booted Config File: cf3:\config.cfg
Last Boot Config Header: # TiMOS-B-14.0.R4 both/i386 Nokia 7750 SR Copyright
Last Saved Config      : cf3:\config.cfg
Max Cfg/BOF Backup Rev : 5

cf3:\config.cfg (which can be changed from this default by modifying the boot options file) will store the current config and the previous 4 versions and while the number of saved configs is modifiable, 5 is most likely enough for most. Anyway, reverting to a previous configuration can be quite a disruptive event, if you copy an older config over cf3:\config.cfg and performing a system reboot. This is where rollback configurations and combining them with transactional (called candidate) configurations become helpful particularly during a complex task like a network migration.

Achyar Nur Andi has a good post discussing the mechanics around rollbacks and candidate configurations at www.achyarnurandi.net, so I will just highlight a few of the main features and how you can enforce the method of router configuration to only use candidate configurations.

The first thing to do is to specify the rollback file prefix (in this case conf-rollback on compact flash 3):

A:SR1# /configure system rollback
A:SR1>config>system>rollback# rollback-location cf3:\conf-rollback
INFO: CLI No checkpoints currently exist at the rollback location.
*A:SR1>config>system>rollback# show system rollback

===============================================================================
Rollback Information
===============================================================================
Rollback Location            : cf3:\conf-rollback
Max Local  Rollback Files    : 10
Max Remote Rollback Files    : 10
Save
  Last Rollback Save Result  : None
  Last Save Completion Time  : N/A
Revert
  In Progress                : No
  Last Revert Initiated User : N/A
  Last Revert Checkpoint File: N/A
  Last Revert Result         : None
  Last Revert Initiated Time : N/A
  Last Revert Completion Time: N/A
Delete
  Last Rollback Delete Result: None

===============================================================================
Rollback Files
===============================================================================
Idx    Suffix    Creation Time            Release           User
         Comment
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================
*A:SR1>config>system>rollback# exit all

We’ll create our first rollback point:

*A:SR1# admin rollback save comment "Baseline Config"
Saving rollback configuration to cf3:\conf-rollback.rb... OK
*A:SR1# show system rollback

===============================================================================
Rollback Information
===============================================================================
Rollback Location            : cf3:\conf-rollback
Max Local  Rollback Files    : 10
Max Remote Rollback Files    : 10
Save
  Last Rollback Save Result  : Successful
  Last Save Completion Time  : 2017/05/23 02:35:38  UTC
Revert
  In Progress                : No
  Last Revert Initiated User : N/A
  Last Revert Checkpoint File: N/A
  Last Revert Result         : None
  Last Revert Initiated Time : N/A
  Last Revert Completion Time: N/A
Delete
  Last Rollback Delete Result: None

===============================================================================
Rollback Files
===============================================================================
Idx    Suffix    Creation Time            Release           User
         Comment
-------------------------------------------------------------------------------
latest .rb       2017/05/23 02:35:38  UTC B-14.0.R4         admin
           Baseline Config
-------------------------------------------------------------------------------
No. of Rollback Files: 1
===============================================================================

There’s only one rollback called latest.rb

For this example, just a simply system name change:
*A:SR1# /configure system name "Wrong Name"Now to compare the current working configuration with the rollback:

*A:Wrong Name# admin rollback compare
Processing current config... 0.010 s
Processing "cf3:\conf-rollback.rb"... 0.020 s
----------------------------------------------
  configure
     system
+        name "Wrong Name"
-        name "SR1"
     exit
  exit
It’s very clear what the differences are. I would just like to highlight that at present, these configuration changes are still immediate – rollbacks on their own just provide a means to manage the change, and doesnt provide any atomic operations yet.

Let’s revert back to our old configuration:

*A:Wrong Name# admin rollback revert latest-rb
Restoring rollback configuration cf3:\rollback-dir.rb
Processing current config... 0.010 s
Processing "cf3:\rollback-dir.rb"... 0.020 s
Resolving dependencies... 0.000 s
Tearing setup down... 0.010 s
Rebuilding setup... 0.000 s
Finished in 0.050 s
*A:SR1#

Using candidate configuration mode, as opposed to the default “immediate” configuration mode does not implement the configuration changes until you commit them, in the event of a failure applying the configuration, the system will back out and re-wind the configuration allowing you the option to discard or modify your changes. Candidate configuration mode is enabled via “candidate edit”. For this example we are going to set the system address on our router, configure an ethernet port, create an IES and attach a VLAN on that port to an IP interface.
A:SR1# candidate edit
A:SR1>edit-cfg# configure router interface "system" address 111.111.111.111/32
A:SR1>edit-cfg# configure port 1/2/3 shutdown
A:SR1>edit-cfg# configure port 1/2/3 ethernet mode access
A:SR1>edit-cfg# configure port 1/2/3 ethernet encap-type dot1q
A:SR1>edit-cfg# configure port 1/2/3 no shutdown
A:SR1>edit-cfg# configure service ies 123 customer 1 create
A:SR1>edit-cfg>config>service>ies# interface TEST create
A:SR1>edit-cfg>config>service>ies>if# address 192.168.1.1/24
A:SR1>edit-cfg>config>service>ies>if# sap 1/2/3:4 create
A:SR1>edit-cfg>config>service>ies>if>sap# back
A:SR1>edit-cfg>config>service>ies>if# back
A:SR1>edit-cfg>config>service>ies# no shutdown
Based on where we are within the configuration tree, we can see the associated configuration changes:
A:SR1>edit-cfg>config>service>ies# candidate view
----------------------------------------------
17:             interface "TEST" create
18:                 address "192.168.1.1/24"
19:                 sap "1/2/3:4" create
20:                 exit
21:             exit
22:*            no shutdown
----------------------------------------------
Or if we get to the root of the configuration tree, we can see all the associated changes that are yet to be applied to the running configuration:
A:SR1>edit-cfg>config>service>ies# exit all
A:SR1>edit-cfg# candidate view
----------------------------------------------
1:  configure
2:      router
3:          interface "system"
4:              address "111.111.111.111/32"
5:          exit
6:      exit
7:      port "1/2/3"
8:          shutdown
9:          ethernet
10:             mode access
11:             encap-type dot1q
12:         exit
13:         no shutdown
14:     exit
15:     service
16:         ies "123" customer 1 create
17:             interface "TEST" create
18:                 address "192.168.1.1/24"
19:                 sap "1/2/3:4" create
20:                 exit
21:             exit
22:*            no shutdown
23:         exit
24:     exit
25: exit
----------------------------------------------
Now we can accept and attempt to push the configuration the router using “candidate commit”
A:SR1>edit-cfg# candidate commit
Processing current config... 0.010 s
Error at line 7: Command 'port "1/2/3"' failed in 'configure'
MINOR: CLI Port "1/2/3" does not exist.
Reverting changes...
Processing current config... 0.010 s
Processing memory checkpoint... 0.000 s
Resolving dependencies... 0.000 s
Tearing setup down... 0.000 s
Rebuilding setup... 0.010 s
Finished in 0.040 s
MINOR: CLI Commit failed and has been reverted.
Since there was an error in the configuration – our router doesn’t have a port 1/2/3 – the configuration failed and the whole new configuration context was backed out allowing the option to correct and reapply, or to reject the changes which is quite a powerful configuration tool and concept. As we know the problem was on line 7, we can specifically edit that line using “candidate replace 7” and replacing the string port “1/2/3” with the proper port which is “1/1/3″
*A:SR1>edit-cfg# candidate replace 7
*A:Replace by: port "1/1/3"
INFO: CLI Added 10 lines: 'port "1/1/3"'.
INFO: CLI Removed 10 lines: 'port "1/2/3"'.
It’s probably worth double checking the revised configuration
*A:SR1>edit-cfg# candidate view
----------------------------------------------
1:  configure
2:      router
3:          interface "system"
4:              address "111.111.111.111/32"
5:          exit
6:      exit
7:      port "1/1/3"
8:          shutdown
9:          ethernet
10:             mode access
11:             encap-type dot1q
12:         exit
13:         no shutdown
14:     exit
15:     service
16:         ies "123" customer 1 create
17:             interface "TEST" create
18:                 address "192.168.2.1/24"
19:                 sap "1/2/3:4" create
20:                 exit
21:             exit
22:*            no shutdown
23:         exit
24:     exit
25: exit
----------------------------------------------
The SAP also requires correction to align with the new port – this is on line 19
*A:SR1>edit-cfg# candidate replace 19
*A:Replace by: sap "1/1/3:4" create
INFO: CLI Added 2 lines: 'sap "1/1/3:4" create'.
INFO: CLI Removed 2 lines: 'sap "1/2/3:4" create'.
Now lets apply the configuration
*A:SR1>edit-cfg# candidate commit
Saving checkpoint file... OK
INFO: CLI Successfully executed 25 lines in 0.000 s.
Configuration mode is still quite handy to view what has been configure by jumping into the right configuration context and doing an info or info detail:
*A:SR1# /configure service
*A:SR1>config>service# info
----------------------------------------------
        customer 1 create
            description "Default customer"
        exit
        ies 1 customer 1 create
            interface "External" create
                address 200.200.200.1/24
                sap 1/1/1 create
                exit
            exit
            no shutdown
        exit
        ies 123 customer 1 create
            interface "TEST" create
                address 192.168.2.1/24
                sap 1/1/3:4 create
                exit
            exit
            no shutdown
        exit
----------------------------------------------
An operational problem can occur if we allow the use of both configuration candidate and immediate configurations such as being able to do
*A:SR1>config>service# ies 123 description "Candidate Config Test"the most likely will end up with people sticking with immediate configuration mode unless they are forced to use candidate configs. Fortunately there it is quite easy to enable this.*A:SR1# /configure system management cli configuration no immediate It doesn’t remove the facility to view configurations, just configuration changes:
*A:SR1# configure service ies 123
*A:SR1>config>service>ies# info
----------------------------------------------
            description "Candidate Config Test"
            interface "TEST" create
                address 192.168.2.1/24
                sap 1/1/3:4 create
                exit
            exit
            no shutdown
----------------------------------------------
If we now attempt a non-candidate mode configuration change:
*A:SR1>config>service>ies# description "New Description"
MINOR: CLI Direct modification of the configuration is not allowed. Use 'candidate edit' for all changes.
We are now forced to use candidate configs:
*A:SR1>config>service>ies# candidate edit
*A:SR1>edit-cfg# configure service ies 123 description "New Description"
*A:SR1>edit-cfg# candidate commit
Processing current config... 0.010 s
Saving checkpoint file... OK
INFO: CLI Successfully executed 7 lines in 0.000 s.

Coupled with the right processes, this is one of the tools to help increase the MTBM (Mean Time Between Mistakes) and reduce the amount of network disruption.

Using BGP Flowspec with Nokia SROS to protect the network edge

This post is about how you can use BGP to augment your network edge protection. Specifically BGP Flow spec can be used to dynamically add entries to an access control list (or ip-filter in Nokia SROS speak). Previous posts have been using ExaBGP as the method to inject traffic, and while it’s good at what it does, I’ve moved on to use GoBGP not because it seems to be quite a high performance BGP implementation (I’m just doing proof of concept work) but because GoBGP’s command line interface is more friendly and better documented than ExaBGP, making it the more versatile tool to generate route advertisements in a lab on the fly (at least for me).

BGP Flowspec Test Topology

The lab topology is designed to demonstrate filtering with flowspec – two routers (SR1 is under my control, while the External router is a gateway to unscrupulous entities that I cannot control) and GoBGP running on a computer in my Network Operations centre.

Before getting into the flowspec piece, I’ll go through the installation process for GoBGP and the Go Language for Ubuntu 16.04 (as the routers I’m using as running on eve-ng) and the computer in the example is bridged to SR1) and then how to develop the initial GoBGP configuration.

The Go Programming language is available via the standard Ubuntu repository
adam@m4600:~$ sudo apt-get install golang-go

GoBGP is pulled down as source code and compiled as part of the download process. An environmental variable needs to be set so Go knows where to pull down code, and where to build the binaries.
adam@m4600:~$ export GOPATH=$HOME/go
Tell go to pull down and build gobgpd (this takes awhile at least on my machine and you don’t get an indication on what it’s doing)
adam@m4600:~$ go get github.com/osrg/gobgp/gobgpd
The gobgp cli is a separate program that talks to gobgpd using the GoBGP API – downloading and building it is pretty quick in comparison to gobgpd:
adam@m4600:~$ go get github.com/osrg/gobgp/gobgp
Once built, we shall copy the generated binaries to the an executable path
adam@m4600:~$ sudo cp $GOPATH/bin/* /usr/local/sbin/
For those of us using the default bash shell, we can enable tab completion for the cli

adam@m4600:~$ sudo cp $GOPATH/src/github.com/osrg/gobgp/tools/completion/*.bash /etc/bash_completion.d/

Once GoBGP is installed, we can develop a configuration file that will be used to set up the GoBGP instance – in this case we’re defining the router itself and SR1 as a neighbor for the IPv4 FlowSpec address family:
A configuration file that will be used by the GoBGP Daemon enabling FlowSpec for IPv4 where we peer against SR1:

adam@m4600:~$ cat gobgp-flowspec.conf
[global.config]
  as = 64512
  router-id = "1.2.3.4"

[[neighbors]]
[neighbors.config]
  neighbor-address = "192.168.1.123"
  peer-as = 64512
[[neighbors.afi-safis]]
  [neighbors.afi-safis.config]
  afi-safi-name = "ipv4-flowspec"

The GoBGP documentation is also a bit nicer than ExaBGP too – additional information on setting up a configuration (including different formats is described in the GoBGP getting started document)

Start the GoBGP Daemon and read our new config:

adam@m4600:~$ sudo gobgpd -f gobgp-flowspec.conf
{"level":"info","msg":"gobgpd started","time":"2017-05-18T22:57:22+10:00"}
{"Topic":"Config","level":"info","msg":"Finished reading the config file","time":"2017-05-18T22:57:22+10:00"}
{"level":"info","msg":"Peer 192.168.1.123 is added","time":"2017-05-18T22:57:22+10:00"}
{"Topic":"Peer","level":"info","msg":"Add a peer configuration for:192.168.1.123","time":"2017-05-18T22:57:22+10:00"}
{"Key":"192.168.1.123","State":"BGP_FSM_OPENCONFIRM","Topic":"Peer","level":"info","msg":"Peer Up","time":"2017-05-18T22:57:40+10:00"}
We can see shortly after gobgpd started, it has already established a peering session with SR1 (192.168.1.123)

Opening up a new shell, we can use the gobgp client to verify that:

adam@m4600:~$ gobgp neighbor
Peer             AS  Up/Down State       |#Received  Accepted
192.168.1.123 64512 00:00:22 Establ      |        0         0
adam@m4600:~$ gobgp neighbor 192.168.1.123
BGP neighbor is 192.168.1.123, remote AS 64512
  BGP version 4, remote router ID 123.123.123.123
  BGP state = established, up for 00:01:53
  BGP OutQ = 0, Flops = 0
  Hold time is 90, keepalive interval is 30 seconds
  Configured hold time is 90, keepalive interval is 30 seconds
  %s
  Neighbor capabilities:
    multiprotocol:
        ipv4-flowspec:  advertised and received
    route-refresh:      advertised and received
    4-octet-as: advertised and received
  Message statistics:
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             4          5
    Route Refresh:          0          0
    Discarded:              0          0
    Total:                  5          6
  Route statistics:
    Advertised:             0
    Received:               0
    Accepted:               0

We can also get SR1 to confirm it too:
*A:SR1# show router bgp summary | match Summ post-lines 100
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
192.168.1.46
               64512        9    0 00h00m54s 0/0/0 (FlowIPv4)
                           13    0
-------------------------------------------------------------------------------

Now that the flowspec session is up and running but no one has exchanged any routes yet, I’ll park this for a moment to create a simple ip-filter (or ACL)

*A:SR1# /configure filter
*A:SR1>config>filter# info
----------------------------------------------
        match-list
            ip-prefix-list "FPL_RFC1918" create
                prefix 10.0.0.0/8
                prefix 172.16.0.0/12
                prefix 192.168.0.0/16
            exit
        exit
        ip-filter 100 create
            default-action forward
            embed-filter flowspec router "Base" offset 1000
            entry 10 create
                match
                    src-ip ip-prefix-list "FPL_RFC1918"
                exit
                action
                    drop
                exit
            exit
            entry 20 create
                match
                    dst-ip ip-prefix-list "FPL_RFC1918"
                exit
                action
                    drop
                exit
            exit
        exit
----------------------------------------------

The match-list ip-prefix-list “FPL_RFC1918” is a way to consolidate a number of network prefixes into a single named list enabling filter configurations to be smaller and more manageable (and easier to debug)

ip-filter 100 is the thing of interest. The default-action is what happens if none of the previous entries are matched – in this case it is a black-list ip-filter, since anything we don’t specifically drop will get forwarded
Each filter entry has a number which is used for the ascending evaluation order (entry 10 before entry 20 here) and usually has a match entry and an action to perform when the match was successful. This filter is relatively simple since all it is doing is blocking IP packets that either have source or destination IP addresses from RFC1918 space.

The interesting thing is the embed-filter flowspec entry. Router “Base” is the base routing instance (as opposed to a VPRN) and offset 1000 means that any flowspec entries will be dynamically added to ip-filter 100 starting at entry 1000 (and with an additional offset determined by flowspec)

This filter will be applied to the IP interface facing the external router – This interface is associated with an Internet Enhanced Service (IES) associated with the SR1’s Global Routing Table (Router “Base”)

*A:SR1>config>service>ies# /configure service ies 1
*A:SR1>config>service>ies# info
----------------------------------------------
            interface "External" create
                address 200.200.200.1/24
                sap 1/1/1 create
                exit
            exit
            no shutdown
----------------------------------------------
*A:SR1>config>service>ies# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     00h20m17s  0
       Loop_1                                                       0
0.0.0.0/0                                     Remote  Static    00h20m02s  5
       200.200.200.2                                                1
123.123.123.123/32                            Local   Local     00h20m17s  0
       system                                                       0
192.168.1.0/24                                Local   Local     00h20m02s  0
       NetOps                                                       0
200.200.200.0/24                              Local   Local     00h20m02s  0
       External                                                     0
-------------------------------------------------------------------------------
No. of Routes: 5
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available

In this contrived example SR1 just has a default route point to the External interface in IES 1.
Before we apply the IP filter – check to see we can reach an RFC1918 IP Address:
A:External# ping 1.1.1.1 source 172.16.0.1 count 2
PING 1.1.1.1 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=1.40ms.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=1.91ms.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min = 1.40ms, avg = 1.66ms, max = 1.91ms, stddev = 0.253ms

Apply the IP Filter in the ingress direction of interface External on SR1:
*A:SR1>config>service>ies# interface "External" sap 1/1/1 ingress filter ip 100
*A:SR1>config>service>ies# info
----------------------------------------------
            interface "External" create
                address 200.200.200.1/24
                sap 1/1/1 create
                    ingress
                        filter ip 100
                    exit
                exit
            exit
            no shutdown

And repeat the test:
*A:External# ping 1.1.1.1 source 172.16.0.1 count 2
PING 1.1.1.1 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 0 packets received, 100% packet loss

We can see the ping failed and if we look at the counters
*A:SR1>config>service>ies# show filter ip 100 counters

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2
Sub-Entries         : 6
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
We can see the ping was unsuccessful because of entry 10 stopping traffic coming back to SR1 (because we weren’t filtering traffic leaving SR1)

Now all of a sudden we find a ping flood (well okay, calling two packets a flood is a rather long bow to draw but anyway) is coming from 100.100.100.100 and hitting 1.1.1.1

A:External# ping 1.1.1.1 source 100.100.100.100 count 2
PING 1.1.1.1 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=1.16ms.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=1.26ms.

These ICMP packets are causing havok to 1.1.1.1, so it would be nice to add some precision filtering to stop that from occuring, while allowing other traffic through – this is where our GoBGP instance and flowspec comes back into the story. We will ask gobgp to announce a flowspec “route” that specifies a discard action (actually a rate-limit of 0 kbps) for traffic from 100.100.100.100 to 1.1.1.1 using ICMP:
adam@m4600:~$ gobgp global rib add -a ipv4-flowspec match source 100.100.100.100/32 destination 1.1.1.1/32 protocol icmp then discard
SR1 can confirm it recieved this route:
*A:SR1>config>service>ies# show router bgp routes flow-ipv4
===============================================================================
 BGP Router ID:123.123.123.123  AS:64512       Local AS:64512
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP FLOW IPV4 Routes
===============================================================================
Flag  Network             Nexthop                 LocalPref       MED
      As-Path
-------------------------------------------------------------------------------
u*>?  --                  0.0.0.0                 100             None
      No As-Path

      Community Action:  rate-limit: 0 kbps
      NLRI Subcomponents:
      Dest Pref : 1.1.1.1/32
      Src Pref  : 100.100.100.100/32
      Ip Proto  : [ == 1 ]
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Flowspec can do quite interesting things, besides rate-limiting traffic to 0kbps (dropping it), it’s possible to use flowspec to tell the router to rate limit flows rather than completely blocking, remark traffic to a different level of priority or even redirect traffic to a VRF by setting the appropriate route targets (this could for example allow the forwarding of traffic into a VPRN or VRF to reach a scrubbing network prior to re-injection of clean traffic back into the network)

Lets see if ip filter has taken this flowspec entry:

*A:SR1>config>service>ies# show filter ip 100

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2/0/0/1 (Fixed/Radius/Cc/Embedded)
Sub-Entries         : 6/0/0/1
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : ip-prefix-list "FPL_RFC1918"
Src. Port           : n/a
Dest. IP            : 0.0.0.0/0
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 0.0.0.0/0
Src. Port           : n/a
Dest. IP            : ip-prefix-list "FPL_RFC1918"
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

Entry               : 1256
Origin              : Inserted by embedded filter fSpec-0 entry 256
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 100.100.100.100/32
Src. Port           : n/a
Dest. IP            : 1.1.1.1/32
Dest. Port          : n/a
Protocol            : 1                            Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
We had specified that the offset for flowspec was 1000, this flowspec entry (number 256) became ip filter 100 entry 256. Time to verify that the ping now should fail:
*A:External# ping 1.1.1.1 source 100.100.100.100 count 2
PING 1.1.1.1 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 0 packets received, 100% packet loss

Yes, and we can verify that it’s only blocking icmp between those endpoints by sending IP packets from another protocol say UDP by doing a traceroute:
*A:External# traceroute 1.1.1.1 source 100.100.100.100
traceroute to 1.1.1.1 from 100.100.100.100, 30 hops max, 40 byte packets
  1  1.1.1.1 (1.1.1.1)    1.74 ms  1.51 ms  1.52 ms
Yes, this is pretty good as far as network based filters (that aren’t performing payload based inspection) are doing, there’s quite a few match conditions that can be used to narrow things down even further. In this contrived example it may be similar effort to configure a filter on the fly but in a production environment, you could hook GoBGP to your route reflector and through a single procedure push updates to all your edge routers in one go. Another good thing about this method, is if the filtering is only temporary, it’s just as easy to remove as it was to put in.
To see what’s currently in the rib for ipv4-flowspec:
adam@m4600:~$ gobgp global rib  -a ipv4-flowspec
    Network                                                              Next Hop             AS_PATH              Age        Attrs
*>  [destination:1.1.1.1/32][source:100.100.100.100/32][protocol:==icmp ]fictitious                                00:12:06   [{Origin: ?} {Extcomms: [discard]}]
Withdraw the announcement:
adam@m4600:~$ gobgp global rib del -a ipv4-flowspec match source 100.100.100.100/32 destination 1.1.1.1/32 protocol icmp then discard
adam@m4600:~$ gobgp global rib  -a ipv4-flowspec                                Network not in table

Verify SR1 ip filter 100 no longer has the entry:
*A:SR1>config>service>ies# show filter ip 100

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2
Sub-Entries         : 6
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : ip-prefix-list "FPL_RFC1918"
Src. Port           : n/a
Dest. IP            : 0.0.0.0/0
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 0.0.0.0/0
Src. Port           : n/a
Dest. IP            : ip-prefix-list "FPL_RFC1918"
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
Entry 1256 has left the building.

Hopefully that was a reasonable introduction to using BGP Flowspec with filtering and GoBGP.

Flowspec is quite often a piece of the puzzle in DDoS mitigation tools, usually there is some kind of automation attached which makes decisions using inputs such a Netflow/IPFIX data, SNMP polling of interfaces to track anomalous loading and in some cases threat intelligence feeds from security vendors.

TACACS+ Authentication with Nokia Service Routers

Nokia SROS supports the use of AAA for a range of tasks, some of the more interesting and complicated are related to subscriber management when the Service Router is acting as a Broadband Network Gateway (BNG) however AAA is also useful for the network operations teams to provide centralised authentication for a fleet of routers where managing individual local accounts is not really something to contemplate.

SROS supports the use of RADIUS or TACACS+ for this management access control and today TACACS+ will be the method used with a linux daemon based on code from http://www.shrubbery.net/tac_plus/ which will be configured to support a Nokia Service Router (however this configuration would be quite Cisco IOS friendly) and the SROS router will use TACACS+ for authentication and identifying what access rights the user has my mapping using profiles.

Nokia SR and TACACS+ Server Test Topology

As you can see above, R1 (instantiated service router in eve-ng) has port 1/1/3 bridged to the internal Ethernet of the computer running eve-ng so both the router interface and the internal Ethernet are on the same IP subnet allowing connectivity to the TACACS+ server that will be run on the laptop.

To install the TACACS+ software, as eve-ng is built on Ubuntu 16.04, installation is as simple as invoking:
root@m4600:~# apt-get install tacacs+
The config was then modified to look like below:

root@m4600:~# cat /etc/tacacs+/tac_plus.conf
# shared secret with TACACS client
key = "tac_secret"
# Set where to send accounting records
accounting syslog;
accounting file = /var/log/tac_plus/tac_plus.acct

acl = mgmt_acl {
        # regex to allow access hosts from 192.168.1.0/24
        permit = 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4])
}

# administrative group, priv-lvl 15 to be mapped to SROS administative profile
group = administrative {
        default service = permit
        expires = "Jan 1 2020"
        acl = mgmt_acl
        service = exec {
         priv-lvl = 15
        }
}
# limited group, priv-lvl 1 to be mapped to SROS limited profile
group = limited {
        default service = permit
        expires = "Jan 1 2020"
        acl = mgmt_acl
        service = exec {
         priv-lvl = 1
        }
}

# our tacacs test accounts
# des password is generated by running tac_pwd on the plaintext
user = testadmin {
        member = administrative
        login = des JZ1fHFoSp.v/E
        # plaintext password = pass
}

user = testlimited {
        member = limited
        login = des O8ZepJOyIIuYo
        # plaintext password = test
}

A couple of the key things here besides the key which is the shared secret between the TACACS+ server and the router is that there are two groups defined administrative and limited, where the only difference is the priv-lvl. With Cisco platforms, this is what is used for TACACS+ uses during the authorisation stage to tell IOS what access rights a user has. SROS is able to map this to a “profile”.

Out of the box, SROS has two built in profiles, administrative (used for most installation and commissioning activities) and default which is somewhat less capable, however it is possible to define specific profiles in line with the roles of your users. In the config above there is a group called limited which will be identified by priv-lvl 1.

On R1 we can define a custom profile in the system security configuration context:

A:R1# /configure system security profile "limited"
A:R1>config>system>security>profile# info
----------------------------------------------
                default-action deny-all
                entry 10
                    match "show router route-table"
                    action permit
                exit
                entry 20
                    match "show users"
                    action permit
                exit
                entry 30
                    match "show system security user"
                    action permit
                exit
                entry 40
                    match "logout"
                    action permit
                exit
This example is certainly quite limited in what can be done due to the default-action deny-all, requiring specific white-listing of commands
To enable TACACS+ support on the router we first need to configure the TACACS server using the aggreed shared secret (configuring the timeout is optional but it specifies how many seconds we shall wait for a response – if the server is down, this is effectively how long you will wait to fall back to local authentication)
A:R1>config>system>security>profile# /configure system security tacplus
*A:R1>config>system>security>tacplus$ server 1 address 192.168.1.47 secret "tac_secret"
*A:R1>config>system>security>tacplus$ timeout 5

Now to create the priv-lvl mapping to profiles:
*A:R1>config>system>security>tacplus$ priv-lvl-map
A:R1>config>system>security>tacplus$ priv-lvl-map
A:R1>config>system>security>tacplus>priv$ priv-lvl 1 "limited"
A:R1>config>system>security>tacplus>priv$ priv-lvl 15 "administrative"

We also need to enable authorisation to be associated with these mappings:
A:R1>config>system>security>tacplus>priv$ back
A:R1>config>system>security>tacplus$ authorization use-priv-lvl

Now to actually enable tacacs authentication, within the password context we specify the authentication order to include the methods we prefer.
*A:R1>config>system>security>tacplus$ /configure system security password
*A:R1>config>system>security>password# authentication-order tacplus local exit-on-reject

If TACACS+ is unavailable, we fall back to local authentication accounts – if we hadn’t include “exit-on-reject”, a failed authentication attempt with TACACS+ (reject) would move onto the next authentication mechanisms (local)

SROS performs a AAA server health check by sending dummy authentication requests to a server and determines if the server is alive based on obtaining a response, this can end up with the authentication logs getting a lot of failed access attempts, however it can be disabled if desired:
*A:R1>config>system>security>password# no health-check
For this testing, I’ll be using telnet, so I need to enable the telnet-server (outside of a lab, I would not suggest this at all!)

*A:R1>config>system>security>password# back
*A:R1>config>system>security# telnet-server

So to recap the router configuration:
*A:R1>config>system>security# info
----------------------------------------------
            telnet-server
            profile "limited"
                default-action deny-all
                entry 10
                    match "show router route-table"
                    action permit
                exit
                entry 20
                    match "show users"
                    action permit
                exit
                entry 30
                    match "show system security user"
                    action permit
                exit
                entry 40
                    match "logout"
                    action permit
                exit
            exit
            password
                authentication-order tacplus local exit-on-reject
                no health-check
            exit
            tacplus
                authorization use-priv-lvl
                priv-lvl-map
                    priv-lvl 1 "limited"
                    priv-lvl 15 "administrative"
                exit
                timeout 5
                server 1 address 192.168.1.47 secret "1mSYRiobfhHAdFA9cZH3wBviQtXKFDld" hash2
            exit

Time to test if this works. Start the tacacs service on (m4600 has the IP of 192.168.1.47 which is what R1 will be communicating with)
root@m4600:~# tac_plus -d 16 -L -C /etc/tacacs+/tac_plus.conf
And start viewing syslog
root@m4600:~# tail -f /var/log/syslog
May 14 14:39:14 m4600 tac_plus[28164]: Reading config
May 14 14:39:14 m4600 tac_plus[28164]: Version F4.0.4.27a Initialized 1
May 14 14:39:14 m4600 tac_plus[28164]: tac_plus server F4.0.4.27a starting
May 14 14:39:14 m4600 tac_plus[28165]: Backgrounded
May 14 14:39:14 m4600 tac_plus[28166]: socket FD 0 AF 2
May 14 14:39:14 m4600 tac_plus[28166]: socket FD 2 AF 10
May 14 14:39:14 m4600 tac_plus[28166]: uid=0 euid=0 gid=0 egid=0 s=-1637085952

Open up another session on m4600 and telnet to 192.168.1.123 using the credentials of testadmin/pass:
May 14 14:39:23 m4600 tac_plus[28201]: connect from 192.168.1.123 [192.168.1.123]
May 14 14:39:23 m4600 tac_plus[28201]: cfg_acl_check(mgmt_acl, 192.168.1.123)
May 14 14:39:23 m4600 tac_plus[28201]: ip 192.168.1.123 matched permit regex 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4]) of acl filter mgmt_acl
May 14 14:39:23 m4600 tac_plus[28201]: host ACLs for user 'testadmin' permit
May 14 14:39:23 m4600 tac_plus[28201]: login query for 'testadmin' port console from 192.168.1.123 accepted
May 14 14:39:23 m4600 tac_plus[28202]: connect from 192.168.1.123 [192.168.1.123]
May 14 14:39:23 m4600 tac_plus[28202]: cfg_acl_check(mgmt_acl, 192.168.1.123)
May 14 14:39:23 m4600 tac_plus[28202]: ip 192.168.1.123 matched permit regex 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4]) of acl filter mgmt_acl
May 14 14:39:23 m4600 tac_plus[28202]: host ACLs for user 'testadmin' permit
May 14 14:39:23 m4600 tac_plus[28202]: authorization query for 'testadmin' console from 192.168.1.123 accepted

Lets go back to the telnet session and check who we are and our access rights:
*A:R1# show users
===============================================================================
User                             Type    Login time             Idle time
  From
===============================================================================
                                 Console       --               0d 00:00:21
  --
testadmin                        Telnet  14MAY2017 04:41:41     0d 00:00:00
  192.168.1.47
-------------------------------------------------------------------------------
Number of users : 1
===============================================================================
*A:R1>config>system>security# show system security user testadmin detail

===============================================================================
Users
===============================================================================
User ID      New User Permissions            Password   Login    Failed   Local
             Pwd console ftp li snmp netconf Expires    Attempts Logins   Conf
-------------------------------------------------------------------------------
testadmin    n   y       n   n  n    n       never      1        0        n
-------------------------------------------------------------------------------
Number of users : 1
===============================================================================

===============================================================================
Temporary User Configuration Detail
===============================================================================
===============================================================================
user id            : testadmin
-------------------------------------------------------------------------------
console parameters
-------------------------------------------------------------------------------
new pw required    : n/a                cannot change pw   : n/a
home directory     :
restricted to home : no
login exec file    :
profile            : administrative
locked-out         : no
===============================================================================

Okay, that’s good. Lets log out and log back in R1 using the credentials of testlimited/test:
May 14 14:43:37 m4600 tac_plus[29058]: connect from 192.168.1.123 [192.168.1.123]
May 14 14:43:37 m4600 tac_plus[29058]: cfg_acl_check(mgmt_acl, 192.168.1.123)
May 14 14:43:37 m4600 tac_plus[29058]: ip 192.168.1.123 matched permit regex 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4]) of acl filter mgmt_acl
May 14 14:43:37 m4600 tac_plus[29058]: host ACLs for user 'testlimited' permit
May 14 14:43:37 m4600 tac_plus[29058]: login query for 'testlimited' port telnet from 192.168.1.123 accepted
May 14 14:43:37 m4600 tac_plus[29059]: connect from 192.168.1.123 [192.168.1.123]
May 14 14:43:37 m4600 tac_plus[29059]: cfg_acl_check(mgmt_acl, 192.168.1.123)
May 14 14:43:37 m4600 tac_plus[29059]: ip 192.168.1.123 matched permit regex 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4]) of acl filter mgmt_acl
May 14 14:43:37 m4600 tac_plus[29059]: host ACLs for user 'testlimited' permit
May 14 14:43:37 m4600 tac_plus[29059]: authorization query for 'testlimited' telnet from 192.168.1.123 accepted

Looks promising from the TACACS server, lets go back to the telnet session and check who we are and our access rights:
*A:R1# show users
===============================================================================
User                             Type    Login time             Idle time
  From
===============================================================================
                                 Console       --               0d 00:04:55
  --
testlimited                      Telnet  14MAY2017 04:43:36     0d 00:00:00
  192.168.1.47
-------------------------------------------------------------------------------
Number of users : 1
===============================================================================
*A:R1# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
192.168.1.0/24                                Local   Local     01h07m25s  0
       TACACS                                                       0
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
*A:R1# admin display-config
MINOR: CLI Command not allowed for this user.

Certainly appears to be a limited user,

*A:R1# show system security user testlimited detail

===============================================================================
Users
===============================================================================
User ID      New User Permissions            Password   Login    Failed   Local
             Pwd console ftp li snmp netconf Expires    Attempts Logins   Conf
-------------------------------------------------------------------------------
testlimited  n   y       n   n  n    n       never      1        0        n
-------------------------------------------------------------------------------
Number of users : 1
===============================================================================

===============================================================================
Temporary User Configuration Detail
===============================================================================
===============================================================================
user id            : testlimited
-------------------------------------------------------------------------------
console parameters
-------------------------------------------------------------------------------
new pw required    : n/a                cannot change pw   : n/a
home directory     :
restricted to home : no
login exec file    :
profile            : limited
locked-out         : no
===============================================================================

Okay, so we’re correctly associated with the limited profile account.

Role based access control is a good idea for managing your network and being able to leverage your existing AAA infrastructure helps make operating a heterogeneous network that little bit easier.

Route aggregation is not always straight forward

Today’s post is a simple 3 router topology based on a true story when route aggregation didn’t appear to working as expected at first glance and some additional thought was required as to why things were behaving that way and what was required to make it do what I wanted.

I’m using 3 x Nokia VSR-Sims running SROS 14.0R4, and while the concepts discussed here are definitely flavoured through a SROS lens, the concepts will be familiar to different router platforms and their associated operating systems.

R1 and R2 are in BGP AS 12 and use ospf to advertise their respective system addresses which are used for their IBGP peerings

configure
    system
        name "R1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    router
        interface "Loop"
            address 100.100.100.1/24
            loopback
            no shutdown
        exit
        interface "R2"
            address 10.1.2.1/27
            port 1/1/1
            no shutdown
        exit
        interface "system"
            address 1.1.1.1/32
            no shutdown
        exit
        autonomous-system 12
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "R2"        
                    no shutdown
                exit
            exit
            no shutdown
        exit
        policy-options
            begin
            prefix-list "PL_LOOP"
                prefix 100.100.100.0/24 exact
            exit
            policy-statement "PS_LOOP_EXP"
                entry 10
                    from
                        protocol direct
                        prefix-list "PL_LOOP"
                    exit
                    action accept
                    exit              
                exit                  
            exit                      
            commit                    
        exit
        bgp
            group "IBGP"
                export "PS_LOOP_EXP"
                peer-as 12
                neighbor 2.2.2.2
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R2"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "R1"
            address 10.1.2.2/27
            port 1/1/1
            no shutdown
        exit
        interface "R3"
            address 10.2.3.2/27
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 2.2.2.2/32
            no shutdown
        exit
        autonomous-system 12          
        router-id 2.2.2.2
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "R1"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
        bgp
            group "IBGP"
                peer-as 12
                neighbor 1.1.1.1
                exit
            exit
            no shutdown
        exit
    exit
exit all

R3 which is in BGP AS 3 will be peering with R2. While we can configure R2, as R3 is in a different AS, we cannot touch it nor modify its configuration. R3 is already configured to peer with R2 and is waiting for R2 to come online.

Our configuration for R2 to establish the BGP Session:

A:R2# configure router bgp 
A:R2>config>router>bgp# group EBGP 
*A:R2>config>router>bgp>group$ neighbor 10.2.3.3 peer-as 3 
*A:R2>config>router>bgp>group$ exit all 

Assuming enough time has passed for BGP to come up, lets get a quick state of play with BGP on R2:

*A:R2# show router bgp summary | match "BGP Sum" post-lines 100  
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.1.1.1
               12          11    0 00h03m30s 1/1/0 (IPv4)
                           10    0           
10.2.3.3
               3            6    0 00h00m34s 8/8/1 (IPv4)
                            5    0           
-------------------------------------------------------------------------------

Right now R2 has active BGP sessions with R1 and R3 – we can see that R2 has received 8 routes from R3 and has sent 1 (from R1). R2 hasn’t yet sent any routes learnt from R3 to R1 however this should happen shortly.

These are the BGP routes that R2 knows of

*A:R2# show router bgp routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
u*>i  3.0.0.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.1.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.2.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.3.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.4.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.5.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.6.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.7.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  100.100.100.0/24                                   100         None
      1.1.1.1                                            None        -
      No As-Path                                                      
-------------------------------------------------------------------------------
Routes : 9
===============================================================================

As this post is about route aggregation, on R2 we want to send a summary route through to R1 (3.0.0.0/21) instead of all the individual routes. To do this we will create the aggregate route and specify it to be a summary-only route and because we can, we will include the AS-Set in the aggregate so R1 knows these came from AS 3
*A:R2# configure router aggregate 3.0.0.0/21 summary-only as-set description "Aggregate from AS3"
*A:R2# show router aggregate detail 

===============================================================================
Legend: G - generate-icmp enabled
===============================================================================
Aggregate Route Table (Router: Base)
===============================================================================
Prefix           : 3.0.0.0/21
Description      : Aggregate from AS3
Summary          : True           AS Set           : True
Aggr AS          : 0              Aggr IP-Address  : 0.0.0.0
Aggr OperState   : Active         
Nexthop Type     : None           Nexthop          :    
Community        :                
-------------------------------------------------------------------------------
No. of Aggregate Routes: 1
==============================================================================="

We can see the aggregate appear in the routing table as a blackhole route from protocol aggregate
*A:R2# show router route-table 

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
1.1.1.1/32                                    Remote  OSPF      00h10m16s  10
       10.1.2.1                                                     100
2.2.2.2/32                                    Local   Local     00h11m14s  0
       system                                                       0
3.0.0.0/21                                    Blackh* Aggr      00h02m13s  130
       Black Hole                                                   0
3.0.0.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.1.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.2.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.3.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.4.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.5.0/24                                    Remote  BGP       00h06m00s  170
       10.2.3.3                                                     0
3.0.6.0/24                                    Remote  BGP       00h06m00s  170
       10.2.3.3                                                     0
3.0.7.0/24                                    Remote  BGP       00h06m00s  170
       10.2.3.3                                                     0
10.1.2.0/27                                   Local   Local     00h11m00s  0
       R1                                                           0
10.2.3.0/27                                   Local   Local     00h11m00s  0
       R3                                                           0
100.100.100.0/24                              Remote  BGP       00h08m53s  170
       10.1.2.1                                                     0
-------------------------------------------------------------------------------
No. of Routes: 14
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
* indicates that the corresponding row element may have been truncated.

As we can see in the routing table, an aggregate route is treated as its own routing protocol, so we need to develop a routing policy to advertise the aggregate to R1
*A:R2# configure router policy-options 
*A:R2>config>router>policy-options# begin 
*A:R2>config>router>policy-options# policy-statement PS_AGGREGATE 
*A:R2>config>router>policy-options>policy-statement$ entry 10 from protocol aggregate 
*A:R2>config>router>policy-options>policy-statement$ entry 10 action accept 
*A:R2>config>router>policy-options>policy-statement>entry>action$ exit 
*A:R2>config>router>policy-options>policy-statement$ info 
----------------------------------------------
                entry 10
                    from
                        protocol aggregate
                    exit
                    action accept
                    exit
                exit
----------------------------------------------
*A:R2>config>router>policy-options>policy-statement$ exit 
*A:R2>config>router>policy-options# commit

We then can use the policy to export to our neighbor (using group IBGP or on the neighbor directly)
*A:R2>config>router>policy-options# /configure router bgp group "IBGP"     
*A:R2>config>router>bgp>group# export "PS_AGGREGATE"

One thing we haven’t done yet is that the EBGP next-hop 10.2.3.3 will not be visible to R1, so we can either add that interface into OSPF (as a passive interface so we don’t attempt to peer with an external router at the IGP level) or have R2 set next-hop-self (I generally prefer this as it keeps the IGP just for internal core links)
*A:R2>config>router>bgp>group# next-hop-self 
*A:R2>config>router>bgp>group# info 
----------------------------------------------
                next-hop-self
                export "PS_AGGREGATE" 
                peer-as 12
                neighbor 1.1.1.1
                exit
----------------------------------------------

Okay, so now R1 should have 3.0.0.0/21 and the job is done, so lets verify this is working on R1
A:R1# show router route-table 

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     00h18m48s  0
       system                                                       0
2.2.2.2/32                                    Remote  OSPF      00h17m57s  10
       10.1.2.2                                                     100
10.1.2.0/27                                   Local   Local     00h18m32s  0
       R2                                                           0
100.100.100.0/24                              Local   Local     00h18m48s  0
       Loop                                                         0
-------------------------------------------------------------------------------
No. of Routes: 4
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

3.0.0.0/21 is not present, so something is wrong here. What is R2 sending to R1?
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
No Matching Entries Found
===============================================================================

Nothing – let’s double check our policy
*A:R2>config>router>bgp>group# show router policy "PS_AGGREGATE" 
    entry 10
        from
            protocol aggregate
        exit
        action accept
        exit
    exit
*A:R2>config>router>bgp>group# show router aggregate 

===============================================================================
Legend: G - generate-icmp enabled
===============================================================================
Aggregates (Router: Base)
===============================================================================
Prefix                                          Aggr IP-Address   Aggr AS
   Summary                                         AS Set          State
     NextHop                                         Community     NextHopType
-------------------------------------------------------------------------------
3.0.0.0/21                                      0.0.0.0           0
   True                                            True            Active
                                                                      None
-------------------------------------------------------------------------------
No. of Aggregates: 1
===============================================================================

Well that looks okay but maybe the aggregate route is wrong
*A:R2>config>router>bgp>group# /admin display-config | match expression "^\ +agg"
        aggregate 3.0.0.0/21 summary-only as-set description "Aggregate from AS3"

Lets try it without including the summary-only option and see if the contributing routes will get advertised to R1.
*A:R2>config>router>bgp>group# /configure router aggregate 3.0.0.0/21 as-set description "Agg AS3 no summary-only"
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
i     3.0.0.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.1.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.3.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.4.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.5.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.6.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.7.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
-------------------------------------------------------------------------------
Routes : 7
===============================================================================

We are only sending 7 routes but we received 8 from R3!
*A:R2>config>router>bgp>group# show router bgp neighbor 10.2.3.3 received-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
u*>i  3.0.0.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.1.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.2.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.3.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.4.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.5.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.6.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.7.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
-------------------------------------------------------------------------------
Routes : 8
===============================================================================

So what is it about 3.0.2.0/24?
*A:R2>config>router>bgp>group# show router bgp routes 3.0.2.0/24 detail 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Original Attributes
 
Network        : 3.0.2.0/24
Nexthop        : 10.2.3.3
Path Id        : None                   
From           : 10.2.3.3
Res. Nexthop   : 10.2.3.3
Local Pref.    : n/a                    Interface Name : R3
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : None
AIGP Metric    : None                   
Connector      : None                 
Community      : no-advertise
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 3.3.3.3
Fwd Class      : None                   Priority       : None
Flags          : Used  Valid  Best  IGP  
Route Source   : External
AS-Path        : 3 
Route Tag      : 0                      
Neighbor-AS    : 3
Orig Validation: NotFound               
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default                
Last Modified  : 03h55m28s              
 
Modified Attributes
 
Network        : 3.0.2.0/24
Nexthop        : 10.2.3.3
Path Id        : None                   
From           : 10.2.3.3
Res. Nexthop   : 10.2.3.3
Local Pref.    : None                   Interface Name : R3
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : None
AIGP Metric    : None                   
Connector      : None
Community      : no-advertise
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 3.3.3.3
Fwd Class      : None                   Priority       : None
Flags          : Used  Valid  Best  IGP  
Route Source   : External
AS-Path        : 3 
Route Tag      : 0                      
Neighbor-AS    : 3
Orig Validation: NotFound               
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default                
Last Modified  : 03h55m30s              
 
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

3.0.2.0/24 has no-advertise attached to it!
One of the things about aggregate routes is that they aggregate the associated communities as well, so the aggregate route will have no-advertise attached to it, so it will not be advertised to R1.
Unfortunately this doesn’t appear in the show “show router aggregate detail” – the community there is the one that is manually added during the creation of the aggregate.
So how can we fix this? Well there are two methods that spring to mind and I am sure that there are more.

Option 1 – Create an import policy on R2 that just drops the no-advertise community on imported routes.
I think this is the easiest option to implement because then the normal aggregate configuration will work.

*A:R2>config>router>bgp>group# /configure router policy-options 
*A:R2>config>router>policy-options# begin 
*A:R2>config>router>policy-options# community NO_ADV members no-advertise 
*A:R2>config>router>policy-options# policy-statement PS_IGNORE_NO_ADV 
*A:R2>config>router>policy-options>policy-statement$ entry 10
*A:R2>config>router>policy-options>policy-statement>entry$ from community NO_ADV
*A:R2>config>router>policy-options>policy-statement>entry$ action accept 
*A:R2>config>router>policy-options>policy-statement>entry>action$ community remove "NO_ADV" 
*A:R2>config>router>policy-options>policy-statement>entry>action$ back 
*A:R2>config>router>policy-options>policy-statement>entry$ back 
*A:R2>config>router>policy-options>policy-statement$ info 
----------------------------------------------
                entry 10
                    from
                        community "NO_ADV"
                    exit
                    action accept
                        community remove "NO_ADV"
                    exit
                exit
----------------------------------------------
*A:R2>config>router>policy-options>policy-statement$ back 
*A:R2>config>router>policy-options# commit 
*A:R2>config>router>policy-options# /configure router bgp group "EBGP" 
*A:R2>config>router>bgp>group# neighbor 10.2.3.3 import "PS_IGNORE_NO_ADV"

Let’s see if that has resolved things:
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
i     3.0.0.0/21                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.0.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.1.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.2.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.3.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.4.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.5.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.6.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.7.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
-------------------------------------------------------------------------------
Routes : 9
===============================================================================

Yes, 3.0.2.0/24 is present and because none of the routes that contribute to the aggregate have no-advertise attached, the aggregate is also advertised to R1. So time to change the Aggregate route so it’s back to summary only:
*A:R2>config>router>bgp>group# /configure router aggregate 3.0.0.0/21 summary-only as-set description "Aggregate from AS3"
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes                                        
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
i     3.0.0.0/21                                         100         None
      2.2.2.2                                            None        -
      3                                                               
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Okay sorted.

Option 2 – Using Static Routes for Aggregation
If we wish to respect the no-advertise binding on 3.0.2.0/24, we can simulate some of the behavior of an aggregate route without caring about no-advertise (or no-export if we are concerned about advertisements outside of our AS).
First we need to remove the import policy on R2 facing R3.

*A:R2>config>router>bgp>group# info 
----------------------------------------------
                neighbor 10.2.3.3
                    import "PS_IGNORE_NO_ADV" 
                    peer-as 3
                exit
----------------------------------------------
*A:R2>config>router>bgp>group# neighbor 10.2.3.3 no import

And remove the aggregate for 3.0.0.0/21
*A:R2>config>router>bgp>group# /configure router no aggregate 3.0.0.0/21
Now we create a static black-hole route with BGP community 12:12 attached to it. We’re attaching the community so we can distinguish between regular static routes and our “aggregate”
A:R2>config>router# static-route-entry 3.0.0.0/21 
*A:R2>config>router>static-route-entry$ black-hole 
*A:R2>config>router>static-route-entry>black-hole$ community 12:12 
*A:R2>config>router>static-route-entry>black-hole$ no shutdown

As a note, SROS Release 14 changed the specific syntax for creating static routes but the concepts generally remain the same for previous SROS versions.
Now we’ll work on the routing policy to advertise our static aggregate route.
First we’ll create a named community that was used for our aggregate:
*A:R2>config>router>static-route-entry$ /configure router policy-options 
*A:R2>config>router>policy-options# begin
*A:R2>config>router>policy-options# community STATIC_AGG members 12:12 

Now we create a prefix list to match the routes that contribute to our aggregate
*A:R2>config>router>policy-options# prefix-list PL_R3_CONTRIB 
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.0.0/21 longer
*A:R2>config>router>policy-options>prefix-list$ exit

Finally we take the existing PS_AGGREGATE and modify it to work with our static aggregate and drop the contributing routes:
*A:R2>config>router>policy-options# policy-statement "PS_AGGREGATE" 
*A:R2>config>router>policy-options>policy-statement# info 
----------------------------------------------
                entry 10
                    from
                        protocol aggregate
                    exit
                    action accept
                    exit
                exit
*A:R2>config>router>policy-options>policy-statement# entry 10 
*A:R2>config>router>policy-options>policy-statement>entry# from protocol static 
*A:R2>config>router>policy-options>policy-statement>entry# from community "STATIC_AGG"
*A:R2>config>router>policy-options>policy-statement>entry# back 
*A:R2>config>router>policy-options>policy-statement# entry 20 
*A:R2>config>router>policy-options>policy-statement>entry$ from prefix-list "PL_R3_CONTRIB" 
*A:R2>config>router>policy-options>policy-statement>entry$ action drop 
*A:R2>config>router>policy-options>policy-statement>entry>action$ exit 
*A:R2>config>router>policy-options>policy-statement>entry$ exit 
*A:R2>config>router>policy-options>policy-statement# info 
----------------------------------------------
                entry 10
                    from
                        protocol static
                        community "STATIC_AGG"
                    exit
                    action accept
                    exit
                exit
                entry 20
                    from
                        prefix-list "PL_R3_CONTRIB"
                    exit
                    action drop
                    exit
                exit
*A:R2>config>router>policy-options>policy-statement# back 
*A:R2>config>router>policy-options# commit

Lets check what R2 is advertising to R1:
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
?     3.0.0.0/21                                         100         None
      2.2.2.2                                            None        -
      No As-Path                                                      
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

There are a couple of issues with this implementation the AS-Path information is lost (nothing we can do about that) but more importantly, this “aggregate” will stay up even if the contributing routes are not present. To overcome this issue, the static route can be associated with a prefix-list which will be used to determine if the static route can become active. It should be noted that although we already have PL_R3_CONTRIB, it cannot be used here as the prefix list in the static route requires specific prefixes to match against. While matching all possible prefixes could be problematic, in most instances simply matching against a few key prefixes will be sufficient:
*A:R2>config>router>bgp>group# /configure router policy-options 
*A:R2>config>router>policy-options# begin
*A:R2>config>router>policy-options# prefix-list PL_R3_STATIC_AGG_OK  
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.0.0/24 
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.7.0/24 
*A:R2>config>router>policy-options>prefix-list$ exit 
*A:R2>config>router>policy-options# commit 

Modify the static route to be up when any of the prefixes in PL_R3_STATIC_AGG_OK are in the routing table:
*A:R2>config>router>policy-options# /configure router static-route-entry 3.0.0.0/21 black-hole prefix-list "PL_R3_STATIC_AGG_OK" any

We can see the route is active and the prefix-list being used to validate:
*A:R2>config>router>policy-options# show router static-route detail 

===============================================================================
Static Route Table (Router: Base)  Family: IPv4
===============================================================================
Prefix           : 3.0.0.0/21
Nexthop          : n/a
Type             : Blackhole
Dynamic BGP      : disabled                        Generate ICMP     : disabled
Interface        : n/a                             Active            : Y
Prefix List      : PL_R3_STATIC_AGG_OK             Prefix List Type  : Any
Metric           : 1                               Preference        : 5
Source Class     : 0                               Dest Class        : 0
Admin  State     : Up                              Tag               : 0
Creation Origin  : manual                          
BFD              : disabled
Community        : 12:12                           
CPE-check        : disabled
-------------------------------------------------------------------------------
No. of Static Routes: 1

===============================================================================

If we shutdown our BGP session to R3, the routes in PL_R3_STATIC_AGG_OK will disappear from the routing table and the static route will be brought out of service
*A:R2>config>router>policy-options# /configure router bgp group "EBGP" 
*A:R2>config>router>bgp>group# shutdown 
*A:R2>config>router>bgp>group# show router static-route detail    

===============================================================================
Static Route Table (Router: Base)  Family: IPv4
===============================================================================
Prefix           : 3.0.0.0/21
Nexthop          : n/a
Type             : Blackhole
Dynamic BGP      : disabled                        Generate ICMP     : disabled
Interface        : n/a                             Active            : N
Prefix List      : PL_R3_STATIC_AGG_OK             Prefix List Type  : Any
Metric           : 1                               Preference        : 5
Source Class     : 0                               Dest Class        : 0
Admin  State     : Up                              Tag               : 0
Creation Origin  : manual                          
BFD              : disabled
Community        : 12:12                           
CPE-check        : disabled
Inactive Reason  : prefix-list match failed
-------------------------------------------------------------------------------
No. of Static Routes: 1

===============================================================================

We can see the static route is down because the prefix-list match has failed and we can confirm that we aren’t advertising this to R1:
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
No Matching Entries Found
===============================================================================

So we’ll restore the EBGP session between R2 and R3 and give it enough time to exchange routes again:
*A:R2>config>router>bgp>group# no shutdown              
*A:R2>config>router>bgp>group# show router bgp routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
u*>i  3.0.0.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.1.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.2.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.3.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.4.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.5.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.6.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.7.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  100.100.100.0/24                                   100         None
      1.1.1.1                                            None        -
      No As-Path                                                      
-------------------------------------------------------------------------------
Routes : 9
===============================================================================

The routes from R3 are back, lets confirm the static blackhole is back in service:
*A:R2>config>router>bgp>group# show router static-route detail 

===============================================================================
Static Route Table (Router: Base)  Family: IPv4
===============================================================================
Prefix           : 3.0.0.0/21
Nexthop          : n/a
Type             : Blackhole
Dynamic BGP      : disabled                        Generate ICMP     : disabled
Interface        : n/a                             Active            : Y
Prefix List      : PL_R3_STATIC_AGG_OK             Prefix List Type  : Any
Metric           : 1                               Preference        : 5
Source Class     : 0                               Dest Class        : 0
Admin  State     : Up                              Tag               : 0
Creation Origin  : manual                          
BFD              : disabled
Community        : 12:12                           
CPE-check        : disabled
-------------------------------------------------------------------------------
No. of Static Routes: 1

===============================================================================

Yes, so we should be offering this to R3 again:
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
?     3.0.0.0/21                                         100         None
      2.2.2.2                                            None        -
      No As-Path                                                      
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Yes, the aggregate route is now conditionally advertised.

While this scenario isn’t likely to occur all the time, based on my experience it is something to consider if things are not working quite as expected.

The case of Nokia Virtual Service Router and the non-unique Chassis MAC Address

So I’m playing with eve-ng and have decided to work on a Layer 2 scenario and a few problems with my emulation environment came up which needed a way forward, which resulted in this rambling tale…

SROS 12.0R6 5 Router Topology

R1, R2 and R3 Will be the MPLS Core with VPLS configured, while R4 and R5 will be Layer 3 CE devices that talk to each other over the VPLS.

The CE Devices are pretty straight forward so we’ll get those up first

R4 is a single-ended configuration with Interface R5 on Port 1/1/1 having the IP 192.168.1.4/27

configure
    system
        name "R4"
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    router 
        interface "R5"
            address 192.168.1.4/27
            port 1/1/1
            no shutdown
        exit
        interface "system"
            no shutdown
        exit
    exit
exit all

R5 is a a little more complex, it has a LAG toward – Interface R4 on LAG-1 with Ports 1/1/1 and 1/1/2 having the IP 192.168.1.5/27

configure
    system
        name "R5"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
            autonegotiate limited
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
            autonegotiate limited
        exit
        no shutdown
    exit
    lag 1                             
        port 1/1/1 
        port 1/1/2 
        lacp active administrative-key 32768 
        no shutdown
    exit
    router 
        interface "R4"
            address 192.168.1.5/27
            port lag-1
            no shutdown
        exit
        interface "system"
            no shutdown
        exit                          
    exit
exit all

Multi-speed Ethernet interfaces when associated with a LAG must have autonegotiate set to limited to control the bundle member speed so they all bundle members operate the same speed

Now to Develop the MPLS Core Configuration on R1, R2 and R3 – this is quite straight forward, we are just going to use OSPF and LDP on the directly connected interfaces:

configure
    system
        name "R1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        shutdown
        ethernet
        exit
    exit
    router 
        interface "R2"
            address 10.1.2.1/27
            port 1/1/1
            no shutdown
        exit
        interface "R3"
            address 10.1.3.1/27
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.1/32
            no shutdown
        exit
        ospf
            area 0.0.0.0              
                interface "system"
                    no shutdown
                exit
                interface "R2"
                    no shutdown
                exit
                interface "R3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "R2"
                exit
                interface "R3"
                exit
            exit
            targeted-session
            exit                      
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R2"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        shutdown
        ethernet
        exit
    exit
    router 
        interface "R1"
            address 10.1.2.2/27
            port 1/1/1
            no shutdown
        exit
        interface "R3"
            address 10.2.3.2/27
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.2/32
            no shutdown
        exit
        ospf
            area 0.0.0.0              
                interface "system"
                    no shutdown
                exit
                interface "R1"
                    no shutdown
                exit
                interface "R3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "R1"
                exit
                interface "R3"
                exit
            exit
            targeted-session
            exit                      
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R3"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        shutdown
        ethernet
        exit
    exit
    router 
        interface "R1"
            address 10.1.3.3/27
            port 1/1/2
            no shutdown
        exit
        interface "R2"
            address 10.2.3.3/27
            port 1/1/3
            no shutdown
        exit
        interface "system"
            address 10.10.10.3/32
            no shutdown
        exit
        ospf
            area 0.0.0.0              
                interface "system"
                    no shutdown
                exit
                interface "R1"
                    no shutdown
                exit
                interface "R2"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "R1"
                exit
                interface "R2"
                exit
            exit
            targeted-session
            exit                      
            no shutdown
        exit
    exit
exit all

The Layer 2 Service that we are going to build is a VPLS and will be using Spoke-SDPs that connected to each adjacent router (an alternate could be to use a full-mesh but I specifically want to test STP operation here)

*A:R1>config>service# info 
----------------------------------------------
        sdp 2 mpls create
            far-end 10.10.10.2
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit
        sdp 3 mpls create
            far-end 10.10.10.3
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit

*A:R2>config>service# info 
----------------------------------------------
        sdp 1 mpls create
            far-end 10.10.10.1
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit
        sdp 3 mpls create
            far-end 10.10.10.3
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit

*A:R3>config>service# info 
----------------------------------------------
        sdp 1 mpls create
            far-end 10.10.10.1
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit
        sdp 2 mpls create
            far-end 10.10.10.2
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit

Verifying the SDPs are up:

A:R1# show service sdp 

============================================================================
Services: Service Destination Points
============================================================================
SdpId  AdmMTU  OprMTU  Far End          Adm  Opr         Del     LSP   Sig
----------------------------------------------------------------------------
2      0       8914    10.10.10.2       Up   Up          MPLS    L     TLDP
3      0       8914    10.10.10.3       Up   Up          MPLS    L     TLDP
----------------------------------------------------------------------------
Number of SDPs : 2
----------------------------------------------------------------------------
Legend: R = RSVP, L = LDP, B = BGP, M = MPLS-TP, n/a = Not Applicable
============================================================================

A:R2# show service sdp 

============================================================================
Services: Service Destination Points
============================================================================
SdpId  AdmMTU  OprMTU  Far End          Adm  Opr         Del     LSP   Sig
----------------------------------------------------------------------------
1      0       8914    10.10.10.1       Up   Up          MPLS    L     TLDP
3      0       8914    10.10.10.3       Up   Up          MPLS    L     TLDP
----------------------------------------------------------------------------
Number of SDPs : 2
----------------------------------------------------------------------------
Legend: R = RSVP, L = LDP, B = BGP, M = MPLS-TP, n/a = Not Applicable
============================================================================

A:R3# show service sdp 

============================================================================
Services: Service Destination Points
============================================================================
SdpId  AdmMTU  OprMTU  Far End          Adm  Opr         Del     LSP   Sig
----------------------------------------------------------------------------
1      0       8914    10.10.10.1       Up   Up          MPLS    L     TLDP
2      0       8914    10.10.10.2       Up   Up          MPLS    L     TLDP
----------------------------------------------------------------------------
Number of SDPs : 2
----------------------------------------------------------------------------
Legend: R = RSVP, L = LDP, B = BGP, M = MPLS-TP, n/a = Not Applicable
============================================================================

With the transport infrastructure in place VPLS 100 without the customer access components can be set up:

*A:R1>config>service>vpls$ pwc 
-------------------------------------------------------------------------------
Present Working Context :
-------------------------------------------------------------------------------
 <root>
  configure 
  service 
  vpls "100" customer 1 create 
-------------------------------------------------------------------------------
A:R1>config>service>vpls$ info 
----------------------------------------------
            stp
                no shutdown
            exit
            spoke-sdp 2:100 create
                no shutdown
            exit
            spoke-sdp 3:100 create
                no shutdown
            exit
            no shutdown

*A:R2>config>service>vpls$ pwc 
-------------------------------------------------------------------------------
Present Working Context :
-------------------------------------------------------------------------------
 <root>
  configure 
  service 
  vpls "100" customer 1 create 
-------------------------------------------------------------------------------
A:R2>config>service>vpls$ info 
----------------------------------------------
            stp
                no shutdown
            exit
            spoke-sdp 1:100 create
                no shutdown
            exit
            spoke-sdp 3:100 create
                no shutdown
            exit
            no shutdown

*A:R3>config>service>vpls$ pwc 
-------------------------------------------------------------------------------
Present Working Context :
-------------------------------------------------------------------------------
 <root>
  configure 
  service 
  vpls "100" customer 1 create 
-------------------------------------------------------------------------------
A:R3>config>service>vpls$ info 
----------------------------------------------
            stp
                no shutdown
            exit
            spoke-sdp 1:100 create
                no shutdown
            exit
            spoke-sdp 2:100 create
                no shutdown
            exit
            no shutdown

Verify that VPLS 100 is up and running:

*A:R1>config>service>*A:R1# show service id 100 base | match Ident post-lines 3 
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sdp:2:100 S(10.10.10.2)                  Spok         0       8914    Up   Up
sdp:3:100 S(10.10.10.3)                  Spok         0       8914    Up   Up

A:R2# show service id 100 base | match Ident post-lines 3 
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sdp:1:100 S(10.10.10.1)                  Spok         0       8914    Up   Up
sdp:3:100 S(10.10.10.3)                  Spok         0       8914    Up   Up

A:R3# show service id 100 base | match Ident post-lines 3 
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sdp:1:100 S(10.10.10.1)                  Spok         0       8914    Up   Up
sdp:2:100 S(10.10.10.2)                  Spok         0       8914    Up   Up

Looks good With 3 routers each connecting to each other using spokes will introduce a bridging loop so we need a loop avoidance mechanism – luckily we enabled STP, so lets see how STP is behaving:

*A:R1# show service id 100 stp                        

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 80:00.da:00:ff:00:00:01  Top. Change Count : 4
Root Bridge        : This Bridge              Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:10:13
Vcp Active Prot.   : N/A                      
Root Port          : N/A                      External RPC      : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
2:100              Up        Designated Forward     2049   True   Pt-pt  Rstp
3:100              Up        Backup     Discard     2050   False  Pt-pt  Rstp
===============================================================================

*A:R2# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 80:00.da:00:ff:00:00:01  Top. Change Count : 3
Root Bridge        : This Bridge              Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:10:47
Vcp Active Prot.   : N/A                      
Root Port          : N/A                      External RPC      : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
1:100              DwnstrmLp Designated Discard     2049   False  Pt-pt  Rstp
3:100              Up        Backup     Discard     2050   False  Pt-pt  Rstp
===============================================================================

*A:R3# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 80:00.da:00:ff:00:00:01  Top. Change Count : 3
Root Bridge        : This Bridge              Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:10:54
Vcp Active Prot.   : N/A                      
Root Port          : N/A                      External RPC      : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
1:100              Up        Designated Forward     2048   False  Pt-pt  Rstp
2:100              Up        Designated Forward     2049   False  Pt-pt  Rstp
===============================================================================

This doesn’t seem right, SDP 1:100 on R2 is saying that the downstream interface is looped and both interfaces are discarding!

If we look at the highlighted lines on each of the router outputs we notice that all Routers in the VPLS have the same Bridge ID, which is definitely a bad thing.

For SROS, the Bridge Id is partly derived from the chassis MAC address:

*A:R1# show chassis detail | match MAC  
  Base MAC address                  : da:00:ff:00:00:01

*A:R2# show chassis detail | match MAC  
  Base MAC address                  : da:00:ff:00:00:01

*A:R3# show chassis detail | match MAC  
  Base MAC address                  : da:00:ff:00:00:01

With real hardware, the Chassis MAC address actually is unique so this problem wont come up – however with the VSRs they’re all the same.

As an asside, the Chassis MAC address is used in a few places besides STP, one is with the SNMP engine id

*A:R1# show chassis detail | match MAC      
  Base MAC address                  : da:00:ff:00:00:01
*A:R1# show system information | match Engine 
SNMP Engine ID         : 0000197f0000da00ff000001
SNMP Engine Boots      : 11

It is possible within the configuration to manually set the Engine ID (I think it would probably be best to do this in production just in case you end up replacing faulty hardware)

With SROS version 14.0R4 a new option for the boot options file (or bof) was introduced which allows the manual setting of the chassis MAC address (followed by a reboot):

*A:R14# bof system-base-mac 00:11:22:33:44:02 
*A:R14# bof save 
Writing BOF to cf3:/bof.cfg ... OK
Completed.
Writing configuration to cf3:\config.cfg
Saving configuration ... OK
Completed.
A:R14# /admin reboot 
Are you sure you want to reboot (y/n)? y

Which is great but this particular set up is using SROS 12.0R6 and that BOF option doesn’t exist an alternate method is required.

For STP we can cast our mind back to remember what the Bridge ID consists of… It’s both the Priority (which by default is 32768) and the Bridge MAC address.

So as a quick and nasty fix, I should just be able to change the STP Priority in VPLS 100 on R1/R2/R3 and resolve the STP problem, it also will allow me to specifically select a root bridge which is probably a good thing to do.
*A:R1# configure service vpls 100 stp priority 4096
*A:R2# configure service vpls 100 stp priority 8192
*A:R3# configure service vpls 100 stp priority 16384
Lets see how things are going now:

*A:R1# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 10:00.da:00:ff:00:00:01  Top. Change Count : 6
Root Bridge        : This Bridge              Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:00:35
Vcp Active Prot.   : N/A                      
Root Port          : N/A                      External RPC      : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
2:100              Up        Designated Forward     2049   False  Pt-pt  Rstp
3:100              Up        Designated Forward     2050   False  Pt-pt  Rstp
===============================================================================

*A:R2# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 20:00.da:00:ff:00:00:01  Top. Change Count : 4
Root Bridge        : 10:00.da:00:ff:00:00:01  Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:01:07
Vcp Active Prot.   : N/A                      
Root Port          : 2049                     External RPC      : 10

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
1:100              Up        Root       Forward     2049   False  Pt-pt  Rstp
3:100              Up        Designated Forward     2050   False  Pt-pt  Rstp
===============================================================================

*A:R3# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 40:00.da:00:ff:00:00:01  Top. Change Count : 4
Root Bridge        : 10:00.da:00:ff:00:00:01  Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:01:52
Vcp Active Prot.   : N/A                      
Root Port          : 2048                     External RPC      : 10

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
1:100              Up        Root       Forward     2048   False  Pt-pt  Rstp
2:100              Up        Alternate  Discard     2049   False  Pt-pt  Rstp
===============================================================================

Success, all routers have different bridge IDs and all agree that R1 is the root and only one port is in discarding state.

Now we will create the CE router attachments (Service Access Points) on the Core starting with R3 which is facing R4 – by default Ethernet ports are in network mode, to be able to bind to a service, the port must be mode access (or hybrid)

*A:R3# /configure port 1/1/1     
*A:R3>config>port# shutdown 
*A:R3>config>port# ethernet mode access 
*A:R3>config>port# ethernet encap-type null 
*A:R3>config>port# no shutdown 
*A:R3>config>port# /configure service vpls 100 
*A:R3>config>service>vpls# sap 1/1/1 create 
*A:R3>config>service>vpls>sap$ show service id 100 base

===============================================================================
Service Basic Information
===============================================================================
Service Id        : 100                 Vpn Id            : 0
Service Type      : VPLS                
Name              : (Not Specified)
Description       : (Not Specified)
Customer Id       : 1                   Creation Origin   : manual
Last Status Change: 04/21/2017 13:20:28 
Last Mgmt Change  : 04/21/2017 13:44:59 
Etree Mode        : Disabled            
Admin State       : Up                  Oper State        : Up
MTU               : 1514                Def. Mesh VC Id   : 100
SAP Count         : 1                   SDP Bind Count    : 2
Snd Flush on Fail : Disabled            Host Conn Verify  : Disabled
Propagate MacFlush: Disabled            Per Svc Hashing   : Disabled
Allow IP Intf Bind: Disabled            
Def. Gateway IP   : None                
Def. Gateway MAC  : None                
Temp Flood Time   : Disabled            Temp Flood        : Inactive
Temp Flood Chg Cnt: 0                   
VSD Domain        : <none>            
 
-------------------------------------------------------------------------------
Service Access & Destination Points
-------------------------------------------------------------------------------
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sap:1/1/1                                null         1514    1514    Up   Up
sdp:1:100 S(10.10.10.1)                  Spok         0       8914    Up   Up
sdp:2:100 S(10.10.10.2)                  Spok         0       8914    Up   Up
===============================================================================

Now things are going to get a little more complicated on R1 and R2 as we are going to establish a Multi-Chassis LAG towards R5. R5 is unaware of the MC-LAG, it is just talking LACP to R1 and R2 thinking they are just one system. R1 and R2 require synchronisation between each other to set up the Active-Standby LAG.

We’ll start by creating regular LAG-1 Facing R5 on R1 and R2 with a single port in each:

*A:R1# /configure port 1/1/3 shutdown                          
*A:R1# /configure port 1/1/3 ethernet mode access 
*A:R1# /configure port 1/1/3 ethernet encap-type null 
*A:R1# /configure port 1/1/3 ethernet autonegotiate limited 
*A:R1# /configure port 1/1/3 no shutdown                    
*A:R1# /configure lag 1 
*A:R1>config>lag$ mode access 
*A:R1>config>lag$ lacp active 
*A:R1>config>lag$ port 1/1/3 
*A:R1>config>lag$ no shutdown

*A:R2# /configure port 1/1/3 shutdown                          
*A:R2# /configure port 1/1/3 ethernet mode access 
*A:R2# /configure port 1/1/3 ethernet encap-type null 
*A:R2# /configure port 1/1/3 ethernet autonegotiate limited 
*A:R2# /configure port 1/1/3 no shutdown                    
*A:R2# /configure lag 1 
*A:R2>config>lag$ mode access 
*A:R2>config>lag$ lacp active 
*A:R2>config>lag$ port 1/1/3 
*A:R2>config>lag$ no shutdown

Now to set up MC-LAG we need to set up a multi-chassis peering between R1 and R2 (multi-chassis redundancy supports more than just MC-LAG):

*A:R1>config>lag# /configure redundancy multi-chassis peer 10.10.10.2 create
*A:R1>config>redundancy>multi-chassis>peer# no shutdown

*A:R2>config>lag# /configure redundancy multi-chassis peer 10.10.10.1 create 
*A:R2>config>redundancy>multi-chassis>peer# no shutdown

Then we create the MC-LAG itself, we require the lacp-key, system-id and priority to be the same on each router:

*A:R1>config>redundancy>multi-chassis>peer# mc-lag
*A:R1>config>redundancy>mc>peer>mc-lag#lag 1 lacp-key 2468 remote-lag 1 system-id 00:00:be:ef:ca:fe system-priority 1000 
*A:R1>config>redundancy>mc>peer>mc-lag#no shutdown

*A:R2>config>redundancy>multi-chassis>peer# mc-lag
*A:R2>config>redundancy>mc>peer>mc-lag#lag 1 lacp-key 2468 remote-lag 1 system-id 00:00:be:ef:ca:fe system-priority 1000 
*A:R2>config>redundancy>mc>peer>mc-lag#no shutdown

Now the MC-LAG should be up and running, first we’ll check the peering

*A:R1>config>redundancy>mc>peer>mc-lag# show redundancy multi-chassis all 

===============================================================================
Multi-Chassis Peers
===============================================================================
Peer IP          Peer Admin      Client    Admin        Oper         State
 Src IP           Auth                                               
-------------------------------------------------------------------------------
10.10.10.2       Enabled         MC-Sync:  --           --           --
 10.10.10.1       None           MC-Ring:  --           --           --
                                 MC-Endpt: --           --           --
                                 MC-Lag:   Enabled      Enabled      --
                                 MC-IPsec: --           --           Disabled
===============================================================================

*A:R2>config>redundancy>mc>peer>mc-lag# show redundancy multi-chassis all 

===============================================================================
Multi-Chassis Peers
===============================================================================
Peer IP          Peer Admin      Client    Admin        Oper         State
 Src IP           Auth                                               
-------------------------------------------------------------------------------
10.10.10.1       Enabled         MC-Sync:  --           --           --
 10.10.10.2       None           MC-Ring:  --           --           --
                                 MC-Endpt: --           --           --
                                 MC-Lag:   Enabled      Enabled      --
                                 MC-IPsec: --           --           Disabled
===============================================================================

Looks promising, lets check our LAG status
*A:R1>config>redundancy>mc>peer>mc-lag# show lag 

===============================================================================
Lag Data
===============================================================================
Lag-id         Adm     Opr     Weighted Threshold Up-Count MC Act/Stdby
-------------------------------------------------------------------------------
1              up      down    No       0         0        standby
-------------------------------------------------------------------------------
Total Lag-ids: 1       Single Chassis: 0        MC Act: 0       MC Stdby: 1
===============================================================================

*A:R2>config>redundancy>mc>peer>mc-lag# show lag 

===============================================================================
Lag Data
===============================================================================
Lag-id         Adm     Opr     Weighted Threshold Up-Count MC Act/Stdby
-------------------------------------------------------------------------------
1              up      down    No       0         0        standby
-------------------------------------------------------------------------------
Total Lag-ids: 1       Single Chassis: 0        MC Act: 0       MC Stdby: 1
===============================================================================

Ummm… both of these are showing that they are in Multi-Chassis Standby

It turns out that within the MC-LAG configuration, the Base Chassis MAC needs to be unique too. While we cannot directly change the Base MAC prior to SROS version 14.0R4 there is actually an alternative method available. if we set the out-of-band management ethernet IP address, this will influence the chassis MAC address.

*A:R1>config>lag# show bof 
===============================================================================
BOF (Memory)
===============================================================================
    primary-image    cf3:\timos\both.tim
    primary-config   cf3:\config.cfg
    autonegotiate
    duplex           full
    speed            100
    wait             3
    persist          off
    no li-local-save
    no li-separate
    console-speed    115200
===============================================================================
*A:R1>config>lag# /bof address 192.168.100.1/24 
*A:R1>config>lag# /bof save 
Writing BOF to cf3:/bof.cfg ... OK
Completed.
*A:R1>config>lag# show bof 
===============================================================================
BOF (Memory)
===============================================================================
    primary-image    cf3:\timos\both.tim
    primary-config   cf3:\config.cfg
    address          192.168.100.1/24 active
    autonegotiate
    duplex           full
    speed            100
    wait             3
    persist          off
    no li-local-save
    no li-separate
    console-speed    115200
===============================================================================

Save and reboot
*A:R1>config>lag# /admin save 
Writing configuration to cf3:\config.cfg
Saving configuration ... OK
Completed.
A:R1>config>lag# /admin reboot 
Are you sure you want to reboot (y/n)? y

We’ll do the same thing with R2 but give it a different IP so the MAC Addresses should be different:
*A:R2>config>lag# /bof address 192.168.100.2/24 
*A:R2>config>lag# /bof save 
Writing BOF to cf3:/bof.cfg ... OK
Completed.
*A:R2>config>lag# /admin save 
Writing configuration to cf3:\config.cfg
Saving configuration ... OK
Completed.
A:R2>config>lag# /admin reboot 
Are you sure you want to reboot (y/n)? y 

After the reboot we can compare R1 and R2’s Base MAC Address
A:R1# show chassis detail | match MAC 
  Base MAC address                  : c8:01:ff:00:00:00

A:R2# show chassis detail | match MAC 
  Base MAC address                  : c8:02:ff:00:00:00

Okay they are different now – has it resolved our MC-LAG issue?
A:R1# show lag 1 port 

===============================================================================
Lag Port States
LACP Status: e - Enabled, d - Disabled
===============================================================================
Lag-id Port-id   Adm   Act/Stdby Opr   Primary  Sub-group     Forced  Priority
-------------------------------------------------------------------------------
1(e)   1/1/3     up    active    up    yes      1             -       32768
===============================================================================

A:R2# show lag 1 port 

===============================================================================
Lag Port States
LACP Status: e - Enabled, d - Disabled
===============================================================================
Lag-id Port-id   Adm   Act/Stdby Opr   Primary  Sub-group     Forced  Priority
-------------------------------------------------------------------------------
1(e)   1/1/3     up    standby   down  yes      1             -       32768
===============================================================================

A:R5# show lag 1 port 

===============================================================================
Lag Port States
LACP Status: e - Enabled, d - Disabled
===============================================================================
Lag-id Port-id   Adm   Act/Stdby Opr   Primary  Sub-group     Forced  Priority
-------------------------------------------------------------------------------
1(e)   1/1/1     up    active    up    yes      1             -       32768
       1/1/2     up    active    down           1             -       32768
===============================================================================

Yes R1, R2 and R5 are in alignment, now lets put the LAG into VPLS 100 on R1 and R2
A:R1# /configure service vpls 100 sap lag-1 create
A:R2# /configure service vpls 100 sap lag-1 create
Lets see if R5 can ping R4
A:R5# ping 192.168.1.4 count 1 
PING 192.168.1.4 56 data bytes
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=12.3ms.

---- 192.168.1.4 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 12.3ms, avg = 12.3ms, max = 12.3ms, stddev = 0.000ms

Success!

Lets check the MAC address table in vpls 100 (Forwarding Data Base):

*A:R1>config>service>vpls>sap$ show service id 100 fdb detail 

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
100       50:00:00:07:00:01 sdp:3:100                L/0      04/21/17 14:47:33
100       da:00:ff:00:01:42 sap:lag-1                L/0      04/21/17 14:52:57
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

*A:R2>config>service>vpls>sap$ show service id 100 fdb detail 

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
100       50:00:00:07:00:01 sdp:1:100                L/90     04/21/17 14:53:01
100       da:00:ff:00:01:42 sdp:1:100                L/90     04/21/17 14:45:05
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

*A:R2>config>service>vpls>sap$ show service id 100 fdb detail

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
100       50:00:00:07:00:01 sap:1/1/1                L/0      04/21/17 14:52:42
100       da:00:ff:00:01:42 sdp:1:100                L/0      04/21/17 14:44:46
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

Now to check out the MC-LAG resiliency, we’ll start a continuous ping on R5 to R4 and then shutdown port 1/1/3 (LAG-1) on R1
*A:R1>config>service>vpls>sap$ /configure port 1/1/3 shutdown
And Check if R2 LAG 1 Port 1/1/3 goes from standby to active
*A:R2>config>service>vpls>sap$ show lag 1 port 

===============================================================================
Lag Port States
LACP Status: e - Enabled, d - Disabled
===============================================================================
Lag-id Port-id   Adm   Act/Stdby Opr   Primary  Sub-group     Forced  Priority
-------------------------------------------------------------------------------
1(e)   1/1/3     up    active    up    yes      1             -       32768
===============================================================================

We can see the interface has come up and there were a few packets lost but the link recovered – we could speed up the link convergence time but I think the general concept has been demonstrated sucessfully.

The moral of the story here – with Virtual SROS systems, it’s worth ensuring you have a unique chassis MAC address!

Working with eve-ng (Active-Backup Bond Interfaces with eth0 and wlan0)

After a lack of updates, its’ time for a new blog post – this post is about linux networking particularly using active-backup bond interfaces for wired and wireless LAN interfaces, which is part of creating my updated virtual network lab environment.

Unetlab which was pretty much an alternative to GNS3 has now evolved into eve-ng which is quite a nice system. Amongst other things is it has a custom linux kernel that doesn’t block L2 Slow protocols like LACP. One of the things I specifically like about Eve besides having the facility to use html5 sessions to handle telnet/VNC consoles (as well as native tools) is that there are some SROS specific modifications that support the distributed VSR models as well as passing SMBIOS parameters etc.

I did a bare metal install pretty much following the process described in http://www.eve-ng.net/index.php/documentation/installation/bare-install but did a few more things.

I installed xcfe4 so I can have a graphical desktop with firefox so I can use eve locally, not just remoting into it.

I also did a few changes to the base install network configuration to allow the use of the copper ethernet as the primary interface but falling back to wireless.

Normally you cannot add a wireless interface into a bridge (normally eve binds eth0 into bridge pnet0 but simply adding wlan0 didn’t work)

Fortunately you can add a bond into a bridge, and the bond is less picky about who joins.

These are the items I added to /etc/network/interfaces

#Bond0 Config
auto bond0
iface bond0 inet manual
    bond-slaves eth0 wlan0
    bond-mode 1
    bond-miimon 100
  • bond-slaves are the link members of the bond (eth0 and wlan0 are my copper and wireless lan interfaces respectively)
  • bond-mode 1 is active-backup – Only one interface at a time will be operational, with the preference to the interface that is configured as bind-primary
  • bond-miimon 100 means that every 100ms the link state is checked
# Wireless interface
allow-hotplug wlan0
iface wlan0 inet manual
    wpa-ssid ReplaceThisWithYourSSID
    wpa-psk ReplaceThisWithYourPresharedKey
    bond-master bond0

I’m not sure if its mandatory but allow-hotplug wlan0 seems to help and bond-master seemed to be required.

The eth0 section was modified to the following

# The primary network interface
allow-hotplug eth0
iface eth0 inet manual
    bond-master bond0
    bond-primary eth0
  • Here allow-hotplug eth0 seems to wake the system to the fact a cable has been connected
  • bond-master bond0 as with wlan0, this appears to be needed
  • bond-primary eth0 means that when both eth0 and wlan0 are up, eth0 should be the one used.

And finally pnet0 was modified to use bond0 instead of eth0

auto pnet0
iface pnet0 inet dhcp
    bridge_ports bond0
    bridge_stp off

So after issuing a “service networking restart”, here’s our verifcation that the bond interface is up:

root@m4600:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: d0:67:e5:57:12:9e
Slave queue ID: 0

Slave Interface: wlan0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 24:77:03:b1:9f:78
Slave queue ID: 0
root@m4600:~# brctl show pnet0
bridge name     bridge id               STP enabled     interfaces
pnet0           8000.d067e557129e       no              bond0
root@m4600:~# ip -4 addr show pnet0
4: pnet0:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.31/24 brd 192.168.1.255 scope global pnet0
       valid_lft forever preferred_lft forever

Quick Network Verification:

root@m4600:~# ping 8.8.8.8 -c 3
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=30.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=24.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=28.3 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 24.924/27.798/30.131/2.164 ms

Summarise the bond status:

root@m4600:~# grep -A 1 "Interface\|Primary" /proc/net/bonding/bond0
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: wlan0
--
Slave Interface: eth0
MII Status: down
--
Slave Interface: wlan0
MII Status: up

Now Pull out the Ethernet cable

root@m4600:~# ping 8.8.8.8 -c 3
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=28.0 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=35.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=27.9 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 27.910/30.537/35.614/3.593 ms

Verify the bond interface is using wlan0

root@m4600:~# grep -A 1 "Interface\|Primary" /proc/net/bonding/bond0
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: wlan0
--
Slave Interface: eth0
MII Status: down
--
Slave Interface: wlan0
MII Status: up

Re Insert the Ethernet cable

root@m4600:~# ping 8.8.8.8 -c 3
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=23.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=21.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=23.1 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 21.968/22.822/23.397/0.628 ms
root@m4600:~# grep -A 1 "Interface\|Primary" /proc/net/bonding/bond0
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: eth0
--
Slave Interface: eth0
MII Status: up
--
Slave Interface: wlan0
MII Status: up

So this is all good. (Actually during this testing, I was SSHed into the laptop and the session didn’t break)

GNS3 with ExaBGP Part 2 – Full Internet Routing Table

Following on the the initial concept for using ExaBGP in part 1 we will kick it up a notch and inject a replica BGP full routing table feed. Thankfully the people at RIPE NCC have something we can use available through their Routing Information Service Raw Data page.

For my example I’m going to pull data from rrc00 which is based in Amsterdam and has a good number of peers. The data archive for rrc00 is located at http://data.ris.ripe.net/rrc00/ and the most straight forward file to download is the latest-bview.gz file.

A note about these bview files – These archives contain BGP data encoded in Multi-Threaded Routing Toolkit (MRT) Routing Information Export Format, which ExaBGP on its own cannot digest. Thankfully this is addressed by a python based MRT Parsing tool which includes a script to generate ExaBGP compatible configs (mrt2exabgp.py).

I recommend installing via a clone the git repositorary as you get the freshest version of the tool.

adam@gns3:~$ sudo -H pip install https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip
Collecting https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip
  Downloading https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip (52kB)
    100% |████████████████████████████████| 61kB 122kB/s
Installing collected packages: mrtparse
  Running setup.py install for mrtparse ... done
Successfully installed mrtparse-1.4

adam@gns3:~$ wget http://data.ris.ripe.net/rrc00/latest-bview.gz

After downloading the 50MB+ file, we can use the mrt2exabgp.py script – details on its operation are here however for my purposes I will just use the -G and -P flags and redirect the output to exabgptable.py which will end becoming a python script that is called fullbgptable.py

My Linux box is a long in the tooth AMD Athlon II X4 630 based system with 16GB RAM, so not precisely a speed demon, and doing this conversion is going to take awhile, so I’m using the time command to see how long the process will actually take, while I grab a coffee.

adam@gns3:~$ time /usr/local/lib/python2.7/dist-packages/mrtparse/examples/mrt2exabgp.py -G -P latest-bview.gz  > fullbgptable.py

real    24m25.654s
user    24m19.396s
sys     0m0.612s
adam@gns3:~$ ls -la fullbgptable.py
-rw-rw-r-- 1 adam adam 35282115 Nov  8 23:54 fullbgptable.py

Okay… so that took nearly half an hour, lets have a quick look at this.

adam@gns3:~$ head fullbgptable.py
#!/usr/bin/env python

import sys
import time

msgs = [
'announce attributes origin IGP as-path [29608 3356 29396 29396 29396 39686 44953 ] med 13 community [3356:2 3356:22 3356:100 3356:123 3356:503 3356:2067 29608:30600] next-hop 79.143.241.12 nlri 93.95.248.0/21',
'announce attributes origin IGP as-path [29608 3356 6453 9498 58682 24389 ] med 13 community [3356:2 3356:22 3356:86 3356:502 3356:666 3356:2066 6453:2000 6453:2200 6453:2204 29608:30600] next-hop 79.143.241.12 nlri 202.56.5.0/24 202.56.6.0/23 202.56.7.0/24',
'announce attributes origin IGP as-path [29608 3356 25795 202773 ] med 13 community [3356:3 3356:22 3356:100 3356:123 3356:575 3356:2003 25795:100 25795:40000 29608:30600] next-hop 79.143.241.12 nlri 185.152.130.0/24 185.152.131.0/24',
'announce attributes origin IGP as-path [29608 6939 31027 44869 203646 ] med 11 community [29608:40090] next-hop 2a01:678::2 nlri 2a03:9aa0::/32',
adam@gns3:~$ tail fullbgptable.py
    msg = msgs.pop(0)
    if isinstance(msg, str):
        sys.stdout.write(msg + '\n')
        sys.stdout.flush()
    else:
        time.sleep(msg)

while True:
    time.sleep(1)

It appears good, so lets create the ini file for exabgp to use.

group SR4 {
    router-id 1.2.3.3;
    neighbor 1.2.3.4 {
        local-address 1.2.3.3;
        local-as 64512;
        peer-as 1234;
    }
    process fullbgp {
        run /usr/bin/python /home/adam/fullbgptable.py;
    }
}

Time to fire up SR4 and start pushing a whole lot of IPv4 routes to it (you may notice in the fullbgptable.py extract above an IPv6 prefix, however since we are only using the IPv4 address family, this will be ignored)

While ExaBGP is still running, lets see how many bgp routes we have received.

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512   21732    0 00h00m32s 21726/0/0 (IPv4)
                            7    0
-------------------------------------------------------------------------------

In a little over 30 seconds we have recieved 21726 routes but none of them are active, lets have a look at what we have been offered.

A:SR4# show router bgp routes
===============================================================================
 BGP Router ID:10.10.10.4       AS:1234        Local AS:1234
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path
-------------------------------------------------------------------------------
i     1.0.4.0/24                                         None        13
      79.143.241.12                                      None        -
      29608 3356 4637 1221 38803 56203
i     1.0.5.0/24                                         None        13
      79.143.241.12                                      None        -
      29608 3356 4637 1221 38803 56203
i     1.0.6.0/24                                         None        13
A:SR4#  show router bgp routes 1.0.7.0/24 detail | match expression "Flags|Next"
Nexthop        : 79.143.241.12
Res. Nexthop   : Unresolved
Flags          : Invalid  IGP  Nexthop-Unresolved
Nexthop        : 79.143.241.12
Res. Nexthop   : Unresolved
Flags          : Invalid  IGP  Nexthop-Unresolved

The IP Nexthop is 79.143.241.12 but since SR4 doesn’t know how to get there, the route is invald.

Lets have a look at how many IP Nexthops are in fullbgptable.py

adam@gns3:~$ grep -Eo 'next-hop [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' fulltable.py | cut -f 2 | sort | uniq -c
   1177 12.0.1.63
     79 146.228.1.3
   1330 176.12.110.8
     58 178.255.145.243
    681 193.0.0.56
    849 193.150.22.1
    153 193.160.39.1
    847 202.12.28.1
   1876 203.119.76.5
    112 203.123.48.6
    543 208.51.134.248
    290 212.25.27.44
    110 213.200.87.254
    157 45.61.0.85
 102315 79.143.241.12

So we can see that there are a number of nexthops with the vast majority associated with 79.143.241.12 (the counts are not precisely the number of individual routes due to the way the fullbgptable.py has consolidated similar prefixes for performance purposes). To support installing these routes into the routing table on SR4 rather than modifying fullbgptable.py to use an alternate nexthop (say 1.2.3.3 which is used by ExaBGP), a quick and dirty workaround is to use a static blackhole route for 79.143.241.12/32 (on IOS this would be the equivalent of a null route)

A:SR4# configure router static-route 79.143.241.12/32 black-hole

And restart ExaBGP (If we really wanted to we could add blackhole routes for the other nexthops, however 79.143.241.12 covers the vast majority of routes).

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512   17966    0 00h00m32s 17950/16879/4204 (IPv4)
                          849    0
-------------------------------------------------------------------------------
A:SR4# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.0.7.0/24                                    Blackh* BGP       00h01m12s  170
       Black Hole                                                   0
1.0.128.0/17                                  Blackh* BGP       00h00m14s  170
       Black Hole                                                   0
1.1.20.0/24                                   Blackh* BGP       00h00m58s  170
       Black Hole                                                   0
1.1.128.0/17                                  Blackh* BGP       00h00m14s  170
       Black Hole                                                   0
1.2.3.0/24                                    Local   Local     00h22m49s  0
       ExaBGP                                                       0
1.2.4.0/24                                    Blackh* BGP       00h00m35s  170
       Black Hole                                                   0
1.2.11.0/24                                   Blackh* BGP       00h00m50s  170
       Black Hole                                                   0
1.2.128.0/17                                  Blackh* BGP       00h00m14s  170

This looks pretty good now (however it looks like picking 1.2.3.0/24 for my link addressing wasn’t the best choice and I should have used something from RFC1918)

While ExaBGP is pushing routes to SR4, lets check the status:

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  347185    0 00h03m00s 347165/318512/269061 (IPv4)
                        44567    0
-------------------------------------------------------------------------------
A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  405376    0 00h03m31s 405354/371121/325132 (IPv4)
                        54221    0
-------------------------------------------------------------------------------
A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  483575    0 00h04m13s 483552/442325/431845 (IPv4)
                        72792    1
-------------------------------------------------------------------------------
A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512      23    0 00h00m10s Disabled
                           10    0
-------------------------------------------------------------------------------

Okay, this isn’t a good thing, the state is showing up as disabled and there are no longer any prefix counts – something serious must have happened. On SROS routers, log 99 is a good first place to investigate potential problems.

A:SR4# show log log-id 99

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
warning: 1 events dropped from log
Memory Log contents  [size=500   next event=43  (not wrapped)]

42 2016/11/08 14:29:56.40 UTC WARNING: BGP #2012 Base Peer 1: 1.2.3.3
"Peer 1: 1.2.3.3: Closing connection: VR 1: Group ExaBGP: Peer 1.2.3.3 not enabled or not in configuration"

41 2016/11/08 14:29:53.09 UTC CRITICAL: BGP #2015 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: out of memory - disabled the peer"

40 2016/11/08 14:29:53.08 UTC WARNING: BGP #2005 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: sending notification: code CEASE subcode OUT_OF_RESR"

39 2016/11/08 14:29:53.08 UTC WARNING: BGP #2002 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: moved from higher state ESTABLISHED to lower state IDLE due to event OUT_OF_MEMORY"

38 2016/11/08 14:25:10.64 UTC MINOR: BGP #2001 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: moved into established state"

We can see that Event 38 was when the BGP session came up, however event 39 brought the session down because we ran out of memory.

In GNS3 SR4 had been allocated 2GB of memory, obviously this is not enough if we expect to take a full BGP feed – lets bump the memory to 3GB and see if that fixes it.

Up SR4's Memory from 2GB to 3GB
Up SR4’s Memory from 2GB to 3GB

Restart SR4 and kick off ExaBGP again…

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  647537    0 00h06m03s 647522/595050/595050 (IPv4)
                       102359    0
-------------------------------------------------------------------------------

Well, the session is alive for longer than previous, and the ExaBGP session console output has gone quiet, so it looks like it has transmitted everything. Lets wait a bit more just to be certain.

A:SR4# show router bgp summary | match Summary post-lines 100
A:SR4# A:SR4>show>router>bgp# show router bgp summary
===============================================================================
 BGP Router ID:10.10.10.4       AS:1234        Local AS:1234
===============================================================================
BGP Admin State         : Up          BGP Oper State              : Up
Total Peer Groups       : 1           Total Peers                 : 1
Total BGP Paths         : 110581      Total Path Memory           : 26928008
Total IPv4 Remote Rts   : 647522      Total IPv4 Rem. Active Rts  : 595050
Total McIPv4 Remote Rts : 0           Total McIPv4 Rem. Active Rts: 0
Total McIPv6 Remote Rts : 0           Total McIPv6 Rem. Active Rts: 0
Total IPv6 Remote Rts   : 0           Total IPv6 Rem. Active Rts  : 0
Total IPv4 Backup Rts   : 0           Total IPv6 Backup Rts       : 0

Total Supressed Rts     : 0           Total Hist. Rts             : 0
Total Decay Rts         : 0

Total VPN Peer Groups   : 0           Total VPN Peers             : 0
Total VPN Local Rts     : 0
Total VPN-IPv4 Rem. Rts : 0           Total VPN-IPv4 Rem. Act. Rts: 0
Total VPN-IPv6 Rem. Rts : 0           Total VPN-IPv6 Rem. Act. Rts: 0
Total VPN-IPv4 Bkup Rts : 0           Total VPN-IPv6 Bkup Rts     : 0

Total VPN Supp. Rts     : 0           Total VPN Hist. Rts         : 0
Total VPN Decay Rts     : 0

Total L2-VPN Rem. Rts   : 0           Total L2VPN Rem. Act. Rts   : 0
Total MVPN-IPv4 Rem Rts : 0           Total MVPN-IPv4 Rem Act Rts : 0
Total MDT-SAFI Rem Rts  : 0           Total MDT-SAFI Rem Act Rts  : 0
Total MSPW Rem Rts      : 0           Total MSPW Rem Act Rts      : 0
Total RouteTgt Rem Rts  : 0           Total RouteTgt Rem Act Rts  : 0
Total McVpnIPv4 Rem Rts : 0           Total McVpnIPv4 Rem Act Rts : 0
Total MVPN-IPv6 Rem Rts : 0           Total MVPN-IPv6 Rem Act Rts : 0
Total EVPN Rem Rts      : 0           Total EVPN Rem Act Rts      : 0
Total FlowIpv4 Rem Rts  : 0           Total FlowIpv4 Rem Act Rts  : 0
Total FlowIpv6 Rem Rts  : 0           Total FlowIpv6 Rem Act Rts  : 0

===============================================================================
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  647547    0 00h10m58s 647522/595050/595050 (IPv4)
                       102369    0
-------------------------------------------------------------------------------

The session is still up and no new routes have been received.

While this post has been using SROS, this example can be applied to any BGP system, just make sure you have sufficient memory in your system as the IPv4 routing table is not small! ExaBGP is a pretty interesting tool and in this case can help make a simulated environment seem more like the real world, the only thing is that this is a static snapshot and there is ongoing churn within the internet.