Carrier Supporting Carrier with SROS

Carrier of Carriers or Carrier supporting Carrier is a hierarchical construct to allow a network provider to provide MPLS connectivity with relatively low complexity. While it is even simpler to build layer 2 circuits over a backhaul network, in some cases routed connectivity may be preferred as you have a common connection point to the far end regions without needing to concern yourself with the provisioning scale and bandwidth management of each path. While this is quite a large posting it is mainly to show the configuration and verification steps. The summary of actions to create a CsC configuration is at the end of the post if you run out of patience.

This Example will have a three router Carrier of Carriers Network Operator (SC-1, SC-2 and SC-3) providing connectivity between a customer (VPRN 200) with two regions – Region A (RA-1, RA-2 and RA-3) and Region B (RB-1, RB-2 and RB-3)

9 Router Carrier Supporting Carrier Topology

Prior to SROS version 14.0R4, a single RIB was used for both labeled and unlabeled prefixes. With 14.0R4, two RIBs were established (actually, two RIBs for IPv4 and another two for IPv6) – this demonstration will have half the routers using 12.0R6 and half running 14.0R8 to highlight the configuration differences. The routers that are running SROS 14.0R8 in this example are configured using the VSR-D model which has the Control Plane Module operating in a VM (VM-CP) independent of the Input/Output Module (VM-DP) each VM-CP and VM-DP is logically connected via Switch Fabric ports – this is transparent to the configuration and operation of the simulated routers.

The initial configurations for Region A without the CSC Uplink on RA-1 as as below. OSPF as the IGP, LDP signalled LSPs are used, RA-3 is the local VPN Route Reflector for AS65000, VPRN 123 is created on each router.

configure
    system
        name "RA-1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-2"
            address 1.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "RA-3"
            address 1.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 1.1.1.1/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-2"
                    no shutdown
                exit
                interface "RA-3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-2"
                exit
                interface "RA-3"
                exit
            exit
            no shutdown
        exit
        bgp
            split-horizon
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.1:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.1/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RA-2"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-1"
            address 1.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "RA-3"
            address 1.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 1.1.1.2/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-1"
                    no shutdown
                exit
                interface "RA-3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-1"
                exit
                interface "RA-3"
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.2:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.2/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RA-3"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-1"
            address 1.1.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "RA-2"
            address 1.2.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "system"
            address 1.1.1.3/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-1"
                    no shutdown
                exit
                interface "RA-2"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-1"
                exit
                interface "RA-2"
                exit
            exit
        exit
        bgp
            group "VPN-RRC"
                family vpn-ipv4
                cluster 1.1.1.3
                peer-as 65000
                neighbor 1.1.1.1
                    description "RA-1"
                exit
                neighbor 1.1.1.2
                    description "RA-2"
                exit
            exit
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.3:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.3/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

Similarly for Region B, the initial configurations without the CSC Uplink on RB-1 as as below. OSPF as the IGP, LDP signalled LSPs, RB-3 is the local VPN Route Reflector for AS65000, VPRN 123 is created on each router. Besides the card/mda differences on the emulated hardware, the primary difference here compared to Region A which is using an older SROS version is the LSP binding (auto-bind) syntax changes to give more control as to the LSP types that may be desired.

configure
    system
        name "RB-1"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-2"
            address 2.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "RB-3"
            address 2.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 2.2.2.1/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-2"
                    no shutdown
                exit
                interface "RB-3"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-2" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-3" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            split-horizon
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.1:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.1/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RB-2"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-1"
            address 2.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "RB-3"
            address 2.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 2.2.2.2/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-1"
                    no shutdown
                exit
                interface "RB-3"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-1" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-3" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.2:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.2/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RB-3"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-1"
            address 2.1.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "RB-2"
            address 2.2.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "system"
            address 2.2.2.3/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-1"
                    no shutdown
                exit
                interface "RB-2"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-1" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-2" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RRC"
                family vpn-ipv4
                cluster 2.2.2.3
                peer-as 65000
                neighbor 2.2.2.1
                    description "RB-1"
                exit
                neighbor 2.2.2.2
                    description "RB-2"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.3:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.3/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

The CSC Operators Network uses IS-IS as the IGP, RSVP-TE signalled LSPs, SC-R3 is the local VPN Route Reflector for AS64512.

configure
    system
        name "SC-R1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R2"
            address 10.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R3"
            address 10.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.1/32
            no shutdown
        exit
        autonomous-system 64512
        isis
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R2"
                to 10.10.10.2
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R3"
                to 10.10.10.3
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                peer-as 64512
                neighbor 10.10.10.3
                    description "SC-R3"
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "SC-R2"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R3"
            address 10.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.2/32
            no shutdown
        exit
        autonomous-system 64512
        isis 0
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R3"
                to 10.10.10.3
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R1"
                to 10.10.10.1
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                peer-as 64512
                neighbor 10.10.10.3
                    description "SC-R3"
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "SC-R3"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R2"
            address 10.2.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.3/32
            no shutdown
        exit
        autonomous-system 64512
        isis
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R1"
                to 10.10.10.1
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R2"
                to 10.10.10.2
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                cluster 10.10.10.3
                peer-as 64512
                neighbor 10.10.10.1
                    description "SC-R1"
                exit
                neighbor 10.10.10.2
                    description "SC-R2"
                exit
            exit
            no shutdown
        exit
    exit
exit all

At this stage, we have three Islands – Region A, Region B and the Carrier of Carriers Network (without the CSC configuration)

RA-1 is the Region A CSC-CE Router (which is also operating as a regular PE Router) which peers with the CSC Network SC-R1 to exchange labeled routes using BGP. Note: Both of these routers are running SROS 12.0R6 and are configured in the “old way”

Create VPRN 200 on SC-R1 and BGP Peering for RA-1:

configure
    router
        policy-options
            begin
            policy-statement "PS_MPBGP_VPN_TO_BGP"
                entry 10
                    from
                        protocol bgp-vpn
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
    exit
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    service
        vprn 200 customer 1 create
            carrier-carrier-vpn
            autonomous-system 64512
            route-distinguisher 10.10.10.1:200
            auto-bind rsvp-te
            vrf-target target:64512:200
            network-interface "RA-1" create
                address 10.1.1.11/24
                port 1/1/3
                no shutdown
            exit
            bgp
                split-horizon
                group "RegionA"
                    family ipv4
                    as-override
                    export "PS_MPBGP_VPN_TO_BGP"
                    peer-as 65000
                    neighbor 10.1.1.1
                        advertise-label ipv4
                    exit
                exit
                no shutdown
            exit
            no shutdown
        exit
    exit
exit all

Of note is the definition of the VPRN type: carrier-carrier-vpn, once enabled regular IP interfaces cannot be created, only “network-interfaces” which must be on Ethernet ports that are of mode network. As both Region A and Region B both use AS 65000, as-override has been used to stop AS-Path loops occuring when advertising prefixes from other regions with the same BGP AS. We are using the IPv4 Address family but are specifically advertising associated MPLS Labels (This particular configuration component is different after SROS Release 14.0R4) Routing policy is defined and used to export VPN routes to RA-1.

Create the BGP Peering for RA-1 with SC-R1:

configure
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.1.1/24
            port 1/1/3
            no shutdown
        exit
        policy-options
            begin
            prefix-list "RegionA"
                prefix 1.1.1.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_BGP_EXP_CSC"
                entry 10
                    from
                        prefix-list "RegionA"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        bgp
            group "CSC"
                export "PS_BGP_EXP_CSC"
                neighbor 10.1.1.11
                    description "SC-R1"
                    peer-as 64512
                    advertise-label ipv4 include-ldp-prefix
                exit
            exit
        exit
    exit all

Routing policy is used to advertise any system addresses from Region A (1.1.1.x/32) with MPLS labels to the CSC

RB-1 is the Region B CSC-CE Router (which is also operating as a regular PE Router) which peers with the CSC Network SC-R2 to exchange labeled routes using BGP. Note: Both of these routers are running SROS 14.0R8 and are configured in the “new way”

Create VPRN 200 on SC-R2 and BGP Peering for RB-1:

configure
    router
        policy-options
            begin
            policy-statement "PS_MPBGP_VPN_TO_BGP"
                entry 10
                    from
                        protocol bgp-vpn
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
    exit
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    service
        vprn 200 customer 1 create
            carrier-carrier-vpn
            autonomous-system 64512
            route-distinguisher 10.10.10.2:200
            auto-bind-tunnel
                resolution-filter
                    rsvp
                exit
                resolution filter
            exit
            vrf-target target:64512:200
            network-interface "RB-1" create
                address 10.2.2.11/24
                port 1/1/3
                no shutdown
            exit
            bgp
                split-horizon
                group "RB"
                    family label-ipv4
                    as-override
                    export "PS_MPBGP_VPN_TO_BGP"
                    peer-as 65000
                    neighbor 10.2.2.1
                    exit
                exit
                no shutdown
            exit
            no shutdown
        exit
    exit
exit all

The main difference compared with the configuration on SC-1 besides the auto-bind-tunnel changes, are that we are specifically peering using the label-ipv4 address family instead of the ipv4 address family and advertising labels.

Create the BGP Peering for RB-1 with SC-R2:

configure
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    router
        interface "SC-R2"
            address 10.2.2.1/24
            port 1/1/3
            no shutdown
        exit
        policy-options
            begin
            prefix-list "RegionB"
                prefix 2.2.2.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_BGP_EXP_CSC"
                entry 10
                    from
                        prefix-list "RegionB"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        bgp
            group "CSC"
                export "PS_BGP_EXP_CSC"
                neighbor 10.2.2.11
                    description "SC-R2"
                    family label-ipv4
                    peer-as 64512
                    advertise-ldp-prefix
                exit
            exit
        exit
    exit all

Routing policy is used to advertise any system addresses from Region B (2.2.2.x/32) with MPLS labels to the CSC

Once all peers are configured, examine the CSC VPRN 200 Routing Tables:

A:SC-R1#  show router 200 route-table

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
1.1.1.2/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
1.1.1.3/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
2.2.2.1/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
2.2.2.2/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
2.2.2.3/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
10.1.1.0/24                                   Local   Local     00h08m29s  0
       RA-1                                                         0
10.2.2.0/24                                   Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
-------------------------------------------------------------------------------
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

A:SC-R2# show router 200 route-table

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
1.1.1.2/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
1.1.1.3/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
2.2.2.1/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
2.2.2.2/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
2.2.2.3/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
10.1.1.0/24                                   Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
10.2.2.0/24                                   Local   Local     02h11m42s  0
       RB-1                                                         0
-------------------------------------------------------------------------------
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

With Region A and Region B connectivity via the CSC we should be able to see the initial list of tunnels on the CSC-CE routers (RA-1 and RA-2):

A:RA-1# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.2/32            ldp   MPLS   -        9        1.1.2.2        100
1.1.1.3/32            ldp   MPLS   -        9        1.1.3.3        100
2.2.2.1/32            bgp   MPLS   -        10       10.1.1.11      1000
2.2.2.2/32            bgp   MPLS   -        10       10.1.1.11      1000
2.2.2.3/32            bgp   MPLS   -        10       10.1.1.11      1000
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-1# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        bgp       MPLS  262301    12       10.2.2.11      1000
1.1.1.2/32        bgp       MPLS  262303    12       10.2.2.11      1000
1.1.1.3/32        bgp       MPLS  262304    12       10.2.2.11      1000
2.2.2.2/32        ldp       MPLS  65540     9        2.1.2.2        100
2.2.2.3/32        ldp       MPLS  65541     9        2.1.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-2# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.2.1        100
1.1.1.3/32            ldp   MPLS   -        9        1.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-3# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.3.1        100
1.1.1.2/32            ldp   MPLS   -        9        1.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
2.2.2.1/32        ldp       MPLS  65596     9        2.1.2.1        100
2.2.2.3/32        ldp       MPLS  65544     9        2.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-3# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
2.2.2.1/32        ldp       MPLS  65597     9        2.1.3.1        100
2.2.2.2/32        ldp       MPLS  65537     9        2.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

RA-2, RA-3, RB-2 and RB-3 need the CSC-CEs to to redistribute the prefixes we learnt from the CSC into OSPF and LDP.
configure
    router
        policy-options
            begin
            prefix-list "RegionB"
                prefix 2.2.2.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_OSPF_LDP_EXP_RB"
                entry 10
                    from
                        prefix-list "RegionB"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        ospf
            asbr
            export "PS_OSPF_LDP_EXP_RB"
        exit
        ldp
            export-tunnel-table "PS_OSPF_LDP_EXP_RB"
        exit
    exit
exit all

configure
    router
        policy-options
            begin
            prefix-list "RegionA"
                prefix 1.1.1.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_OSPF_LDP_EXP_RA"
                entry 10
                    from
                        prefix-list "RegionA"
                    exit
                    action accept
                    exit
                exit
                default-action drop
                exit
            exit
            commit
        exit
        ospf
            asbr
            export "PS_OSPF_LDP_EXP_RA"
        exit
        ldp
            export-tunnel-table "PS_OSPF_LDP_EXP_RA"
        exit
Remember: SROS specifically requires OSPF in the GRT to be defined as an ASBR if you intend to export routes.
A:RA-2# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.2.1        100
1.1.1.3/32            ldp   MPLS   -        9        1.2.3.3        100
2.2.2.1/32            ldp   MPLS   -        9        1.1.2.1        1
2.2.2.2/32            ldp   MPLS   -        9        1.1.2.1        1
2.2.2.3/32            ldp   MPLS   -        9        1.1.2.1        1
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-3# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.3.1        100
1.1.1.2/32            ldp   MPLS   -        9        1.2.3.2        100
2.2.2.1/32            ldp   MPLS   -        9        1.1.3.1        1
2.2.2.2/32            ldp   MPLS   -        9        1.1.3.1        1
2.2.2.3/32            ldp   MPLS   -        9        1.1.3.1        1
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        ldp       MPLS  65619     9        2.1.2.1        1
1.1.1.2/32        ldp       MPLS  65620     9        2.1.2.1        1
1.1.1.3/32        ldp       MPLS  65621     9        2.1.2.1        1
2.2.2.1/32        ldp       MPLS  65596     9        2.1.2.1        100
2.2.2.3/32        ldp       MPLS  65544     9        2.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-3# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        ldp       MPLS  65620     9        2.1.3.1        1
1.1.1.2/32        ldp       MPLS  65621     9        2.1.3.1        1
1.1.1.3/32        ldp       MPLS  65622     9        2.1.3.1        1
2.2.2.1/32        ldp       MPLS  65597     9        2.1.3.1        100
2.2.2.2/32        ldp       MPLS  65537     9        2.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================
Okay, we have tunnels between all routers in both Regions A and B.
The final step to get VPRN connectivity between Region A and Region B is to peer the VPN Route Reflectors (RA-3 and RB-3)
configure
        bgp
            group "VPN-RR-RegionB"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
        exit
    exit all

configure
        bgp
            group "VPN-RR-RegionA"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
        exit
    exit all

We should be able to now verify this with VPRN 123
A:RA-1# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Local   Local     00h50m13s  0
       Loop                                                         0
123.1.1.2/32                                  Remote  BGP VPN   00h44m50s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h45m49s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.1 (tunneled:BGP)                                       0
123.2.2.2/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.2 (tunneled:BGP)                                       0
123.2.2.3/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.3 (tunneled:BGP)                                       0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-1# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=0.245ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.245ms, avg = 0.245ms, max = 0.245ms, stddev = 0.000ms
A:RA-1# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=1.26ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.26ms, avg = 1.26ms, max = 1.26ms, stddev = 0.000ms
A:RA-1# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=1.37ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.37ms, avg = 1.37ms, max = 1.37ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=3.46ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.46ms, avg = 3.46ms, max = 3.46ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=3.92ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.92ms, avg = 3.92ms, max = 3.92ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=3.83ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.83ms, avg = 3.83ms, max = 3.83ms, stddev = 0.000ms

A:RA-2# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h46m12s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Local   Local     00h48m34s  0
       Loop                                                         0
123.1.1.3/32                                  Remote  BGP VPN   00h46m12s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-2# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=1.12ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.12ms, avg = 1.12ms, max = 1.12ms, stddev = 0.000ms
A:RA-2# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=0.171ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.171ms, avg = 0.171ms, max = 0.171ms, stddev = 0.000ms
A:RA-2# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=1.31ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.31ms, avg = 1.31ms, max = 1.31ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=4.12ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.12ms, avg = 4.12ms, max = 4.12ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=7.76ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 7.76ms, avg = 7.76ms, max = 7.76ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=4.53ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.53ms, avg = 4.53ms, max = 4.53ms, stddev = 0.000ms

A:RA-3# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h47m48s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h46m56s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Local   Local     00h49m05s  0
       Loop                                                         0
123.2.2.1/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-3>config>router# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=1.17ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.17ms, avg = 1.17ms, max = 1.17ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=1.07ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.07ms, avg = 1.07ms, max = 1.07ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=0.166ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.166ms, avg = 0.166ms, max = 0.166ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=6.13ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 6.13ms, avg = 6.13ms, max = 6.13ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=5.04ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.04ms, avg = 5.04ms, max = 5.04ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=4.37ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.37ms, avg = 4.37ms, max = 4.37ms, stddev = 0.000ms

A:RB-1# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.1 (tunneled:BGP)                                       0
123.1.1.2/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.2 (tunneled:BGP)                                       0
123.1.1.3/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.3 (tunneled:BGP)                                       0
123.2.2.1/32                                  Local   Local     02h35m54s  0
       Loop                                                         0
123.2.2.2/32                                  Remote  BGP VPN   02h11m21s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   02h11m21s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-1# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=4.42ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.42ms, avg = 4.42ms, max = 4.42ms, stddev = 0.000ms
A:RB-1# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=3.89ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.89ms, avg = 3.89ms, max = 3.89ms, stddev = 0.000ms
A:RB-1# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=5.97ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.97ms, avg = 5.97ms, max = 5.97ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=0.118ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.118ms, avg = 0.118ms, max = 0.118ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=1.96ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.96ms, avg = 1.96ms, max = 1.96ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=1.80ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.80ms, avg = 1.80ms, max = 1.80ms, stddev = 0.000ms

A:RB-2# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   02h11m55s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Local   Local     03h04m35s  0
       Loop                                                         0
123.2.2.3/32                                  Remote  BGP VPN   02h36m00s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-2# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=3.43ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.43ms, avg = 3.43ms, max = 3.43ms, stddev = 0.000ms
A:RB-2# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=5.63ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.63ms, avg = 5.63ms, max = 5.63ms, stddev = 0.000ms
A:RB-2# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=4.40ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.40ms, avg = 4.40ms, max = 4.40ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=1.85ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.85ms, avg = 1.85ms, max = 1.85ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=0.150ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.150ms, avg = 0.150ms, max = 0.150ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=1.90ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.90ms, avg = 1.90ms, max = 1.90ms, stddev = 0.000ms

A:RB-3# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   02h13m05s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   02h37m03s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Local   Local     02h37m07s  0
       Loop                                                         0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-3# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=3.83ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.83ms, avg = 3.83ms, max = 3.83ms, stddev = 0.000ms
A:RB-3# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=4.38ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.38ms, avg = 4.38ms, max = 4.38ms, stddev = 0.000ms
A:RB-3# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=4.47ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.47ms, avg = 4.47ms, max = 4.47ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=1.97ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.97ms, avg = 1.97ms, max = 1.97ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=1.88ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.88ms, avg = 1.88ms, max = 1.88ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=0.269ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.269ms, avg = 0.269ms, max = 0.269ms, stddev = 0.000ms
While this is quite a long post, the actual configuration itself for the CSC is straight forward:

  • The CSC VPRN must be set to carrier-carrier-vpn
  • The CSC VPRN uses Ethernet ports with mode network even though its for customers
  • The CSC-PE and CSC-CE exchange labeled routes
  • The CSC-CE redistributes between OSPF/LDP and BGP

Changes to the way the RIB works on SROS and how that impacts CsC configurations was also briefly discussed.

Using BGP Flowspec with Nokia SROS to protect the network edge

This post is about how you can use BGP to augment your network edge protection. Specifically BGP Flow spec can be used to dynamically add entries to an access control list (or ip-filter in Nokia SROS speak). Previous posts have been using ExaBGP as the method to inject traffic, and while it’s good at what it does, I’ve moved on to use GoBGP not because it seems to be quite a high performance BGP implementation (I’m just doing proof of concept work) but because GoBGP’s command line interface is more friendly and better documented than ExaBGP, making it the more versatile tool to generate route advertisements in a lab on the fly (at least for me).

BGP Flowspec Test Topology

The lab topology is designed to demonstrate filtering with flowspec – two routers (SR1 is under my control, while the External router is a gateway to unscrupulous entities that I cannot control) and GoBGP running on a computer in my Network Operations centre.

Before getting into the flowspec piece, I’ll go through the installation process for GoBGP and the Go Language for Ubuntu 16.04 (as the routers I’m using as running on eve-ng) and the computer in the example is bridged to SR1) and then how to develop the initial GoBGP configuration.

The Go Programming language is available via the standard Ubuntu repository
adam@m4600:~$ sudo apt-get install golang-go

GoBGP is pulled down as source code and compiled as part of the download process. An environmental variable needs to be set so Go knows where to pull down code, and where to build the binaries.
adam@m4600:~$ export GOPATH=$HOME/go
Tell go to pull down and build gobgpd (this takes awhile at least on my machine and you don’t get an indication on what it’s doing)
adam@m4600:~$ go get github.com/osrg/gobgp/gobgpd
The gobgp cli is a separate program that talks to gobgpd using the GoBGP API – downloading and building it is pretty quick in comparison to gobgpd:
adam@m4600:~$ go get github.com/osrg/gobgp/gobgp
Once built, we shall copy the generated binaries to the an executable path
adam@m4600:~$ sudo cp $GOPATH/bin/* /usr/local/sbin/
For those of us using the default bash shell, we can enable tab completion for the cli

adam@m4600:~$ sudo cp $GOPATH/src/github.com/osrg/gobgp/tools/completion/*.bash /etc/bash_completion.d/

Once GoBGP is installed, we can develop a configuration file that will be used to set up the GoBGP instance – in this case we’re defining the router itself and SR1 as a neighbor for the IPv4 FlowSpec address family:
A configuration file that will be used by the GoBGP Daemon enabling FlowSpec for IPv4 where we peer against SR1:

adam@m4600:~$ cat gobgp-flowspec.conf
[global.config]
  as = 64512
  router-id = "1.2.3.4"

[[neighbors]]
[neighbors.config]
  neighbor-address = "192.168.1.123"
  peer-as = 64512
[[neighbors.afi-safis]]
  [neighbors.afi-safis.config]
  afi-safi-name = "ipv4-flowspec"

The GoBGP documentation is also a bit nicer than ExaBGP too – additional information on setting up a configuration (including different formats is described in the GoBGP getting started document)

Start the GoBGP Daemon and read our new config:

adam@m4600:~$ sudo gobgpd -f gobgp-flowspec.conf
{"level":"info","msg":"gobgpd started","time":"2017-05-18T22:57:22+10:00"}
{"Topic":"Config","level":"info","msg":"Finished reading the config file","time":"2017-05-18T22:57:22+10:00"}
{"level":"info","msg":"Peer 192.168.1.123 is added","time":"2017-05-18T22:57:22+10:00"}
{"Topic":"Peer","level":"info","msg":"Add a peer configuration for:192.168.1.123","time":"2017-05-18T22:57:22+10:00"}
{"Key":"192.168.1.123","State":"BGP_FSM_OPENCONFIRM","Topic":"Peer","level":"info","msg":"Peer Up","time":"2017-05-18T22:57:40+10:00"}
We can see shortly after gobgpd started, it has already established a peering session with SR1 (192.168.1.123)

Opening up a new shell, we can use the gobgp client to verify that:

adam@m4600:~$ gobgp neighbor
Peer             AS  Up/Down State       |#Received  Accepted
192.168.1.123 64512 00:00:22 Establ      |        0         0
adam@m4600:~$ gobgp neighbor 192.168.1.123
BGP neighbor is 192.168.1.123, remote AS 64512
  BGP version 4, remote router ID 123.123.123.123
  BGP state = established, up for 00:01:53
  BGP OutQ = 0, Flops = 0
  Hold time is 90, keepalive interval is 30 seconds
  Configured hold time is 90, keepalive interval is 30 seconds
  %s
  Neighbor capabilities:
    multiprotocol:
        ipv4-flowspec:  advertised and received
    route-refresh:      advertised and received
    4-octet-as: advertised and received
  Message statistics:
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             4          5
    Route Refresh:          0          0
    Discarded:              0          0
    Total:                  5          6
  Route statistics:
    Advertised:             0
    Received:               0
    Accepted:               0

We can also get SR1 to confirm it too:
*A:SR1# show router bgp summary | match Summ post-lines 100
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
192.168.1.46
               64512        9    0 00h00m54s 0/0/0 (FlowIPv4)
                           13    0
-------------------------------------------------------------------------------

Now that the flowspec session is up and running but no one has exchanged any routes yet, I’ll park this for a moment to create a simple ip-filter (or ACL)

*A:SR1# /configure filter
*A:SR1>config>filter# info
----------------------------------------------
        match-list
            ip-prefix-list "FPL_RFC1918" create
                prefix 10.0.0.0/8
                prefix 172.16.0.0/12
                prefix 192.168.0.0/16
            exit
        exit
        ip-filter 100 create
            default-action forward
            embed-filter flowspec router "Base" offset 1000
            entry 10 create
                match
                    src-ip ip-prefix-list "FPL_RFC1918"
                exit
                action
                    drop
                exit
            exit
            entry 20 create
                match
                    dst-ip ip-prefix-list "FPL_RFC1918"
                exit
                action
                    drop
                exit
            exit
        exit
----------------------------------------------

The match-list ip-prefix-list “FPL_RFC1918” is a way to consolidate a number of network prefixes into a single named list enabling filter configurations to be smaller and more manageable (and easier to debug)

ip-filter 100 is the thing of interest. The default-action is what happens if none of the previous entries are matched – in this case it is a black-list ip-filter, since anything we don’t specifically drop will get forwarded
Each filter entry has a number which is used for the ascending evaluation order (entry 10 before entry 20 here) and usually has a match entry and an action to perform when the match was successful. This filter is relatively simple since all it is doing is blocking IP packets that either have source or destination IP addresses from RFC1918 space.

The interesting thing is the embed-filter flowspec entry. Router “Base” is the base routing instance (as opposed to a VPRN) and offset 1000 means that any flowspec entries will be dynamically added to ip-filter 100 starting at entry 1000 (and with an additional offset determined by flowspec)

This filter will be applied to the IP interface facing the external router – This interface is associated with an Internet Enhanced Service (IES) associated with the SR1’s Global Routing Table (Router “Base”)

*A:SR1>config>service>ies# /configure service ies 1
*A:SR1>config>service>ies# info
----------------------------------------------
            interface "External" create
                address 200.200.200.1/24
                sap 1/1/1 create
                exit
            exit
            no shutdown
----------------------------------------------
*A:SR1>config>service>ies# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     00h20m17s  0
       Loop_1                                                       0
0.0.0.0/0                                     Remote  Static    00h20m02s  5
       200.200.200.2                                                1
123.123.123.123/32                            Local   Local     00h20m17s  0
       system                                                       0
192.168.1.0/24                                Local   Local     00h20m02s  0
       NetOps                                                       0
200.200.200.0/24                              Local   Local     00h20m02s  0
       External                                                     0
-------------------------------------------------------------------------------
No. of Routes: 5
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available

In this contrived example SR1 just has a default route point to the External interface in IES 1.
Before we apply the IP filter – check to see we can reach an RFC1918 IP Address:
A:External# ping 1.1.1.1 source 172.16.0.1 count 2
PING 1.1.1.1 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=1.40ms.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=1.91ms.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min = 1.40ms, avg = 1.66ms, max = 1.91ms, stddev = 0.253ms

Apply the IP Filter in the ingress direction of interface External on SR1:
*A:SR1>config>service>ies# interface "External" sap 1/1/1 ingress filter ip 100
*A:SR1>config>service>ies# info
----------------------------------------------
            interface "External" create
                address 200.200.200.1/24
                sap 1/1/1 create
                    ingress
                        filter ip 100
                    exit
                exit
            exit
            no shutdown

And repeat the test:
*A:External# ping 1.1.1.1 source 172.16.0.1 count 2
PING 1.1.1.1 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 0 packets received, 100% packet loss

We can see the ping failed and if we look at the counters
*A:SR1>config>service>ies# show filter ip 100 counters

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2
Sub-Entries         : 6
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
We can see the ping was unsuccessful because of entry 10 stopping traffic coming back to SR1 (because we weren’t filtering traffic leaving SR1)

Now all of a sudden we find a ping flood (well okay, calling two packets a flood is a rather long bow to draw but anyway) is coming from 100.100.100.100 and hitting 1.1.1.1

A:External# ping 1.1.1.1 source 100.100.100.100 count 2
PING 1.1.1.1 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=1.16ms.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=1.26ms.

These ICMP packets are causing havok to 1.1.1.1, so it would be nice to add some precision filtering to stop that from occuring, while allowing other traffic through – this is where our GoBGP instance and flowspec comes back into the story. We will ask gobgp to announce a flowspec “route” that specifies a discard action (actually a rate-limit of 0 kbps) for traffic from 100.100.100.100 to 1.1.1.1 using ICMP:
adam@m4600:~$ gobgp global rib add -a ipv4-flowspec match source 100.100.100.100/32 destination 1.1.1.1/32 protocol icmp then discard
SR1 can confirm it recieved this route:
*A:SR1>config>service>ies# show router bgp routes flow-ipv4
===============================================================================
 BGP Router ID:123.123.123.123  AS:64512       Local AS:64512
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP FLOW IPV4 Routes
===============================================================================
Flag  Network             Nexthop                 LocalPref       MED
      As-Path
-------------------------------------------------------------------------------
u*>?  --                  0.0.0.0                 100             None
      No As-Path

      Community Action:  rate-limit: 0 kbps
      NLRI Subcomponents:
      Dest Pref : 1.1.1.1/32
      Src Pref  : 100.100.100.100/32
      Ip Proto  : [ == 1 ]
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Flowspec can do quite interesting things, besides rate-limiting traffic to 0kbps (dropping it), it’s possible to use flowspec to tell the router to rate limit flows rather than completely blocking, remark traffic to a different level of priority or even redirect traffic to a VRF by setting the appropriate route targets (this could for example allow the forwarding of traffic into a VPRN or VRF to reach a scrubbing network prior to re-injection of clean traffic back into the network)

Lets see if ip filter has taken this flowspec entry:

*A:SR1>config>service>ies# show filter ip 100

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2/0/0/1 (Fixed/Radius/Cc/Embedded)
Sub-Entries         : 6/0/0/1
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : ip-prefix-list "FPL_RFC1918"
Src. Port           : n/a
Dest. IP            : 0.0.0.0/0
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 0.0.0.0/0
Src. Port           : n/a
Dest. IP            : ip-prefix-list "FPL_RFC1918"
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

Entry               : 1256
Origin              : Inserted by embedded filter fSpec-0 entry 256
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 100.100.100.100/32
Src. Port           : n/a
Dest. IP            : 1.1.1.1/32
Dest. Port          : n/a
Protocol            : 1                            Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
We had specified that the offset for flowspec was 1000, this flowspec entry (number 256) became ip filter 100 entry 256. Time to verify that the ping now should fail:
*A:External# ping 1.1.1.1 source 100.100.100.100 count 2
PING 1.1.1.1 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.

---- 1.1.1.1 PING Statistics ----
2 packets transmitted, 0 packets received, 100% packet loss

Yes, and we can verify that it’s only blocking icmp between those endpoints by sending IP packets from another protocol say UDP by doing a traceroute:
*A:External# traceroute 1.1.1.1 source 100.100.100.100
traceroute to 1.1.1.1 from 100.100.100.100, 30 hops max, 40 byte packets
  1  1.1.1.1 (1.1.1.1)    1.74 ms  1.51 ms  1.52 ms
Yes, this is pretty good as far as network based filters (that aren’t performing payload based inspection) are doing, there’s quite a few match conditions that can be used to narrow things down even further. In this contrived example it may be similar effort to configure a filter on the fly but in a production environment, you could hook GoBGP to your route reflector and through a single procedure push updates to all your edge routers in one go. Another good thing about this method, is if the filtering is only temporary, it’s just as easy to remove as it was to put in.
To see what’s currently in the rib for ipv4-flowspec:
adam@m4600:~$ gobgp global rib  -a ipv4-flowspec
    Network                                                              Next Hop             AS_PATH              Age        Attrs
*>  [destination:1.1.1.1/32][source:100.100.100.100/32][protocol:==icmp ]fictitious                                00:12:06   [{Origin: ?} {Extcomms: [discard]}]
Withdraw the announcement:
adam@m4600:~$ gobgp global rib del -a ipv4-flowspec match source 100.100.100.100/32 destination 1.1.1.1/32 protocol icmp then discard
adam@m4600:~$ gobgp global rib  -a ipv4-flowspec                                Network not in table

Verify SR1 ip filter 100 no longer has the entry:
*A:SR1>config>service>ies# show filter ip 100

===============================================================================
IP Filter
===============================================================================
Filter Id           : 100                          Applied        : Yes
Scope               : Template                     Def. Action    : Forward
System filter       : Unchained
Radius Ins Pt       : n/a
CrCtl. Ins Pt       : n/a
RadSh. Ins Pt       : n/a
PccRl. Ins Pt       : n/a
Entries             : 2
Sub-Entries         : 6
Description         : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry               : 10
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : ip-prefix-list "FPL_RFC1918"
Src. Port           : n/a
Dest. IP            : 0.0.0.0/0
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 2 pkts (204 bytes)
Egr. Matches        : 0 pkts

Entry               : 20
Description         : (Not Specified)
Log Id              : n/a
Src. IP             : 0.0.0.0/0
Src. Port           : n/a
Dest. IP            : ip-prefix-list "FPL_RFC1918"
Dest. Port          : n/a
Protocol            : Undefined                    Dscp           : Undefined
ICMP Type           : Undefined                    ICMP Code      : Undefined
Fragment            : Off                          Src Route Opt  : Off
Sampling            : Off                          Int. Sampling  : On
IP-Option           : 0/0                          Multiple Option: Off
TCP-syn             : Off                          TCP-ack        : Off
Option-pres         : Off
Egress PBR          : Disabled
Primary Action      : Drop
Ing. Matches        : 0 pkts
Egr. Matches        : 0 pkts

===============================================================================
Entry 1256 has left the building.

Hopefully that was a reasonable introduction to using BGP Flowspec with filtering and GoBGP.

Flowspec is quite often a piece of the puzzle in DDoS mitigation tools, usually there is some kind of automation attached which makes decisions using inputs such a Netflow/IPFIX data, SNMP polling of interfaces to track anomalous loading and in some cases threat intelligence feeds from security vendors.

Route aggregation is not always straight forward

Today’s post is a simple 3 router topology based on a true story when route aggregation didn’t appear to working as expected at first glance and some additional thought was required as to why things were behaving that way and what was required to make it do what I wanted.

I’m using 3 x Nokia VSR-Sims running SROS 14.0R4, and while the concepts discussed here are definitely flavoured through a SROS lens, the concepts will be familiar to different router platforms and their associated operating systems.

R1 and R2 are in BGP AS 12 and use ospf to advertise their respective system addresses which are used for their IBGP peerings

configure
    system
        name "R1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    router
        interface "Loop"
            address 100.100.100.1/24
            loopback
            no shutdown
        exit
        interface "R2"
            address 10.1.2.1/27
            port 1/1/1
            no shutdown
        exit
        interface "system"
            address 1.1.1.1/32
            no shutdown
        exit
        autonomous-system 12
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "R2"        
                    no shutdown
                exit
            exit
            no shutdown
        exit
        policy-options
            begin
            prefix-list "PL_LOOP"
                prefix 100.100.100.0/24 exact
            exit
            policy-statement "PS_LOOP_EXP"
                entry 10
                    from
                        protocol direct
                        prefix-list "PL_LOOP"
                    exit
                    action accept
                    exit              
                exit                  
            exit                      
            commit                    
        exit
        bgp
            group "IBGP"
                export "PS_LOOP_EXP"
                peer-as 12
                neighbor 2.2.2.2
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R2"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "R1"
            address 10.1.2.2/27
            port 1/1/1
            no shutdown
        exit
        interface "R3"
            address 10.2.3.2/27
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 2.2.2.2/32
            no shutdown
        exit
        autonomous-system 12          
        router-id 2.2.2.2
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "R1"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
        bgp
            group "IBGP"
                peer-as 12
                neighbor 1.1.1.1
                exit
            exit
            no shutdown
        exit
    exit
exit all

R3 which is in BGP AS 3 will be peering with R2. While we can configure R2, as R3 is in a different AS, we cannot touch it nor modify its configuration. R3 is already configured to peer with R2 and is waiting for R2 to come online.

Our configuration for R2 to establish the BGP Session:

A:R2# configure router bgp 
A:R2>config>router>bgp# group EBGP 
*A:R2>config>router>bgp>group$ neighbor 10.2.3.3 peer-as 3 
*A:R2>config>router>bgp>group$ exit all 

Assuming enough time has passed for BGP to come up, lets get a quick state of play with BGP on R2:

*A:R2# show router bgp summary | match "BGP Sum" post-lines 100  
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.1.1.1
               12          11    0 00h03m30s 1/1/0 (IPv4)
                           10    0           
10.2.3.3
               3            6    0 00h00m34s 8/8/1 (IPv4)
                            5    0           
-------------------------------------------------------------------------------

Right now R2 has active BGP sessions with R1 and R3 – we can see that R2 has received 8 routes from R3 and has sent 1 (from R1). R2 hasn’t yet sent any routes learnt from R3 to R1 however this should happen shortly.

These are the BGP routes that R2 knows of

*A:R2# show router bgp routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
u*>i  3.0.0.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.1.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.2.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.3.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.4.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.5.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.6.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.7.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  100.100.100.0/24                                   100         None
      1.1.1.1                                            None        -
      No As-Path                                                      
-------------------------------------------------------------------------------
Routes : 9
===============================================================================

As this post is about route aggregation, on R2 we want to send a summary route through to R1 (3.0.0.0/21) instead of all the individual routes. To do this we will create the aggregate route and specify it to be a summary-only route and because we can, we will include the AS-Set in the aggregate so R1 knows these came from AS 3
*A:R2# configure router aggregate 3.0.0.0/21 summary-only as-set description "Aggregate from AS3"
*A:R2# show router aggregate detail 

===============================================================================
Legend: G - generate-icmp enabled
===============================================================================
Aggregate Route Table (Router: Base)
===============================================================================
Prefix           : 3.0.0.0/21
Description      : Aggregate from AS3
Summary          : True           AS Set           : True
Aggr AS          : 0              Aggr IP-Address  : 0.0.0.0
Aggr OperState   : Active         
Nexthop Type     : None           Nexthop          :    
Community        :                
-------------------------------------------------------------------------------
No. of Aggregate Routes: 1
==============================================================================="

We can see the aggregate appear in the routing table as a blackhole route from protocol aggregate
*A:R2# show router route-table 

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
1.1.1.1/32                                    Remote  OSPF      00h10m16s  10
       10.1.2.1                                                     100
2.2.2.2/32                                    Local   Local     00h11m14s  0
       system                                                       0
3.0.0.0/21                                    Blackh* Aggr      00h02m13s  130
       Black Hole                                                   0
3.0.0.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.1.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.2.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.3.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.4.0/24                                    Remote  BGP       00h05m59s  170
       10.2.3.3                                                     0
3.0.5.0/24                                    Remote  BGP       00h06m00s  170
       10.2.3.3                                                     0
3.0.6.0/24                                    Remote  BGP       00h06m00s  170
       10.2.3.3                                                     0
3.0.7.0/24                                    Remote  BGP       00h06m00s  170
       10.2.3.3                                                     0
10.1.2.0/27                                   Local   Local     00h11m00s  0
       R1                                                           0
10.2.3.0/27                                   Local   Local     00h11m00s  0
       R3                                                           0
100.100.100.0/24                              Remote  BGP       00h08m53s  170
       10.1.2.1                                                     0
-------------------------------------------------------------------------------
No. of Routes: 14
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
* indicates that the corresponding row element may have been truncated.

As we can see in the routing table, an aggregate route is treated as its own routing protocol, so we need to develop a routing policy to advertise the aggregate to R1
*A:R2# configure router policy-options 
*A:R2>config>router>policy-options# begin 
*A:R2>config>router>policy-options# policy-statement PS_AGGREGATE 
*A:R2>config>router>policy-options>policy-statement$ entry 10 from protocol aggregate 
*A:R2>config>router>policy-options>policy-statement$ entry 10 action accept 
*A:R2>config>router>policy-options>policy-statement>entry>action$ exit 
*A:R2>config>router>policy-options>policy-statement$ info 
----------------------------------------------
                entry 10
                    from
                        protocol aggregate
                    exit
                    action accept
                    exit
                exit
----------------------------------------------
*A:R2>config>router>policy-options>policy-statement$ exit 
*A:R2>config>router>policy-options# commit

We then can use the policy to export to our neighbor (using group IBGP or on the neighbor directly)
*A:R2>config>router>policy-options# /configure router bgp group "IBGP"     
*A:R2>config>router>bgp>group# export "PS_AGGREGATE"

One thing we haven’t done yet is that the EBGP next-hop 10.2.3.3 will not be visible to R1, so we can either add that interface into OSPF (as a passive interface so we don’t attempt to peer with an external router at the IGP level) or have R2 set next-hop-self (I generally prefer this as it keeps the IGP just for internal core links)
*A:R2>config>router>bgp>group# next-hop-self 
*A:R2>config>router>bgp>group# info 
----------------------------------------------
                next-hop-self
                export "PS_AGGREGATE" 
                peer-as 12
                neighbor 1.1.1.1
                exit
----------------------------------------------

Okay, so now R1 should have 3.0.0.0/21 and the job is done, so lets verify this is working on R1
A:R1# show router route-table 

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     00h18m48s  0
       system                                                       0
2.2.2.2/32                                    Remote  OSPF      00h17m57s  10
       10.1.2.2                                                     100
10.1.2.0/27                                   Local   Local     00h18m32s  0
       R2                                                           0
100.100.100.0/24                              Local   Local     00h18m48s  0
       Loop                                                         0
-------------------------------------------------------------------------------
No. of Routes: 4
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

3.0.0.0/21 is not present, so something is wrong here. What is R2 sending to R1?
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
No Matching Entries Found
===============================================================================

Nothing – let’s double check our policy
*A:R2>config>router>bgp>group# show router policy "PS_AGGREGATE" 
    entry 10
        from
            protocol aggregate
        exit
        action accept
        exit
    exit
*A:R2>config>router>bgp>group# show router aggregate 

===============================================================================
Legend: G - generate-icmp enabled
===============================================================================
Aggregates (Router: Base)
===============================================================================
Prefix                                          Aggr IP-Address   Aggr AS
   Summary                                         AS Set          State
     NextHop                                         Community     NextHopType
-------------------------------------------------------------------------------
3.0.0.0/21                                      0.0.0.0           0
   True                                            True            Active
                                                                      None
-------------------------------------------------------------------------------
No. of Aggregates: 1
===============================================================================

Well that looks okay but maybe the aggregate route is wrong
*A:R2>config>router>bgp>group# /admin display-config | match expression "^\ +agg"
        aggregate 3.0.0.0/21 summary-only as-set description "Aggregate from AS3"

Lets try it without including the summary-only option and see if the contributing routes will get advertised to R1.
*A:R2>config>router>bgp>group# /configure router aggregate 3.0.0.0/21 as-set description "Agg AS3 no summary-only"
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
i     3.0.0.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.1.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.3.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.4.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.5.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.6.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.7.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
-------------------------------------------------------------------------------
Routes : 7
===============================================================================

We are only sending 7 routes but we received 8 from R3!
*A:R2>config>router>bgp>group# show router bgp neighbor 10.2.3.3 received-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
u*>i  3.0.0.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.1.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.2.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.3.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.4.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.5.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.6.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.7.0/24                                         n/a         None
      10.2.3.3                                           None        -
      3                                                               
-------------------------------------------------------------------------------
Routes : 8
===============================================================================

So what is it about 3.0.2.0/24?
*A:R2>config>router>bgp>group# show router bgp routes 3.0.2.0/24 detail 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Original Attributes
 
Network        : 3.0.2.0/24
Nexthop        : 10.2.3.3
Path Id        : None                   
From           : 10.2.3.3
Res. Nexthop   : 10.2.3.3
Local Pref.    : n/a                    Interface Name : R3
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : None
AIGP Metric    : None                   
Connector      : None                 
Community      : no-advertise
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 3.3.3.3
Fwd Class      : None                   Priority       : None
Flags          : Used  Valid  Best  IGP  
Route Source   : External
AS-Path        : 3 
Route Tag      : 0                      
Neighbor-AS    : 3
Orig Validation: NotFound               
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default                
Last Modified  : 03h55m28s              
 
Modified Attributes
 
Network        : 3.0.2.0/24
Nexthop        : 10.2.3.3
Path Id        : None                   
From           : 10.2.3.3
Res. Nexthop   : 10.2.3.3
Local Pref.    : None                   Interface Name : R3
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : None
AIGP Metric    : None                   
Connector      : None
Community      : no-advertise
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 3.3.3.3
Fwd Class      : None                   Priority       : None
Flags          : Used  Valid  Best  IGP  
Route Source   : External
AS-Path        : 3 
Route Tag      : 0                      
Neighbor-AS    : 3
Orig Validation: NotFound               
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default                
Last Modified  : 03h55m30s              
 
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

3.0.2.0/24 has no-advertise attached to it!
One of the things about aggregate routes is that they aggregate the associated communities as well, so the aggregate route will have no-advertise attached to it, so it will not be advertised to R1.
Unfortunately this doesn’t appear in the show “show router aggregate detail” – the community there is the one that is manually added during the creation of the aggregate.
So how can we fix this? Well there are two methods that spring to mind and I am sure that there are more.

Option 1 – Create an import policy on R2 that just drops the no-advertise community on imported routes.
I think this is the easiest option to implement because then the normal aggregate configuration will work.

*A:R2>config>router>bgp>group# /configure router policy-options 
*A:R2>config>router>policy-options# begin 
*A:R2>config>router>policy-options# community NO_ADV members no-advertise 
*A:R2>config>router>policy-options# policy-statement PS_IGNORE_NO_ADV 
*A:R2>config>router>policy-options>policy-statement$ entry 10
*A:R2>config>router>policy-options>policy-statement>entry$ from community NO_ADV
*A:R2>config>router>policy-options>policy-statement>entry$ action accept 
*A:R2>config>router>policy-options>policy-statement>entry>action$ community remove "NO_ADV" 
*A:R2>config>router>policy-options>policy-statement>entry>action$ back 
*A:R2>config>router>policy-options>policy-statement>entry$ back 
*A:R2>config>router>policy-options>policy-statement$ info 
----------------------------------------------
                entry 10
                    from
                        community "NO_ADV"
                    exit
                    action accept
                        community remove "NO_ADV"
                    exit
                exit
----------------------------------------------
*A:R2>config>router>policy-options>policy-statement$ back 
*A:R2>config>router>policy-options# commit 
*A:R2>config>router>policy-options# /configure router bgp group "EBGP" 
*A:R2>config>router>bgp>group# neighbor 10.2.3.3 import "PS_IGNORE_NO_ADV"

Let’s see if that has resolved things:
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
i     3.0.0.0/21                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.0.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.1.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.2.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.3.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.4.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.5.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.6.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
i     3.0.7.0/24                                         100         None
      2.2.2.2                                            None        -
      3                                                               
-------------------------------------------------------------------------------
Routes : 9
===============================================================================

Yes, 3.0.2.0/24 is present and because none of the routes that contribute to the aggregate have no-advertise attached, the aggregate is also advertised to R1. So time to change the Aggregate route so it’s back to summary only:
*A:R2>config>router>bgp>group# /configure router aggregate 3.0.0.0/21 summary-only as-set description "Aggregate from AS3"
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes                                        
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
i     3.0.0.0/21                                         100         None
      2.2.2.2                                            None        -
      3                                                               
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Okay sorted.

Option 2 – Using Static Routes for Aggregation
If we wish to respect the no-advertise binding on 3.0.2.0/24, we can simulate some of the behavior of an aggregate route without caring about no-advertise (or no-export if we are concerned about advertisements outside of our AS).
First we need to remove the import policy on R2 facing R3.

*A:R2>config>router>bgp>group# info 
----------------------------------------------
                neighbor 10.2.3.3
                    import "PS_IGNORE_NO_ADV" 
                    peer-as 3
                exit
----------------------------------------------
*A:R2>config>router>bgp>group# neighbor 10.2.3.3 no import

And remove the aggregate for 3.0.0.0/21
*A:R2>config>router>bgp>group# /configure router no aggregate 3.0.0.0/21
Now we create a static black-hole route with BGP community 12:12 attached to it. We’re attaching the community so we can distinguish between regular static routes and our “aggregate”
A:R2>config>router# static-route-entry 3.0.0.0/21 
*A:R2>config>router>static-route-entry$ black-hole 
*A:R2>config>router>static-route-entry>black-hole$ community 12:12 
*A:R2>config>router>static-route-entry>black-hole$ no shutdown

As a note, SROS Release 14 changed the specific syntax for creating static routes but the concepts generally remain the same for previous SROS versions.
Now we’ll work on the routing policy to advertise our static aggregate route.
First we’ll create a named community that was used for our aggregate:
*A:R2>config>router>static-route-entry$ /configure router policy-options 
*A:R2>config>router>policy-options# begin
*A:R2>config>router>policy-options# community STATIC_AGG members 12:12 

Now we create a prefix list to match the routes that contribute to our aggregate
*A:R2>config>router>policy-options# prefix-list PL_R3_CONTRIB 
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.0.0/21 longer
*A:R2>config>router>policy-options>prefix-list$ exit

Finally we take the existing PS_AGGREGATE and modify it to work with our static aggregate and drop the contributing routes:
*A:R2>config>router>policy-options# policy-statement "PS_AGGREGATE" 
*A:R2>config>router>policy-options>policy-statement# info 
----------------------------------------------
                entry 10
                    from
                        protocol aggregate
                    exit
                    action accept
                    exit
                exit
*A:R2>config>router>policy-options>policy-statement# entry 10 
*A:R2>config>router>policy-options>policy-statement>entry# from protocol static 
*A:R2>config>router>policy-options>policy-statement>entry# from community "STATIC_AGG"
*A:R2>config>router>policy-options>policy-statement>entry# back 
*A:R2>config>router>policy-options>policy-statement# entry 20 
*A:R2>config>router>policy-options>policy-statement>entry$ from prefix-list "PL_R3_CONTRIB" 
*A:R2>config>router>policy-options>policy-statement>entry$ action drop 
*A:R2>config>router>policy-options>policy-statement>entry>action$ exit 
*A:R2>config>router>policy-options>policy-statement>entry$ exit 
*A:R2>config>router>policy-options>policy-statement# info 
----------------------------------------------
                entry 10
                    from
                        protocol static
                        community "STATIC_AGG"
                    exit
                    action accept
                    exit
                exit
                entry 20
                    from
                        prefix-list "PL_R3_CONTRIB"
                    exit
                    action drop
                    exit
                exit
*A:R2>config>router>policy-options>policy-statement# back 
*A:R2>config>router>policy-options# commit

Lets check what R2 is advertising to R1:
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
?     3.0.0.0/21                                         100         None
      2.2.2.2                                            None        -
      No As-Path                                                      
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

There are a couple of issues with this implementation the AS-Path information is lost (nothing we can do about that) but more importantly, this “aggregate” will stay up even if the contributing routes are not present. To overcome this issue, the static route can be associated with a prefix-list which will be used to determine if the static route can become active. It should be noted that although we already have PL_R3_CONTRIB, it cannot be used here as the prefix list in the static route requires specific prefixes to match against. While matching all possible prefixes could be problematic, in most instances simply matching against a few key prefixes will be sufficient:
*A:R2>config>router>bgp>group# /configure router policy-options 
*A:R2>config>router>policy-options# begin
*A:R2>config>router>policy-options# prefix-list PL_R3_STATIC_AGG_OK  
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.0.0/24 
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.7.0/24 
*A:R2>config>router>policy-options>prefix-list$ exit 
*A:R2>config>router>policy-options# commit 

Modify the static route to be up when any of the prefixes in PL_R3_STATIC_AGG_OK are in the routing table:
*A:R2>config>router>policy-options# /configure router static-route-entry 3.0.0.0/21 black-hole prefix-list "PL_R3_STATIC_AGG_OK" any

We can see the route is active and the prefix-list being used to validate:
*A:R2>config>router>policy-options# show router static-route detail 

===============================================================================
Static Route Table (Router: Base)  Family: IPv4
===============================================================================
Prefix           : 3.0.0.0/21
Nexthop          : n/a
Type             : Blackhole
Dynamic BGP      : disabled                        Generate ICMP     : disabled
Interface        : n/a                             Active            : Y
Prefix List      : PL_R3_STATIC_AGG_OK             Prefix List Type  : Any
Metric           : 1                               Preference        : 5
Source Class     : 0                               Dest Class        : 0
Admin  State     : Up                              Tag               : 0
Creation Origin  : manual                          
BFD              : disabled
Community        : 12:12                           
CPE-check        : disabled
-------------------------------------------------------------------------------
No. of Static Routes: 1

===============================================================================

If we shutdown our BGP session to R3, the routes in PL_R3_STATIC_AGG_OK will disappear from the routing table and the static route will be brought out of service
*A:R2>config>router>policy-options# /configure router bgp group "EBGP" 
*A:R2>config>router>bgp>group# shutdown 
*A:R2>config>router>bgp>group# show router static-route detail    

===============================================================================
Static Route Table (Router: Base)  Family: IPv4
===============================================================================
Prefix           : 3.0.0.0/21
Nexthop          : n/a
Type             : Blackhole
Dynamic BGP      : disabled                        Generate ICMP     : disabled
Interface        : n/a                             Active            : N
Prefix List      : PL_R3_STATIC_AGG_OK             Prefix List Type  : Any
Metric           : 1                               Preference        : 5
Source Class     : 0                               Dest Class        : 0
Admin  State     : Up                              Tag               : 0
Creation Origin  : manual                          
BFD              : disabled
Community        : 12:12                           
CPE-check        : disabled
Inactive Reason  : prefix-list match failed
-------------------------------------------------------------------------------
No. of Static Routes: 1

===============================================================================

We can see the static route is down because the prefix-list match has failed and we can confirm that we aren’t advertising this to R1:
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
No Matching Entries Found
===============================================================================

So we’ll restore the EBGP session between R2 and R3 and give it enough time to exchange routes again:
*A:R2>config>router>bgp>group# no shutdown              
*A:R2>config>router>bgp>group# show router bgp routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
u*>i  3.0.0.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.1.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.2.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.3.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.4.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.5.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.6.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  3.0.7.0/24                                         None        None
      10.2.3.3                                           None        -
      3                                                               
u*>i  100.100.100.0/24                                   100         None
      1.1.1.1                                            None        -
      No As-Path                                                      
-------------------------------------------------------------------------------
Routes : 9
===============================================================================

The routes from R3 are back, lets confirm the static blackhole is back in service:
*A:R2>config>router>bgp>group# show router static-route detail 

===============================================================================
Static Route Table (Router: Base)  Family: IPv4
===============================================================================
Prefix           : 3.0.0.0/21
Nexthop          : n/a
Type             : Blackhole
Dynamic BGP      : disabled                        Generate ICMP     : disabled
Interface        : n/a                             Active            : Y
Prefix List      : PL_R3_STATIC_AGG_OK             Prefix List Type  : Any
Metric           : 1                               Preference        : 5
Source Class     : 0                               Dest Class        : 0
Admin  State     : Up                              Tag               : 0
Creation Origin  : manual                          
BFD              : disabled
Community        : 12:12                           
CPE-check        : disabled
-------------------------------------------------------------------------------
No. of Static Routes: 1

===============================================================================

Yes, so we should be offering this to R3 again:
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes 
===============================================================================
 BGP Router ID:2.2.2.2          AS:12          Local AS:12         
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path                                                        
-------------------------------------------------------------------------------
?     3.0.0.0/21                                         100         None
      2.2.2.2                                            None        -
      No As-Path                                                      
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Yes, the aggregate route is now conditionally advertised.

While this scenario isn’t likely to occur all the time, based on my experience it is something to consider if things are not working quite as expected.