Carrier Supporting Carrier with SROS

Carrier of Carriers or Carrier supporting Carrier is a hierarchical construct to allow a network provider to provide MPLS connectivity with relatively low complexity. While it is even simpler to build layer 2 circuits over a backhaul network, in some cases routed connectivity may be preferred as you have a common connection point to the far end regions without needing to concern yourself with the provisioning scale and bandwidth management of each path. While this is quite a large posting it is mainly to show the configuration and verification steps. The summary of actions to create a CsC configuration is at the end of the post if you run out of patience.

This Example will have a three router Carrier of Carriers Network Operator (SC-1, SC-2 and SC-3) providing connectivity between a customer (VPRN 200) with two regions – Region A (RA-1, RA-2 and RA-3) and Region B (RB-1, RB-2 and RB-3)

9 Router Carrier Supporting Carrier Topology

Prior to SROS version 14.0R4, a single RIB was used for both labeled and unlabeled prefixes. With 14.0R4, two RIBs were established (actually, two RIBs for IPv4 and another two for IPv6) – this demonstration will have half the routers using 12.0R6 and half running 14.0R8 to highlight the configuration differences. The routers that are running SROS 14.0R8 in this example are configured using the VSR-D model which has the Control Plane Module operating in a VM (VM-CP) independent of the Input/Output Module (VM-DP) each VM-CP and VM-DP is logically connected via Switch Fabric ports – this is transparent to the configuration and operation of the simulated routers.

The initial configurations for Region A without the CSC Uplink on RA-1 as as below. OSPF as the IGP, LDP signalled LSPs are used, RA-3 is the local VPN Route Reflector for AS65000, VPRN 123 is created on each router.

configure
    system
        name "RA-1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-2"
            address 1.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "RA-3"
            address 1.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 1.1.1.1/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-2"
                    no shutdown
                exit
                interface "RA-3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-2"
                exit
                interface "RA-3"
                exit
            exit
            no shutdown
        exit
        bgp
            split-horizon
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.1:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.1/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RA-2"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-1"
            address 1.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "RA-3"
            address 1.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 1.1.1.2/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-1"
                    no shutdown
                exit
                interface "RA-3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-1"
                exit
                interface "RA-3"
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.2:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.2/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RA-3"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RA-1"
            address 1.1.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "RA-2"
            address 1.2.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "system"
            address 1.1.1.3/32
            no shutdown
        exit
        autonomous-system 65000
        ospf
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RA-1"
                    no shutdown
                exit
                interface "RA-2"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "RA-1"
                exit
                interface "RA-2"
                exit
            exit
        exit
        bgp
            group "VPN-RRC"
                family vpn-ipv4
                cluster 1.1.1.3
                peer-as 65000
                neighbor 1.1.1.1
                    description "RA-1"
                exit
                neighbor 1.1.1.2
                    description "RA-2"
                exit
            exit
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 1.1.1.3:123
            auto-bind ldp
            vrf-target target:65000:123
            interface "Loop" create
                address 123.1.1.3/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

Similarly for Region B, the initial configurations without the CSC Uplink on RB-1 as as below. OSPF as the IGP, LDP signalled LSPs, RB-3 is the local VPN Route Reflector for AS65000, VPRN 123 is created on each router. Besides the card/mda differences on the emulated hardware, the primary difference here compared to Region A which is using an older SROS version is the LSP binding (auto-bind) syntax changes to give more control as to the LSP types that may be desired.

configure
    system
        name "RB-1"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-2"
            address 2.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "RB-3"
            address 2.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 2.2.2.1/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-2"
                    no shutdown
                exit
                interface "RB-3"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-2" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-3" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            split-horizon
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.1:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.1/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RB-2"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-1"
            address 2.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "RB-3"
            address 2.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 2.2.2.2/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-1"
                    no shutdown
                exit
                interface "RB-3"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-1" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-3" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RR"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.2:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.2/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "RB-3"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "RB-1"
            address 2.1.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "RB-2"
            address 2.2.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "system"
            address 2.2.2.3/32
            no shutdown
        exit
        autonomous-system 65000
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
                interface "RB-1"
                    no shutdown
                exit
                interface "RB-2"
                    no shutdown
                exit
            exit
            no shutdown
        exit
        ldp
            interface-parameters
                interface "RB-1" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
                interface "RB-2" dual-stack
                    ipv4
                        fec-type-capability
                            prefix-ipv6 disable
                            p2mp-ipv6 disable
                        exit
                        no shutdown
                    exit
                    no shutdown
                exit
            exit
            no shutdown
        exit
        bgp
            group "VPN-RRC"
                family vpn-ipv4
                cluster 2.2.2.3
                peer-as 65000
                neighbor 2.2.2.1
                    description "RB-1"
                exit
                neighbor 2.2.2.2
                    description "RB-2"
                exit
            exit
            no shutdown
        exit
    exit
    service
        vprn 123 customer 1 create
            route-distinguisher 2.2.2.3:123
            auto-bind-tunnel
                resolution-filter
                    ldp
                exit
                resolution filter
            exit
            vrf-target target:65000:123
            interface "Loop" create
                address 123.2.2.3/32
                loopback
            exit
            no shutdown
        exit
    exit
exit all

The CSC Operators Network uses IS-IS as the IGP, RSVP-TE signalled LSPs, SC-R3 is the local VPN Route Reflector for AS64512.

configure
    system
        name "SC-R1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R2"
            address 10.1.2.1/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R3"
            address 10.1.3.1/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.1/32
            no shutdown
        exit
        autonomous-system 64512
        isis
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R2"
                to 10.10.10.2
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R3"
                to 10.10.10.3
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                peer-as 64512
                neighbor 10.10.10.3
                    description "SC-R3"
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "SC-R2"
    exit
    sfm 1
        sfm-type m-sfm5-12
        no shutdown
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m10-1gb-xp-sfp
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.2.2/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R3"
            address 10.2.3.2/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.2/32
            no shutdown
        exit
        autonomous-system 64512
        isis 0
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R3"
                to 10.10.10.3
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R1"
                to 10.10.10.1
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R3"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                peer-as 64512
                neighbor 10.10.10.3
                    description "SC-R3"
                exit
            exit
            no shutdown
        exit
    exit
exit all

configure
    system
        name "SC-R3"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.3.3/29
            port 1/1/1
            no shutdown
        exit
        interface "SC-R2"
            address 10.2.3.3/29
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.3/32
            no shutdown
        exit
        autonomous-system 64512
        isis
            level-capability level-2
            area-id 49.0001
            traffic-engineering
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            no shutdown
        exit
        mpls
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            path "LOOSE_HOPS"
                no shutdown
            exit
            lsp "SC-R1"
                to 10.10.10.1
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            lsp "SC-R2"
                to 10.10.10.2
                cspf
                adspec
                fast-reroute facility
                exit
                primary "LOOSE_HOPS"
                exit
                no shutdown
            exit
            no shutdown
        exit
        rsvp
            interface "system"
                no shutdown
            exit
            interface "SC-R1"
                no shutdown
            exit
            interface "SC-R2"
                no shutdown
            exit
            no shutdown
        exit
        bgp
            group "IBGP"
                family vpn-ipv4
                cluster 10.10.10.3
                peer-as 64512
                neighbor 10.10.10.1
                    description "SC-R1"
                exit
                neighbor 10.10.10.2
                    description "SC-R2"
                exit
            exit
            no shutdown
        exit
    exit
exit all

At this stage, we have three Islands – Region A, Region B and the Carrier of Carriers Network (without the CSC configuration)

RA-1 is the Region A CSC-CE Router (which is also operating as a regular PE Router) which peers with the CSC Network SC-R1 to exchange labeled routes using BGP. Note: Both of these routers are running SROS 12.0R6 and are configured in the “old way”

Create VPRN 200 on SC-R1 and BGP Peering for RA-1:

configure
    router
        policy-options
            begin
            policy-statement "PS_MPBGP_VPN_TO_BGP"
                entry 10
                    from
                        protocol bgp-vpn
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
    exit
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    service
        vprn 200 customer 1 create
            carrier-carrier-vpn
            autonomous-system 64512
            route-distinguisher 10.10.10.1:200
            auto-bind rsvp-te
            vrf-target target:64512:200
            network-interface "RA-1" create
                address 10.1.1.11/24
                port 1/1/3
                no shutdown
            exit
            bgp
                split-horizon
                group "RegionA"
                    family ipv4
                    as-override
                    export "PS_MPBGP_VPN_TO_BGP"
                    peer-as 65000
                    neighbor 10.1.1.1
                        advertise-label ipv4
                    exit
                exit
                no shutdown
            exit
            no shutdown
        exit
    exit
exit all

Of note is the definition of the VPRN type: carrier-carrier-vpn, once enabled regular IP interfaces cannot be created, only “network-interfaces” which must be on Ethernet ports that are of mode network. As both Region A and Region B both use AS 65000, as-override has been used to stop AS-Path loops occuring when advertising prefixes from other regions with the same BGP AS. We are using the IPv4 Address family but are specifically advertising associated MPLS Labels (This particular configuration component is different after SROS Release 14.0R4) Routing policy is defined and used to export VPN routes to RA-1.

Create the BGP Peering for RA-1 with SC-R1:

configure
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    router
        interface "SC-R1"
            address 10.1.1.1/24
            port 1/1/3
            no shutdown
        exit
        policy-options
            begin
            prefix-list "RegionA"
                prefix 1.1.1.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_BGP_EXP_CSC"
                entry 10
                    from
                        prefix-list "RegionA"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        bgp
            group "CSC"
                export "PS_BGP_EXP_CSC"
                neighbor 10.1.1.11
                    description "SC-R1"
                    peer-as 64512
                    advertise-label ipv4 include-ldp-prefix
                exit
            exit
        exit
    exit all

Routing policy is used to advertise any system addresses from Region A (1.1.1.x/32) with MPLS labels to the CSC

RB-1 is the Region B CSC-CE Router (which is also operating as a regular PE Router) which peers with the CSC Network SC-R2 to exchange labeled routes using BGP. Note: Both of these routers are running SROS 14.0R8 and are configured in the “new way”

Create VPRN 200 on SC-R2 and BGP Peering for RB-1:

configure
    router
        policy-options
            begin
            policy-statement "PS_MPBGP_VPN_TO_BGP"
                entry 10
                    from
                        protocol bgp-vpn
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
    exit
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    service
        vprn 200 customer 1 create
            carrier-carrier-vpn
            autonomous-system 64512
            route-distinguisher 10.10.10.2:200
            auto-bind-tunnel
                resolution-filter
                    rsvp
                exit
                resolution filter
            exit
            vrf-target target:64512:200
            network-interface "RB-1" create
                address 10.2.2.11/24
                port 1/1/3
                no shutdown
            exit
            bgp
                split-horizon
                group "RB"
                    family label-ipv4
                    as-override
                    export "PS_MPBGP_VPN_TO_BGP"
                    peer-as 65000
                    neighbor 10.2.2.1
                    exit
                exit
                no shutdown
            exit
            no shutdown
        exit
    exit
exit all

The main difference compared with the configuration on SC-1 besides the auto-bind-tunnel changes, are that we are specifically peering using the label-ipv4 address family instead of the ipv4 address family and advertising labels.

Create the BGP Peering for RB-1 with SC-R2:

configure
    port 1/1/3
        ethernet
            mode network
        exit
        no shutdown
    exit
    router
        interface "SC-R2"
            address 10.2.2.1/24
            port 1/1/3
            no shutdown
        exit
        policy-options
            begin
            prefix-list "RegionB"
                prefix 2.2.2.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_BGP_EXP_CSC"
                entry 10
                    from
                        prefix-list "RegionB"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        bgp
            group "CSC"
                export "PS_BGP_EXP_CSC"
                neighbor 10.2.2.11
                    description "SC-R2"
                    family label-ipv4
                    peer-as 64512
                    advertise-ldp-prefix
                exit
            exit
        exit
    exit all

Routing policy is used to advertise any system addresses from Region B (2.2.2.x/32) with MPLS labels to the CSC

Once all peers are configured, examine the CSC VPRN 200 Routing Tables:

A:SC-R1#  show router 200 route-table

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
1.1.1.2/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
1.1.1.3/32                                    Remote  BGP       00h07m55s  170
       10.1.1.1                                                     0
2.2.2.1/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
2.2.2.2/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
2.2.2.3/32                                    Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
10.1.1.0/24                                   Local   Local     00h08m29s  0
       RA-1                                                         0
10.2.2.0/24                                   Remote  BGP VPN   00h07m28s  170
       10.10.10.2 (tunneled:RSVP:1)                                 0
-------------------------------------------------------------------------------
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

A:SC-R2# show router 200 route-table

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
1.1.1.2/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
1.1.1.3/32                                    Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
2.2.2.1/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
2.2.2.2/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
2.2.2.3/32                                    Remote  BGP_LABEL 02h02m23s  170
       10.2.2.1                                                     0
10.1.1.0/24                                   Remote  BGP VPN   00h07m20s  170
       10.10.10.1 (tunneled:RSVP:2)                                 0
10.2.2.0/24                                   Local   Local     02h11m42s  0
       RB-1                                                         0
-------------------------------------------------------------------------------
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

With Region A and Region B connectivity via the CSC we should be able to see the initial list of tunnels on the CSC-CE routers (RA-1 and RA-2):

A:RA-1# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.2/32            ldp   MPLS   -        9        1.1.2.2        100
1.1.1.3/32            ldp   MPLS   -        9        1.1.3.3        100
2.2.2.1/32            bgp   MPLS   -        10       10.1.1.11      1000
2.2.2.2/32            bgp   MPLS   -        10       10.1.1.11      1000
2.2.2.3/32            bgp   MPLS   -        10       10.1.1.11      1000
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-1# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        bgp       MPLS  262301    12       10.2.2.11      1000
1.1.1.2/32        bgp       MPLS  262303    12       10.2.2.11      1000
1.1.1.3/32        bgp       MPLS  262304    12       10.2.2.11      1000
2.2.2.2/32        ldp       MPLS  65540     9        2.1.2.2        100
2.2.2.3/32        ldp       MPLS  65541     9        2.1.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-2# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.2.1        100
1.1.1.3/32            ldp   MPLS   -        9        1.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-3# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.3.1        100
1.1.1.2/32            ldp   MPLS   -        9        1.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
2.2.2.1/32        ldp       MPLS  65596     9        2.1.2.1        100
2.2.2.3/32        ldp       MPLS  65544     9        2.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-3# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
2.2.2.1/32        ldp       MPLS  65597     9        2.1.3.1        100
2.2.2.2/32        ldp       MPLS  65537     9        2.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

RA-2, RA-3, RB-2 and RB-3 need the CSC-CEs to to redistribute the prefixes we learnt from the CSC into OSPF and LDP.
configure
    router
        policy-options
            begin
            prefix-list "RegionB"
                prefix 2.2.2.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_OSPF_LDP_EXP_RB"
                entry 10
                    from
                        prefix-list "RegionB"
                    exit
                    action accept
                    exit
                exit
                default-action reject
            exit
            commit
        exit
        ospf
            asbr
            export "PS_OSPF_LDP_EXP_RB"
        exit
        ldp
            export-tunnel-table "PS_OSPF_LDP_EXP_RB"
        exit
    exit
exit all

configure
    router
        policy-options
            begin
            prefix-list "RegionA"
                prefix 1.1.1.0/24 prefix-length-range 32-32
            exit
            policy-statement "PS_OSPF_LDP_EXP_RA"
                entry 10
                    from
                        prefix-list "RegionA"
                    exit
                    action accept
                    exit
                exit
                default-action drop
                exit
            exit
            commit
        exit
        ospf
            asbr
            export "PS_OSPF_LDP_EXP_RA"
        exit
        ldp
            export-tunnel-table "PS_OSPF_LDP_EXP_RA"
        exit
Remember: SROS specifically requires OSPF in the GRT to be defined as an ASBR if you intend to export routes.
A:RA-2# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.2.1        100
1.1.1.3/32            ldp   MPLS   -        9        1.2.3.3        100
2.2.2.1/32            ldp   MPLS   -        9        1.1.2.1        1
2.2.2.2/32            ldp   MPLS   -        9        1.1.2.1        1
2.2.2.3/32            ldp   MPLS   -        9        1.1.2.1        1
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RA-3# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination           Owner Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32            ldp   MPLS   -        9        1.1.3.1        100
1.1.1.2/32            ldp   MPLS   -        9        1.2.3.2        100
2.2.2.1/32            ldp   MPLS   -        9        1.1.3.1        1
2.2.2.2/32            ldp   MPLS   -        9        1.1.3.1        1
2.2.2.3/32            ldp   MPLS   -        9        1.1.3.1        1
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        ldp       MPLS  65619     9        2.1.2.1        1
1.1.1.2/32        ldp       MPLS  65620     9        2.1.2.1        1
1.1.1.3/32        ldp       MPLS  65621     9        2.1.2.1        1
2.2.2.1/32        ldp       MPLS  65596     9        2.1.2.1        100
2.2.2.3/32        ldp       MPLS  65544     9        2.2.3.3        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================

A:RB-3# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination       Owner     Encap TunnelId  Pref     Nexthop        Metric
-------------------------------------------------------------------------------
1.1.1.1/32        ldp       MPLS  65620     9        2.1.3.1        1
1.1.1.2/32        ldp       MPLS  65621     9        2.1.3.1        1
1.1.1.3/32        ldp       MPLS  65622     9        2.1.3.1        1
2.2.2.1/32        ldp       MPLS  65597     9        2.1.3.1        100
2.2.2.2/32        ldp       MPLS  65537     9        2.2.3.2        100
-------------------------------------------------------------------------------
Flags: B = BGP backup route available
       E = inactive best-external BGP route
===============================================================================
Okay, we have tunnels between all routers in both Regions A and B.
The final step to get VPRN connectivity between Region A and Region B is to peer the VPN Route Reflectors (RA-3 and RB-3)
configure
        bgp
            group "VPN-RR-RegionB"
                family vpn-ipv4
                peer-as 65000
                neighbor 2.2.2.3
                    description "RB-3"
                exit
            exit
        exit
    exit all

configure
        bgp
            group "VPN-RR-RegionA"
                family vpn-ipv4
                peer-as 65000
                neighbor 1.1.1.3
                    description "RA-3"
                exit
            exit
        exit
    exit all

We should be able to now verify this with VPRN 123
A:RA-1# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Local   Local     00h50m13s  0
       Loop                                                         0
123.1.1.2/32                                  Remote  BGP VPN   00h44m50s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h45m49s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.1 (tunneled:BGP)                                       0
123.2.2.2/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.2 (tunneled:BGP)                                       0
123.2.2.3/32                                  Remote  BGP VPN   00h00m26s  170
       2.2.2.3 (tunneled:BGP)                                       0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-1# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=0.245ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.245ms, avg = 0.245ms, max = 0.245ms, stddev = 0.000ms
A:RA-1# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=1.26ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.26ms, avg = 1.26ms, max = 1.26ms, stddev = 0.000ms
A:RA-1# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=1.37ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.37ms, avg = 1.37ms, max = 1.37ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=3.46ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.46ms, avg = 3.46ms, max = 3.46ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=3.92ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.92ms, avg = 3.92ms, max = 3.92ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=3.83ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.83ms, avg = 3.83ms, max = 3.83ms, stddev = 0.000ms

A:RA-2# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h46m12s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Local   Local     00h48m34s  0
       Loop                                                         0
123.1.1.3/32                                  Remote  BGP VPN   00h46m12s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   00h01m31s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-2# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=1.12ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.12ms, avg = 1.12ms, max = 1.12ms, stddev = 0.000ms
A:RA-2# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=0.171ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.171ms, avg = 0.171ms, max = 0.171ms, stddev = 0.000ms
A:RA-2# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=1.31ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.31ms, avg = 1.31ms, max = 1.31ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=4.12ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.12ms, avg = 4.12ms, max = 4.12ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=7.76ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 7.76ms, avg = 7.76ms, max = 7.76ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=4.53ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.53ms, avg = 4.53ms, max = 4.53ms, stddev = 0.000ms

A:RA-3# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h47m48s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h46m56s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Local   Local     00h49m05s  0
       Loop                                                         0
123.2.2.1/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   00h02m37s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RA-3>config>router# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=1.17ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.17ms, avg = 1.17ms, max = 1.17ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=1.07ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.07ms, avg = 1.07ms, max = 1.07ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=0.166ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.166ms, avg = 0.166ms, max = 0.166ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=6.13ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 6.13ms, avg = 6.13ms, max = 6.13ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=5.04ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.04ms, avg = 5.04ms, max = 5.04ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=4.37ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.37ms, avg = 4.37ms, max = 4.37ms, stddev = 0.000ms

A:RB-1# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.1 (tunneled:BGP)                                       0
123.1.1.2/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.2 (tunneled:BGP)                                       0
123.1.1.3/32                                  Remote  BGP VPN   00h03m33s  170
       1.1.1.3 (tunneled:BGP)                                       0
123.2.2.1/32                                  Local   Local     02h35m54s  0
       Loop                                                         0
123.2.2.2/32                                  Remote  BGP VPN   02h11m21s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Remote  BGP VPN   02h11m21s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-1# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=4.42ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.42ms, avg = 4.42ms, max = 4.42ms, stddev = 0.000ms
A:RB-1# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=3.89ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.89ms, avg = 3.89ms, max = 3.89ms, stddev = 0.000ms
A:RB-1# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=5.97ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.97ms, avg = 5.97ms, max = 5.97ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=0.118ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.118ms, avg = 0.118ms, max = 0.118ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=1.96ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.96ms, avg = 1.96ms, max = 1.96ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=1.80ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.80ms, avg = 1.80ms, max = 1.80ms, stddev = 0.000ms

A:RB-2# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h04m13s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   02h11m55s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Local   Local     03h04m35s  0
       Loop                                                         0
123.2.2.3/32                                  Remote  BGP VPN   02h36m00s  170
       2.2.2.3 (tunneled)                                           0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-2# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=3.43ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.43ms, avg = 3.43ms, max = 3.43ms, stddev = 0.000ms
A:RB-2# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=5.63ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.63ms, avg = 5.63ms, max = 5.63ms, stddev = 0.000ms
A:RB-2# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=4.40ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.40ms, avg = 4.40ms, max = 4.40ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=1.85ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.85ms, avg = 1.85ms, max = 1.85ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=0.150ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.150ms, avg = 0.150ms, max = 0.150ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=1.90ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.90ms, avg = 1.90ms, max = 1.90ms, stddev = 0.000ms

A:RB-3# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
123.1.1.1/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.1 (tunneled)                                           0
123.1.1.2/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.2 (tunneled)                                           0
123.1.1.3/32                                  Remote  BGP VPN   00h05m23s  170
       1.1.1.3 (tunneled)                                           0
123.2.2.1/32                                  Remote  BGP VPN   02h13m05s  170
       2.2.2.1 (tunneled)                                           0
123.2.2.2/32                                  Remote  BGP VPN   02h37m03s  170
       2.2.2.2 (tunneled)                                           0
123.2.2.3/32                                  Local   Local     02h37m07s  0
       Loop                                                         0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
A:RB-3# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=3.83ms.

---- 123.1.1.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.83ms, avg = 3.83ms, max = 3.83ms, stddev = 0.000ms
A:RB-3# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=4.38ms.

---- 123.1.1.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.38ms, avg = 4.38ms, max = 4.38ms, stddev = 0.000ms
A:RB-3# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=4.47ms.

---- 123.1.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.47ms, avg = 4.47ms, max = 4.47ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=1.97ms.

---- 123.2.2.1 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.97ms, avg = 1.97ms, max = 1.97ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=1.88ms.

---- 123.2.2.2 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.88ms, avg = 1.88ms, max = 1.88ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=0.269ms.

---- 123.2.2.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.269ms, avg = 0.269ms, max = 0.269ms, stddev = 0.000ms
While this is quite a long post, the actual configuration itself for the CSC is straight forward:

  • The CSC VPRN must be set to carrier-carrier-vpn
  • The CSC VPRN uses Ethernet ports with mode network even though its for customers
  • The CSC-PE and CSC-CE exchange labeled routes
  • The CSC-CE redistributes between OSPF/LDP and BGP

Changes to the way the RIB works on SROS and how that impacts CsC configurations was also briefly discussed.

Resolving OSPF MTU problems with SROS

OSPF is a popular Interior Gateway Routing Protocol and in many instances it “just works” for a lot of situations, however care must be taken even in simple deployments. An issue that comes up from time to time is with regards to the maximum transmission unit (MTU). The network topology is a three router topology where I only have direct control a Nokia SROS based router.

OSPF MTU Test Topology

TL;DR – OSPF neighbor in ExchStart – you need to increase your MTU, OSPF neighbor in Exchange – you need to decrease your MTU. Keep reading to see how you can identify and resolve the MTU issues on Nokia Routers with SROS.

Below is the configuration of SR (the router under our administrative control):

configure
    system
        name "SR"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
            mode access
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
            mode access
        exit
        no shutdown
    exit
#--------------------------------------------------
echo "Router (Network Side) Configuration"
#--------------------------------------------------
    router Base
        interface "system"
            address 1.1.1.1/32
            no shutdown
        exit
#--------------------------------------------------
echo "OSPFv2 Configuration"
#--------------------------------------------------
        ospf 0
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit

#--------------------------------------------------
echo "Service Configuration"
#--------------------------------------------------
    service
        customer 1 create
            description "Default customer"
        exit
        ies 100 customer 1 create
            description "PEER1"
            interface "PEER1" create
                address 10.1.2.1/27
                sap 1/1/1 create
                exit
            exit
            no shutdown
        exit
        ies 200 customer 1 create
            description "PEER2"
            interface "PEER2" create
                address 10.1.3.1/27
                sap 1/1/2 create
                exit
            exit
            no shutdown
        exit
    exit
#--------------------------------------------------
echo "Router (Service Side) Configuration"
#--------------------------------------------------
    router
        ospf 0
            area 0.0.0.1
                interface "PEER1"
                    no shutdown
                exit
                interface "PEER2"
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
exit all

One thing to note is that the Peer routers are attached to an Internet Enhanced Service (IES) and not part of the OSPF Backbone Area – from a stored configuration perspective there is a distinction between core network and customer configurations but from a protocol pespective things are the same. IES Interfaces that are bound to Service Access Points (SAPs) which must be changed from the default mode of network – in this case we are using access, however hybrid is an option as well.

As this post is about resolving issues, obviously things are not working as straight forward as expected.

A:SR# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     00h40m51s  0
       system                                                       0
10.1.2.0/27                                   Local   Local     00h33m32s  0
       PEER1                                                        0
10.1.3.0/27                                   Local   Local     00h34m02s  0
       PEER2                                                        0
-------------------------------------------------------------------------------
No. of Routes: 3
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

So nothing from OSPF is in the routing table, while its possible (but unlikely) that our peers aren’t advertising anything, an alternate explanation is that it could be a connectivity issue, so we’ll ping each peer router first
A:SR# ping 10.1.2.2 count 3
PING 10.1.2.2 56 data bytes
64 bytes from 10.1.2.2: icmp_seq=1 ttl=64 time=1.28ms.
64 bytes from 10.1.2.2: icmp_seq=2 ttl=64 time=1.15ms.
64 bytes from 10.1.2.2: icmp_seq=3 ttl=64 time=1.08ms.

---- 10.1.2.2 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.08ms, avg = 1.17ms, max = 1.28ms, stddev = 0.081ms
A:SR# ping 10.1.3.3 count 3
PING 10.1.3.3 56 data bytes
64 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.51ms.
64 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.31ms.
64 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.24ms.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.24ms, avg = 1.35ms, max = 1.51ms, stddev = 0.116ms
Okay so IP connectivity is established, lets check the OSPF interface state
A:SR# show router ospf interface

===============================================================================
Rtr Base OSPFv2 Instance 0 Interfaces
===============================================================================
If Name               Area Id         Designated Rtr  Bkup Desig Rtr  Adm  Oper
-------------------------------------------------------------------------------
system                0.0.0.0         1.1.1.1         0.0.0.0         Up   DR
PEER1                 0.0.0.1         1.1.1.1         100.100.100.100 Up   DR
PEER2                 0.0.0.1         1.1.1.1         0.0.0.0         Up   DR
-------------------------------------------------------------------------------
No. of OSPF Interfaces: 3
===============================================================================

From first glance PEER1 seems okay but PEER2 doesn’t have a BDR and since we are using the default ospf interface type (broadcast) we would expect that to see both the DR and BDR – lets get some more details
A:SR# show router ospf interface "PEER2" detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Interface "PEER2" (detail)
===============================================================================
-------------------------------------------------------------------------------
Configuration
-------------------------------------------------------------------------------
IP Address       : 10.1.3.1
Area Id          : 0.0.0.1              Priority         : 1
Hello Intrvl     : 10 sec               Rtr Dead Intrvl  : 40 sec
Retrans Intrvl   : 5 sec                Poll Intrvl      : 120 sec
Cfg Metric       : 0                    Advert Subnet    : True
Transit Delay    : 1                    Cfg IF Type      : None
Passive          : False                Cfg MTU          : 0
LSA-filter-out   : None                 Adv Rtr Capab    : Yes
LFA              : Include              LFA NH Template  :
RIB-priority     : None
Auth Type        : None
-------------------------------------------------------------------------------
State
-------------------------------------------------------------------------------
Admin Status     : Enabled              Oper State       : Designated Rtr
Designated Rtr   : 1.1.1.1              Backup Desig Rtr : 0.0.0.0
IF Type          : Broadcast            Network Type     : Stub
Oper MTU         : 1500                 Last Enabled     : 06/02/2017 01:44:23
Oper Metric      : 100                  Bfd Enabled      : No
Te Metric        : 100                  Te State         : Down
Admin Groups     : None
Ldp Sync         : outOfService         Ldp Sync Wait    : Disabled
Ldp Timer State  : Disabled             Ldp Tm Left      : 0
-------------------------------------------------------------------------------
Statistics
-------------------------------------------------------------------------------
Nbr Count        : 0                    If Events        : 2
Tot Rx Packets   : 0                    Tot Tx Packets   : 76
Rx Hellos        : 0                    Tx Hellos        : 76
Rx DBDs          : 0                    Tx DBDs          : 0
Rx LSRs          : 0                    Tx LSRs          : 0
Rx LSUs          : 0                    Tx LSUs          : 0
Rx LS Acks       : 0                    Tx LS Acks       : 0
Retransmits      : 0                    Discards         : 78
Bad Networks     : 0                    Bad Virt Links   : 0
Bad Areas        : 78                   Bad Dest Addrs   : 0
Bad Auth Types   : 0                    Auth Failures    : 0
Bad Neighbors    : 0                    Bad Pkt Types    : 0
Bad Lengths      : 0                    Bad Hello Int.   : 0
Bad Dead Int.    : 0                    Bad Options      : 0
Bad Versions     : 0                    Bad Checksums    : 0
LSA Count        : 0                    LSA Checksum     : 0x0
===============================================================================
Okay, we can see that there are discards which align with the Bad Area Count – this means that PEER2 doesn’t believe it’s part of OSPF Area 1.

Log-id 99 is automatically configured on Nokia SROS devices to capture a number of event messages however it can get a bit overwhelming to find something specific. Fortunately there are ways to reduce the output by specifying the application (OSPF) and something that may be part of the log message itself we want to see (PEER2)

A:SR# show log log-id 99 application OSPF message PEER2

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=944  (wrapped)]

941 2017/06/02 02:10:15.55 UTC WARNING: OSPF #2043 Base VR:  1 OSPFv2 (0)
"LCL_RTR_ID 1.1.1.1: Conflicting configuration areaMismatch on interface PEER2 from 10.1.3.3 in hello"
So while we have identified a problem – OSPF Area MisMatch, we need to overcome it – remembering we cant configure PEER2 (the person that manages it is on a training course and cannot be contacted, while your project manager is wanting solutions, not problems..)

This is where using show and debug commands can help identify and resolve issues – SROS is quite powerful with its debugging tools and while they can be used in production, it is always best to attempt to narrow down what you are attempting to collect – firstly we need to create a debug log if one doesn’t already exist – for this example I’m just logging to a circular memory buffer but it could go to SNMP, syslog or a file if necessary.

A:SR# configure log log-id 10
*A:SR>config>log>log-id$ from debug-trace
*A:SR>config>log>log-id$ to memory
*A:SR>config>log>log-id$ no shutdown
*A:SR>config>log>log-id$ back
*A:SR>config>log# info
----------------------------------------------
        log-id 10
            from debug-trace
            to memory
            no shutdown
        exit
----------------------------------------------
Now to set up the debug – we know it’s from interface PEER2 and the log message kindly told us the packet type (in hello)..
*A:SR>config>log# /debug router ospf packet hello "PEER2"
*A:SR>config>log# show debug
debug
    router "Base"
        ospf
            packet hello "PEER2"
        exit
    exit
exit
Router Base is the global routing table of the router, the debug can reference other services e.g. a VPRN if necessary by changing the router – After a few seconds (OSPF hello packets will come every 10 seconds or so) we can look in log 10 to see what was received.
*A:SR>config>log# show log log-id 10

===============================================================================
Event Log 10
===============================================================================
Description : (Not Specified)
Memory Log contents  [size=100   next event=10  (not wrapped)]

9 2017/06/02 02:19:27.10 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT

>> Outgoing OSPF packet on I/F PEER2 area 0.0.0.1
OSPF Version      : 2
Router Id         : 1.1.1.1
Area Id           : 0.0.0.1
Checksum          : ecb9
Auth Type         : Null
Auth Key          : 00 00 00 00 00 00 00 00
Packet Type       : HELLO
Packet Length     : 44 "

8 2017/06/02 02:19:26.55 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT DROPPED
area mismatch"

7 2017/06/02 02:19:26.54 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT

>> Incoming OSPF packet on I/F PEER2 area 0.0.0.2
OSPF Version      : 2
Router Id         : 200.200.200.200
Area Id           : 0.0.0.2
Checksum          : 5d27
Auth Type         : Null
Auth Key          : 00 00 00 00 00 00 00 00
Packet Type       : HELLO
Packet Length     : 44 "
SR is configured with PEER2 in Area 1 but it should be in Area 2, lets fix that
*A:SR>config>log# /configure router ospf
*A:SR>config>router>ospf# info
----------------------------------------------
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
            exit
            area 0.0.0.1
                interface "PEER1"
                    no shutdown
                exit
                interface "PEER2"
                    no shutdown
                exit
            exit
            no shutdown
----------------------------------------------
*A:SR>config>router>ospf# area 1 interface "PEER2" shutdown
*A:SR>config>router>ospf# area 1 no interface "PEER2"
*A:SR>config>router>ospf# area 2 interface "PEER2" no shutdown
*A:SR>config>router>ospf# info
----------------------------------------------
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
            exit
            area 0.0.0.1
                interface "PEER1"
                    no shutdown
                exit
            exit
            area 0.0.0.2
                interface "PEER2"
                    no shutdown
                exit
            exit
            no shutdown
----------------------------------------------
Now see if that fixes that problem.
*A:SR>config>router>ospf# show router ospf interface

===============================================================================
Rtr Base OSPFv2 Instance 0 Interfaces
===============================================================================
If Name               Area Id         Designated Rtr  Bkup Desig Rtr  Adm  Oper
-------------------------------------------------------------------------------
system                0.0.0.0         1.1.1.1         0.0.0.0         Up   DR
PEER1                 0.0.0.1         1.1.1.1         100.100.100.100 Up   DR
PEER2                 0.0.0.2         200.200.200.200 1.1.1.1         Up   BDR
-------------------------------------------------------------------------------
No. of OSPF Interfaces: 3
===============================================================================
Yes we can see both the DR and BDR for our OSPF peers but before we move on, we should stop the debug activity
*A:SR>config>router>ospf# /debug router no ospf
*A:SR>config>router>ospf# show debug
debug
exit
Now lets see if OSPF routing exchange is occurring.
*A:SR>config>router>ospf# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     01h15m46s  0
       system                                                       0
10.1.2.0/27                                   Local   Local     01h08m27s  0
       PEER1                                                        0
10.1.3.0/27                                   Local   Local     01h08m57s  0
       PEER2                                                        0
-------------------------------------------------------------------------------
No. of Routes: 3
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
Well that isn’t fixed yet (which should be no surprise as this is about MTU issues) so lets move onto the next phase and examine the state of our OSPF neighbors
*A:SR>config>router>ospf# show router ospf neighbor

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors
===============================================================================
Interface-Name                   Rtr Id          State      Pri  RetxQ   TTL
   Area-Id
-------------------------------------------------------------------------------
PEER1                            100.100.100.100 ExchStart  1    0       34
   0.0.0.1
PEER2                            200.200.200.200 Exchange   1    0       32
   0.0.0.2
-------------------------------------------------------------------------------
No. of Neighbors: 2
===============================================================================
A router that is stuck in ExchStart or Exchange is a hallmark of OSPF MTU related problems.
Let’s start working on PEER1.
*A:SR>config>router>ospf# show router ospf neighbor "PEER1" detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface "PEER1" (detail)
===============================================================================
-------------------------------------------------------------------------------
Neighbor : 10.1.2.2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Neighbor Rtr Id : 100.100.100.100  Interface: PEER1
-------------------------------------------------------------------------------
Neighbor IP Addr : 10.1.2.2
Local IF IP Addr : 10.1.2.1
Area Id          : 0.0.0.1
Designated Rtr   : 1.1.1.1              Backup Desig Rtr : 100.100.100.100
Neighbor State   : ExchStart            Priority         : 1
Retrans Q Length : 0                    Options          : - E - -  -  - - --
Events           : 1068                 Last Event Time  : 06/02/2017 02:59:34
Up Time          : 0d 01:11:00          Time Before Dead : 38 sec
GR Helper        : Not Helping          GR Helper Age    : 0 sec
GR Exit Reason   : None                 GR Restart Reason: Unknown (0)
Bad Nbr States   : 0                    LSA Inst fails   : 0
Bad Seq Nums     : 0                    Bad MTUs         : 1066
Bad Packets      : 0                    LSA not in LSDB  : 0
Option Mismatches: 0                    Nbr Duplicates   : 0
Num Restarts     : 0                    Last Restart at  : Never
===============================================================================
There are quite a few Bad MTUs being reported – While some vendors have an option to ignore the OSPF MTU, there are quite a number of MTU implications that can occur within the core when you consider various tunnel options that this is not provided.

Before we start to change things lets see if our trusty log 99 to see says anything about this:

*A:SR>config>router>ospf# show log log-id 99 application OSPF message PEER1

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=1473  (wrapped)]

1472 2017/06/02 02:40:17.67 UTC WARNING: OSPF #2043 Base VR:  1 OSPFv2 (0)
"LCL_RTR_ID 1.1.1.1: Conflicting configuration mtuMismatch on interface PEER1 from 10.1.2.2 in dbDescript"
We can use another debug to determine what the actual MTU should be (as before with the area mismatch, log 99 gave us a hint as to the packet type we should be investigating):
*A:SR>config>router>ospf# /debug router ospf packet dbdescr ingress "PEER1"
Clear the log and see what we are recieving:
*A:SR>config>router>ospf# /clear log 10
*A:SR>config>router>ospf# /show log log-id 10

===============================================================================
Event Log 10
===============================================================================
Description : (Not Specified)
Memory Log contents  [size=100   next event=3  (not wrapped)]

2 2017/06/02 02:55:17.67 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT DROPPED
MTU mismatch"

1 2017/06/02 02:55:17.67 UTC MINOR: DEBUG #2001 Base OSPFv2
"OSPFv2: PKT

>> Incoming OSPF packet on I/F PEER1 area 0.0.0.1
OSPF Version      : 2
Router Id         : 100.100.100.100
Area Id           : 0.0.0.1
Checksum          : e35a
Auth Type         : Null
Auth Key          : 00 00 00 00 00 00 00 00
Packet Type       : DB_DESC
Packet Length     : 32

Interface MTU     : 1504
Options           : 000042
Flags             : 7   INIT MORE MAST
Sequence Num      : 2514
"

Okay, so PEER1 requires an MTU of 1504, lets modify that within the OSPF configuration:
*A:SR>config>router>ospf# info
----------------------------------------------
            area 0.0.0.0
                interface "system"
                    no shutdown
                exit
            exit
            area 0.0.0.1
                interface "PEER1"
                    no shutdown
                exit
            exit
            area 0.0.0.2
                interface "PEER2"
                    no shutdown
                exit
            exit
            no shutdown
----------------------------------------------
*A:SR>config>router>ospf# area 1 interface "PEER1" mtu 1504
When applying a configuration it is good to verify things are working as expected:
*A:SR>config>router>ospf# show router ospf interface "PEER1" detail | match MTU
Passive          : False                Cfg MTU          : 1504
Oper MTU         : 1500                 Last Enabled     : 06/02/2017 01:44:23
Although we configured the MTU to be 1504, the Operational MTU is 1500 (This is because the IP MTU is 1500 so OSPF cant be given a larger MTU on this interface)
*A:SR>config>router>ospf# show router interface "PEER1" detail | match MTU
IP MTU           : (default)
IP Oper MTU      : 1500

When Ethernet Ports are configured as mode access and left at the default encapsulation (null) the Ethernet port MTU is 1514 bytes (to support a 1500 byte IP MTU and 14 bytes of Ethernet Header – FCS is not included in MTU calculations)
*A:SR>config>router>ospf# show port 1/1/1 | match MTU
Physical Link      : Yes                        MTU              : 1514
To get a 1504 byte IP MTU, we can just add 4 bytes to the Port Ethernet MTU
*A:SR>config>router>ospf# /configure port 1/1/1 ethernet mtu 1518
*A:SR>config>router>ospf# show router interface "PEER1" detail | match MTU
IP MTU           : (default)
IP Oper MTU      : 1504
*A:SR>config>router>ospf# show router ospf interface "PEER1" detail | match MTU
Passive          : False                Cfg MTU          : 1504
Oper MTU         : 1504                 Last Enabled     : 06/02/2017 01:44:23
This should mean that the OSPF neighbor will now perform the database exchange and enter the Full state.
*A:SR>config>router>ospf# show router ospf neighbor "PEER1"

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface "PEER1"
===============================================================================
Interface-Name                   Rtr Id          State      Pri  RetxQ   TTL
   Area-Id
-------------------------------------------------------------------------------
PEER1                            100.100.100.100 Full       1    0       34
   0.0.0.1
-------------------------------------------------------------------------------
No. of Neighbors: 1
===============================================================================
*A:SR>config>router>ospf# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     02h10m36s  0
       system                                                       0
10.1.2.0/27                                   Local   Local     02h03m16s  0
       PEER1                                                        0
10.1.3.0/27                                   Local   Local     02h03m46s  0
       PEER2                                                        0
100.100.100.100/32                            Remote  OSPF      00h03m18s  10
       10.1.2.2                                                     100
-------------------------------------------------------------------------------
No. of Routes: 4
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
The OSPF issue with PEER1 appears to have been resolved so back to PEER2.

*A:SR>config>router>ospf# show router ospf neighbor "PEER2" detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface "PEER2" (detail)
===============================================================================
-------------------------------------------------------------------------------
Neighbor : 10.1.3.3
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Neighbor Rtr Id : 200.200.200.200  Interface: PEER2
-------------------------------------------------------------------------------
Neighbor IP Addr : 10.1.3.3
Local IF IP Addr : 10.1.3.1
Area Id          : 0.0.0.2
Designated Rtr   : 200.200.200.200      Backup Desig Rtr : 1.1.1.1
Neighbor State   : Exchange             Priority         : 1
Retrans Q Length : 3                    Options          : - E - -  -  - O --
Events           : 3                    Last Event Time  : 06/02/2017 02:22:36
Up Time          : 0d 01:01:10          Time Before Dead : 37 sec
GR Helper        : Not Helping          GR Helper Age    : 0 sec
GR Exit Reason   : None                 GR Restart Reason: Unknown (0)
Bad Nbr States   : 0                    LSA Inst fails   : 0
Bad Seq Nums     : 0                    Bad MTUs         : 0
Bad Packets      : 0                    LSA not in LSDB  : 0
Option Mismatches: 0                    Nbr Duplicates   : 917
Num Restarts     : 0                    Last Restart at  : Never
===============================================================================
There are no Bad MTUs being reported here, all we can see is that we are forever in Exchange state – lets check log 99 to see if anything at all related to PEER2 is present
*A:SR>config>router>ospf# show log log-id 99 message PEER2

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=2033  (wrapped)]
There is nothing present (the older events have wrapped around since we are only keeping the last 500 events)
What I have found is when OSPF neighbors are stuck in ExchStart, your router is the one with the MTU too small but while the router that is stuck in Exchange is the one with the MTU that is too big for its peer.
To work out what the smaller MTU should be, we’ll send ping packets of various lengths to work out what is the biggest unfragmented packet that can be sent to PEER2. Note: when we send a ping and specify the size, we are actually calling out what the ICMP payload size should be, so we need to ensure for IPv4 we consider the 20 byte IP header and 8 byte ICMP header – so an IP interface with an IP-MTU of 1500 would work for a ping with a payload size of 1472 but would fail at 1473.
We can test this concept on a known quantity (PEER1 which has an IP MTU of 1504) we should be able to get a ping payload of 1476 through okay but 1477 should fail – make sure we set the DF bit!
*A:SR>config>router>ospf# ping 10.1.2.2 size 1476 do-not-fragment count 3
PING 10.1.2.2 1476 data bytes
1484 bytes from 10.1.2.2: icmp_seq=1 ttl=64 time=1.34ms.
1484 bytes from 10.1.2.2: icmp_seq=2 ttl=64 time=1.27ms.
1484 bytes from 10.1.2.2: icmp_seq=3 ttl=64 time=1.16ms.

---- 10.1.2.2 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.16ms, avg = 1.26ms, max = 1.34ms, stddev = 0.072ms
*A:SR>config>router>ospf# ping 10.1.2.2 size 1477 do-not-fragment count 3
PING 10.1.2.2 1477 data bytes

---- 10.1.2.2 PING Statistics ----
3 packets transmitted, 3 packets bounced, 0 packets received, 100% packet loss
This works as expected, so the concept appears sound.

*A:SR>config>router>ospf# show router interface "PEER2" detail | match MTU
IP MTU           : (default)
IP Oper MTU      : 1500
We know that we have a ceiling of 1500 and we know the MTU must be lower than this. But just to be certain, we’ll try based on a 1500 byte IP packet anyway
*A:SR>config>router>ospf# ping 10.1.3.3 size 1472 do-not-fragment count 3
PING 10.1.3.3 1472 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.
Request timed out. icmp_seq=3.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 0 packets received, 100% packet loss
Unsurprising, the Peer MTU is less than 1500 bytes, lets try a slightly smaller payload
*A:SR>config>router>ospf# ping 10.1.3.3 size 1462 do-not-fragment count 3
PING 10.1.3.3 1462 data bytes
1470 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.44ms.
1470 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.36ms.
1470 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.34ms.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.34ms, avg = 1.38ms, max = 1.44ms, stddev = 0.042ms
Okay, time to divide and conquer to determine the largest payload that gets through
*A:SR>config>router>ospf# ping 10.1.3.3 size 1467 do-not-fragment count 3
PING 10.1.3.3 1467 data bytes
1475 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.24ms.
1475 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.30ms.
1475 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=2.16ms.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.24ms, avg = 1.57ms, max = 2.16ms, stddev = 0.417ms
*A:SR>config>router>ospf# ping 10.1.3.3 size 1469 do-not-fragment count 3
PING 10.1.3.3 1469 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.
Request timed out. icmp_seq=3.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 0 packets received, 100% packet loss
*A:SR>config>router>ospf# ping 10.1.3.3 size 1468 do-not-fragment count 3
PING 10.1.3.3 1468 data bytes
1476 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.19ms.
1476 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.30ms.
1476 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.26ms.

---- 10.1.3.3 PING Statistics ----
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.19ms, avg = 1.25ms, max = 1.30ms, stddev = 0.043ms
An ICMP payload of 1468 fits within an IP packet with a size of 1496 – adjust the OSPF MTU to 1496 and see if that results in getting a full adjacency.
*A:SR>config>router>ospf# area 2 interface "PEER2" mtu 1496
*A:SR>config>router>ospf# show router ospf interface "PEER2" detail | match MTU
Passive          : False                Cfg MTU          : 1496
Oper MTU         : 1496                 Last Enabled     : 06/02/2017 02:22:36
*A:SR>config>router>ospf# show router ospf neighbor "PEER2"

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface "PEER2"
===============================================================================
Interface-Name                   Rtr Id          State      Pri  RetxQ   TTL
   Area-Id
-------------------------------------------------------------------------------
PEER2                            200.200.200.200 Full       1    0       35
   0.0.0.2
-------------------------------------------------------------------------------
No. of Neighbors: 1
===============================================================================
The adjacency is up – lets see what routes we have learnt
*A:SR>config>router>ospf# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.1.1.1/32                                    Local   Local     02h40m46s  0
       system                                                       0
10.1.2.0/27                                   Local   Local     02h33m27s  0
       PEER1                                                        0
10.1.3.0/27                                   Local   Local     02h33m56s  0
       PEER2                                                        0
100.100.100.100/32                            Remote  OSPF      00h33m29s  10
       10.1.2.2                                                     100
200.200.200.200/32                            Remote  OSPF      00h01m27s  10
       10.1.3.3                                                     100
-------------------------------------------------------------------------------
No. of Routes: 5
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

We now have learnt routes from PEER1 and PEER2, time for a quick dataplane verification:
*A:SR>config>router>ospf# ping 100.100.100.100 source 1.1.1.1 count 1
PING 100.100.100.100 56 data bytes
64 bytes from 100.100.100.100: icmp_seq=1 ttl=64 time=1.38ms.

---- 100.100.100.100 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.38ms, avg = 1.38ms, max = 1.38ms, stddev = 0.000ms
*A:SR>config>router>ospf# ping 200.200.200.200 source 1.1.1.1 count 1
PING 200.200.200.200 56 data bytes
64 bytes from 200.200.200.200: icmp_seq=1 ttl=64 time=1.14ms.

---- 200.200.200.200 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.14ms, avg = 1.14ms, max = 1.14ms, stddev = 0.000ms
We now have successful routing exchange and data plane reachability.