ThingNetwork.io

Simplified Provisioning of RSVP-TE Signaled LSPs with Nokia SROS

This post will demonstrate some RSVP-TE LSP concepts using Nokia SROS. It will start off with the basic method of establishing a RSVP-TE Signaled LSP and a cursory glance of some of the TE options and then will jump into a more useful method to cope with larger deployments, depending on your impatience, it may could be worthwhile to jump directly to the “LSP Templates” section, however if you would like a bit of background, grab a cup of coffee and read on.

MPLS Label Switched Paths (LSPs) established using Label Distribution Protocol (LDP) signaling is fine for many applications and a primary advantage is that it is very quick to deploy and is widely supported across a range of device, however the Interior Gateway Routing Protocol (Typically OSPF or IS-IS) dictates the path taken by the LSP which may not always be desired particularly if you wish to steer particular flows one way or another. LSPs signaled using Resource Reservation Protocol with Traffic Engineering Extensions (RSVP-TE) introduces a lot more fine grained control (however it is equally valid to let the LSP follow the path of the IGP as well) however a disadvantage using such LSPs is that it can become quite cumbersome to build each one particularly if you have many to manage, or there is a lot of ongoing change.

The following 4 router partial mesh will be used to support this the configurations in this post.

We’ll start with the initial configurations on the routers (simulated 7750 SR12s) which will including everything except for the MPLS related configuration.
[codegroup][php tab=”R1 Initial”]configure
system
name “R1”
exit
sfm 1
sfm-type m-sfm5-12
no shutdown
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m10-1gb-xp-sfp
no shutdown
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
port 1/1/3
ethernet
exit
no shutdown
exit
router
interface “R2”
address 10.1.2.1/24
port 1/1/2
no shutdown
exit
interface “R3”
address 10.1.3.1/24
port 1/1/3
no shutdown
exit
interface “system”
address 10.10.10.1/32
no shutdown
exit
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R2”
no shutdown
exit
interface “R3″
no shutdown
exit
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”R2 Initial”]configure
system
name “R2”
exit
sfm 1
sfm-type m-sfm5-12
no shutdown
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m10-1gb-xp-sfp
no shutdown
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
port 1/1/3
ethernet
exit
no shutdown
exit
port 1/1/4
ethernet
exit
no shutdown
exit
router
interface “R1”
address 10.1.2.2/24
port 1/1/2
no shutdown
exit
interface “R3”
address 10.2.3.2/24
port 1/1/4
no shutdown
exit
interface “R4”
address 10.2.4.2/24
port 1/1/3
no shutdown
exit
interface “system”
address 10.10.10.2/32
no shutdown
exit
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R1”
no shutdown
exit
interface “R3”
no shutdown
exit
interface “R4″
no shutdown
exit
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”R3 Initial”]configure
system
name “R3”
exit
sfm 1
sfm-type m-sfm5-12
no shutdown
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m10-1gb-xp-sfp
no shutdown
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
port 1/1/3
ethernet
exit
no shutdown
exit
port 1/1/4
ethernet
exit
no shutdown
exit
router
interface “R1”
address 10.1.3.3/24
port 1/1/3
no shutdown
exit
interface “R2”
address 10.2.3.3/24
port 1/1/4
no shutdown
exit
interface “R4”
address 10.3.4.3/24
port 1/1/2
no shutdown
exit
interface “system”
address 10.10.10.3/32
no shutdown
exit
ospf 0
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R1”
no shutdown
exit
interface “R2”
no shutdown
exit
interface “R4″
no shutdown
exit
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”R4 Initial”]configure
system
name “R4”
exit
sfm 1
sfm-type m-sfm5-12
no shutdown
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m10-1gb-xp-sfp
no shutdown
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
port 1/1/3
ethernet
exit
no shutdown
exit
router
interface “R2”
address 10.2.4.4/24
port 1/1/3
no shutdown
exit
interface “R3”
address 10.3.4.4/24
port 1/1/2
no shutdown
exit
interface “system”
address 10.10.10.4/32
no shutdown
exit
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R3”
no shutdown
exit
interface “R2”
no shutdown
exit
exit
no shutdown
exit
exit
exit all[/php][/codegroup]There are 3 initial steps to enable RSVP-TE on Nokia SROS platforms:
1: Add the router interfaces into the MPLS Process (including the system interface) and enable the process. The follow will be applied on R1[php]configure
router
mpls
interface “system”
no shutdown
exit
interface “R2”
no shutdown
exit
interface “R3”
no shutdown
exit
no shutdown
exit[/php]
2: RSVP automatically adds the interfaces inserted into MPLS into its process, however RSVP is shutdown by default and required enabling
[php] rsvp
no shutdown
exit[/php]
3: As we are intending to use constrained paths which are not limited to that of the shortest path, the IGP traffic engineering extensions need enabling
[php] ospf
traffic-engineering
exit
exit
exit all[/php]

Once done we are ready to build MPLS paths and assign them to LSPs but before we do that, here is the equivalent configuration applied to R2, R3 and R4
[codegroup][php tab=”R2 Enable RSVP-TE”]configure
router
mpls
interface “system”
no shutdown
exit
interface “R1”
no shutdown
exit
interface “R3”
no shutdown
exit
interface “R4″
no shutdown
exit
no shutdown
exit
rsvp
no shutdown
exit
ospf
traffic-engineering
exit
exit
exit all[/php][php tab=”R3 Enable RSVP-TE”]configure
router
mpls
interface “system”
no shutdown
exit
interface “R1”
no shutdown
exit
interface “R2”
no shutdown
exit
interface “R4″
no shutdown
exit
no shutdown
exit
rsvp
no shutdown
exit
ospf
traffic-engineering
exit
exit
exit all[/php][php tab=”R4 Enable RSVP-TE”]configure
router
mpls
interface “system”
no shutdown
exit
interface “R2”
no shutdown
exit
interface “R3”
no shutdown
exit
no shutdown
exit
rsvp
no shutdown
exit
ospf
traffic-engineering
exit
exit
exit all[/php][/codegroup]

The first LSP to be created will be configured on R1 and destined to R3, however rather than use the directly connected path, the path will go via the outer Square (R1 -> R2 -> R4 -> R3). On R1 we will first define the path between R1 and R3 using strict hops where each hop is the system address of the next router:[php]/configure
router
mpls
path “R1-R3”
hop 1 10.10.10.2 strict
hop 2 10.10.10.4 strict
hop 3 10.10.10.3 strict
no shutdown
exit[/php] Once the path is defined, the LSP “LSP_R1-R3” will be created ensuring that a constraint based shortest based path can be used and using “R1-R3” as the primary path:[php] lsp “LSP_R1-R3”
to 10.10.10.3
cspf
primary “R1-R3”
exit
no shutdown
exit
exit
exit
exit all[/php]Once configured and enable, we can verify if the LSP is up and running:[php]*A:R1# show router mpls lsp

===============================================================================
MPLS LSPs (Originating)
===============================================================================
LSP Name To Tun Fastfail Adm Opr
Id Config
——————————————————————————-
LSP_R1-R3 10.10.10.3 3 No Up Up
——————————————————————————-
LSPs : 1
===============================================================================[/php]Yes it is, however that summary doesn’t provide details on the path, so a more detailed command will be called[php]*A:R1# show router mpls lsp “LSP_R1-R3” path detail | match Explicit post-lines 100
Explicit Hops :
10.10.10.2(S) -> 10.10.10.4(S) -> 10.10.10.3(S)
Actual Hops :
10.1.2.1 (10.10.10.1) Record Label : N/A
-> 10.1.2.2 (10.10.10.2) Record Label : 262143
-> 10.2.4.4 (10.10.10.4) Record Label : 262139
-> 10.3.4.3 (10.10.10.3) Record Label : 262143
Computed Hops :
10.1.2.1(S)
-> 10.1.2.2(S)
-> 10.2.4.4(S)
-> 10.3.4.3(S)
Resignal Eligible: False
Last Resignal : n/a CSPF Metric : 300
===============================================================================[/php]The path is taken as specified, and we can also confirm this on the transit (MPLS Label Switch Routers) and terminating (Egress MPLS Label Edge Router)[php]*A:R2# show router mpls lsp transit
===============================================================================
MPLS LSPs (Transit)
===============================================================================
Legend : @ – Active Detour
===============================================================================
From To In I/F Out I/F State LSP Name
——————————————————————————-
10.10.10.1 10.10.10.3 1/1/2 1/1/3 Up LSP_R1-R3::R1-R3
——————————————————————————-
LSPs : 1
——————————————————————————-[/php][php]*A:R4# show router mpls lsp transit
===============================================================================
MPLS LSPs (Transit)
===============================================================================
Legend : @ – Active Detour
===============================================================================
From To In I/F Out I/F State LSP Name
——————————————————————————-
10.10.10.1 10.10.10.3 1/1/3 1/1/2 Up LSP_R1-R3::R1-R3
——————————————————————————-
LSPs : 1
——————————————————————————-[/php][php]*A:R3# show router mpls lsp terminate
===============================================================================
MPLS LSPs (Terminate)
===============================================================================
Legend : @ – Active Detour
===============================================================================
From To In I/F Out I/F State LSP Name
——————————————————————————-
10.10.10.1 10.10.10.3 1/1/2 n/a Up LSP_R1-R3::R1-R3
——————————————————————————-
LSPs : 1
——————————————————————————-[/php]As MPLS LSPs are unidirectional, we would need to provision a corresponding LSP on R3 to terminate on R1, to have something more than just a demonstration – however I’ll skip that step and shutdown the LSP on R1 and do something a little different.[php]*A:R1# /configure router mpls lsp “LSP_R1-R3” shutdown[/php]Firstly we are going to define an admin-group or link color which can be used to give input into the specific hops the path of an LSP will take (rules can be made based on the presence or non-presence of certain admin-groups however with 32 values available, care needs to be given on how these are allocated)[php]/configure
router
if-attribute
admin-group “R1-R2” value 5
exit[/php]Then this admin-group will be associated to the R2 facing interface within the MPLS context[php] mpls
interface “R2”
admin-group “R1-R2”
exit[/php]Now to create an unconstrained path (by simply creating and administratively enabling the path, providing nothing else limits the selection criteria, the IGP path will be used for the LSP)[php] path “PATH_LOOSE”
no shutdown
exit[/php]First we will create a LSP to R1 that is not constrained at all:[php] lsp “LSP_R1-R2-U”
to 10.10.10.2
cspf
primary “PATH_LOOSE”
exit
no shutdown
exit[/php] and now a constrained path that is configured in the same manner, except that it requires the path to exclude links with the R1-R2 admin-group[php] lsp “LSP_R1-R2-C”
to 10.10.10.2
cspf
exclude “R1-R2”
primary “PATH_LOOSE”
exit
no shutdown
exit[/php]To compare the two different paths taken for each LSP:[php]*A:R1>config>router>mpls# show router mpls lsp “LSP_R1-R2-U” path detail | match Actual post-lines 100
Actual Hops :
10.1.2.1 (10.10.10.1) Record Label : N/A
-> 10.1.2.2 (10.10.10.2) Record Label : 262143
Computed Hops :
10.1.2.1(S)
-> 10.1.2.2(S)
Resignal Eligible: False
Last Resignal : n/a CSPF Metric : 100
===============================================================================
*A:R1>config>router>mpls# show router mpls lsp “LSP_R1-R2-C” path detail | match Actual post-lines 100
Actual Hops :
10.1.3.1 (10.10.10.1) Record Label : N/A
-> 10.1.3.3 (10.10.10.3) Record Label : 262143
-> 10.2.3.2 (10.10.10.2) Record Label : 262142
Computed Hops :
10.1.3.1(S)
-> 10.1.3.3(S)
-> 10.2.3.2(S)
Resignal Eligible: False
Last Resignal : n/a CSPF Metric : 200
===============================================================================[/php]The output shows that the constrained path avoided the R1-R2 link due to the admin-group “R1-R2” attached to the interface and the directive to exclude links with that admin-group. R2 can be used to confirm the entry of each LSP as well with the terminate directive:[php]*A:R2# show router mpls lsp terminate
===============================================================================
MPLS LSPs (Terminate)
===============================================================================
Legend : @ – Active Detour
===============================================================================
From To In I/F Out I/F State LSP Name
——————————————————————————-
10.10.10.1 10.10.10.2 1/1/2 n/a Up LSP_R1-R2-U::PATH_LOOSE
10.10.10.1 10.10.10.2 1/1/4 n/a Up LSP_R1-R2-C::PATH_LOOSE
——————————————————————————-
LSPs : 2
——————————————————————————-[/php]
This post has not touched upon secondary or standby paths, Fast Reroute, bandwidth reservation, set up or hold priority or a myriad of other features that may be considered within a production environment, however even when using admin-groups as a tool to steer traffic, the setup of LSPs can become quite tiresome in very short order, if you require a full-mesh of LSPs to be defined, each time you add a router to the network, you have to revist each router and build an LSP to the new one (similar to the IBGP problem before concepts like route reflectors reduced the configuration overhead) – larger configurations are more prone to errors and can make the time to deploy nodes more painful which may result in deciding that LDP signaled LSPs are a reasonable trade off.

LSP-Templates

LSP-Templates are one way to help take back the configuration overhead for initial builds, and to make it easier when adding nodes to an existing network. This example will be performed on R2. Define the a loose path with the name “PATH_LOOSE” as was done on R1 and create a Point-to-Point Mesh LSP-Template named AUTOLSP, that will use that path and be enabled for cspf:[php]configure
router
mpls
path “PATH_LOOSE”
no shutdown
exit
lsp-template “AUTOLSP” mesh-p2p
default-path “PATH_LOOSE”
cspf
no shutdown
exit
exit all[/php]To leverage the LSP-Template, we need to create a routing-policy which will be used to determine what nodes we will wish to establish a LSP with. In this case each nodes system address is in the 10.10.10.0/24 network, so we’ll create a match for host routes within that network using the prefix list and policy statement below.[php]configure
router
policy-options
begin
prefix-list “PL_MPLSNETWORK”
prefix 10.10.10.0/24 prefix-length-range 32-32
exit
policy-statement “PS_AUTOMESH_RSVP”
entry 10
from
prefix-list “PL_MPLSNETWORK”
exit
action accept
exit
exit
exit
commit
exit all[/php]The OSPF router LSA will be associated with the system addresses (since we haven’t changed the router ids):
[php]*A:R2# show router ospf database type router

===============================================================================
Rtr Base OSPFv2 Instance 0 Link State Database (type: Router)
===============================================================================
Type Area Id Link State Id Adv Rtr Id Age Sequence Cksum
——————————————————————————-
Router 0.0.0.0 10.10.10.1 10.10.10.1 1698 0x8000000a 0xaf19
Router 0.0.0.0 10.10.10.2 10.10.10.2 1114 0x8000000c 0xbb6a
Router 0.0.0.0 10.10.10.3 10.10.10.3 67 0x8000000d 0x1efb
Router 0.0.0.0 10.10.10.4 10.10.10.4 101 0x8000000b 0x4d5c
——————————————————————————-
No. of LSAs: 4
===============================================================================[/php]Back within the MPLS configuration context, auto-created lsps using the lsp-template AUTOLSP and the routing policy PS_AUTOMESH_RSVP will be enabled:[php]configure
router
mpls
auto-lsp lsp-template “AUTOLSP” policy “PS_AUTOMESH_RSVP”
exit
exit all[/php]Now to determine what LSPs have been automatically created:[php]*A:R2# show router mpls lsp auto-lsp mesh-p2p

===============================================================================
MPLS Auto-LSP
===============================================================================
LSP Name Type Fastfail Admin Oper
Config State State
——————————————————————————-
AUTOLSP-10.10.10.1-61453 MeshP2P Yes Up Up
AUTOLSP-10.10.10.3-61454 MeshP2P Yes Up Up
AUTOLSP-10.10.10.4-61455 MeshP2P Yes Up Up
——————————————————————————-
Auto-LSPs : 3
===============================================================================[/php]The LSP name is system derived from the lsp-template name, the destination router, and the tunnel id:[php]*A:R2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
——————————————————————————-
10.10.10.1/32 rsvp MPLS 61453 7 10.1.2.1 100
10.10.10.3/32 rsvp MPLS 61454 7 10.2.3.3 100
10.10.10.4/32 rsvp MPLS 61455 7 10.2.4.4 100
——————————————————————————-
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================[/php]The same configuration can be applied to R1, R3 and R4, when consolidated it looks like this:[php]configure
router
policy-options
begin
prefix-list “PL_MPLSNETWORK”
prefix 10.10.10.0/24 prefix-length-range 32-32
exit
policy-statement “PS_AUTOMESH_RSVP”
entry 10
from
prefix-list “PL_MPLSNETWORK”
exit
action accept
exit
exit
exit
commit
exit
mpls
path “PATH_LOOSE”
no shutdown
exit
lsp-template “AUTOLSP” mesh-p2p
default-path “PATH_LOOSE”
cspf
no shutdown
exit
auto-lsp lsp-template “AUTOLSP” policy “PS_AUTOMESH_RSVP”
exit
exit
exit all[/php]
What I really like about this, is that it brings configuration consistency, relatively small configurations that can do some good things, particularly if you have a well thought out deployment strategy – e.g. if you deploy another router, e.g. R5 with the system address of 10.10.10.5/32, R1-R4 will automatically attempt to establish LSPs to it which resulting in lower touch provisioning when adding new network elements to your network. LSP-Templates have been around on the 7×50 platform for some time, however it is only recently that the software on the 7705 SAR platforms have had this feature introduced allowing simplified RSVP-TE provisioning on large, medium and small platforms.

Please note that the specific LSP-Template that was used in this example is very simple and only used to demonstrate the basic concept – in a live deployment additional attributes such as fast-reroute, adspec and admin-groups if appropriate should be considered.

Carrier Supporting Carrier with SROS

Carrier of Carriers or Carrier supporting Carrier is a hierarchical construct to allow a network provider to provide MPLS connectivity with relatively low complexity. While it is even simpler to build layer 2 circuits over a backhaul network, in some cases routed connectivity may be preferred as you have a common connection point to the far end regions without needing to concern yourself with the provisioning scale and bandwidth management of each path. While this is quite a large posting it is mainly to show the configuration and verification steps. The summary of actions to create a CsC configuration is at the end of the post if you run out of patience.

This Example will have a three router Carrier of Carriers Network Operator (SC-1, SC-2 and SC-3) providing connectivity between a customer (VPRN 200) with two regions – Region A (RA-1, RA-2 and RA-3) and Region B (RB-1, RB-2 and RB-3)

9 Router Carrier Supporting Carrier Topology

Prior to SROS version 14.0R4, a single RIB was used for both labeled and unlabeled prefixes. With 14.0R4, two RIBs were established (actually, two RIBs for IPv4 and another two for IPv6) – this demonstration will have half the routers using 12.0R6 and half running 14.0R8 to highlight the configuration differences. The routers that are running SROS 14.0R8 in this example are configured using the VSR-D model which has the Control Plane Module operating in a VM (VM-CP) independent of the Input/Output Module (VM-DP) each VM-CP and VM-DP is logically connected via Switch Fabric ports – this is transparent to the configuration and operation of the simulated routers.

The initial configurations for Region A without the CSC Uplink on RA-1 as as below. OSPF as the IGP, LDP signalled LSPs are used, RA-3 is the local VPN Route Reflector for AS65000, VPRN 123 is created on each router.
[codegroup]
[php tab=”RA-1 Initial”]configure
system
name “RA-1”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “RA-2”
address 1.1.2.1/29
port 1/1/1
no shutdown
exit
interface “RA-3”
address 1.1.3.1/29
port 1/1/2
no shutdown
exit
interface “system”
address 1.1.1.1/32
no shutdown
exit
autonomous-system 65000
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “RA-2”
no shutdown
exit
interface “RA-3”
no shutdown
exit
exit
exit
ldp
interface-parameters
interface “RA-2”
exit
interface “RA-3”
exit
exit
no shutdown
exit
bgp
split-horizon
group “VPN-RR”
family vpn-ipv4
peer-as 65000
neighbor 1.1.1.3
description “RA-3”
exit
exit
no shutdown
exit
exit
service
vprn 123 customer 1 create
route-distinguisher 1.1.1.1:123
auto-bind ldp
vrf-target target:65000:123
interface “Loop” create
address 123.1.1.1/32
loopback
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”RA-2 Initial”]configure
system
name “RA-2”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “RA-1”
address 1.1.2.2/29
port 1/1/1
no shutdown
exit
interface “RA-3”
address 1.2.3.2/29
port 1/1/2
no shutdown
exit
interface “system”
address 1.1.1.2/32
no shutdown
exit
autonomous-system 65000
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “RA-1”
no shutdown
exit
interface “RA-3”
no shutdown
exit
exit
exit
ldp
interface-parameters
interface “RA-1”
exit
interface “RA-3”
exit
exit
no shutdown
exit
bgp
group “VPN-RR”
family vpn-ipv4
peer-as 65000
neighbor 1.1.1.3
description “RA-3”
exit
exit
no shutdown
exit
exit
service
vprn 123 customer 1 create
route-distinguisher 1.1.1.2:123
auto-bind ldp
vrf-target target:65000:123
interface “Loop” create
address 123.1.1.2/32
loopback
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”RA-3 (RR) Initial”]configure
system
name “RA-3”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “RA-1”
address 1.1.3.3/29
port 1/1/2
no shutdown
exit
interface “RA-2”
address 1.2.3.3/29
port 1/1/1
no shutdown
exit
interface “system”
address 1.1.1.3/32
no shutdown
exit
autonomous-system 65000
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “RA-1”
no shutdown
exit
interface “RA-2”
no shutdown
exit
exit
exit
ldp
interface-parameters
interface “RA-1”
exit
interface “RA-2”
exit
exit
exit
bgp
group “VPN-RRC”
family vpn-ipv4
cluster 1.1.1.3
peer-as 65000
neighbor 1.1.1.1
description “RA-1”
exit
neighbor 1.1.1.2
description “RA-2”
exit
exit
exit
exit
service
vprn 123 customer 1 create
route-distinguisher 1.1.1.3:123
auto-bind ldp
vrf-target target:65000:123
interface “Loop” create
address 123.1.1.3/32
loopback
exit
no shutdown
exit
exit
exit all[/php]
[/codegroup]
Similarly for Region B, the initial configurations without the CSC Uplink on RB-1 as as below. OSPF as the IGP, LDP signalled LSPs, RB-3 is the local VPN Route Reflector for AS65000, VPRN 123 is created on each router. Besides the card/mda differences on the emulated hardware, the primary difference here compared to Region A which is using an older SROS version is the LSP binding (auto-bind) syntax changes to give more control as to the LSP types that may be desired.
[codegroup]
[php tab=”RB-1 Initial”]configure
system
name “RB-1”
exit
sfm 1
sfm-type m-sfm5-12
no shutdown
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m10-1gb-xp-sfp
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “RB-2”
address 2.1.2.1/29
port 1/1/1
no shutdown
exit
interface “RB-3”
address 2.1.3.1/29
port 1/1/2
no shutdown
exit
interface “system”
address 2.2.2.1/32
no shutdown
exit
autonomous-system 65000
ospf 0
area 0.0.0.0
interface “system”
no shutdown
exit
interface “RB-2”
no shutdown
exit
interface “RB-3”
no shutdown
exit
exit
no shutdown
exit
ldp
interface-parameters
interface “RB-2” dual-stack
ipv4
fec-type-capability
prefix-ipv6 disable
p2mp-ipv6 disable
exit
no shutdown
exit
no shutdown
exit
interface “RB-3” dual-stack
ipv4
fec-type-capability
prefix-ipv6 disable
p2mp-ipv6 disable
exit
no shutdown
exit
no shutdown
exit
exit
no shutdown
exit
bgp
split-horizon
group “VPN-RR”
family vpn-ipv4
peer-as 65000
neighbor 2.2.2.3
description “RB-3”
exit
exit
no shutdown
exit
exit
service
vprn 123 customer 1 create
route-distinguisher 2.2.2.1:123
auto-bind-tunnel
resolution-filter
ldp
exit
resolution filter
exit
vrf-target target:65000:123
interface “Loop” create
address 123.2.2.1/32
loopback
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”RB-2 Initial”]configure
system
name “RB-2”
exit
sfm 1
sfm-type m-sfm5-12
no shutdown
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m10-1gb-xp-sfp
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “RB-1”
address 2.1.2.2/29
port 1/1/1
no shutdown
exit
interface “RB-3”
address 2.2.3.2/29
port 1/1/2
no shutdown
exit
interface “system”
address 2.2.2.2/32
no shutdown
exit
autonomous-system 65000
ospf 0
area 0.0.0.0
interface “system”
no shutdown
exit
interface “RB-1”
no shutdown
exit
interface “RB-3”
no shutdown
exit
exit
no shutdown
exit
ldp
interface-parameters
interface “RB-1” dual-stack
ipv4
fec-type-capability
prefix-ipv6 disable
p2mp-ipv6 disable
exit
no shutdown
exit
no shutdown
exit
interface “RB-3” dual-stack
ipv4
fec-type-capability
prefix-ipv6 disable
p2mp-ipv6 disable
exit
no shutdown
exit
no shutdown
exit
exit
no shutdown
exit
bgp
group “VPN-RR”
family vpn-ipv4
peer-as 65000
neighbor 2.2.2.3
description “RB-3”
exit
exit
no shutdown
exit
exit
service
vprn 123 customer 1 create
route-distinguisher 2.2.2.2:123
auto-bind-tunnel
resolution-filter
ldp
exit
resolution filter
exit
vrf-target target:65000:123
interface “Loop” create
address 123.2.2.2/32
loopback
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”RB-3 (RR) Initial”]configure
system
name “RB-3”
exit
sfm 1
sfm-type m-sfm5-12
no shutdown
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m10-1gb-xp-sfp
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “RB-1”
address 2.1.3.3/29
port 1/1/2
no shutdown
exit
interface “RB-2”
address 2.2.3.3/29
port 1/1/1
no shutdown
exit
interface “system”
address 2.2.2.3/32
no shutdown
exit
autonomous-system 65000
ospf 0
area 0.0.0.0
interface “system”
no shutdown
exit
interface “RB-1”
no shutdown
exit
interface “RB-2”
no shutdown
exit
exit
no shutdown
exit
ldp
interface-parameters
interface “RB-1” dual-stack
ipv4
fec-type-capability
prefix-ipv6 disable
p2mp-ipv6 disable
exit
no shutdown
exit
no shutdown
exit
interface “RB-2” dual-stack
ipv4
fec-type-capability
prefix-ipv6 disable
p2mp-ipv6 disable
exit
no shutdown
exit
no shutdown
exit
exit
no shutdown
exit
bgp
group “VPN-RRC”
family vpn-ipv4
cluster 2.2.2.3
peer-as 65000
neighbor 2.2.2.1
description “RB-1”
exit
neighbor 2.2.2.2
description “RB-2”
exit
exit
no shutdown
exit
exit
service
vprn 123 customer 1 create
route-distinguisher 2.2.2.3:123
auto-bind-tunnel
resolution-filter
ldp
exit
resolution filter
exit
vrf-target target:65000:123
interface “Loop” create
address 123.2.2.3/32
loopback
exit
no shutdown
exit
exit
exit all[/php]
[/codegroup]
The CSC Operators Network uses IS-IS as the IGP, RSVP-TE signalled LSPs, SC-R3 is the local VPN Route Reflector for AS64512.
[codegroup]
[php tab=”SC-R1 Initial”]configure
system
name “SC-R1”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “SC-R2”
address 10.1.2.1/29
port 1/1/1
no shutdown
exit
interface “SC-R3”
address 10.1.3.1/29
port 1/1/2
no shutdown
exit
interface “system”
address 10.10.10.1/32
no shutdown
exit
autonomous-system 64512
isis
level-capability level-2
area-id 49.0001
traffic-engineering
interface “system”
no shutdown
exit
interface “SC-R2”
no shutdown
exit
interface “SC-R3”
no shutdown
exit
no shutdown
exit
mpls
interface “system”
no shutdown
exit
interface “SC-R2”
no shutdown
exit
interface “SC-R3”
no shutdown
exit
path “LOOSE_HOPS”
no shutdown
exit
lsp “SC-R2”
to 10.10.10.2
cspf
adspec
fast-reroute facility
exit
primary “LOOSE_HOPS”
exit
no shutdown
exit
lsp “SC-R3”
to 10.10.10.3
cspf
adspec
fast-reroute facility
exit
primary “LOOSE_HOPS”
exit
no shutdown
exit
no shutdown
exit
rsvp
interface “system”
no shutdown
exit
interface “SC-R2”
no shutdown
exit
interface “SC-R3”
no shutdown
exit
no shutdown
exit
bgp
group “IBGP”
family vpn-ipv4
peer-as 64512
neighbor 10.10.10.3
description “SC-R3″
exit
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”SC-R2 Initial”]configure
system
name “SC-R2”
exit
sfm 1
sfm-type m-sfm5-12
no shutdown
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m10-1gb-xp-sfp
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “SC-R1”
address 10.1.2.2/29
port 1/1/1
no shutdown
exit
interface “SC-R3”
address 10.2.3.2/29
port 1/1/2
no shutdown
exit
interface “system”
address 10.10.10.2/32
no shutdown
exit
autonomous-system 64512
isis 0
level-capability level-2
area-id 49.0001
traffic-engineering
interface “system”
no shutdown
exit
interface “SC-R1”
no shutdown
exit
interface “SC-R3”
no shutdown
exit
no shutdown
exit
mpls
interface “system”
no shutdown
exit
interface “SC-R1”
no shutdown
exit
interface “SC-R3”
no shutdown
exit
path “LOOSE_HOPS”
no shutdown
exit
lsp “SC-R3”
to 10.10.10.3
cspf
adspec
fast-reroute facility
exit
primary “LOOSE_HOPS”
exit
no shutdown
exit
lsp “SC-R1”
to 10.10.10.1
cspf
adspec
fast-reroute facility
exit
primary “LOOSE_HOPS”
exit
no shutdown
exit
no shutdown
exit
rsvp
interface “system”
no shutdown
exit
interface “SC-R1”
no shutdown
exit
interface “SC-R3”
no shutdown
exit
no shutdown
exit
bgp
group “IBGP”
family vpn-ipv4
peer-as 64512
neighbor 10.10.10.3
description “SC-R3″
exit
exit
no shutdown
exit
exit
exit all[/php]
[php tab=”SC-R3 (RR) Initial”]configure
system
name “SC-R3”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “SC-R1”
address 10.1.3.3/29
port 1/1/1
no shutdown
exit
interface “SC-R2”
address 10.2.3.3/29
port 1/1/2
no shutdown
exit
interface “system”
address 10.10.10.3/32
no shutdown
exit
autonomous-system 64512
isis
level-capability level-2
area-id 49.0001
traffic-engineering
interface “system”
no shutdown
exit
interface “SC-R1”
no shutdown
exit
interface “SC-R2”
no shutdown
exit
no shutdown
exit
mpls
interface “system”
no shutdown
exit
interface “SC-R1”
no shutdown
exit
interface “SC-R2”
no shutdown
exit
path “LOOSE_HOPS”
no shutdown
exit
lsp “SC-R1”
to 10.10.10.1
cspf
adspec
fast-reroute facility
exit
primary “LOOSE_HOPS”
exit
no shutdown
exit
lsp “SC-R2”
to 10.10.10.2
cspf
adspec
fast-reroute facility
exit
primary “LOOSE_HOPS”
exit
no shutdown
exit
no shutdown
exit
rsvp
interface “system”
no shutdown
exit
interface “SC-R1”
no shutdown
exit
interface “SC-R2”
no shutdown
exit
no shutdown
exit
bgp
group “IBGP”
family vpn-ipv4
cluster 10.10.10.3
peer-as 64512
neighbor 10.10.10.1
description “SC-R1”
exit
neighbor 10.10.10.2
description “SC-R2”
exit
exit
no shutdown
exit
exit
exit all[/php][/codegroup]

At this stage, we have three Islands – Region A, Region B and the Carrier of Carriers Network (without the CSC configuration)

RA-1 is the Region A CSC-CE Router (which is also operating as a regular PE Router) which peers with the CSC Network SC-R1 to exchange labeled routes using BGP. Note: Both of these routers are running SROS 12.0R6 and are configured in the “old way”

Create VPRN 200 on SC-R1 and BGP Peering for RA-1:
[php]configure
router
policy-options
begin
policy-statement “PS_MPBGP_VPN_TO_BGP”
entry 10
from
protocol bgp-vpn
exit
action accept
exit
exit
default-action reject
exit
commit
exit
exit
port 1/1/3
ethernet
mode network
exit
no shutdown
exit
service
vprn 200 customer 1 create
carrier-carrier-vpn
autonomous-system 64512
route-distinguisher 10.10.10.1:200
auto-bind rsvp-te
vrf-target target:64512:200
network-interface “RA-1” create
address 10.1.1.11/24
port 1/1/3
no shutdown
exit
bgp
split-horizon
group “RegionA”
family ipv4
as-override
export “PS_MPBGP_VPN_TO_BGP”
peer-as 65000
neighbor 10.1.1.1
advertise-label ipv4
exit
exit
no shutdown
exit
no shutdown
exit
exit
exit all[/php]
Of note is the definition of the VPRN type: carrier-carrier-vpn, once enabled regular IP interfaces cannot be created, only “network-interfaces” which must be on Ethernet ports that are of mode network. As both Region A and Region B both use AS 65000, as-override has been used to stop AS-Path loops occuring when advertising prefixes from other regions with the same BGP AS. We are using the IPv4 Address family but are specifically advertising associated MPLS Labels (This particular configuration component is different after SROS Release 14.0R4) Routing policy is defined and used to export VPN routes to RA-1.

Create the BGP Peering for RA-1 with SC-R1:
[php]configure
port 1/1/3
ethernet
mode network
exit
no shutdown
exit
router
interface “SC-R1”
address 10.1.1.1/24
port 1/1/3
no shutdown
exit
policy-options
begin
prefix-list “RegionA”
prefix 1.1.1.0/24 prefix-length-range 32-32
exit
policy-statement “PS_BGP_EXP_CSC”
entry 10
from
prefix-list “RegionA”
exit
action accept
exit
exit
default-action reject
exit
commit
exit
bgp
group “CSC”
export “PS_BGP_EXP_CSC”
neighbor 10.1.1.11
description “SC-R1”
peer-as 64512
advertise-label ipv4 include-ldp-prefix
exit
exit
exit
exit all[/php]
Routing policy is used to advertise any system addresses from Region A (1.1.1.x/32) with MPLS labels to the CSC

RB-1 is the Region B CSC-CE Router (which is also operating as a regular PE Router) which peers with the CSC Network SC-R2 to exchange labeled routes using BGP. Note: Both of these routers are running SROS 14.0R8 and are configured in the “new way”

Create VPRN 200 on SC-R2 and BGP Peering for RB-1:
[php]configure
router
policy-options
begin
policy-statement “PS_MPBGP_VPN_TO_BGP”
entry 10
from
protocol bgp-vpn
exit
action accept
exit
exit
default-action reject
exit
commit
exit
exit
port 1/1/3
ethernet
mode network
exit
no shutdown
exit
service
vprn 200 customer 1 create
carrier-carrier-vpn
autonomous-system 64512
route-distinguisher 10.10.10.2:200
auto-bind-tunnel
resolution-filter
rsvp
exit
resolution filter
exit
vrf-target target:64512:200
network-interface “RB-1” create
address 10.2.2.11/24
port 1/1/3
no shutdown
exit
bgp
split-horizon
group “RB”
family label-ipv4
as-override
export “PS_MPBGP_VPN_TO_BGP”
peer-as 65000
neighbor 10.2.2.1
exit
exit
no shutdown
exit
no shutdown
exit
exit
exit all[/php]
The main difference compared with the configuration on SC-1 besides the auto-bind-tunnel changes, are that we are specifically peering using the label-ipv4 address family instead of the ipv4 address family and advertising labels.

Create the BGP Peering for RB-1 with SC-R2:
[php]configure
port 1/1/3
ethernet
mode network
exit
no shutdown
exit
router
interface “SC-R2”
address 10.2.2.1/24
port 1/1/3
no shutdown
exit
policy-options
begin
prefix-list “RegionB”
prefix 2.2.2.0/24 prefix-length-range 32-32
exit
policy-statement “PS_BGP_EXP_CSC”
entry 10
from
prefix-list “RegionB”
exit
action accept
exit
exit
default-action reject
exit
commit
exit
bgp
group “CSC”
export “PS_BGP_EXP_CSC”
neighbor 10.2.2.11
description “SC-R2″
family label-ipv4
peer-as 64512
advertise-ldp-prefix
exit
exit
exit
exit all[/php]
Routing policy is used to advertise any system addresses from Region B (2.2.2.x/32) with MPLS labels to the CSC

Once all peers are configured, examine the CSC VPRN 200 Routing Tables:
[codegroup][php tab=”SC-R1 VPRN 200″]A:SC-R1# show router 200 route-table

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
1.1.1.1/32 Remote BGP 00h07m55s 170
10.1.1.1 0
1.1.1.2/32 Remote BGP 00h07m55s 170
10.1.1.1 0
1.1.1.3/32 Remote BGP 00h07m55s 170
10.1.1.1 0
2.2.2.1/32 Remote BGP VPN 00h07m28s 170
10.10.10.2 (tunneled:RSVP:1) 0
2.2.2.2/32 Remote BGP VPN 00h07m28s 170
10.10.10.2 (tunneled:RSVP:1) 0
2.2.2.3/32 Remote BGP VPN 00h07m28s 170
10.10.10.2 (tunneled:RSVP:1) 0
10.1.1.0/24 Local Local 00h08m29s 0
RA-1 0
10.2.2.0/24 Remote BGP VPN 00h07m28s 170
10.10.10.2 (tunneled:RSVP:1) 0
——————————————————————————-
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================[/php]
[php tab=”SC-R2 VPRN 200″]A:SC-R2# show router 200 route-table

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
1.1.1.1/32 Remote BGP VPN 00h07m20s 170
10.10.10.1 (tunneled:RSVP:2) 0
1.1.1.2/32 Remote BGP VPN 00h07m20s 170
10.10.10.1 (tunneled:RSVP:2) 0
1.1.1.3/32 Remote BGP VPN 00h07m20s 170
10.10.10.1 (tunneled:RSVP:2) 0
2.2.2.1/32 Remote BGP_LABEL 02h02m23s 170
10.2.2.1 0
2.2.2.2/32 Remote BGP_LABEL 02h02m23s 170
10.2.2.1 0
2.2.2.3/32 Remote BGP_LABEL 02h02m23s 170
10.2.2.1 0
10.1.1.0/24 Remote BGP VPN 00h07m20s 170
10.10.10.1 (tunneled:RSVP:2) 0
10.2.2.0/24 Local Local 02h11m42s 0
RB-1 0
——————————————————————————-
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================[/php][/codegroup]

With Region A and Region B connectivity via the CSC we should be able to see the initial list of tunnels on the CSC-CE routers (RA-1 and RA-2):[codegroup]
[php tab=”RA-1 (CSC-CE) Tunnel Table”]A:RA-1# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
——————————————————————————-
1.1.1.2/32 ldp MPLS – 9 1.1.2.2 100
1.1.1.3/32 ldp MPLS – 9 1.1.3.3 100
2.2.2.1/32 bgp MPLS – 10 10.1.1.11 1000
2.2.2.2/32 bgp MPLS – 10 10.1.1.11 1000
2.2.2.3/32 bgp MPLS – 10 10.1.1.11 1000
——————————————————————————-
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================[/php]

[php tab=”RB-1 (CSC-CE) Tunnel Table”]A:RB-1# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
——————————————————————————-
1.1.1.1/32 ldp MPLS – 9 1.1.2.1 100
1.1.1.3/32 ldp MPLS – 9 1.2.3.3 100
——————————————————————————-
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================[/php]
[php tab=”RA-3 Tunnel Table”]A:RA-3# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
——————————————————————————-
1.1.1.1/32 ldp MPLS – 9 1.1.3.1 100
1.1.1.2/32 ldp MPLS – 9 1.2.3.2 100
——————————————————————————-
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================[/php]
[php tab=”RB-2 Tunnel Table”]A:RB-2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
——————————————————————————-
2.2.2.1/32 ldp MPLS 65597 9 2.1.3.1 100
2.2.2.2/32 ldp MPLS 65537 9 2.2.3.2 100
——————————————————————————-
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================[/php][/codegroup]
RA-2, RA-3, RB-2 and RB-3 need the CSC-CEs to to redistribute the prefixes we learnt from the CSC into OSPF and LDP.[codegroup]
[php tab=”RA-1 (CSC-CE) Redistribution”]configure
router
policy-options
begin
prefix-list “RegionB”
prefix 2.2.2.0/24 prefix-length-range 32-32
exit
policy-statement “PS_OSPF_LDP_EXP_RB”
entry 10
from
prefix-list “RegionB”
exit
action accept
exit
exit
default-action reject
exit
commit
exit
ospf
asbr
export “PS_OSPF_LDP_EXP_RB”
exit
ldp
export-tunnel-table “PS_OSPF_LDP_EXP_RB”
exit
exit
exit all[/php]
[php tab=”RB-1 (CSC-CE) Redistribution”]configure
router
policy-options
begin
prefix-list “RegionA”
prefix 1.1.1.0/24 prefix-length-range 32-32
exit
policy-statement “PS_OSPF_LDP_EXP_RA”
entry 10
from
prefix-list “RegionA”
exit
action accept
exit
exit
default-action drop
exit
exit
commit
exit
ospf
asbr
export “PS_OSPF_LDP_EXP_RA”
exit
ldp
export-tunnel-table “PS_OSPF_LDP_EXP_RA”
exit[/php][/codegroup]Remember: SROS specifically requires OSPF in the GRT to be defined as an ASBR if you intend to export routes.
[codegroup][php tab=”RA-2 Tunnel Table”]A:RA-2# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
——————————————————————————-
1.1.1.1/32 ldp MPLS – 9 1.1.2.1 100
1.1.1.3/32 ldp MPLS – 9 1.2.3.3 100
2.2.2.1/32 ldp MPLS – 9 1.1.2.1 1
2.2.2.2/32 ldp MPLS – 9 1.1.2.1 1
2.2.2.3/32 ldp MPLS – 9 1.1.2.1 1
——————————————————————————-
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================[/php]
[php tab=”RA-3 Tunnel Table”]A:RA-3# show router tunnel-table

===============================================================================
Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
——————————————————————————-
1.1.1.1/32 ldp MPLS – 9 1.1.3.1 100
1.1.1.2/32 ldp MPLS – 9 1.2.3.2 100
2.2.2.1/32 ldp MPLS – 9 1.1.3.1 1
2.2.2.2/32 ldp MPLS – 9 1.1.3.1 1
2.2.2.3/32 ldp MPLS – 9 1.1.3.1 1
——————————————————————————-
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================[/php]
[php tab=”RB-2 Tunnel Table”]A:RB-2# show router tunnel-table

===============================================================================
IPv4 Tunnel Table (Router: Base)
===============================================================================
Destination Owner Encap TunnelId Pref Nexthop Metric
——————————————————————————-
1.1.1.1/32 ldp MPLS 65620 9 2.1.3.1 1
1.1.1.2/32 ldp MPLS 65621 9 2.1.3.1 1
1.1.1.3/32 ldp MPLS 65622 9 2.1.3.1 1
2.2.2.1/32 ldp MPLS 65597 9 2.1.3.1 100
2.2.2.2/32 ldp MPLS 65537 9 2.2.3.2 100
——————————————————————————-
Flags: B = BGP backup route available
E = inactive best-external BGP route
===============================================================================[/php][/codegroup]Okay, we have tunnels between all routers in both Regions A and B.
The final step to get VPRN connectivity between Region A and Region B is to peer the VPN Route Reflectors (RA-3 and RB-3)[codegroup][php tab=”RA-3 RR Peering”]configure
bgp
group “VPN-RR-RegionB”
family vpn-ipv4
peer-as 65000
neighbor 2.2.2.3
description “RB-3″
exit
exit
exit
exit all[/php]
[php tab=”RB-3 RR Peering”]configure
bgp
group “VPN-RR-RegionA”
family vpn-ipv4
peer-as 65000
neighbor 1.1.1.3
description “RA-3″
exit
exit
exit
exit all[/php][/codegroup]
We should be able to now verify this with VPRN 123
[codegroup][php tab=”RA-1 Verify”]
A:RA-1# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
123.1.1.1/32 Local Local 00h50m13s 0
Loop 0
123.1.1.2/32 Remote BGP VPN 00h44m50s 170
1.1.1.2 (tunneled) 0
123.1.1.3/32 Remote BGP VPN 00h45m49s 170
1.1.1.3 (tunneled) 0
123.2.2.1/32 Remote BGP VPN 00h00m26s 170
2.2.2.1 (tunneled:BGP) 0
123.2.2.2/32 Remote BGP VPN 00h00m26s 170
2.2.2.2 (tunneled:BGP) 0
123.2.2.3/32 Remote BGP VPN 00h00m26s 170
2.2.2.3 (tunneled:BGP) 0
——————————————————————————-
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
A:RA-1# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=0.245ms.

—- 123.1.1.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.245ms, avg = 0.245ms, max = 0.245ms, stddev = 0.000ms
A:RA-1# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=1.26ms.

—- 123.1.1.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.26ms, avg = 1.26ms, max = 1.26ms, stddev = 0.000ms
A:RA-1# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=1.37ms.

—- 123.1.1.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.37ms, avg = 1.37ms, max = 1.37ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=3.46ms.

—- 123.2.2.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.46ms, avg = 3.46ms, max = 3.46ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=3.92ms.

—- 123.2.2.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.92ms, avg = 3.92ms, max = 3.92ms, stddev = 0.000ms
A:RA-1# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=3.83ms.

—- 123.2.2.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.83ms, avg = 3.83ms, max = 3.83ms, stddev = 0.000ms[/php]
[php tab=”RA-2 Verify”]A:RA-2# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
123.1.1.1/32 Remote BGP VPN 00h46m12s 170
1.1.1.1 (tunneled) 0
123.1.1.2/32 Local Local 00h48m34s 0
Loop 0
123.1.1.3/32 Remote BGP VPN 00h46m12s 170
1.1.1.3 (tunneled) 0
123.2.2.1/32 Remote BGP VPN 00h01m31s 170
2.2.2.1 (tunneled) 0
123.2.2.2/32 Remote BGP VPN 00h01m31s 170
2.2.2.2 (tunneled) 0
123.2.2.3/32 Remote BGP VPN 00h01m31s 170
2.2.2.3 (tunneled) 0
——————————————————————————-
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
A:RA-2# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=1.12ms.

—- 123.1.1.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.12ms, avg = 1.12ms, max = 1.12ms, stddev = 0.000ms
A:RA-2# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=0.171ms.

—- 123.1.1.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.171ms, avg = 0.171ms, max = 0.171ms, stddev = 0.000ms
A:RA-2# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=1.31ms.

—- 123.1.1.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.31ms, avg = 1.31ms, max = 1.31ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=4.12ms.

—- 123.2.2.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.12ms, avg = 4.12ms, max = 4.12ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=7.76ms.

—- 123.2.2.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 7.76ms, avg = 7.76ms, max = 7.76ms, stddev = 0.000ms
A:RA-2# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=4.53ms.

—- 123.2.2.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.53ms, avg = 4.53ms, max = 4.53ms, stddev = 0.000ms[/php]
[php tab=”RA-3 Verify”]A:RA-3# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
123.1.1.1/32 Remote BGP VPN 00h47m48s 170
1.1.1.1 (tunneled) 0
123.1.1.2/32 Remote BGP VPN 00h46m56s 170
1.1.1.2 (tunneled) 0
123.1.1.3/32 Local Local 00h49m05s 0
Loop 0
123.2.2.1/32 Remote BGP VPN 00h02m37s 170
2.2.2.1 (tunneled) 0
123.2.2.2/32 Remote BGP VPN 00h02m37s 170
2.2.2.2 (tunneled) 0
123.2.2.3/32 Remote BGP VPN 00h02m37s 170
2.2.2.3 (tunneled) 0
——————————————————————————-
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
A:RA-3>config>router# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=1.17ms.

—- 123.1.1.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.17ms, avg = 1.17ms, max = 1.17ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=1.07ms.

—- 123.1.1.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.07ms, avg = 1.07ms, max = 1.07ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=0.166ms.

—- 123.1.1.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.166ms, avg = 0.166ms, max = 0.166ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=6.13ms.

—- 123.2.2.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 6.13ms, avg = 6.13ms, max = 6.13ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=5.04ms.

—- 123.2.2.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.04ms, avg = 5.04ms, max = 5.04ms, stddev = 0.000ms
A:RA-3>config>router# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=4.37ms.

—- 123.2.2.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.37ms, avg = 4.37ms, max = 4.37ms, stddev = 0.000ms[/php]
[php tab=”RB-1 Verify”]A:RB-1# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
123.1.1.1/32 Remote BGP VPN 00h03m33s 170
1.1.1.1 (tunneled:BGP) 0
123.1.1.2/32 Remote BGP VPN 00h03m33s 170
1.1.1.2 (tunneled:BGP) 0
123.1.1.3/32 Remote BGP VPN 00h03m33s 170
1.1.1.3 (tunneled:BGP) 0
123.2.2.1/32 Local Local 02h35m54s 0
Loop 0
123.2.2.2/32 Remote BGP VPN 02h11m21s 170
2.2.2.2 (tunneled) 0
123.2.2.3/32 Remote BGP VPN 02h11m21s 170
2.2.2.3 (tunneled) 0
——————————————————————————-
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
A:RB-1# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=4.42ms.

—- 123.1.1.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.42ms, avg = 4.42ms, max = 4.42ms, stddev = 0.000ms
A:RB-1# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=3.89ms.

—- 123.1.1.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.89ms, avg = 3.89ms, max = 3.89ms, stddev = 0.000ms
A:RB-1# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=5.97ms.

—- 123.1.1.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.97ms, avg = 5.97ms, max = 5.97ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=0.118ms.

—- 123.2.2.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.118ms, avg = 0.118ms, max = 0.118ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=1.96ms.

—- 123.2.2.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.96ms, avg = 1.96ms, max = 1.96ms, stddev = 0.000ms
A:RB-1# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=1.80ms.

—- 123.2.2.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.80ms, avg = 1.80ms, max = 1.80ms, stddev = 0.000ms[/php]
[php tab=”RB-2 Verify”]A:RB-2# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
123.1.1.1/32 Remote BGP VPN 00h04m13s 170
1.1.1.1 (tunneled) 0
123.1.1.2/32 Remote BGP VPN 00h04m13s 170
1.1.1.2 (tunneled) 0
123.1.1.3/32 Remote BGP VPN 00h04m13s 170
1.1.1.3 (tunneled) 0
123.2.2.1/32 Remote BGP VPN 02h11m55s 170
2.2.2.1 (tunneled) 0
123.2.2.2/32 Local Local 03h04m35s 0
Loop 0
123.2.2.3/32 Remote BGP VPN 02h36m00s 170
2.2.2.3 (tunneled) 0
——————————————————————————-
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
A:RB-2# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=3.43ms.

—- 123.1.1.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.43ms, avg = 3.43ms, max = 3.43ms, stddev = 0.000ms
A:RB-2# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=5.63ms.

—- 123.1.1.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 5.63ms, avg = 5.63ms, max = 5.63ms, stddev = 0.000ms
A:RB-2# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=4.40ms.

—- 123.1.1.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.40ms, avg = 4.40ms, max = 4.40ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=1.85ms.

—- 123.2.2.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.85ms, avg = 1.85ms, max = 1.85ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=0.150ms.

—- 123.2.2.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.150ms, avg = 0.150ms, max = 0.150ms, stddev = 0.000ms
A:RB-2# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=1.90ms.

—- 123.2.2.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.90ms, avg = 1.90ms, max = 1.90ms, stddev = 0.000ms[/php]
[php tab=”RB-3 Verify”]A:RB-3# show router 123 route-table

===============================================================================
Route Table (Service: 123)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
123.1.1.1/32 Remote BGP VPN 00h05m23s 170
1.1.1.1 (tunneled) 0
123.1.1.2/32 Remote BGP VPN 00h05m23s 170
1.1.1.2 (tunneled) 0
123.1.1.3/32 Remote BGP VPN 00h05m23s 170
1.1.1.3 (tunneled) 0
123.2.2.1/32 Remote BGP VPN 02h13m05s 170
2.2.2.1 (tunneled) 0
123.2.2.2/32 Remote BGP VPN 02h37m03s 170
2.2.2.2 (tunneled) 0
123.2.2.3/32 Local Local 02h37m07s 0
Loop 0
——————————————————————————-
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
A:RB-3# ping router 123 123.1.1.1 count 1
PING 123.1.1.1 56 data bytes
64 bytes from 123.1.1.1: icmp_seq=1 ttl=64 time=3.83ms.

—- 123.1.1.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.83ms, avg = 3.83ms, max = 3.83ms, stddev = 0.000ms
A:RB-3# ping router 123 123.1.1.2 count 1
PING 123.1.1.2 56 data bytes
64 bytes from 123.1.1.2: icmp_seq=1 ttl=64 time=4.38ms.

—- 123.1.1.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.38ms, avg = 4.38ms, max = 4.38ms, stddev = 0.000ms
A:RB-3# ping router 123 123.1.1.3 count 1
PING 123.1.1.3 56 data bytes
64 bytes from 123.1.1.3: icmp_seq=1 ttl=64 time=4.47ms.

—- 123.1.1.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 4.47ms, avg = 4.47ms, max = 4.47ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.1 count 1
PING 123.2.2.1 56 data bytes
64 bytes from 123.2.2.1: icmp_seq=1 ttl=64 time=1.97ms.

—- 123.2.2.1 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.97ms, avg = 1.97ms, max = 1.97ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.2 count 1
PING 123.2.2.2 56 data bytes
64 bytes from 123.2.2.2: icmp_seq=1 ttl=64 time=1.88ms.

—- 123.2.2.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.88ms, avg = 1.88ms, max = 1.88ms, stddev = 0.000ms
A:RB-3# ping router 123 123.2.2.3 count 1
PING 123.2.2.3 56 data bytes
64 bytes from 123.2.2.3: icmp_seq=1 ttl=64 time=0.269ms.

—- 123.2.2.3 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 0.269ms, avg = 0.269ms, max = 0.269ms, stddev = 0.000ms[/php][/codegroup]While this is quite a long post, the actual configuration itself for the CSC is straight forward:

The CSC VPRN must be set to carrier-carrier-vpn
The CSC VPRN uses Ethernet ports with mode network even though its for customers
The CSC-PE and CSC-CE exchange labeled routes
The CSC-CE redistributes between OSPF/LDP and BGP

Changes to the way the RIB works on SROS and how that impacts CsC configurations was also briefly discussed.

Resolving OSPF MTU problems with SROS

OSPF is a popular Interior Gateway Routing Protocol and in many instances it “just works” for a lot of situations, however care must be taken even in simple deployments. An issue that comes up from time to time is with regards to the maximum transmission unit (MTU). The network topology is a three router topology where I only have direct control a Nokia SROS based router.

TL;DR – OSPF neighbor in ExchStart – you need to increase your MTU, OSPF neighbor in Exchange – you need to decrease your MTU. Keep reading to see how you can identify and resolve the MTU issues on Nokia Routers with SROS.

Below is the configuration of SR (the router under our administrative control):
[php]configure
system
name “SR”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
mode access
exit
no shutdown
exit
port 1/1/2
ethernet
mode access
exit
no shutdown
exit
#————————————————–
echo “Router (Network Side) Configuration”
#————————————————–
router Base
interface “system”
address 1.1.1.1/32
no shutdown
exit
#————————————————–
echo “OSPFv2 Configuration”
#————————————————–
ospf 0
area 0.0.0.0
interface “system”
no shutdown
exit
exit
no shutdown
exit
exit

#————————————————–
echo “Service Configuration”
#————————————————–
service
customer 1 create
description “Default customer”
exit
ies 100 customer 1 create
description “PEER1”
interface “PEER1” create
address 10.1.2.1/27
sap 1/1/1 create
exit
exit
no shutdown
exit
ies 200 customer 1 create
description “PEER2”
interface “PEER2” create
address 10.1.3.1/27
sap 1/1/2 create
exit
exit
no shutdown
exit
exit
#————————————————–
echo “Router (Service Side) Configuration”
#————————————————–
router
ospf 0
area 0.0.0.1
interface “PEER1”
no shutdown
exit
interface “PEER2″
no shutdown
exit
exit
no shutdown
exit
exit
exit all
[/php]

One thing to note is that the Peer routers are attached to an Internet Enhanced Service (IES) and not part of the OSPF Backbone Area – from a stored configuration perspective there is a distinction between core network and customer configurations but from a protocol pespective things are the same. IES Interfaces that are bound to Service Access Points (SAPs) which must be changed from the default mode of network – in this case we are using access, however hybrid is an option as well.

As this post is about resolving issues, obviously things are not working as straight forward as expected.
[php]A:SR# show router route-table

—- 10.1.2.2 PING Statistics —-
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.08ms, avg = 1.17ms, max = 1.28ms, stddev = 0.081ms
A:SR# ping 10.1.3.3 count 3
PING 10.1.3.3 56 data bytes
64 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.51ms.
64 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.31ms.
64 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.24ms.

—- 10.1.3.3 PING Statistics —-
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.24ms, avg = 1.35ms, max = 1.51ms, stddev = 0.116ms[/php]Okay so IP connectivity is established, lets check the OSPF interface state
[php highlight=”10″]A:SR# show router ospf interface

===============================================================================
Rtr Base OSPFv2 Instance 0 Interfaces
===============================================================================
If Name Area Id Designated Rtr Bkup Desig Rtr Adm Oper
——————————————————————————-
system 0.0.0.0 1.1.1.1 0.0.0.0 Up DR
PEER1 0.0.0.1 1.1.1.1 100.100.100.100 Up DR
PEER2 0.0.0.1 1.1.1.1 0.0.0.0 Up DR
——————————————————————————-
No. of OSPF Interfaces: 3
===============================================================================[/php]
From first glance PEER1 seems okay but PEER2 doesn’t have a BDR and since we are using the default ospf interface type (broadcast) we would expect that to see both the DR and BDR – lets get some more details
[php highlight=”42,44”]A:SR# show router ospf interface “PEER2” detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Interface “PEER2” (detail)
===============================================================================
——————————————————————————-
Configuration
——————————————————————————-
IP Address : 10.1.3.1
Area Id : 0.0.0.1 Priority : 1
Hello Intrvl : 10 sec Rtr Dead Intrvl : 40 sec
Retrans Intrvl : 5 sec Poll Intrvl : 120 sec
Cfg Metric : 0 Advert Subnet : True
Transit Delay : 1 Cfg IF Type : None
Passive : False Cfg MTU : 0
LSA-filter-out : None Adv Rtr Capab : Yes
LFA : Include LFA NH Template :
RIB-priority : None
Auth Type : None
——————————————————————————-
State
——————————————————————————-
Admin Status : Enabled Oper State : Designated Rtr
Designated Rtr : 1.1.1.1 Backup Desig Rtr : 0.0.0.0
IF Type : Broadcast Network Type : Stub
Oper MTU : 1500 Last Enabled : 06/02/2017 01:44:23
Oper Metric : 100 Bfd Enabled : No
Te Metric : 100 Te State : Down
Admin Groups : None
Ldp Sync : outOfService Ldp Sync Wait : Disabled
Ldp Timer State : Disabled Ldp Tm Left : 0
——————————————————————————-
Statistics
——————————————————————————-
Nbr Count : 0 If Events : 2
Tot Rx Packets : 0 Tot Tx Packets : 76
Rx Hellos : 0 Tx Hellos : 76
Rx DBDs : 0 Tx DBDs : 0
Rx LSRs : 0 Tx LSRs : 0
Rx LSUs : 0 Tx LSUs : 0
Rx LS Acks : 0 Tx LS Acks : 0
Retransmits : 0 Discards : 78
Bad Networks : 0 Bad Virt Links : 0
Bad Areas : 78 Bad Dest Addrs : 0
Bad Auth Types : 0 Auth Failures : 0
Bad Neighbors : 0 Bad Pkt Types : 0
Bad Lengths : 0 Bad Hello Int. : 0
Bad Dead Int. : 0 Bad Options : 0
Bad Versions : 0 Bad Checksums : 0
LSA Count : 0 LSA Checksum : 0x0
===============================================================================[/php]Okay, we can see that there are discards which align with the Bad Area Count – this means that PEER2 doesn’t believe it’s part of OSPF Area 1.

Log-id 99 is automatically configured on Nokia SROS devices to capture a number of event messages however it can get a bit overwhelming to find something specific. Fortunately there are ways to reduce the output by specifying the application (OSPF) and something that may be part of the log message itself we want to see (PEER2)
[php]A:SR# show log log-id 99 application OSPF message PEER2

941 2017/06/02 02:10:15.55 UTC WARNING: OSPF #2043 Base VR: 1 OSPFv2 (0)
“LCL_RTR_ID 1.1.1.1: Conflicting configuration areaMismatch on interface PEER2 from 10.1.3.3 in hello”[/php]So while we have identified a problem – OSPF Area MisMatch, we need to overcome it – remembering we cant configure PEER2 (the person that manages it is on a training course and cannot be contacted, while your project manager is wanting solutions, not problems..)

This is where using show and debug commands can help identify and resolve issues – SROS is quite powerful with its debugging tools and while they can be used in production, it is always best to attempt to narrow down what you are attempting to collect – firstly we need to create a debug log if one doesn’t already exist – for this example I’m just logging to a circular memory buffer but it could go to SNMP, syslog or a file if necessary.
[php]A:SR# configure log log-id 10
*A:SR>config>log>log-id$ from debug-trace
*A:SR>config>log>log-id$ to memory
*A:SR>config>log>log-id$ no shutdown
*A:SR>config>log>log-id$ back
*A:SR>config>log# info
———————————————-
log-id 10
from debug-trace
to memory
no shutdown
exit
———————————————-[/php]Now to set up the debug – we know it’s from interface PEER2 and the log message kindly told us the packet type (in hello)..
[php]*A:SR>config>log# /debug router ospf packet hello “PEER2”
*A:SR>config>log# show debug
debug
router “Base”
ospf
packet hello “PEER2″
exit
exit
exit[/php]Router Base is the global routing table of the router, the debug can reference other services e.g. a VPRN if necessary by changing the router – After a few seconds (OSPF hello packets will come every 10 seconds or so) we can look in log 10 to see what was received.
[php highlight=”15,24,32”]*A:SR>config>log# show log log-id 10

===============================================================================
Event Log 10
===============================================================================
Description : (Not Specified)
Memory Log contents [size=100 next event=10 (not wrapped)]

9 2017/06/02 02:19:27.10 UTC MINOR: DEBUG #2001 Base OSPFv2
“OSPFv2: PKT

>> Outgoing OSPF packet on I/F PEER2 area 0.0.0.1
OSPF Version : 2
Router Id : 1.1.1.1
Area Id : 0.0.0.1
Checksum : ecb9
Auth Type : Null
Auth Key : 00 00 00 00 00 00 00 00
Packet Type : HELLO
Packet Length : 44 ”

8 2017/06/02 02:19:26.55 UTC MINOR: DEBUG #2001 Base OSPFv2
“OSPFv2: PKT DROPPED
area mismatch”

7 2017/06/02 02:19:26.54 UTC MINOR: DEBUG #2001 Base OSPFv2
“OSPFv2: PKT

>> Incoming OSPF packet on I/F PEER2 area 0.0.0.2
OSPF Version : 2
Router Id : 200.200.200.200
Area Id : 0.0.0.2
Checksum : 5d27
Auth Type : Null
Auth Key : 00 00 00 00 00 00 00 00
Packet Type : HELLO
Packet Length : 44 “[/php]SR is configured with PEER2 in Area 1 but it should be in Area 2, lets fix that
[php]*A:SR>config>log# /configure router ospf
*A:SR>config>router>ospf# info
———————————————-
area 0.0.0.0
interface “system”
no shutdown
exit
exit
area 0.0.0.1
interface “PEER1”
no shutdown
exit
interface “PEER2”
no shutdown
exit
exit
no shutdown
———————————————-
*A:SR>config>router>ospf# area 1 interface “PEER2” shutdown
*A:SR>config>router>ospf# area 1 no interface “PEER2”
*A:SR>config>router>ospf# area 2 interface “PEER2” no shutdown
*A:SR>config>router>ospf# info
———————————————-
area 0.0.0.0
interface “system”
no shutdown
exit
exit
area 0.0.0.1
interface “PEER1”
no shutdown
exit
exit
area 0.0.0.2
interface “PEER2″
no shutdown
exit
exit
no shutdown
———————————————-[/php]Now see if that fixes that problem.
[php]*A:SR>config>router>ospf# show router ospf interface

===============================================================================
Rtr Base OSPFv2 Instance 0 Interfaces
===============================================================================
If Name Area Id Designated Rtr Bkup Desig Rtr Adm Oper
——————————————————————————-
system 0.0.0.0 1.1.1.1 0.0.0.0 Up DR
PEER1 0.0.0.1 1.1.1.1 100.100.100.100 Up DR
PEER2 0.0.0.2 200.200.200.200 1.1.1.1 Up BDR
——————————————————————————-
No. of OSPF Interfaces: 3
===============================================================================[/php]Yes we can see both the DR and BDR for our OSPF peers but before we move on, we should stop the debug activity
[php]*A:SR>config>router>ospf# /debug router no ospf
*A:SR>config>router>ospf# show debug
debug
exit[/php]Now lets see if OSPF routing exchange is occurring.
[php]*A:SR>config>router>ospf# show router route-table

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors
===============================================================================
Interface-Name Rtr Id State Pri RetxQ TTL
Area-Id
——————————————————————————-
PEER1 100.100.100.100 ExchStart 1 0 34
0.0.0.1
PEER2 200.200.200.200 Exchange 1 0 32
0.0.0.2
——————————————————————————-
No. of Neighbors: 2
===============================================================================[/php]A router that is stuck in ExchStart or Exchange is a hallmark of OSPF MTU related problems.
Let’s start working on PEER1.[php highlight=”16,23”]*A:SR>config>router>ospf# show router ospf neighbor “PEER1” detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface “PEER1” (detail)
===============================================================================
——————————————————————————-
Neighbor : 10.1.2.2
——————————————————————————-
——————————————————————————-
Neighbor Rtr Id : 100.100.100.100 Interface: PEER1
——————————————————————————-
Neighbor IP Addr : 10.1.2.2
Local IF IP Addr : 10.1.2.1
Area Id : 0.0.0.1
Designated Rtr : 1.1.1.1 Backup Desig Rtr : 100.100.100.100
Neighbor State : ExchStart Priority : 1
Retrans Q Length : 0 Options : – E – – – – – —
Events : 1068 Last Event Time : 06/02/2017 02:59:34
Up Time : 0d 01:11:00 Time Before Dead : 38 sec
GR Helper : Not Helping GR Helper Age : 0 sec
GR Exit Reason : None GR Restart Reason: Unknown (0)
Bad Nbr States : 0 LSA Inst fails : 0
Bad Seq Nums : 0 Bad MTUs : 1066
Bad Packets : 0 LSA not in LSDB : 0
Option Mismatches: 0 Nbr Duplicates : 0
Num Restarts : 0 Last Restart at : Never
===============================================================================[/php]There are quite a few Bad MTUs being reported – While some vendors have an option to ignore the OSPF MTU, there are quite a number of MTU implications that can occur within the core when you consider various tunnel options that this is not provided.

Before we start to change things lets see if our trusty log 99 to see says anything about this:[php]*A:SR>config>router>ospf# show log log-id 99 application OSPF message PEER1

1472 2017/06/02 02:40:17.67 UTC WARNING: OSPF #2043 Base VR: 1 OSPFv2 (0)
“LCL_RTR_ID 1.1.1.1: Conflicting configuration mtuMismatch on interface PEER1 from 10.1.2.2 in dbDescript”[/php]We can use another debug to determine what the actual MTU should be (as before with the area mismatch, log 99 gave us a hint as to the packet type we should be investigating):
[php]*A:SR>config>router>ospf# /debug router ospf packet dbdescr ingress “PEER1″[/php]
Clear the log and see what we are recieving:
[php highlight=”27”]*A:SR>config>router>ospf# /clear log 10
*A:SR>config>router>ospf# /show log log-id 10

2 2017/06/02 02:55:17.67 UTC MINOR: DEBUG #2001 Base OSPFv2
“OSPFv2: PKT DROPPED
MTU mismatch”

1 2017/06/02 02:55:17.67 UTC MINOR: DEBUG #2001 Base OSPFv2
“OSPFv2: PKT

>> Incoming OSPF packet on I/F PEER1 area 0.0.0.1
OSPF Version : 2
Router Id : 100.100.100.100
Area Id : 0.0.0.1
Checksum : e35a
Auth Type : Null
Auth Key : 00 00 00 00 00 00 00 00
Packet Type : DB_DESC
Packet Length : 32

Interface MTU : 1504
Options : 000042
Flags : 7 INIT MORE MAST
Sequence Num : 2514
“[/php]
Okay, so PEER1 requires an MTU of 1504, lets modify that within the OSPF configuration:[php]*A:SR>config>router>ospf# info
———————————————-
area 0.0.0.0
interface “system”
no shutdown
exit
exit
area 0.0.0.1
interface “PEER1”
no shutdown
exit
exit
area 0.0.0.2
interface “PEER2”
no shutdown
exit
exit
no shutdown
———————————————-
*A:SR>config>router>ospf# area 1 interface “PEER1″ mtu 1504[/php]When applying a configuration it is good to verify things are working as expected:
[php highlight=”2,3”]*A:SR>config>router>ospf# show router ospf interface “PEER1” detail | match MTU
Passive : False Cfg MTU : 1504
Oper MTU : 1500 Last Enabled : 06/02/2017 01:44:23[/php]Although we configured the MTU to be 1504, the Operational MTU is 1500 (This is because the IP MTU is 1500 so OSPF cant be given a larger MTU on this interface)
[php]*A:SR>config>router>ospf# show router interface “PEER1” detail | match MTU
IP MTU : (default)
IP Oper MTU : 1500[/php]
When Ethernet Ports are configured as mode access and left at the default encapsulation (null) the Ethernet port MTU is 1514 bytes (to support a 1500 byte IP MTU and 14 bytes of Ethernet Header – FCS is not included in MTU calculations)
[php]*A:SR>config>router>ospf# show port 1/1/1 | match MTU
Physical Link : Yes MTU : 1514[/php]To get a 1504 byte IP MTU, we can just add 4 bytes to the Port Ethernet MTU[php]*A:SR>config>router>ospf# /configure port 1/1/1 ethernet mtu 1518
*A:SR>config>router>ospf# show router interface “PEER1” detail | match MTU
IP MTU : (default)
IP Oper MTU : 1504
*A:SR>config>router>ospf# show router ospf interface “PEER1” detail | match MTU
Passive : False Cfg MTU : 1504
Oper MTU : 1504 Last Enabled : 06/02/2017 01:44:23[/php]This should mean that the OSPF neighbor will now perform the database exchange and enter the Full state.[php]*A:SR>config>router>ospf# show router ospf neighbor “PEER1”

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface “PEER1″
===============================================================================
Interface-Name Rtr Id State Pri RetxQ TTL
Area-Id
——————————————————————————-
PEER1 100.100.100.100 Full 1 0 34
0.0.0.1
——————————————————————————-
No. of Neighbors: 1
===============================================================================
*A:SR>config>router>ospf# show router route-table

[php highlight=”16,23”]*A:SR>config>router>ospf# show router ospf neighbor “PEER2” detail

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface “PEER2″ (detail)
===============================================================================
——————————————————————————-
Neighbor : 10.1.3.3
——————————————————————————-
——————————————————————————-
Neighbor Rtr Id : 200.200.200.200 Interface: PEER2
——————————————————————————-
Neighbor IP Addr : 10.1.3.3
Local IF IP Addr : 10.1.3.1
Area Id : 0.0.0.2
Designated Rtr : 200.200.200.200 Backup Desig Rtr : 1.1.1.1
Neighbor State : Exchange Priority : 1
Retrans Q Length : 3 Options : – E – – – – O —
Events : 3 Last Event Time : 06/02/2017 02:22:36
Up Time : 0d 01:01:10 Time Before Dead : 37 sec
GR Helper : Not Helping GR Helper Age : 0 sec
GR Exit Reason : None GR Restart Reason: Unknown (0)
Bad Nbr States : 0 LSA Inst fails : 0
Bad Seq Nums : 0 Bad MTUs : 0
Bad Packets : 0 LSA not in LSDB : 0
Option Mismatches: 0 Nbr Duplicates : 917
Num Restarts : 0 Last Restart at : Never
===============================================================================[/php]There are no Bad MTUs being reported here, all we can see is that we are forever in Exchange state – lets check log 99 to see if anything at all related to PEER2 is present
[php linenumbers=”True”]*A:SR>config>router>ospf# show log log-id 99 message PEER2

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents [size=500 next event=2033 (wrapped)][/php]There is nothing present (the older events have wrapped around since we are only keeping the last 500 events)
What I have found is when OSPF neighbors are stuck in ExchStart, your router is the one with the MTU too small but while the router that is stuck in Exchange is the one with the MTU that is too big for its peer.
To work out what the smaller MTU should be, we’ll send ping packets of various lengths to work out what is the biggest unfragmented packet that can be sent to PEER2. Note: when we send a ping and specify the size, we are actually calling out what the ICMP payload size should be, so we need to ensure for IPv4 we consider the 20 byte IP header and 8 byte ICMP header – so an IP interface with an IP-MTU of 1500 would work for a ping with a payload size of 1472 but would fail at 1473.
We can test this concept on a known quantity (PEER1 which has an IP MTU of 1504) we should be able to get a ping payload of 1476 through okay but 1477 should fail – make sure we set the DF bit!
[php]*A:SR>config>router>ospf# ping 10.1.2.2 size 1476 do-not-fragment count 3
PING 10.1.2.2 1476 data bytes
1484 bytes from 10.1.2.2: icmp_seq=1 ttl=64 time=1.34ms.
1484 bytes from 10.1.2.2: icmp_seq=2 ttl=64 time=1.27ms.
1484 bytes from 10.1.2.2: icmp_seq=3 ttl=64 time=1.16ms.

—- 10.1.2.2 PING Statistics —-
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.16ms, avg = 1.26ms, max = 1.34ms, stddev = 0.072ms
*A:SR>config>router>ospf# ping 10.1.2.2 size 1477 do-not-fragment count 3
PING 10.1.2.2 1477 data bytes

—- 10.1.2.2 PING Statistics —-
3 packets transmitted, 3 packets bounced, 0 packets received, 100% packet loss[/php]This works as expected, so the concept appears sound.

[php]*A:SR>config>router>ospf# show router interface “PEER2” detail | match MTU
IP MTU : (default)
IP Oper MTU : 1500[/php]We know that we have a ceiling of 1500 and we know the MTU must be lower than this. But just to be certain, we’ll try based on a 1500 byte IP packet anyway[php]*A:SR>config>router>ospf# ping 10.1.3.3 size 1472 do-not-fragment count 3
PING 10.1.3.3 1472 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.
Request timed out. icmp_seq=3.

—- 10.1.3.3 PING Statistics —-
3 packets transmitted, 0 packets received, 100% packet loss[/php] Unsurprising, the Peer MTU is less than 1500 bytes, lets try a slightly smaller payload[php]*A:SR>config>router>ospf# ping 10.1.3.3 size 1462 do-not-fragment count 3
PING 10.1.3.3 1462 data bytes
1470 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.44ms.
1470 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.36ms.
1470 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.34ms.

—- 10.1.3.3 PING Statistics —-
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.34ms, avg = 1.38ms, max = 1.44ms, stddev = 0.042ms[/php]Okay, time to divide and conquer to determine the largest payload that gets through[php]*A:SR>config>router>ospf# ping 10.1.3.3 size 1467 do-not-fragment count 3
PING 10.1.3.3 1467 data bytes
1475 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.24ms.
1475 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.30ms.
1475 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=2.16ms.

—- 10.1.3.3 PING Statistics —-
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.24ms, avg = 1.57ms, max = 2.16ms, stddev = 0.417ms
*A:SR>config>router>ospf# ping 10.1.3.3 size 1469 do-not-fragment count 3
PING 10.1.3.3 1469 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.
Request timed out. icmp_seq=3.

—- 10.1.3.3 PING Statistics —-
3 packets transmitted, 0 packets received, 100% packet loss
*A:SR>config>router>ospf# ping 10.1.3.3 size 1468 do-not-fragment count 3
PING 10.1.3.3 1468 data bytes
1476 bytes from 10.1.3.3: icmp_seq=1 ttl=64 time=1.19ms.
1476 bytes from 10.1.3.3: icmp_seq=2 ttl=64 time=1.30ms.
1476 bytes from 10.1.3.3: icmp_seq=3 ttl=64 time=1.26ms.

—- 10.1.3.3 PING Statistics —-
3 packets transmitted, 3 packets received, 0.00% packet loss
round-trip min = 1.19ms, avg = 1.25ms, max = 1.30ms, stddev = 0.043ms[/php]An ICMP payload of 1468 fits within an IP packet with a size of 1496 – adjust the OSPF MTU to 1496 and see if that results in getting a full adjacency.
[php]*A:SR>config>router>ospf# area 2 interface “PEER2” mtu 1496
*A:SR>config>router>ospf# show router ospf interface “PEER2” detail | match MTU
Passive : False Cfg MTU : 1496
Oper MTU : 1496 Last Enabled : 06/02/2017 02:22:36
*A:SR>config>router>ospf# show router ospf neighbor “PEER2”

===============================================================================
Rtr Base OSPFv2 Instance 0 Neighbors for Interface “PEER2”
===============================================================================
Interface-Name Rtr Id State Pri RetxQ TTL
Area-Id
——————————————————————————-
PEER2 200.200.200.200 Full 1 0 35
0.0.0.2
——————————————————————————-
No. of Neighbors: 1
===============================================================================[/php]The adjacency is up – lets see what routes we have learnt
[php]*A:SR>config>router>ospf# show router route-table

—- 100.100.100.100 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.38ms, avg = 1.38ms, max = 1.38ms, stddev = 0.000ms
*A:SR>config>router>ospf# ping 200.200.200.200 source 1.1.1.1 count 1
PING 200.200.200.200 56 data bytes
64 bytes from 200.200.200.200: icmp_seq=1 ttl=64 time=1.14ms.

—- 200.200.200.200 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 1.14ms, avg = 1.14ms, max = 1.14ms, stddev = 0.000ms
[/php] We now have successful routing exchange and data plane reachability.

Transactional (Candidate) configurations with Nokia SROS

While it’s not a new feature for SROS, the introduction of transactional configurations with SROS was not a day one feature (unlike JunOS), so it may be less known to existing users.

Firstly before getting into rollback configurations, I would like to point out that SROS has been saving multiple copies of configurations via rotation based mechanism for quite some time without needing specific activation.

[php highlight=”6″]A:SR1# show system information | match expression “Config|Backup”
Config Source : primary
Last Booted Config File: cf3:\config.cfg
Last Boot Config Header: # TiMOS-B-14.0.R4 both/i386 Nokia 7750 SR Copyright
Last Saved Config : cf3:\config.cfg
Max Cfg/BOF Backup Rev : 5[/php]

cf3:\config.cfg (which can be changed from this default by modifying the boot options file) will store the current config and the previous 4 versions and while the number of saved configs is modifiable, 5 is most likely enough for most. Anyway, reverting to a previous configuration can be quite a disruptive event, if you copy an older config over cf3:\config.cfg and performing a system reboot. This is where rollback configurations and combining them with transactional (called candidate) configurations become helpful particularly during a complex task like a network migration.

Achyar Nur Andi has a good post discussing the mechanics around rollbacks and candidate configurations at www.achyarnurandi.net, so I will just highlight a few of the main features and how you can enforce the method of router configuration to only use candidate configurations.

The first thing to do is to specify the rollback file prefix (in this case conf-rollback on compact flash 3):
[php]A:SR1# /configure system rollback
A:SR1>config>system>rollback# rollback-location cf3:\conf-rollback
INFO: CLI No checkpoints currently exist at the rollback location.
*A:SR1>config>system>rollback# show system rollback

===============================================================================
Rollback Information
===============================================================================
Rollback Location : cf3:\conf-rollback
Max Local Rollback Files : 10
Max Remote Rollback Files : 10
Save
Last Rollback Save Result : None
Last Save Completion Time : N/A
Revert
In Progress : No
Last Revert Initiated User : N/A
Last Revert Checkpoint File: N/A
Last Revert Result : None
Last Revert Initiated Time : N/A
Last Revert Completion Time: N/A
Delete
Last Rollback Delete Result: None

===============================================================================
Rollback Files
===============================================================================
Idx Suffix Creation Time Release User
Comment
——————————————————————————-
No Matching Entries
===============================================================================
*A:SR1>config>system>rollback# exit all[/php]

We’ll create our first rollback point:
[php highlight=”30,31″]*A:SR1# admin rollback save comment “Baseline Config”
Saving rollback configuration to cf3:\conf-rollback.rb… OK
*A:SR1# show system rollback

===============================================================================
Rollback Information
===============================================================================
Rollback Location : cf3:\conf-rollback
Max Local Rollback Files : 10
Max Remote Rollback Files : 10
Save
Last Rollback Save Result : Successful
Last Save Completion Time : 2017/05/23 02:35:38 UTC
Revert
In Progress : No
Last Revert Initiated User : N/A
Last Revert Checkpoint File: N/A
Last Revert Result : None
Last Revert Initiated Time : N/A
Last Revert Completion Time: N/A
Delete
Last Rollback Delete Result: None

===============================================================================
Rollback Files
===============================================================================
Idx Suffix Creation Time Release User
Comment
——————————————————————————-
latest .rb 2017/05/23 02:35:38 UTC B-14.0.R4 admin
Baseline Config
——————————————————————————-
No. of Rollback Files: 1
===============================================================================[/php]
There’s only one rollback called latest.rb

For this example, just a simply system name change:
[php]*A:SR1# /configure system name “Wrong Name”[/php]Now to compare the current working configuration with the rollback:[php]*A:Wrong Name# admin rollback compare
Processing current config… 0.010 s
Processing “cf3:\conf-rollback.rb”… 0.020 s
———————————————-
configure
system
+ name “Wrong Name”
– name “SR1”
exit
exit[/php]It’s very clear what the differences are. I would just like to highlight that at present, these configuration changes are still immediate – rollbacks on their own just provide a means to manage the change, and doesnt provide any atomic operations yet.

Let’s revert back to our old configuration:[php]*A:Wrong Name# admin rollback revert latest-rb
Restoring rollback configuration cf3:\rollback-dir.rb
Processing current config… 0.010 s
Processing “cf3:\rollback-dir.rb”… 0.020 s
Resolving dependencies… 0.000 s
Tearing setup down… 0.010 s
Rebuilding setup… 0.000 s
Finished in 0.050 s
*A:SR1#[/php]
Using candidate configuration mode, as opposed to the default “immediate” configuration mode does not implement the configuration changes until you commit them, in the event of a failure applying the configuration, the system will back out and re-wind the configuration allowing you the option to discard or modify your changes. Candidate configuration mode is enabled via “candidate edit”. For this example we are going to set the system address on our router, configure an ethernet port, create an IES and attach a VLAN on that port to an IP interface.
[php]A:SR1# candidate edit
A:SR1>edit-cfg# configure router interface “system” address 111.111.111.111/32
A:SR1>edit-cfg# configure port 1/2/3 shutdown
A:SR1>edit-cfg# configure port 1/2/3 ethernet mode access
A:SR1>edit-cfg# configure port 1/2/3 ethernet encap-type dot1q
A:SR1>edit-cfg# configure port 1/2/3 no shutdown
A:SR1>edit-cfg# configure service ies 123 customer 1 create
A:SR1>edit-cfg>config>service>ies# interface TEST create
A:SR1>edit-cfg>config>service>ies>if# address 192.168.1.1/24
A:SR1>edit-cfg>config>service>ies>if# sap 1/2/3:4 create
A:SR1>edit-cfg>config>service>ies>if>sap# back
A:SR1>edit-cfg>config>service>ies>if# back
A:SR1>edit-cfg>config>service>ies# no shutdown[/php]Based on where we are within the configuration tree, we can see the associated configuration changes:
[php]A:SR1>edit-cfg>config>service>ies# candidate view
———————————————-
17: interface “TEST” create
18: address “192.168.1.1/24”
19: sap “1/2/3:4” create
20: exit
21: exit
22:* no shutdown
———————————————-[/php]Or if we get to the root of the configuration tree, we can see all the associated changes that are yet to be applied to the running configuration:[php]A:SR1>edit-cfg>config>service>ies# exit all
A:SR1>edit-cfg# candidate view
———————————————-
1: configure
2: router
3: interface “system”
4: address “111.111.111.111/32”
5: exit
6: exit
7: port “1/2/3”
8: shutdown
9: ethernet
10: mode access
11: encap-type dot1q
12: exit
13: no shutdown
14: exit
15: service
16: ies “123” customer 1 create
17: interface “TEST” create
18: address “192.168.1.1/24”
19: sap “1/2/3:4” create
20: exit
21: exit
22:* no shutdown
23: exit
24: exit
25: exit
———————————————-[/php]Now we can accept and attempt to push the configuration the router using “candidate commit”[php highlight=”3,4”]A:SR1>edit-cfg# candidate commit
Processing current config… 0.010 s
Error at line 7: Command ‘port “1/2/3″‘ failed in ‘configure’
MINOR: CLI Port “1/2/3” does not exist.
Reverting changes…
Processing current config… 0.010 s
Processing memory checkpoint… 0.000 s
Resolving dependencies… 0.000 s
Tearing setup down… 0.000 s
Rebuilding setup… 0.010 s
Finished in 0.040 s
MINOR: CLI Commit failed and has been reverted.[/php]Since there was an error in the configuration – our router doesn’t have a port 1/2/3 – the configuration failed and the whole new configuration context was backed out allowing the option to correct and reapply, or to reject the changes which is quite a powerful configuration tool and concept. As we know the problem was on line 7, we can specifically edit that line using “candidate replace 7” and replacing the string port “1/2/3” with the proper port which is “1/1/3″[php]*A:SR1>edit-cfg# candidate replace 7
*A:Replace by: port “1/1/3”
INFO: CLI Added 10 lines: ‘port “1/1/3″‘.
INFO: CLI Removed 10 lines: ‘port “1/2/3″‘.[/php]It’s probably worth double checking the revised configuration[php highlight=”9,21”]*A:SR1>edit-cfg# candidate view
———————————————-
1: configure
2: router
3: interface “system”
4: address “111.111.111.111/32”
5: exit
6: exit
7: port “1/1/3”
8: shutdown
9: ethernet
10: mode access
11: encap-type dot1q
12: exit
13: no shutdown
14: exit
15: service
16: ies “123” customer 1 create
17: interface “TEST” create
18: address “192.168.2.1/24”
19: sap “1/2/3:4” create
20: exit
21: exit
22:* no shutdown
23: exit
24: exit
25: exit
———————————————-[/php]The SAP also requires correction to align with the new port – this is on line 19[php]*A:SR1>edit-cfg# candidate replace 19
*A:Replace by: sap “1/1/3:4” create
INFO: CLI Added 2 lines: ‘sap “1/1/3:4” create’.
INFO: CLI Removed 2 lines: ‘sap “1/2/3:4” create’.[/php]Now lets apply the configuration[php]*A:SR1>edit-cfg# candidate commit
Saving checkpoint file… OK
INFO: CLI Successfully executed 25 lines in 0.000 s.[/php]Configuration mode is still quite handy to view what has been configure by jumping into the right configuration context and doing an info or info detail:[php]*A:SR1# /configure service
*A:SR1>config>service# info
———————————————-
customer 1 create
description “Default customer”
exit
ies 1 customer 1 create
interface “External” create
address 200.200.200.1/24
sap 1/1/1 create
exit
exit
no shutdown
exit
ies 123 customer 1 create
interface “TEST” create
address 192.168.2.1/24
sap 1/1/3:4 create
exit
exit
no shutdown
exit
———————————————-[/php]An operational problem can occur if we allow the use of both configuration candidate and immediate configurations such as being able to do
[php]*A:SR1>config>service# ies 123 description “Candidate Config Test”[/php]the most likely will end up with people sticking with immediate configuration mode unless they are forced to use candidate configs. Fortunately there it is quite easy to enable this.[php]*A:SR1# /configure system management cli configuration no immediate[/php] It doesn’t remove the facility to view configurations, just configuration changes:[php]*A:SR1# configure service ies 123
*A:SR1>config>service>ies# info
———————————————-
description “Candidate Config Test”
interface “TEST” create
address 192.168.2.1/24
sap 1/1/3:4 create
exit
exit
no shutdown
———————————————-[/php]If we now attempt a non-candidate mode configuration change:
[php]*A:SR1>config>service>ies# description “New Description”
MINOR: CLI Direct modification of the configuration is not allowed. Use ‘candidate edit’ for all changes.[/php]We are now forced to use candidate configs:
[php]*A:SR1>config>service>ies# candidate edit
*A:SR1>edit-cfg# configure service ies 123 description “New Description”
*A:SR1>edit-cfg# candidate commit
Processing current config… 0.010 s
Saving checkpoint file… OK
INFO: CLI Successfully executed 7 lines in 0.000 s.[/php]
Coupled with the right processes, this is one of the tools to help increase the MTBM (Mean Time Between Mistakes) and reduce the amount of network disruption.

Using BGP Flowspec with Nokia SROS to protect the network edge

This post is about how you can use BGP to augment your network edge protection. Specifically BGP Flow spec can be used to dynamically add entries to an access control list (or ip-filter in Nokia SROS speak). Previous posts have been using ExaBGP as the method to inject traffic, and while it’s good at what it does, I’ve moved on to use GoBGP not because it seems to be quite a high performance BGP implementation (I’m just doing proof of concept work) but because GoBGP’s command line interface is more friendly and better documented than ExaBGP, making it the more versatile tool to generate route advertisements in a lab on the fly (at least for me).

The lab topology is designed to demonstrate filtering with flowspec – two routers (SR1 is under my control, while the External router is a gateway to unscrupulous entities that I cannot control) and GoBGP running on a computer in my Network Operations centre.

Before getting into the flowspec piece, I’ll go through the installation process for GoBGP and the Go Language for Ubuntu 16.04 (as the routers I’m using as running on eve-ng) and the computer in the example is bridged to SR1) and then how to develop the initial GoBGP configuration.

The Go Programming language is available via the standard Ubuntu repository
[php]adam@m4600:~$ sudo apt-get install golang-go[/php]

GoBGP is pulled down as source code and compiled as part of the download process. An environmental variable needs to be set so Go knows where to pull down code, and where to build the binaries.
[php]adam@m4600:~$ export GOPATH=$HOME/go[/php]
Tell go to pull down and build gobgpd (this takes awhile at least on my machine and you don’t get an indication on what it’s doing)
[php]adam@m4600:~$ go get github.com/osrg/gobgp/gobgpd[/php]
The gobgp cli is a separate program that talks to gobgpd using the GoBGP API – downloading and building it is pretty quick in comparison to gobgpd:
[php]adam@m4600:~$ go get github.com/osrg/gobgp/gobgp[/php]
Once built, we shall copy the generated binaries to the an executable path
[php]adam@m4600:~$ sudo cp $GOPATH/bin/* /usr/local/sbin/[/php]
For those of us using the default bash shell, we can enable tab completion for the cli
[php]adam@m4600:~$ sudo cp $GOPATH/src/github.com/osrg/gobgp/tools/completion/*.bash /etc/bash_completion.d/
[/php]

Once GoBGP is installed, we can develop a configuration file that will be used to set up the GoBGP instance – in this case we’re defining the router itself and SR1 as a neighbor for the IPv4 FlowSpec address family:
A configuration file that will be used by the GoBGP Daemon enabling FlowSpec for IPv4 where we peer against SR1:[php]
adam@m4600:~$ cat gobgp-flowspec.conf
[global.config]
as = 64512
router-id = “1.2.3.4”

[[neighbors]]
[neighbors.config]
neighbor-address = “192.168.1.123”
peer-as = 64512
[[neighbors.afi-safis]]
[neighbors.afi-safis.config]
afi-safi-name = “ipv4-flowspec”[/php]
The GoBGP documentation is also a bit nicer than ExaBGP too – additional information on setting up a configuration (including different formats is described in the GoBGP getting started document)

Start the GoBGP Daemon and read our new config:
[php]adam@m4600:~$ sudo gobgpd -f gobgp-flowspec.conf
{“level”:”info”,”msg”:”gobgpd started”,”time”:”2017-05-18T22:57:22+10:00″}
{“Topic”:”Config”,”level”:”info”,”msg”:”Finished reading the config file”,”time”:”2017-05-18T22:57:22+10:00″}
{“level”:”info”,”msg”:”Peer 192.168.1.123 is added”,”time”:”2017-05-18T22:57:22+10:00″}
{“Topic”:”Peer”,”level”:”info”,”msg”:”Add a peer configuration for:192.168.1.123″,”time”:”2017-05-18T22:57:22+10:00″}
{“Key”:”192.168.1.123″,”State”:”BGP_FSM_OPENCONFIRM”,”Topic”:”Peer”,”level”:”info”,”msg”:”Peer Up”,”time”:”2017-05-18T22:57:40+10:00″}[/php]We can see shortly after gobgpd started, it has already established a peering session with SR1 (192.168.1.123)

Opening up a new shell, we can use the gobgp client to verify that:
[php]adam@m4600:~$ gobgp neighbor
Peer AS Up/Down State |#Received Accepted
192.168.1.123 64512 00:00:22 Establ | 0 0
adam@m4600:~$ gobgp neighbor 192.168.1.123
BGP neighbor is 192.168.1.123, remote AS 64512
BGP version 4, remote router ID 123.123.123.123
BGP state = established, up for 00:01:53
BGP OutQ = 0, Flops = 0
Hold time is 90, keepalive interval is 30 seconds
Configured hold time is 90, keepalive interval is 30 seconds
%s
Neighbor capabilities:
multiprotocol:
ipv4-flowspec: advertised and received
route-refresh: advertised and received
4-octet-as: advertised and received
Message statistics:
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 0 0
Keepalives: 4 5
Route Refresh: 0 0
Discarded: 0 0
Total: 5 6
Route statistics:
Advertised: 0
Received: 0
Accepted: 0[/php]
We can also get SR1 to confirm it too:
[php]*A:SR1# show router bgp summary | match Summ post-lines 100
BGP Summary
===============================================================================
Legend : D – Dynamic Neighbor
===============================================================================
Neighbor
Description
AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
——————————————————————————-
192.168.1.46
64512 9 0 00h00m54s 0/0/0 (FlowIPv4)
13 0
——————————————————————————-[/php]

Now that the flowspec session is up and running but no one has exchanged any routes yet, I’ll park this for a moment to create a simple ip-filter (or ACL)[php linenumbers=”True” highlight=”5,13,14,22″]*A:SR1# /configure filter
*A:SR1>config>filter# info
———————————————-
match-list
ip-prefix-list “FPL_RFC1918” create
prefix 10.0.0.0/8
prefix 172.16.0.0/12
prefix 192.168.0.0/16
exit
exit
ip-filter 100 create
default-action forward
embed-filter flowspec router “Base” offset 1000
entry 10 create
match
src-ip ip-prefix-list “FPL_RFC1918”
exit
action
drop
exit
exit
entry 20 create
match
dst-ip ip-prefix-list “FPL_RFC1918”
exit
action
drop
exit
exit
exit
———————————————-[/php]

The match-list ip-prefix-list “FPL_RFC1918” is a way to consolidate a number of network prefixes into a single named list enabling filter configurations to be smaller and more manageable (and easier to debug)

ip-filter 100 is the thing of interest. The default-action is what happens if none of the previous entries are matched – in this case it is a black-list ip-filter, since anything we don’t specifically drop will get forwarded
Each filter entry has a number which is used for the ascending evaluation order (entry 10 before entry 20 here) and usually has a match entry and an action to perform when the match was successful. This filter is relatively simple since all it is doing is blocking IP packets that either have source or destination IP addresses from RFC1918 space.

The interesting thing is the embed-filter flowspec entry. Router “Base” is the base routing instance (as opposed to a VPRN) and offset 1000 means that any flowspec entries will be dynamically added to ip-filter 100 starting at entry 1000 (and with an additional offset determined by flowspec)

This filter will be applied to the IP interface facing the external router – This interface is associated with an Internet Enhanced Service (IES) associated with the SR1’s Global Routing Table (Router “Base”)
[php highlight=”21,22″]*A:SR1>config>service>ies# /configure service ies 1
*A:SR1>config>service>ies# info
———————————————-
interface “External” create
address 200.200.200.1/24
sap 1/1/1 create
exit
exit
no shutdown
———————————————-
*A:SR1>config>service>ies# show router route-table

—- 1.1.1.1 PING Statistics —-
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min = 1.40ms, avg = 1.66ms, max = 1.91ms, stddev = 0.253ms[/php]
Apply the IP Filter in the ingress direction of interface External on SR1:
[php]*A:SR1>config>service>ies# interface “External” sap 1/1/1 ingress filter ip 100
*A:SR1>config>service>ies# info
———————————————-
interface “External” create
address 200.200.200.1/24
sap 1/1/1 create
ingress
filter ip 100
exit
exit
exit
no shutdown
[/php]
And repeat the test:
[php]*A:External# ping 1.1.1.1 source 172.16.0.1 count 2
PING 1.1.1.1 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.

—- 1.1.1.1 PING Statistics —-
2 packets transmitted, 0 packets received, 100% packet loss[/php]
We can see the ping failed and if we look at the counters
[php highlight=”20″]*A:SR1>config>service>ies# show filter ip 100 counters

Entry : 20
Ing. Matches : 0 pkts
Egr. Matches : 0 pkts

===============================================================================[/php]We can see the ping was unsuccessful because of entry 10 stopping traffic coming back to SR1 (because we weren’t filtering traffic leaving SR1)

Now all of a sudden we find a ping flood (well okay, calling two packets a flood is a rather long bow to draw but anyway) is coming from 100.100.100.100 and hitting 1.1.1.1[php]A:External# ping 1.1.1.1 source 100.100.100.100 count 2
PING 1.1.1.1 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=1.16ms.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=1.26ms.[/php]
These ICMP packets are causing havok to 1.1.1.1, so it would be nice to add some precision filtering to stop that from occuring, while allowing other traffic through – this is where our GoBGP instance and flowspec comes back into the story. We will ask gobgp to announce a flowspec “route” that specifies a discard action (actually a rate-limit of 0 kbps) for traffic from 100.100.100.100 to 1.1.1.1 using ICMP:
[python]adam@m4600:~$ gobgp global rib add -a ipv4-flowspec match source 100.100.100.100/32 destination 1.1.1.1/32 protocol icmp then discard[/python]
SR1 can confirm it recieved this route:
[php]*A:SR1>config>service>ies# show router bgp routes flow-ipv4
===============================================================================
BGP Router ID:123.123.123.123 AS:64512 Local AS:64512
===============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete

===============================================================================
BGP FLOW IPV4 Routes
===============================================================================
Flag Network Nexthop LocalPref MED
As-Path
——————————————————————————-
u*>? — 0.0.0.0 100 None
No As-Path

Community Action: rate-limit: 0 kbps
NLRI Subcomponents:
Dest Pref : 1.1.1.1/32
Src Pref : 100.100.100.100/32
Ip Proto : [ == 1 ]
——————————————————————————-
Routes : 1
===============================================================================[/php]
Flowspec can do quite interesting things, besides rate-limiting traffic to 0kbps (dropping it), it’s possible to use flowspec to tell the router to rate limit flows rather than completely blocking, remark traffic to a different level of priority or even redirect traffic to a VRF by setting the appropriate route targets (this could for example allow the forwarding of traffic into a VPRN or VRF to reach a scrubbing network prior to re-injection of clean traffic back into the network)

Lets see if ip filter has taken this flowspec entry:[php highlight=”57,58″]*A:SR1>config>service>ies# show filter ip 100

===============================================================================
IP Filter
===============================================================================
Filter Id : 100 Applied : Yes
Scope : Template Def. Action : Forward
System filter : Unchained
Radius Ins Pt : n/a
CrCtl. Ins Pt : n/a
RadSh. Ins Pt : n/a
PccRl. Ins Pt : n/a
Entries : 2/0/0/1 (Fixed/Radius/Cc/Embedded)
Sub-Entries : 6/0/0/1
Description : (Not Specified)
——————————————————————————-
Filter Match Criteria : IP
——————————————————————————-
Entry : 10
Description : (Not Specified)
Log Id : n/a
Src. IP : ip-prefix-list “FPL_RFC1918”
Src. Port : n/a
Dest. IP : 0.0.0.0/0
Dest. Port : n/a
Protocol : Undefined Dscp : Undefined
ICMP Type : Undefined ICMP Code : Undefined
Fragment : Off Src Route Opt : Off
Sampling : Off Int. Sampling : On
IP-Option : 0/0 Multiple Option: Off
TCP-syn : Off TCP-ack : Off
Option-pres : Off
Egress PBR : Disabled
Primary Action : Drop
Ing. Matches : 2 pkts (204 bytes)
Egr. Matches : 0 pkts

Entry : 20
Description : (Not Specified)
Log Id : n/a
Src. IP : 0.0.0.0/0
Src. Port : n/a
Dest. IP : ip-prefix-list “FPL_RFC1918”
Dest. Port : n/a
Protocol : Undefined Dscp : Undefined
ICMP Type : Undefined ICMP Code : Undefined
Fragment : Off Src Route Opt : Off
Sampling : Off Int. Sampling : On
IP-Option : 0/0 Multiple Option: Off
TCP-syn : Off TCP-ack : Off
Option-pres : Off
Egress PBR : Disabled
Primary Action : Drop
Ing. Matches : 0 pkts
Egr. Matches : 0 pkts

Entry : 1256
Origin : Inserted by embedded filter fSpec-0 entry 256
Description : (Not Specified)
Log Id : n/a
Src. IP : 100.100.100.100/32
Src. Port : n/a
Dest. IP : 1.1.1.1/32
Dest. Port : n/a
Protocol : 1 Dscp : Undefined
ICMP Type : Undefined ICMP Code : Undefined
Fragment : Off Src Route Opt : Off
Sampling : Off Int. Sampling : On
IP-Option : 0/0 Multiple Option: Off
TCP-syn : Off TCP-ack : Off
Option-pres : Off
Egress PBR : Disabled
Primary Action : Drop
Ing. Matches : 0 pkts
Egr. Matches : 0 pkts

===============================================================================[/php]We had specified that the offset for flowspec was 1000, this flowspec entry (number 256) became ip filter 100 entry 256. Time to verify that the ping now should fail:
[php]*A:External# ping 1.1.1.1 source 100.100.100.100 count 2
PING 1.1.1.1 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.

—- 1.1.1.1 PING Statistics —-
2 packets transmitted, 0 packets received, 100% packet loss[/php]
Yes, and we can verify that it’s only blocking icmp between those endpoints by sending IP packets from another protocol say UDP by doing a traceroute:
[php]*A:External# traceroute 1.1.1.1 source 100.100.100.100
traceroute to 1.1.1.1 from 100.100.100.100, 30 hops max, 40 byte packets
1 1.1.1.1 (1.1.1.1) 1.74 ms 1.51 ms 1.52 ms[/php]Yes, this is pretty good as far as network based filters (that aren’t performing payload based inspection) are doing, there’s quite a few match conditions that can be used to narrow things down even further. In this contrived example it may be similar effort to configure a filter on the fly but in a production environment, you could hook GoBGP to your route reflector and through a single procedure push updates to all your edge routers in one go. Another good thing about this method, is if the filtering is only temporary, it’s just as easy to remove as it was to put in.
To see what’s currently in the rib for ipv4-flowspec:
[php]adam@m4600:~$ gobgp global rib -a ipv4-flowspec
Network Next Hop AS_PATH Age Attrs
*> [destination:1.1.1.1/32][source:100.100.100.100/32][protocol:==icmp ]fictitious 00:12:06 [{Origin: ?} {Extcomms: [discard]}][/php]Withdraw the announcement:
[php]adam@m4600:~$ gobgp global rib del -a ipv4-flowspec match source 100.100.100.100/32 destination 1.1.1.1/32 protocol icmp then discard
adam@m4600:~$ gobgp global rib -a ipv4-flowspec Network not in table[/php]
Verify SR1 ip filter 100 no longer has the entry:
[php]*A:SR1>config>service>ies# show filter ip 100

===============================================================================
IP Filter
===============================================================================
Filter Id : 100 Applied : Yes
Scope : Template Def. Action : Forward
System filter : Unchained
Radius Ins Pt : n/a
CrCtl. Ins Pt : n/a
RadSh. Ins Pt : n/a
PccRl. Ins Pt : n/a
Entries : 2
Sub-Entries : 6
Description : (Not Specified)
——————————————————————————-
Filter Match Criteria : IP
——————————————————————————-
Entry : 10
Description : (Not Specified)
Log Id : n/a
Src. IP : ip-prefix-list “FPL_RFC1918”
Src. Port : n/a
Dest. IP : 0.0.0.0/0
Dest. Port : n/a
Protocol : Undefined Dscp : Undefined
ICMP Type : Undefined ICMP Code : Undefined
Fragment : Off Src Route Opt : Off
Sampling : Off Int. Sampling : On
IP-Option : 0/0 Multiple Option: Off
TCP-syn : Off TCP-ack : Off
Option-pres : Off
Egress PBR : Disabled
Primary Action : Drop
Ing. Matches : 2 pkts (204 bytes)
Egr. Matches : 0 pkts

===============================================================================[/php]Entry 1256 has left the building.

Hopefully that was a reasonable introduction to using BGP Flowspec with filtering and GoBGP.

Flowspec is quite often a piece of the puzzle in DDoS mitigation tools, usually there is some kind of automation attached which makes decisions using inputs such a Netflow/IPFIX data, SNMP polling of interfaces to track anomalous loading and in some cases threat intelligence feeds from security vendors.

TACACS+ Authentication with Nokia Service Routers

Nokia SROS supports the use of AAA for a range of tasks, some of the more interesting and complicated are related to subscriber management when the Service Router is acting as a Broadband Network Gateway (BNG) however AAA is also useful for the network operations teams to provide centralised authentication for a fleet of routers where managing individual local accounts is not really something to contemplate.

SROS supports the use of RADIUS or TACACS+ for this management access control and today TACACS+ will be the method used with a linux daemon based on code from http://www.shrubbery.net/tac_plus/ which will be configured to support a Nokia Service Router (however this configuration would be quite Cisco IOS friendly) and the SROS router will use TACACS+ for authentication and identifying what access rights the user has my mapping using profiles.

Nokia SR and TACACS+ Server Test Topology

As you can see above, R1 (instantiated service router in eve-ng) has port 1/1/3 bridged to the internal Ethernet of the computer running eve-ng so both the router interface and the internal Ethernet are on the same IP subnet allowing connectivity to the TACACS+ server that will be run on the laptop.

To install the TACACS+ software, as eve-ng is built on Ubuntu 16.04, installation is as simple as invoking:
[php]root@m4600:~# apt-get install tacacs+[/php]
The config was then modified to look like below:
[php]root@m4600:~# cat /etc/tacacs+/tac_plus.conf
# shared secret with TACACS client
key = “tac_secret”
# Set where to send accounting records
accounting syslog;
accounting file = /var/log/tac_plus/tac_plus.acct

acl = mgmt_acl {
# regex to allow access hosts from 192.168.1.0/24
permit = 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4])
}

# administrative group, priv-lvl 15 to be mapped to SROS administative profile
group = administrative {
default service = permit
expires = “Jan 1 2020”
acl = mgmt_acl
service = exec {
priv-lvl = 15
}
}
# limited group, priv-lvl 1 to be mapped to SROS limited profile
group = limited {
default service = permit
expires = “Jan 1 2020”
acl = mgmt_acl
service = exec {
priv-lvl = 1
}
}

# our tacacs test accounts
# des password is generated by running tac_pwd on the plaintext
user = testadmin {
member = administrative
login = des JZ1fHFoSp.v/E
# plaintext password = pass
}

user = testlimited {
member = limited
login = des O8ZepJOyIIuYo
# plaintext password = test
}[/php]

A couple of the key things here besides the key which is the shared secret between the TACACS+ server and the router is that there are two groups defined administrative and limited, where the only difference is the priv-lvl. With Cisco platforms, this is what is used for TACACS+ uses during the authorisation stage to tell IOS what access rights a user has. SROS is able to map this to a “profile”.

Out of the box, SROS has two built in profiles, administrative (used for most installation and commissioning activities) and default which is somewhat less capable, however it is possible to define specific profiles in line with the roles of your users. In the config above there is a group called limited which will be identified by priv-lvl 1.

On R1 we can define a custom profile in the system security configuration context:
[php]A:R1# /configure system security profile “limited”
A:R1>config>system>security>profile# info
———————————————-
default-action deny-all
entry 10
match “show router route-table”
action permit
exit
entry 20
match “show users”
action permit
exit
entry 30
match “show system security user”
action permit
exit
entry 40
match “logout”
action permit
exit
[/php] This example is certainly quite limited in what can be done due to the default-action deny-all, requiring specific white-listing of commands
To enable TACACS+ support on the router we first need to configure the TACACS server using the aggreed shared secret (configuring the timeout is optional but it specifies how many seconds we shall wait for a response – if the server is down, this is effectively how long you will wait to fall back to local authentication)
[php]A:R1>config>system>security>profile# /configure system security tacplus
*A:R1>config>system>security>tacplus$ server 1 address 192.168.1.47 secret “tac_secret”
*A:R1>config>system>security>tacplus$ timeout 5[/php]
Now to create the priv-lvl mapping to profiles:
[php]*A:R1>config>system>security>tacplus$ priv-lvl-map
A:R1>config>system>security>tacplus$ priv-lvl-map
A:R1>config>system>security>tacplus>priv$ priv-lvl 1 “limited”
A:R1>config>system>security>tacplus>priv$ priv-lvl 15 “administrative”[/php]
We also need to enable authorisation to be associated with these mappings:
[php]A:R1>config>system>security>tacplus>priv$ back
A:R1>config>system>security>tacplus$ authorization use-priv-lvl[/php]
Now to actually enable tacacs authentication, within the password context we specify the authentication order to include the methods we prefer.
[php]*A:R1>config>system>security>tacplus$ /configure system security password
*A:R1>config>system>security>password# authentication-order tacplus local exit-on-reject[/php]
If TACACS+ is unavailable, we fall back to local authentication accounts – if we hadn’t include “exit-on-reject”, a failed authentication attempt with TACACS+ (reject) would move onto the next authentication mechanisms (local)

SROS performs a AAA server health check by sending dummy authentication requests to a server and determines if the server is alive based on obtaining a response, this can end up with the authentication logs getting a lot of failed access attempts, however it can be disabled if desired:
[php]*A:R1>config>system>security>password# no health-check[/php]
For this testing, I’ll be using telnet, so I need to enable the telnet-server (outside of a lab, I would not suggest this at all!)
[php]*A:R1>config>system>security>password# back
*A:R1>config>system>security# telnet-server[/php]
So to recap the router configuration:
[php]*A:R1>config>system>security# info
———————————————-
telnet-server
profile “limited”
default-action deny-all
entry 10
match “show router route-table”
action permit
exit
entry 20
match “show users”
action permit
exit
entry 30
match “show system security user”
action permit
exit
entry 40
match “logout”
action permit
exit
exit
password
authentication-order tacplus local exit-on-reject
no health-check
exit
tacplus
authorization use-priv-lvl
priv-lvl-map
priv-lvl 1 “limited”
priv-lvl 15 “administrative”
exit
timeout 5
server 1 address 192.168.1.47 secret “1mSYRiobfhHAdFA9cZH3wBviQtXKFDld” hash2
exit
[/php]
Time to test if this works. Start the tacacs service on (m4600 has the IP of 192.168.1.47 which is what R1 will be communicating with)
[php]root@m4600:~# tac_plus -d 16 -L -C /etc/tacacs+/tac_plus.conf[/php]
And start viewing syslog
[php]root@m4600:~# tail -f /var/log/syslog
May 14 14:39:14 m4600 tac_plus[28164]: Reading config
May 14 14:39:14 m4600 tac_plus[28164]: Version F4.0.4.27a Initialized 1
May 14 14:39:14 m4600 tac_plus[28164]: tac_plus server F4.0.4.27a starting
May 14 14:39:14 m4600 tac_plus[28165]: Backgrounded
May 14 14:39:14 m4600 tac_plus[28166]: socket FD 0 AF 2
May 14 14:39:14 m4600 tac_plus[28166]: socket FD 2 AF 10
May 14 14:39:14 m4600 tac_plus[28166]: uid=0 euid=0 gid=0 egid=0 s=-1637085952[/php]
Open up another session on m4600 and telnet to 192.168.1.123 using the credentials of testadmin/pass:
[php]May 14 14:39:23 m4600 tac_plus[28201]: connect from 192.168.1.123 [192.168.1.123]
May 14 14:39:23 m4600 tac_plus[28201]: cfg_acl_check(mgmt_acl, 192.168.1.123)
May 14 14:39:23 m4600 tac_plus[28201]: ip 192.168.1.123 matched permit regex 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4]) of acl filter mgmt_acl
May 14 14:39:23 m4600 tac_plus[28201]: host ACLs for user ‘testadmin’ permit
May 14 14:39:23 m4600 tac_plus[28201]: login query for ‘testadmin’ port console from 192.168.1.123 accepted
May 14 14:39:23 m4600 tac_plus[28202]: connect from 192.168.1.123 [192.168.1.123]
May 14 14:39:23 m4600 tac_plus[28202]: cfg_acl_check(mgmt_acl, 192.168.1.123)
May 14 14:39:23 m4600 tac_plus[28202]: ip 192.168.1.123 matched permit regex 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4]) of acl filter mgmt_acl
May 14 14:39:23 m4600 tac_plus[28202]: host ACLs for user ‘testadmin’ permit
May 14 14:39:23 m4600 tac_plus[28202]: authorization query for ‘testadmin’ console from 192.168.1.123 accepted[/php]
Lets go back to the telnet session and check who we are and our access rights:
[php highlight=”30,38″]*A:R1# show users
===============================================================================
User Type Login time Idle time
From
===============================================================================
Console — 0d 00:00:21
—
testadmin Telnet 14MAY2017 04:41:41 0d 00:00:00
192.168.1.47
——————————————————————————-
Number of users : 1
===============================================================================
*A:R1>config>system>security# show system security user testadmin detail

===============================================================================
Users
===============================================================================
User ID New User Permissions Password Login Failed Local
Pwd console ftp li snmp netconf Expires Attempts Logins Conf
——————————————————————————-
testadmin n y n n n n never 1 0 n
——————————————————————————-
Number of users : 1
===============================================================================

===============================================================================
Temporary User Configuration Detail
===============================================================================
===============================================================================
user id : testadmin
——————————————————————————-
console parameters
——————————————————————————-
new pw required : n/a cannot change pw : n/a
home directory :
restricted to home : no
login exec file :
profile : administrative
locked-out : no
===============================================================================[/php]
Okay, that’s good. Lets log out and log back in R1 using the credentials of testlimited/test:
[php]May 14 14:43:37 m4600 tac_plus[29058]: connect from 192.168.1.123 [192.168.1.123]
May 14 14:43:37 m4600 tac_plus[29058]: cfg_acl_check(mgmt_acl, 192.168.1.123)
May 14 14:43:37 m4600 tac_plus[29058]: ip 192.168.1.123 matched permit regex 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4]) of acl filter mgmt_acl
May 14 14:43:37 m4600 tac_plus[29058]: host ACLs for user ‘testlimited’ permit
May 14 14:43:37 m4600 tac_plus[29058]: login query for ‘testlimited’ port telnet from 192.168.1.123 accepted
May 14 14:43:37 m4600 tac_plus[29059]: connect from 192.168.1.123 [192.168.1.123]
May 14 14:43:37 m4600 tac_plus[29059]: cfg_acl_check(mgmt_acl, 192.168.1.123)
May 14 14:43:37 m4600 tac_plus[29059]: ip 192.168.1.123 matched permit regex 192\.168\.1\.([1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-4]) of acl filter mgmt_acl
May 14 14:43:37 m4600 tac_plus[29059]: host ACLs for user ‘testlimited’ permit
May 14 14:43:37 m4600 tac_plus[29059]: authorization query for ‘testlimited’ telnet from 192.168.1.123 accepted[/php]
Looks promising from the TACACS server, lets go back to the telnet session and check who we are and our access rights:
[php]*A:R1# show users
===============================================================================
User Type Login time Idle time
From
===============================================================================
Console — 0d 00:04:55
—
testlimited Telnet 14MAY2017 04:43:36 0d 00:00:00
192.168.1.47
——————————————————————————-
Number of users : 1
===============================================================================
*A:R1# show router route-table

[php highlight=”18,26″]*A:R1# show system security user testlimited detail

===============================================================================
Users
===============================================================================
User ID New User Permissions Password Login Failed Local
Pwd console ftp li snmp netconf Expires Attempts Logins Conf
——————————————————————————-
testlimited n y n n n n never 1 0 n
——————————————————————————-
Number of users : 1
===============================================================================

Role based access control is a good idea for managing your network and being able to leverage your existing AAA infrastructure helps make operating a heterogeneous network that little bit easier.

Route aggregation is not always straight forward

Today’s post is a simple 3 router topology based on a true story when route aggregation didn’t appear to working as expected at first glance and some additional thought was required as to why things were behaving that way and what was required to make it do what I wanted.

I’m using 3 x Nokia VSR-Sims running SROS 14.0R4, and while the concepts discussed here are definitely flavoured through a SROS lens, the concepts will be familiar to different router platforms and their associated operating systems.

R1 and R2 are in BGP AS 12 and use ospf to advertise their respective system addresses which are used for their IBGP peerings

[codegroup]
[php linenumbers=”false” tab=”R1 Config”]
configure
system
name “R1”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
router
interface “Loop”
address 100.100.100.1/24
loopback
no shutdown
exit
interface “R2”
address 10.1.2.1/27
port 1/1/1
no shutdown
exit
interface “system”
address 1.1.1.1/32
no shutdown
exit
autonomous-system 12
ospf 0
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R2”
no shutdown
exit
exit
no shutdown
exit
policy-options
begin
prefix-list “PL_LOOP”
prefix 100.100.100.0/24 exact
exit
policy-statement “PS_LOOP_EXP”
entry 10
from
protocol direct
prefix-list “PL_LOOP”
exit
action accept
exit
exit
exit
commit
exit
bgp
group “IBGP”
export “PS_LOOP_EXP”
peer-as 12
neighbor 2.2.2.2
exit
exit
no shutdown
exit
exit
exit all
[/php]
[php linenumbers=”false” tab=”R2 Initial Config”]
configure
system
name “R2”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
router
interface “R1”
address 10.1.2.2/27
port 1/1/1
no shutdown
exit
interface “R3”
address 10.2.3.2/27
port 1/1/2
no shutdown
exit
interface “system”
address 2.2.2.2/32
no shutdown
exit
autonomous-system 12
router-id 2.2.2.2
ospf 0
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R1”
no shutdown
exit
exit
no shutdown
exit
exit
bgp
group “IBGP”
peer-as 12
neighbor 1.1.1.1
exit
exit
no shutdown
exit
exit
exit all
[/php]
[/codegroup]

R3 which is in BGP AS 3 will be peering with R2. While we can configure R2, as R3 is in a different AS, we cannot touch it nor modify its configuration. R3 is already configured to peer with R2 and is waiting for R2 to come online.

Our configuration for R2 to establish the BGP Session:

[php]A:R2# configure router bgp
A:R2>config>router>bgp# group EBGP
*A:R2>config>router>bgp>group$ neighbor 10.2.3.3 peer-as 3
*A:R2>config>router>bgp>group$ exit all [/php]

Assuming enough time has passed for BGP to come up, lets get a quick state of play with BGP on R2:

[php]*A:R2# show router bgp summary | match “BGP Sum” post-lines 100
BGP Summary
===============================================================================
Legend : D – Dynamic Neighbor
===============================================================================
Neighbor
Description
AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
——————————————————————————-
1.1.1.1
12 11 0 00h03m30s 1/1/0 (IPv4)
10 0
10.2.3.3
3 6 0 00h00m34s 8/8/1 (IPv4)
5 0
——————————————————————————-[/php]

Right now R2 has active BGP sessions with R1 and R3 – we can see that R2 has received 8 routes from R3 and has sent 1 (from R1). R2 hasn’t yet sent any routes learnt from R3 to R1 however this should happen shortly.

These are the BGP routes that R2 knows of
[php]*A:R2# show router bgp routes
===============================================================================
BGP Router ID:2.2.2.2 AS:12 Local AS:12
===============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete

============================================================= BGP IPv4 Routes
============================================================= Flag Network Nexthop (Router) As-Path
————————̵ u*>i 3.0.0.0/24 None 10.2.3.3 None 3
u*>i 3.0.1.0/24 None 10.2.3.3 None 3
u*>i 3.0.2.0/24 None 10.2.3.3 None 3
u*>i 3.0.3.0/24 None 10.2.3.3 None 3
u*>i 3.0.4.0/24 None 10.2.3.3 None 3
u*>i 3.0.5.0/24 None 10.2.3.3 None 3
u*>i 3.0.6.0/24 None 10.2.3.3 None 3
u*>i 3.0.7.0/24 None 10.2.3.3 None 3
u*>i 100.100.100.0/24 100 1.1.1.1 None No As-Path
————————̵ Routes : 9
============================================================= As this post is about route aggregation, on R2 we want [php]*A:R2# configure router aggregate 3.0.0.0/21 summary-only *A:R2# show router aggregate detail ==================
==================
LocalPref MED
Path-Id Label
2;—————————————————-
None
–
None
–
None
–
None
–
None
–
None
–
None
–
None
–
None
–
2;—————————————————-
==================[/php]
to send a summary route through to R1 (3.0.0.0/21) instead of all the individual routes. To do this we will create the aggregate route and specify it to be a summary-only route and because we can, we will include the AS-Set in the aggregate so R1 knows these came from AS 3
as-set description “Aggregate from AS3”

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
——————————————————————————-
1.1.1.1/32 Remote OSPF 00h10m16s 10
10.1.2.1 100
2.2.2.2/32 Local Local 00h11m14s 0
system 0
3.0.0.0/21 Blackh* Aggr 00h02m13s 130
Black Hole 0
3.0.0.0/24 Remote BGP 00h05m59s 170
10.2.3.3 0
3.0.1.0/24 Remote BGP 00h05m59s 170
10.2.3.3 0
3.0.2.0/24 Remote BGP 00h05m59s 170
10.2.3.3 0
3.0.3.0/24 Remote BGP 00h05m59s 170
10.2.3.3 0
3.0.4.0/24 Remote BGP 00h05m59s 170
10.2.3.3 0
3.0.5.0/24 Remote BGP 00h06m00s 170
10.2.3.3 0
3.0.6.0/24 Remote BGP 00h06m00s 170
10.2.3.3 0
3.0.7.0/24 Remote BGP 00h06m00s 170
10.2.3.3 0
10.1.2.0/27 Local Local 00h11m00s 0
R1 0
10.2.3.0/27 Local Local 00h11m00s 0
R3 0
100.100.100.0/24 Remote BGP 00h08m53s 170
10.1.2.1 0
——————————————————————————-
No. of Routes: 14
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
* indicates that the corresponding row element may have been truncated.[/php]
As we can see in the routing table, an aggregate route is treated as its own routing protocol, so we need to develop a routing policy to advertise the aggregate to R1
[php]*A:R2# configure router policy-options
*A:R2>config>router>policy-options# begin
*A:R2>config>router>policy-options# policy-statement PS_AGGREGATE
*A:R2>config>router>policy-options>policy-statement$ entry 10 from protocol aggregate
*A:R2>config>router>policy-options>policy-statement$ entry 10 action accept
*A:R2>config>router>policy-options>policy-statement>entry>action$ exit
*A:R2>config>router>policy-options>policy-statement$ info
———————————————-
entry 10
from
protocol aggregate
exit
action accept
exit
exit
———————————————-
*A:R2>config>router>policy-options>policy-statement$ exit
*A:R2>config>router>policy-options# commit[/php]
We then can use the policy to export to our neighbor (using group IBGP or on the neighbor directly)
[php]*A:R2>config>router>policy-options# /configure router bgp group “IBGP”
*A:R2>config>router>bgp>group# export “PS_AGGREGATE”[/php]
One thing we haven’t done yet is that the EBGP next-hop 10.2.3.3 will not be visible to R1, so we can either add that interface into OSPF (as a passive interface so we don’t attempt to peer with an external router at the IGP level) or have R2 set next-hop-self (I generally prefer this as it keeps the IGP just for internal core links)
[php]*A:R2>config>router>bgp>group# next-hop-self
*A:R2>config>router>bgp>group# info
———————————————-
next-hop-self
export “PS_AGGREGATE”
peer-as 12
neighbor 1.1.1.1
exit
———————————————-[/php]
Okay, so now R1 should have 3.0.0.0/21 and the job is done, so lets verify this is working on R1
[php]A:R1# show router route-table

===============================================================================
Legend: G – generate-icmp enabled
===============================================================================
Aggregates (Router: Base)
===============================================================================
Prefix Aggr IP-Address Aggr AS
Summary AS Set State
NextHop Community NextHopType
——————————————————————————-
3.0.0.0/21 0.0.0.0 0
True True Active
None
——————————————————————————-
No. of Aggregates: 1
===============================================================================[/php]
Well that looks okay but maybe the aggregate route is wrong
[php]*A:R2>config>router>bgp>group# /admin display-config | match expression “^\ +agg”
aggregate 3.0.0.0/21 summary-only as-set description “Aggregate from AS3″[/php]
Lets try it without including the summary-only option and see if the contributing routes will get advertised to R1.
[php]*A:R2>config>router>bgp>group# /configure router aggregate 3.0.0.0/21 as-set description “Agg AS3 no summary-only”
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes
===============================================================================
BGP Router ID:2.2.2.2 AS:12 Local AS:12
===============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Original Attributes

Network : 3.0.2.0/24
Nexthop : 10.2.3.3
Path Id : None
From : 10.2.3.3
Res. Nexthop : 10.2.3.3
Local Pref. : n/a Interface Name : R3
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
AIGP Metric : None
Connector : None
Community : no-advertise
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 3.3.3.3
Fwd Class : None Priority : None
Flags : Used Valid Best IGP
Route Source : External
AS-Path : 3
Route Tag : 0
Neighbor-AS : 3
Orig Validation: NotFound
Source Class : 0 Dest Class : 0
Add Paths Send : Default
Last Modified : 03h55m28s

Modified Attributes

Network : 3.0.2.0/24
Nexthop : 10.2.3.3
Path Id : None
From : 10.2.3.3
Res. Nexthop : 10.2.3.3
Local Pref. : None Interface Name : R3
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
AIGP Metric : None
Connector : None
Community : no-advertise
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 3.3.3.3
Fwd Class : None Priority : None
Flags : Used Valid Best IGP
Route Source : External
AS-Path : 3
Route Tag : 0
Neighbor-AS : 3
Orig Validation: NotFound
Source Class : 0 Dest Class : 0
Add Paths Send : Default
Last Modified : 03h55m30s

——————————————————————————-
——————————————————————————-
Routes : 1
===============================================================================[/php]
3.0.2.0/24 has no-advertise attached to it!
One of the things about aggregate routes is that they aggregate the associated communities as well, so the aggregate route will have no-advertise attached to it, so it will not be advertised to R1.
Unfortunately this doesn’t appear in the show “show router aggregate detail” – the community there is the one that is manually added during the creation of the aggregate.
So how can we fix this? Well there are two methods that spring to mind and I am sure that there are more.

Option 1 – Create an import policy on R2 that just drops the no-advertise community on imported routes.
I think this is the easiest option to implement because then the normal aggregate configuration will work.
[php]*A:R2>config>router>bgp>group# /configure router policy-options
*A:R2>config>router>policy-options# begin
*A:R2>config>router>policy-options# community NO_ADV members no-advertise
*A:R2>config>router>policy-options# policy-statement PS_IGNORE_NO_ADV
*A:R2>config>router>policy-options>policy-statement$ entry 10
*A:R2>config>router>policy-options>policy-statement>entry$ from community NO_ADV
*A:R2>config>router>policy-options>policy-statement>entry$ action accept
*A:R2>config>router>policy-options>policy-statement>entry>action$ community remove “NO_ADV”
*A:R2>config>router>policy-options>policy-statement>entry>action$ back
*A:R2>config>router>policy-options>policy-statement>entry$ back
*A:R2>config>router>policy-options>policy-statement$ info
———————————————-
entry 10
from
community “NO_ADV”
exit
action accept
community remove “NO_ADV”
exit
exit
———————————————-
*A:R2>config>router>policy-options>policy-statement$ back
*A:R2>config>router>policy-options# commit
*A:R2>config>router>policy-options# /configure router bgp group “EBGP”
*A:R2>config>router>bgp>group# neighbor 10.2.3.3 import “PS_IGNORE_NO_ADV”[/php]
Let’s see if that has resolved things:
[php]*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes
===============================================================================
BGP Router ID:2.2.2.2 AS:12 Local AS:12
===============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag Network LocalPref MED
Nexthop (Router) Path-Id Label
As-Path
——————————————————————————-
i 3.0.0.0/21 100 None
2.2.2.2 None –
3
i 3.0.0.0/24 100 None
2.2.2.2 None –
3
i 3.0.1.0/24 100 None
2.2.2.2 None –
3
i 3.0.2.0/24 100 None
2.2.2.2 None –
3
i 3.0.3.0/24 100 None
2.2.2.2 None –
3
i 3.0.4.0/24 100 None
2.2.2.2 None –
3
i 3.0.5.0/24 100 None
2.2.2.2 None –
3
i 3.0.6.0/24 100 None
2.2.2.2 None –
3
i 3.0.7.0/24 100 None
2.2.2.2 None –
3
——————————————————————————-
Routes : 9
===============================================================================[/php]
Yes, 3.0.2.0/24 is present and because none of the routes that contribute to the aggregate have no-advertise attached, the aggregate is also advertised to R1. So time to change the Aggregate route so it’s back to summary only:
[php]*A:R2>config>router>bgp>group# /configure router aggregate 3.0.0.0/21 summary-only as-set description “Aggregate from AS3”
*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes
===============================================================================
BGP Router ID:2.2.2.2 AS:12 Local AS:12
===============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete

Option 2 – Using Static Routes for Aggregation
If we wish to respect the no-advertise binding on 3.0.2.0/24, we can simulate some of the behavior of an aggregate route without caring about no-advertise (or no-export if we are concerned about advertisements outside of our AS).
First we need to remove the import policy on R2 facing R3.
[php]*A:R2>config>router>bgp>group# info
———————————————-
neighbor 10.2.3.3
import “PS_IGNORE_NO_ADV”
peer-as 3
exit
———————————————-
*A:R2>config>router>bgp>group# neighbor 10.2.3.3 no import[/php]
And remove the aggregate for 3.0.0.0/21
[php]*A:R2>config>router>bgp>group# /configure router no aggregate 3.0.0.0/21 [/php]
Now we create a static black-hole route with BGP community 12:12 attached to it. We’re attaching the community so we can distinguish between regular static routes and our “aggregate”
[php]A:R2>config>router# static-route-entry 3.0.0.0/21
*A:R2>config>router>static-route-entry$ black-hole
*A:R2>config>router>static-route-entry>black-hole$ community 12:12
*A:R2>config>router>static-route-entry>black-hole$ no shutdown[/php]
As a note, SROS Release 14 changed the specific syntax for creating static routes but the concepts generally remain the same for previous SROS versions.
Now we’ll work on the routing policy to advertise our static aggregate route.
First we’ll create a named community that was used for our aggregate:
[php]*A:R2>config>router>static-route-entry$ /configure router policy-options
*A:R2>config>router>policy-options# begin
*A:R2>config>router>policy-options# community STATIC_AGG members 12:12 [/php]
Now we create a prefix list to match the routes that contribute to our aggregate
[php]*A:R2>config>router>policy-options# prefix-list PL_R3_CONTRIB
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.0.0/21 longer
*A:R2>config>router>policy-options>prefix-list$ exit[/php]
Finally we take the existing PS_AGGREGATE and modify it to work with our static aggregate and drop the contributing routes:
[php]*A:R2>config>router>policy-options# policy-statement “PS_AGGREGATE”
*A:R2>config>router>policy-options>policy-statement# info
———————————————-
entry 10
from
protocol aggregate
exit
action accept
exit
exit
*A:R2>config>router>policy-options>policy-statement# entry 10
*A:R2>config>router>policy-options>policy-statement>entry# from protocol static
*A:R2>config>router>policy-options>policy-statement>entry# from community “STATIC_AGG”
*A:R2>config>router>policy-options>policy-statement>entry# back
*A:R2>config>router>policy-options>policy-statement# entry 20
*A:R2>config>router>policy-options>policy-statement>entry$ from prefix-list “PL_R3_CONTRIB”
*A:R2>config>router>policy-options>policy-statement>entry$ action drop
*A:R2>config>router>policy-options>policy-statement>entry>action$ exit
*A:R2>config>router>policy-options>policy-statement>entry$ exit
*A:R2>config>router>policy-options>policy-statement# info
———————————————-
entry 10
from
protocol static
community “STATIC_AGG”
exit
action accept
exit
exit
entry 20
from
prefix-list “PL_R3_CONTRIB”
exit
action drop
exit
exit
*A:R2>config>router>policy-options>policy-statement# back
*A:R2>config>router>policy-options# commit[/php]
Lets check what R2 is advertising to R1:
[php]*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes
===============================================================================
BGP Router ID:2.2.2.2 AS:12 Local AS:12
===============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag Network LocalPref MED
Nexthop (Router) Path-Id Label
As-Path
——————————————————————————-
? 3.0.0.0/21 100 None
2.2.2.2 None –
No As-Path
——————————————————————————-
Routes : 1
===============================================================================[/php]
There are a couple of issues with this implementation the AS-Path information is lost (nothing we can do about that) but more importantly, this “aggregate” will stay up even if the contributing routes are not present. To overcome this issue, the static route can be associated with a prefix-list which will be used to determine if the static route can become active. It should be noted that although we already have PL_R3_CONTRIB, it cannot be used here as the prefix list in the static route requires specific prefixes to match against. While matching all possible prefixes could be problematic, in most instances simply matching against a few key prefixes will be sufficient:
[php]*A:R2>config>router>bgp>group# /configure router policy-options
*A:R2>config>router>policy-options# begin
*A:R2>config>router>policy-options# prefix-list PL_R3_STATIC_AGG_OK
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.0.0/24
*A:R2>config>router>policy-options>prefix-list$ prefix 3.0.7.0/24
*A:R2>config>router>policy-options>prefix-list$ exit
*A:R2>config>router>policy-options# commit [/php]
Modify the static route to be up when any of the prefixes in PL_R3_STATIC_AGG_OK are in the routing table:
[php]*A:R2>config>router>policy-options# /configure router static-route-entry 3.0.0.0/21 black-hole prefix-list “PL_R3_STATIC_AGG_OK” any
[/php]
We can see the route is active and the prefix-list being used to validate:
[php highlight=”9,10″]*A:R2>config>router>policy-options# show router static-route detail

===============================================================================
Static Route Table (Router: Base) Family: IPv4
===============================================================================
Prefix : 3.0.0.0/21
Nexthop : n/a
Type : Blackhole
Dynamic BGP : disabled Generate ICMP : disabled
Interface : n/a Active : Y
Prefix List : PL_R3_STATIC_AGG_OK Prefix List Type : Any
Metric : 1 Preference : 5
Source Class : 0 Dest Class : 0
Admin State : Up Tag : 0
Creation Origin : manual
BFD : disabled
Community : 12:12
CPE-check : disabled
——————————————————————————-
No. of Static Routes: 1

===============================================================================[/php]
If we shutdown our BGP session to R3, the routes in PL_R3_STATIC_AGG_OK will disappear from the routing table and the static route will be brought out of service
[php highlight=”12,13,21″]*A:R2>config>router>policy-options# /configure router bgp group “EBGP”
*A:R2>config>router>bgp>group# shutdown
*A:R2>config>router>bgp>group# show router static-route detail

===============================================================================
Static Route Table (Router: Base) Family: IPv4
===============================================================================
Prefix : 3.0.0.0/21
Nexthop : n/a
Type : Blackhole
Dynamic BGP : disabled Generate ICMP : disabled
Interface : n/a Active : N
Prefix List : PL_R3_STATIC_AGG_OK Prefix List Type : Any
Metric : 1 Preference : 5
Source Class : 0 Dest Class : 0
Admin State : Up Tag : 0
Creation Origin : manual
BFD : disabled
Community : 12:12
CPE-check : disabled
Inactive Reason : prefix-list match failed
——————————————————————————-
No. of Static Routes: 1

===============================================================================[/php]
We can see the static route is down because the prefix-list match has failed and we can confirm that we aren’t advertising this to R1:
[php]*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes
===============================================================================
BGP Router ID:2.2.2.2 AS:12 Local AS:12
===============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete

===============================================================================[/php]
Yes, so we should be offering this to R3 again:
[php]*A:R2>config>router>bgp>group# show router bgp neighbor 1.1.1.1 advertised-routes
===============================================================================
BGP Router ID:2.2.2.2 AS:12 Local AS:12
===============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete

While this scenario isn’t likely to occur all the time, based on my experience it is something to consider if things are not working quite as expected.

The case of Nokia Virtual Service Router and the non-unique Chassis MAC Address

So I’m playing with eve-ng and have decided to work on a Layer 2 scenario and a few problems with my emulation environment came up which needed a way forward, which resulted in this rambling tale…

R1, R2 and R3 Will be the MPLS Core with VPLS configured, while R4 and R5 will be Layer 3 CE devices that talk to each other over the VPLS.

The CE Devices are pretty straight forward so we’ll get those up first

R4 is a single-ended configuration with Interface R5 on Port 1/1/1 having the IP 192.168.1.4/27
[python linenumbers=”false” tab=”R4 CE Config”]
configure
system
name “R4”
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
router
interface “R5”
address 192.168.1.4/27
port 1/1/1
no shutdown
exit
interface “system”
no shutdown
exit
exit
exit all
[/python]

R5 is a a little more complex, it has a LAG toward – Interface R4 on LAG-1 with Ports 1/1/1 and 1/1/2 having the IP 192.168.1.5/27
[python linenumbers=”false” tab=”R5 CE Config”]
configure
system
name “R5”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
autonegotiate limited
exit
no shutdown
exit
port 1/1/2
ethernet
autonegotiate limited
exit
no shutdown
exit
lag 1
port 1/1/1
port 1/1/2
lacp active administrative-key 32768
no shutdown
exit
router
interface “R4”
address 192.168.1.5/27
port lag-1
no shutdown
exit
interface “system”
no shutdown
exit
exit
exit all
[/python]
Multi-speed Ethernet interfaces when associated with a LAG must have autonegotiate set to limited to control the bundle member speed so they all bundle members operate the same speed

Now to Develop the MPLS Core Configuration on R1, R2 and R3 – this is quite straight forward, we are just going to use OSPF and LDP on the directly connected interfaces:

[codegroup]
[python linenumbers=”false” tab=”R1 Core Base Config”]
configure
system
name “R1”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
port 1/1/3
shutdown
ethernet
exit
exit
router
interface “R2”
address 10.1.2.1/27
port 1/1/1
no shutdown
exit
interface “R3”
address 10.1.3.1/27
port 1/1/2
no shutdown
exit
interface “system”
address 10.10.10.1/32
no shutdown
exit
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R2”
no shutdown
exit
interface “R3”
no shutdown
exit
exit
exit
ldp
interface-parameters
interface “R2”
exit
interface “R3″
exit
exit
targeted-session
exit
no shutdown
exit
exit
exit all
[/python]
[python linenumbers=”false” tab=”R2 Core Base Config”]
configure
system
name “R2”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
port 1/1/3
shutdown
ethernet
exit
exit
router
interface “R1”
address 10.1.2.2/27
port 1/1/1
no shutdown
exit
interface “R3”
address 10.2.3.2/27
port 1/1/2
no shutdown
exit
interface “system”
address 10.10.10.2/32
no shutdown
exit
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R1”
no shutdown
exit
interface “R3”
no shutdown
exit
exit
exit
ldp
interface-parameters
interface “R1”
exit
interface “R3″
exit
exit
targeted-session
exit
no shutdown
exit
exit
exit all
[/python]
[python linenumbers=”false” tab=”R3 Core Base Config”]
configure
system
name “R3”
exit
card 1
card-type iom3-xp-b
mda 1
mda-type m5-1gb-sfp-b
no shutdown
exit
no shutdown
exit
port 1/1/1
ethernet
exit
no shutdown
exit
port 1/1/2
ethernet
exit
no shutdown
exit
port 1/1/3
shutdown
ethernet
exit
exit
router
interface “R1”
address 10.1.3.3/27
port 1/1/2
no shutdown
exit
interface “R2”
address 10.2.3.3/27
port 1/1/3
no shutdown
exit
interface “system”
address 10.10.10.3/32
no shutdown
exit
ospf
area 0.0.0.0
interface “system”
no shutdown
exit
interface “R1”
no shutdown
exit
interface “R2”
no shutdown
exit
exit
exit
ldp
interface-parameters
interface “R1”
exit
interface “R2″
exit
exit
targeted-session
exit
no shutdown
exit
exit
exit all
[/python][/codegroup]
The Layer 2 Service that we are going to build is a VPLS and will be using Spoke-SDPs that connected to each adjacent router (an alternate could be to use a full-mesh but I specifically want to test STP operation here)
[codegroup]
[python linenumbers=”false” tab=”R1 SDP to R2 and R3″]
*A:R1>config>service# info
———————————————-
sdp 2 mpls create
far-end 10.10.10.2
ldp
keep-alive
shutdown
exit
no shutdown
exit
sdp 3 mpls create
far-end 10.10.10.3
ldp
keep-alive
shutdown
exit
no shutdown
exit
[/python]
[python linenumbers=”false” tab=”R2 SDP to R1 and R3″]
*A:R2>config>service# info
———————————————-
sdp 1 mpls create
far-end 10.10.10.1
ldp
keep-alive
shutdown
exit
no shutdown
exit
sdp 3 mpls create
far-end 10.10.10.3
ldp
keep-alive
shutdown
exit
no shutdown
exit
[/python]
[python linenumbers=”false” tab=”R3 SDP to R1 and R2″]
*A:R3>config>service# info
———————————————-
sdp 1 mpls create
far-end 10.10.10.1
ldp
keep-alive
shutdown
exit
no shutdown
exit
sdp 2 mpls create
far-end 10.10.10.2
ldp
keep-alive
shutdown
exit
no shutdown
exit
[/python][/codegroup]
Verifying the SDPs are up:
[codegroup]
[python linenumbers=”false” tab=”R1 SDP State”]
A:R1# show service sdp

============================================================================
Services: Service Destination Points
============================================================================
SdpId AdmMTU OprMTU Far End Adm Opr Del LSP Sig
—————————————————————————-
1 0 8914 10.10.10.1 Up Up MPLS L TLDP
2 0 8914 10.10.10.2 Up Up MPLS L TLDP
—————————————————————————-
Number of SDPs : 2
—————————————————————————-
Legend: R = RSVP, L = LDP, B = BGP, M = MPLS-TP, n/a = Not Applicable
============================================================================
[/python]
[/codegroup]
With the transport infrastructure in place VPLS 100 without the customer access components can be set up:
[codegroup]
[python linenumbers=”false” tab=”Initial R1 VPLS 100 Config”]
*A:R1>config>service>vpls$ pwc
——————————————————————————-
Present Working Context :
——————————————————————————-

configure
service
vpls “100” customer 1 create
——————————————————————————-
A:R1>config>service>vpls$ info
———————————————-
stp
no shutdown
exit
spoke-sdp 2:100 create
no shutdown
exit
spoke-sdp 3:100 create
no shutdown
exit
no shutdown
[/python]
[python linenumbers=”false” tab=”Initial R2 VPLS 100 Config”]
*A:R2>config>service>vpls$ pwc
——————————————————————————-
Present Working Context :
——————————————————————————-

configure
service
vpls “100” customer 1 create
——————————————————————————-
A:R2>config>service>vpls$ info
———————————————-
stp
no shutdown
exit
spoke-sdp 1:100 create
no shutdown
exit
spoke-sdp 3:100 create
no shutdown
exit
no shutdown
[/python]
[python linenumbers=”false” tab=”Initial R3 VPLS 100 Config”]
*A:R3>config>service>vpls$ pwc
——————————————————————————-
Present Working Context :
——————————————————————————-

configure
service
vpls “100” customer 1 create
——————————————————————————-
A:R3>config>service>vpls$ info
———————————————-
stp
no shutdown
exit
spoke-sdp 1:100 create
no shutdown
exit
spoke-sdp 2:100 create
no shutdown
exit
no shutdown
[/python]
[/codegroup]
Verify that VPLS 100 is up and running:
[codegroup]
[python linenumbers=”false” tab=”R1 VPLS 100 Spoke SDP State”]
*A:R1>config>service>*A:R1# show service id 100 base | match Ident post-lines 3
Identifier Type AdmMTU OprMTU Adm Opr
——————————————————————————-
sdp:2:100 S(10.10.10.2) Spok 0 8914 Up Up
sdp:3:100 S(10.10.10.3) Spok 0 8914 Up Up
[/python]
[python linenumbers=”false” tab=”R2 VPLS 100 Spoke SDP State”]
A:R2# show service id 100 base | match Ident post-lines 3
Identifier Type AdmMTU OprMTU Adm Opr
——————————————————————————-
sdp:1:100 S(10.10.10.1) Spok 0 8914 Up Up
sdp:3:100 S(10.10.10.3) Spok 0 8914 Up Up
[/python]
[python linenumbers=”false” tab=”R3 VPLS 100 Spoke SDP State”]
A:R3# show service id 100 base | match Ident post-lines 3
Identifier Type AdmMTU OprMTU Adm Opr
——————————————————————————-
sdp:1:100 S(10.10.10.1) Spok 0 8914 Up Up
sdp:2:100 S(10.10.10.2) Spok 0 8914 Up Up
[/python]
[/codegroup]
Looks good With 3 routers each connecting to each other using spokes will introduce a bridging loop so we need a loop avoidance mechanism – luckily we enabled STP, so lets see how STP is behaving:
[codegroup]
[python linenumbers=”false” tab=”R1 VPLS 100 STP State” highlight=”6-7″]
*A:R1# show service id 100 stp

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id : 80:00.da:00:ff:00:00:01 Top. Change Count : 4
Root Bridge : This Bridge Stp Oper State : Up
Primary Bridge : N/A Topology Change : Inactive
Mode : Rstp Last Top. Change : 0d 00:10:13
Vcp Active Prot. : N/A
Root Port : N/A External RPC : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id Oper- Port- Port- Port- Oper- Link- Active
State Role State Num Edge Type Prot.
——————————————————————————-
2:100 Up Designated Forward 2049 True Pt-pt Rstp
3:100 Up Backup Discard 2050 False Pt-pt Rstp
===============================================================================
[/python]
[python linenumbers=”false” tab=”R2 VPLS 100 STP State” highlight=”6-7″]
*A:R2# show service id 100 stp

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id : 80:00.da:00:ff:00:00:01 Top. Change Count : 3
Root Bridge : This Bridge Stp Oper State : Up
Primary Bridge : N/A Topology Change : Inactive
Mode : Rstp Last Top. Change : 0d 00:10:47
Vcp Active Prot. : N/A
Root Port : N/A External RPC : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id Oper- Port- Port- Port- Oper- Link- Active
State Role State Num Edge Type Prot.
——————————————————————————-
1:100 DwnstrmLp Designated Discard 2049 False Pt-pt Rstp
3:100 Up Backup Discard 2050 False Pt-pt Rstp
===============================================================================
[/python]
[python linenumbers=”false” tab=”R3 VPLS 100 STP State” highlight=”6-7″]
*A:R3# show service id 100 stp

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id : 80:00.da:00:ff:00:00:01 Top. Change Count : 3
Root Bridge : This Bridge Stp Oper State : Up
Primary Bridge : N/A Topology Change : Inactive
Mode : Rstp Last Top. Change : 0d 00:10:54
Vcp Active Prot. : N/A
Root Port : N/A External RPC : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id Oper- Port- Port- Port- Oper- Link- Active
State Role State Num Edge Type Prot.
——————————————————————————-
1:100 Up Designated Forward 2048 False Pt-pt Rstp
2:100 Up Designated Forward 2049 False Pt-pt Rstp
===============================================================================
[/python][/codegroup]
This doesn’t seem right, SDP 1:100 on R2 is saying that the downstream interface is looped and both interfaces are discarding!

If we look at the highlighted lines on each of the router outputs we notice that all Routers in the VPLS have the same Bridge ID, which is definitely a bad thing.

For SROS, the Bridge Id is partly derived from the chassis MAC address:
[python linenumbers=”false”]*A:R1# show chassis detail | match MAC
Base MAC address : da:00:ff:00:00:01[/python]
[python linenumbers=”false”]*A:R2# show chassis detail | match MAC
Base MAC address : da:00:ff:00:00:01[/python]
[python linenumbers=”false”]*A:R3# show chassis detail | match MAC
Base MAC address : da:00:ff:00:00:01[/python]
With real hardware, the Chassis MAC address actually is unique so this problem wont come up – however with the VSRs they’re all the same.

As an asside, the Chassis MAC address is used in a few places besides STP, one is with the SNMP engine id
[python linenumbers=”false” highlight=”2,4″]*A:R1# show chassis detail | match MAC
Base MAC address : da:00:ff:00:00:01
*A:R1# show system information | match Engine
SNMP Engine ID : 0000197f0000da00ff000001
SNMP Engine Boots : 11[/python]

It is possible within the configuration to manually set the Engine ID (I think it would probably be best to do this in production just in case you end up replacing faulty hardware)

With SROS version 14.0R4 a new option for the boot options file (or bof) was introduced which allows the manual setting of the chassis MAC address (followed by a reboot):
[python linenumbers=”false”]*A:R14# bof system-base-mac 00:11:22:33:44:02
*A:R14# bof save
Writing BOF to cf3:/bof.cfg … OK
Completed.
Writing configuration to cf3:\config.cfg
Saving configuration … OK
Completed.
A:R14# /admin reboot
Are you sure you want to reboot (y/n)? y[/python]
Which is great but this particular set up is using SROS 12.0R6 and that BOF option doesn’t exist an alternate method is required.

For STP we can cast our mind back to remember what the Bridge ID consists of… It’s both the Priority (which by default is 32768) and the Bridge MAC address.

So as a quick and nasty fix, I should just be able to change the STP Priority in VPLS 100 on R1/R2/R3 and resolve the STP problem, it also will allow me to specifically select a root bridge which is probably a good thing to do.
[python linenumbers=”false” tab= “R1 VPLS 100 STP”]*A:R1# configure service vpls 100 stp priority 4096[/python]
[python linenumbers=”false” tab= “R2 VPLS 100 STP”]*A:R2# configure service vpls 100 stp priority 8192[/python]
[python linenumbers=”false” tab= “R3 VPLS 100 STP”]*A:R3# configure service vpls 100 stp priority 16384[/python]
Lets see how things are going now:

[codegroup]
[python linenumbers=”false” tab= “R1 VPLS 100 STP”]*A:R1# show service id 100 stp

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id : 10:00.da:00:ff:00:00:01 Top. Change Count : 6
Root Bridge : This Bridge Stp Oper State : Up
Primary Bridge : N/A Topology Change : Inactive
Mode : Rstp Last Top. Change : 0d 00:00:35
Vcp Active Prot. : N/A
Root Port : N/A External RPC : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id Oper- Port- Port- Port- Oper- Link- Active
State Role State Num Edge Type Prot.
——————————————————————————-
2:100 Up Designated Forward 2049 False Pt-pt Rstp
3:100 Up Designated Forward 2050 False Pt-pt Rstp
===============================================================================[/python]
[python linenumbers=”false” tab= “R2 VPLS 100 STP”]*A:R2# show service id 100 stp

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id : 20:00.da:00:ff:00:00:01 Top. Change Count : 4
Root Bridge : 10:00.da:00:ff:00:00:01 Stp Oper State : Up
Primary Bridge : N/A Topology Change : Inactive
Mode : Rstp Last Top. Change : 0d 00:01:07
Vcp Active Prot. : N/A
Root Port : 2049 External RPC : 10

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id Oper- Port- Port- Port- Oper- Link- Active
State Role State Num Edge Type Prot.
——————————————————————————-
1:100 Up Root Forward 2049 False Pt-pt Rstp
3:100 Up Designated Forward 2050 False Pt-pt Rstp
===============================================================================[/python]
[python linenumbers=”false” tab= “R3 VPLS 100 STP”]*A:R3# show service id 100 stp

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id : 40:00.da:00:ff:00:00:01 Top. Change Count : 4
Root Bridge : 10:00.da:00:ff:00:00:01 Stp Oper State : Up
Primary Bridge : N/A Topology Change : Inactive
Mode : Rstp Last Top. Change : 0d 00:01:52
Vcp Active Prot. : N/A
Root Port : 2048 External RPC : 10

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id Oper- Port- Port- Port- Oper- Link- Active
State Role State Num Edge Type Prot.
——————————————————————————-
1:100 Up Root Forward 2048 False Pt-pt Rstp
2:100 Up Alternate Discard 2049 False Pt-pt Rstp
===============================================================================[/python][/codegroup]
Success, all routers have different bridge IDs and all agree that R1 is the root and only one port is in discarding state.

Now we will create the CE router attachments (Service Access Points) on the Core starting with R3 which is facing R4 – by default Ethernet ports are in network mode, to be able to bind to a service, the port must be mode access (or hybrid)
[python linenumbers=”false”]*A:R3# /configure port 1/1/1
*A:R3>config>port# shutdown
*A:R3>config>port# ethernet mode access
*A:R3>config>port# ethernet encap-type null
*A:R3>config>port# no shutdown
*A:R3>config>port# /configure service vpls 100
*A:R3>config>service>vpls# sap 1/1/1 create
*A:R3>config>service>vpls>sap$ show service id 100 base

===============================================================================
Service Basic Information
===============================================================================
Service Id : 100 Vpn Id : 0
Service Type : VPLS
Name : (Not Specified)
Description : (Not Specified)
Customer Id : 1 Creation Origin : manual
Last Status Change: 04/21/2017 13:20:28
Last Mgmt Change : 04/21/2017 13:44:59
Etree Mode : Disabled
Admin State : Up Oper State : Up
MTU : 1514 Def. Mesh VC Id : 100
SAP Count : 1 SDP Bind Count : 2
Snd Flush on Fail : Disabled Host Conn Verify : Disabled
Propagate MacFlush: Disabled Per Svc Hashing : Disabled
Allow IP Intf Bind: Disabled
Def. Gateway IP : None
Def. Gateway MAC : None
Temp Flood Time : Disabled Temp Flood : Inactive
Temp Flood Chg Cnt: 0
VSD Domain :

——————————————————————————-
Service Access & Destination Points
——————————————————————————-
Identifier Type AdmMTU OprMTU Adm Opr
——————————————————————————-
sap:1/1/1 null 1514 1514 Up Up
sdp:1:100 S(10.10.10.1) Spok 0 8914 Up Up
sdp:2:100 S(10.10.10.2) Spok 0 8914 Up Up
===============================================================================[/python]
Now things are going to get a little more complicated on R1 and R2 as we are going to establish a Multi-Chassis LAG towards R5. R5 is unaware of the MC-LAG, it is just talking LACP to R1 and R2 thinking they are just one system. R1 and R2 require synchronisation between each other to set up the Active-Standby LAG.

We’ll start by creating regular LAG-1 Facing R5 on R1 and R2 with a single port in each:
[codegroup]
[python linenumbers=”false” tab=”R1″]*A:R1# /configure port 1/1/3 shutdown
*A:R1# /configure port 1/1/3 ethernet mode access
*A:R1# /configure port 1/1/3 ethernet encap-type null
*A:R1# /configure port 1/1/3 ethernet autonegotiate limited
*A:R1# /configure port 1/1/3 no shutdown
*A:R1# /configure lag 1
*A:R1>config>lag$ mode access
*A:R1>config>lag$ lacp active
*A:R1>config>lag$ port 1/1/3
*A:R1>config>lag$ no shutdown[/python]
[python linenumbers=”false” tab=”R2″]*A:R2# /configure port 1/1/3 shutdown
*A:R2# /configure port 1/1/3 ethernet mode access
*A:R2# /configure port 1/1/3 ethernet encap-type null
*A:R2# /configure port 1/1/3 ethernet autonegotiate limited
*A:R2# /configure port 1/1/3 no shutdown
*A:R2# /configure lag 1
*A:R2>config>lag$ mode access
*A:R2>config>lag$ lacp active
*A:R2>config>lag$ port 1/1/3
*A:R2>config>lag$ no shutdown[/python]
[/codegroup]
Now to set up MC-LAG we need to set up a multi-chassis peering between R1 and R2 (multi-chassis redundancy supports more than just MC-LAG):
[codegroup]
[python linenumbers=”false” tab=”R1 MC Peer with R2″]*A:R1>config>lag# /configure redundancy multi-chassis peer 10.10.10.2 create
*A:R1>config>redundancy>multi-chassis>peer# no shutdown[/python]
[python linenumbers=”false” tab=”R2 MC Peer with R1″]*A:R2>config>lag# /configure redundancy multi-chassis peer 10.10.10.1 create
*A:R2>config>redundancy>multi-chassis>peer# no shutdown[/python]
[/codegroup]
Then we create the MC-LAG itself, we require the lacp-key, system-id and priority to be the same on each router:
[codegroup]
[python linenumbers=”false” tab=”R1 MC-LAG to R5″]*A:R1>config>redundancy>multi-chassis>peer# mc-lag
*A:R1>config>redundancy>mc>peer>mc-lag#lag 1 lacp-key 2468 remote-lag 1 system-id 00:00:be:ef:ca:fe system-priority 1000
*A:R1>config>redundancy>mc>peer>mc-lag#no shutdown[/python]
[python linenumbers=”false” tab=”R2 MC-LAG to R5″]*A:R2>config>redundancy>multi-chassis>peer# mc-lag
*A:R2>config>redundancy>mc>peer>mc-lag#lag 1 lacp-key 2468 remote-lag 1 system-id 00:00:be:ef:ca:fe system-priority 1000
*A:R2>config>redundancy>mc>peer>mc-lag#no shutdown[/python]
[/codegroup]
Now the MC-LAG should be up and running, first we’ll check the peering
[python linenumbers=”false”]*A:R1>config>redundancy>mc>peer>mc-lag# show redundancy multi-chassis all

===============================================================================
Multi-Chassis Peers
===============================================================================
Peer IP Peer Admin Client Admin Oper State
Src IP Auth
——————————————————————————-
10.10.10.2 Enabled MC-Sync: — — —
10.10.10.1 None MC-Ring: — — —
MC-Endpt: — — —
MC-Lag: Enabled Enabled —
MC-IPsec: — — Disabled
===============================================================================[/python]
[python linenumbers=”false”]*A:R2>config>redundancy>mc>peer>mc-lag# show redundancy multi-chassis all

===============================================================================
Multi-Chassis Peers
===============================================================================
Peer IP Peer Admin Client Admin Oper State
Src IP Auth
——————————————————————————-
10.10.10.1 Enabled MC-Sync: — — —
10.10.10.2 None MC-Ring: — — —
MC-Endpt: — — —
MC-Lag: Enabled Enabled —
MC-IPsec: — — Disabled
===============================================================================[/python]
Looks promising, lets check our LAG status
[codegroup][python linenumbers=”false” tab=”R1 LAG Status”]*A:R1>config>redundancy>mc>peer>mc-lag# show lag

===============================================================================
Lag Data
===============================================================================
Lag-id Adm Opr Weighted Threshold Up-Count MC Act/Stdby
——————————————————————————-
1 up down No 0 0 standby
——————————————————————————-
Total Lag-ids: 1 Single Chassis: 0 MC Act: 0 MC Stdby: 1
===============================================================================[/python]
[python linenumbers=”false” tab=”R2 LAG Status”]*A:R2>config>redundancy>mc>peer>mc-lag# show lag

===============================================================================
Lag Data
===============================================================================
Lag-id Adm Opr Weighted Threshold Up-Count MC Act/Stdby
——————————————————————————-
1 up down No 0 0 standby
——————————————————————————-
Total Lag-ids: 1 Single Chassis: 0 MC Act: 0 MC Stdby: 1
===============================================================================[/python][/codegroup]
Ummm… both of these are showing that they are in Multi-Chassis Standby

It turns out that within the MC-LAG configuration, the Base Chassis MAC needs to be unique too. While we cannot directly change the Base MAC prior to SROS version 14.0R4 there is actually an alternative method available. if we set the out-of-band management ethernet IP address, this will influence the chassis MAC address.
[python linenumbers=”false”]*A:R1>config>lag# show bof
===============================================================================
BOF (Memory)
===============================================================================
primary-image cf3:\timos\both.tim
primary-config cf3:\config.cfg
autonegotiate
duplex full
speed 100
wait 3
persist off
no li-local-save
no li-separate
console-speed 115200
===============================================================================
*A:R1>config>lag# /bof address 192.168.100.1/24
*A:R1>config>lag# /bof save
Writing BOF to cf3:/bof.cfg … OK
Completed.
*A:R1>config>lag# show bof
===============================================================================
BOF (Memory)
===============================================================================
primary-image cf3:\timos\both.tim
primary-config cf3:\config.cfg
address 192.168.100.1/24 active
autonegotiate
duplex full
speed 100
wait 3
persist off
no li-local-save
no li-separate
console-speed 115200
===============================================================================[/python]
Save and reboot
[python linenumbers=”false”]*A:R1>config>lag# /admin save
Writing configuration to cf3:\config.cfg
Saving configuration … OK
Completed.
A:R1>config>lag# /admin reboot
Are you sure you want to reboot (y/n)? y[/python]
We’ll do the same thing with R2 but give it a different IP so the MAC Addresses should be different:
[python linenumbers=”false”]*A:R2>config>lag# /bof address 192.168.100.2/24
*A:R2>config>lag# /bof save
Writing BOF to cf3:/bof.cfg … OK
Completed.
*A:R2>config>lag# /admin save
Writing configuration to cf3:\config.cfg
Saving configuration … OK
Completed.
A:R2>config>lag# /admin reboot
Are you sure you want to reboot (y/n)? y [/python]
After the reboot we can compare R1 and R2’s Base MAC Address
[python linenumbers=”false”]A:R1# show chassis detail | match MAC
Base MAC address : c8:01:ff:00:00:00[/python]
[python linenumbers=”false”]A:R2# show chassis detail | match MAC
Base MAC address : c8:02:ff:00:00:00[/python]
Okay they are different now – has it resolved our MC-LAG issue?
[codegroup][python linenumbers=”false” tab=”R1 LAG Port”]A:R1# show lag 1 port

[python linenumbers=”false” tab=”R1 LAG Port”]A:R2# show lag 1 port

===============================================================================
Lag Port States
LACP Status: e – Enabled, d – Disabled
===============================================================================
Lag-id Port-id Adm Act/Stdby Opr Primary Sub-group Forced Priority
——————————————————————————-
1(e) 1/1/1 up active up yes 1 – 32768
1/1/2 up active down 1 – 32768
===============================================================================[/python][/codegroup]
Yes R1, R2 and R5 are in alignment, now lets put the LAG into VPLS 100 on R1 and R2
[python linenumbers=”false”]A:R1# /configure service vpls 100 sap lag-1 create[/python]
[python linenumbers=”false”]A:R2# /configure service vpls 100 sap lag-1 create[/python]
Lets see if R5 can ping R4
[python linenumbers=”false”]A:R5# ping 192.168.1.4 count 1
PING 192.168.1.4 56 data bytes
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=12.3ms.

—- 192.168.1.4 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 12.3ms, avg = 12.3ms, max = 12.3ms, stddev = 0.000ms[/python]
Success!

Lets check the MAC address table in vpls 100 (Forwarding Data Base):
[codegroup]
[python linenumbers=”false” tab=”R1 FDB” highlight “9,10”]*A:R1>config>service>vpls>sap$ show service id 100 fdb detail

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId MAC Source-Identifier Type Last Change
Age
——————————————————————————-
100 50:00:00:07:00:01 sdp:3:100 L/0 04/21/17 14:47:33
100 da:00:ff:00:01:42 sap:lag-1 L/0 04/21/17 14:52:57
——————————————————————————-
No. of MAC Entries: 2
——————————————————————————-
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================[/python]
[python linenumbers=”false” tab=”R2 FDB” highlight “9,10”]*A:R2>config>service>vpls>sap$ show service id 100 fdb detail

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId MAC Source-Identifier Type Last Change
Age
——————————————————————————-
100 50:00:00:07:00:01 sdp:1:100 L/90 04/21/17 14:53:01
100 da:00:ff:00:01:42 sdp:1:100 L/90 04/21/17 14:45:05
——————————————————————————-
No. of MAC Entries: 2
——————————————————————————-
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================[/python]
[python linenumbers=”false” tab=”R3 FDB” highlight “9,10”]*A:R2>config>service>vpls>sap$ show service id 100 fdb detail

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId MAC Source-Identifier Type Last Change
Age
——————————————————————————-
100 50:00:00:07:00:01 sap:1/1/1 L/0 04/21/17 14:52:42
100 da:00:ff:00:01:42 sdp:1:100 L/0 04/21/17 14:44:46
——————————————————————————-
No. of MAC Entries: 2
——————————————————————————-
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================[/python][/codegroup]
Now to check out the MC-LAG resiliency, we’ll start a continuous ping on R5 to R4 and then shutdown port 1/1/3 (LAG-1) on R1
[python linenumbers=”false”]*A:R1>config>service>vpls>sap$ /configure port 1/1/3 shutdown[/python]
And Check if R2 LAG 1 Port 1/1/3 goes from standby to active
[python linenumbers=”false”]*A:R2>config>service>vpls>sap$ show lag 1 port

===============================================================================
Lag Port States
LACP Status: e – Enabled, d – Disabled
===============================================================================
Lag-id Port-id Adm Act/Stdby Opr Primary Sub-group Forced Priority
——————————————————————————-
1(e) 1/1/3 up active up yes 1 – 32768
===============================================================================[/python]
We can see the interface has come up and there were a few packets lost but the link recovered – we could speed up the link convergence time but I think the general concept has been demonstrated sucessfully.

The moral of the story here – with Virtual SROS systems, it’s worth ensuring you have a unique chassis MAC address!

Working with eve-ng (Active-Backup Bond Interfaces with eth0 and wlan0)

After a lack of updates, its’ time for a new blog post – this post is about linux networking particularly using active-backup bond interfaces for wired and wireless LAN interfaces, which is part of creating my updated virtual network lab environment.

Unetlab which was pretty much an alternative to GNS3 has now evolved into eve-ng which is quite a nice system. Amongst other things is it has a custom linux kernel that doesn’t block L2 Slow protocols like LACP. One of the things I specifically like about Eve besides having the facility to use html5 sessions to handle telnet/VNC consoles (as well as native tools) is that there are some SROS specific modifications that support the distributed VSR models as well as passing SMBIOS parameters etc.

I did a bare metal install pretty much following the process described in http://www.eve-ng.net/index.php/documentation/installation/bare-install but did a few more things.

I installed xcfe4 so I can have a graphical desktop with firefox so I can use eve locally, not just remoting into it.

I also did a few changes to the base install network configuration to allow the use of the copper ethernet as the primary interface but falling back to wireless.

Normally you cannot add a wireless interface into a bridge (normally eve binds eth0 into bridge pnet0 but simply adding wlan0 didn’t work)

Fortunately you can add a bond into a bridge, and the bond is less picky about who joins.

These are the items I added to /etc/network/interfaces

#Bond0 Config
auto bond0
iface bond0 inet manual
    bond-slaves eth0 wlan0
    bond-mode 1
    bond-miimon 100

bond-slaves are the link members of the bond (eth0 and wlan0 are my copper and wireless lan interfaces respectively)
bond-mode 1 is active-backup – Only one interface at a time will be operational, with the preference to the interface that is configured as bind-primary
bond-miimon 100 means that every 100ms the link state is checked

# Wireless interface
allow-hotplug wlan0
iface wlan0 inet manual
    wpa-ssid ReplaceThisWithYourSSID
    wpa-psk ReplaceThisWithYourPresharedKey
    bond-master bond0

I’m not sure if its mandatory but allow-hotplug wlan0 seems to help and bond-master seemed to be required.

The eth0 section was modified to the following

# The primary network interface
allow-hotplug eth0
iface eth0 inet manual
    bond-master bond0
    bond-primary eth0

Here allow-hotplug eth0 seems to wake the system to the fact a cable has been connected
bond-master bond0 as with wlan0, this appears to be needed
bond-primary eth0 means that when both eth0 and wlan0 are up, eth0 should be the one used.

And finally pnet0 was modified to use bond0 instead of eth0

auto pnet0
iface pnet0 inet dhcp
    bridge_ports bond0
    bridge_stp off

So after issuing a “service networking restart”, here’s our verifcation that the bond interface is up:

root@m4600:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: d0:67:e5:57:12:9e
Slave queue ID: 0

Slave Interface: wlan0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 24:77:03:b1:9f:78
Slave queue ID: 0

root@m4600:~# brctl show pnet0
bridge name     bridge id               STP enabled     interfaces
pnet0           8000.d067e557129e       no              bond0

root@m4600:~# ip -4 addr show pnet0
4: pnet0:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.31/24 brd 192.168.1.255 scope global pnet0
       valid_lft forever preferred_lft forever

Quick Network Verification:

root@m4600:~# ping 8.8.8.8 -c 3
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=30.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=24.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=28.3 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 24.924/27.798/30.131/2.164 ms

Summarise the bond status:

root@m4600:~# grep -A 1 "Interface\|Primary" /proc/net/bonding/bond0
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: wlan0
--
Slave Interface: eth0
MII Status: down
--
Slave Interface: wlan0
MII Status: up

Now Pull out the Ethernet cable

root@m4600:~# ping 8.8.8.8 -c 3
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=28.0 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=35.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=27.9 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 27.910/30.537/35.614/3.593 ms

Verify the bond interface is using wlan0

root@m4600:~# grep -A 1 "Interface\|Primary" /proc/net/bonding/bond0
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: wlan0
--
Slave Interface: eth0
MII Status: down
--
Slave Interface: wlan0
MII Status: up

Re Insert the Ethernet cable

root@m4600:~# ping 8.8.8.8 -c 3
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=23.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=21.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=23.1 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 21.968/22.822/23.397/0.628 ms
root@m4600:~# grep -A 1 "Interface\|Primary" /proc/net/bonding/bond0
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: eth0
--
Slave Interface: eth0
MII Status: up
--
Slave Interface: wlan0
MII Status: up

So this is all good. (Actually during this testing, I was SSHed into the laptop and the session didn’t break)

GNS3 with ExaBGP Part 2 – Full Internet Routing Table

Following on the the initial concept for using ExaBGP in part 1 we will kick it up a notch and inject a replica BGP full routing table feed. Thankfully the people at RIPE NCC have something we can use available through their Routing Information Service Raw Data page.

For my example I’m going to pull data from rrc00 which is based in Amsterdam and has a good number of peers. The data archive for rrc00 is located at http://data.ris.ripe.net/rrc00/ and the most straight forward file to download is the latest-bview.gz file.

A note about these bview files – These archives contain BGP data encoded in Multi-Threaded Routing Toolkit (MRT) Routing Information Export Format, which ExaBGP on its own cannot digest. Thankfully this is addressed by a python based MRT Parsing tool which includes a script to generate ExaBGP compatible configs (mrt2exabgp.py).

I recommend installing via a clone the git repositorary as you get the freshest version of the tool.

adam@gns3:~$ sudo -H pip install https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip
Collecting https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip
  Downloading https://github.com/YoshiyukiYamauchi/mrtparse/archive/master.zip (52kB)
    100% |████████████████████████████████| 61kB 122kB/s
Installing collected packages: mrtparse
  Running setup.py install for mrtparse ... done
Successfully installed mrtparse-1.4


adam@gns3:~$ wget http://data.ris.ripe.net/rrc00/latest-bview.gz

After downloading the 50MB+ file, we can use the mrt2exabgp.py script – details on its operation are here however for my purposes I will just use the -G and -P flags and redirect the output to exabgptable.py which will end becoming a python script that is called fullbgptable.py

My Linux box is a long in the tooth AMD Athlon II X4 630 based system with 16GB RAM, so not precisely a speed demon, and doing this conversion is going to take awhile, so I’m using the time command to see how long the process will actually take, while I grab a coffee.

adam@gns3:~$ time /usr/local/lib/python2.7/dist-packages/mrtparse/examples/mrt2exabgp.py -G -P latest-bview.gz  > fullbgptable.py

real    24m25.654s
user    24m19.396s
sys     0m0.612s
adam@gns3:~$ ls -la fullbgptable.py
-rw-rw-r-- 1 adam adam 35282115 Nov  8 23:54 fullbgptable.py

Okay… so that took nearly half an hour, lets have a quick look at this.

adam@gns3:~$ head fullbgptable.py
#!/usr/bin/env python

import sys
import time

msgs = [
'announce attributes origin IGP as-path [29608 3356 29396 29396 29396 39686 44953 ] med 13 community [3356:2 3356:22 3356:100 3356:123 3356:503 3356:2067 29608:30600] next-hop 79.143.241.12 nlri 93.95.248.0/21',
'announce attributes origin IGP as-path [29608 3356 6453 9498 58682 24389 ] med 13 community [3356:2 3356:22 3356:86 3356:502 3356:666 3356:2066 6453:2000 6453:2200 6453:2204 29608:30600] next-hop 79.143.241.12 nlri 202.56.5.0/24 202.56.6.0/23 202.56.7.0/24',
'announce attributes origin IGP as-path [29608 3356 25795 202773 ] med 13 community [3356:3 3356:22 3356:100 3356:123 3356:575 3356:2003 25795:100 25795:40000 29608:30600] next-hop 79.143.241.12 nlri 185.152.130.0/24 185.152.131.0/24',
'announce attributes origin IGP as-path [29608 6939 31027 44869 203646 ] med 11 community [29608:40090] next-hop 2a01:678::2 nlri 2a03:9aa0::/32',
adam@gns3:~$ tail fullbgptable.py
    msg = msgs.pop(0)
    if isinstance(msg, str):
        sys.stdout.write(msg + '\n')
        sys.stdout.flush()
    else:
        time.sleep(msg)

while True:
    time.sleep(1)

It appears good, so lets create the ini file for exabgp to use.

group SR4 {
    router-id 1.2.3.3;
    neighbor 1.2.3.4 {
        local-address 1.2.3.3;
        local-as 64512;
        peer-as 1234;
    }
    process fullbgp {
        run /usr/bin/python /home/adam/fullbgptable.py;
    }
}

Time to fire up SR4 and start pushing a whole lot of IPv4 routes to it (you may notice in the fullbgptable.py extract above an IPv6 prefix, however since we are only using the IPv4 address family, this will be ignored)

While ExaBGP is still running, lets see how many bgp routes we have received.

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512   21732    0 00h00m32s 21726/0/0 (IPv4)
                            7    0
-------------------------------------------------------------------------------

In a little over 30 seconds we have recieved 21726 routes but none of them are active, lets have a look at what we have been offered.

A:SR4# show router bgp routes
===============================================================================
 BGP Router ID:10.10.10.4       AS:1234        Local AS:1234
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP IPv4 Routes
===============================================================================
Flag  Network                                            LocalPref   MED
      Nexthop (Router)                                   Path-Id     Label
      As-Path
-------------------------------------------------------------------------------
i     1.0.4.0/24                                         None        13
      79.143.241.12                                      None        -
      29608 3356 4637 1221 38803 56203
i     1.0.5.0/24                                         None        13
      79.143.241.12                                      None        -
      29608 3356 4637 1221 38803 56203
i     1.0.6.0/24                                         None        13
A:SR4#  show router bgp routes 1.0.7.0/24 detail | match expression "Flags|Next"
Nexthop        : 79.143.241.12
Res. Nexthop   : Unresolved
Flags          : Invalid  IGP  Nexthop-Unresolved
Nexthop        : 79.143.241.12
Res. Nexthop   : Unresolved
Flags          : Invalid  IGP  Nexthop-Unresolved

The IP Nexthop is 79.143.241.12 but since SR4 doesn’t know how to get there, the route is invald.

Lets have a look at how many IP Nexthops are in fullbgptable.py

adam@gns3:~$ grep -Eo 'next-hop [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' fulltable.py | cut -f 2 | sort | uniq -c
   1177 12.0.1.63
     79 146.228.1.3
   1330 176.12.110.8
     58 178.255.145.243
    681 193.0.0.56
    849 193.150.22.1
    153 193.160.39.1
    847 202.12.28.1
   1876 203.119.76.5
    112 203.123.48.6
    543 208.51.134.248
    290 212.25.27.44
    110 213.200.87.254
    157 45.61.0.85
 102315 79.143.241.12

So we can see that there are a number of nexthops with the vast majority associated with 79.143.241.12 (the counts are not precisely the number of individual routes due to the way the fullbgptable.py has consolidated similar prefixes for performance purposes). To support installing these routes into the routing table on SR4 rather than modifying fullbgptable.py to use an alternate nexthop (say 1.2.3.3 which is used by ExaBGP), a quick and dirty workaround is to use a static blackhole route for 79.143.241.12/32 (on IOS this would be the equivalent of a null route)

A:SR4# configure router static-route 79.143.241.12/32 black-hole

And restart ExaBGP (If we really wanted to we could add blackhole routes for the other nexthops, however 79.143.241.12 covers the vast majority of routes).

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512   17966    0 00h00m32s 17950/16879/4204 (IPv4)
                          849    0
-------------------------------------------------------------------------------
A:SR4# show router route-table

===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
1.0.7.0/24                                    Blackh* BGP       00h01m12s  170
       Black Hole                                                   0
1.0.128.0/17                                  Blackh* BGP       00h00m14s  170
       Black Hole                                                   0
1.1.20.0/24                                   Blackh* BGP       00h00m58s  170
       Black Hole                                                   0
1.1.128.0/17                                  Blackh* BGP       00h00m14s  170
       Black Hole                                                   0
1.2.3.0/24                                    Local   Local     00h22m49s  0
       ExaBGP                                                       0
1.2.4.0/24                                    Blackh* BGP       00h00m35s  170
       Black Hole                                                   0
1.2.11.0/24                                   Blackh* BGP       00h00m50s  170
       Black Hole                                                   0
1.2.128.0/17                                  Blackh* BGP       00h00m14s  170

This looks pretty good now (however it looks like picking 1.2.3.0/24 for my link addressing wasn’t the best choice and I should have used something from RFC1918)

While ExaBGP is pushing routes to SR4, lets check the status:

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  347185    0 00h03m00s 347165/318512/269061 (IPv4)
                        44567    0
-------------------------------------------------------------------------------

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  405376    0 00h03m31s 405354/371121/325132 (IPv4)
                        54221    0
-------------------------------------------------------------------------------

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  483575    0 00h04m13s 483552/442325/431845 (IPv4)
                        72792    1
-------------------------------------------------------------------------------

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512      23    0 00h00m10s Disabled
                           10    0
-------------------------------------------------------------------------------

Okay, this isn’t a good thing, the state is showing up as disabled and there are no longer any prefix counts – something serious must have happened. On SROS routers, log 99 is a good first place to investigate potential problems.

A:SR4# show log log-id 99

===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
warning: 1 events dropped from log
Memory Log contents  [size=500   next event=43  (not wrapped)]

42 2016/11/08 14:29:56.40 UTC WARNING: BGP #2012 Base Peer 1: 1.2.3.3
"Peer 1: 1.2.3.3: Closing connection: VR 1: Group ExaBGP: Peer 1.2.3.3 not enabled or not in configuration"

41 2016/11/08 14:29:53.09 UTC CRITICAL: BGP #2015 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: out of memory - disabled the peer"

40 2016/11/08 14:29:53.08 UTC WARNING: BGP #2005 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: sending notification: code CEASE subcode OUT_OF_RESR"

39 2016/11/08 14:29:53.08 UTC WARNING: BGP #2002 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: moved from higher state ESTABLISHED to lower state IDLE due to event OUT_OF_MEMORY"

38 2016/11/08 14:25:10.64 UTC MINOR: BGP #2001 Base Peer 1: 1.2.3.3
"VR 1: Group ExaBGP: Peer 1.2.3.3: moved into established state"

We can see that Event 38 was when the BGP session came up, however event 39 brought the session down because we ran out of memory.

In GNS3 SR4 had been allocated 2GB of memory, obviously this is not enough if we expect to take a full BGP feed – lets bump the memory to 3GB and see if that fixes it.

Up SR4's Memory from 2GB to 3GB — Up SR4’s Memory from 2GB to 3GB

Restart SR4 and kick off ExaBGP again…

A:SR4# show router bgp summary | match Summary post-lines 100
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  647537    0 00h06m03s 647522/595050/595050 (IPv4)
                       102359    0
-------------------------------------------------------------------------------

Well, the session is alive for longer than previous, and the ExaBGP session console output has gone quiet, so it looks like it has transmitted everything. Lets wait a bit more just to be certain.

A:SR4# show router bgp summary | match Summary post-lines 100
A:SR4# A:SR4>show>router>bgp# show router bgp summary
===============================================================================
 BGP Router ID:10.10.10.4       AS:1234        Local AS:1234
===============================================================================
BGP Admin State         : Up          BGP Oper State              : Up
Total Peer Groups       : 1           Total Peers                 : 1
Total BGP Paths         : 110581      Total Path Memory           : 26928008
Total IPv4 Remote Rts   : 647522      Total IPv4 Rem. Active Rts  : 595050
Total McIPv4 Remote Rts : 0           Total McIPv4 Rem. Active Rts: 0
Total McIPv6 Remote Rts : 0           Total McIPv6 Rem. Active Rts: 0
Total IPv6 Remote Rts   : 0           Total IPv6 Rem. Active Rts  : 0
Total IPv4 Backup Rts   : 0           Total IPv6 Backup Rts       : 0

Total Supressed Rts     : 0           Total Hist. Rts             : 0
Total Decay Rts         : 0

Total VPN Peer Groups   : 0           Total VPN Peers             : 0
Total VPN Local Rts     : 0
Total VPN-IPv4 Rem. Rts : 0           Total VPN-IPv4 Rem. Act. Rts: 0
Total VPN-IPv6 Rem. Rts : 0           Total VPN-IPv6 Rem. Act. Rts: 0
Total VPN-IPv4 Bkup Rts : 0           Total VPN-IPv6 Bkup Rts     : 0

Total VPN Supp. Rts     : 0           Total VPN Hist. Rts         : 0
Total VPN Decay Rts     : 0

Total L2-VPN Rem. Rts   : 0           Total L2VPN Rem. Act. Rts   : 0
Total MVPN-IPv4 Rem Rts : 0           Total MVPN-IPv4 Rem Act Rts : 0
Total MDT-SAFI Rem Rts  : 0           Total MDT-SAFI Rem Act Rts  : 0
Total MSPW Rem Rts      : 0           Total MSPW Rem Act Rts      : 0
Total RouteTgt Rem Rts  : 0           Total RouteTgt Rem Act Rts  : 0
Total McVpnIPv4 Rem Rts : 0           Total McVpnIPv4 Rem Act Rts : 0
Total MVPN-IPv6 Rem Rts : 0           Total MVPN-IPv6 Rem Act Rts : 0
Total EVPN Rem Rts      : 0           Total EVPN Rem Act Rts      : 0
Total FlowIpv4 Rem Rts  : 0           Total FlowIpv4 Rem Act Rts  : 0
Total FlowIpv6 Rem Rts  : 0           Total FlowIpv6 Rem Act Rts  : 0

===============================================================================
BGP Summary
===============================================================================
Neighbor
Description
                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
                      PktSent OutQ
-------------------------------------------------------------------------------
1.2.3.3
                64512  647547    0 00h10m58s 647522/595050/595050 (IPv4)
                       102369    0
-------------------------------------------------------------------------------

The session is still up and no new routes have been received.

While this post has been using SROS, this example can be applied to any BGP system, just make sure you have sufficient memory in your system as the IPv4 routing table is not small! ExaBGP is a pretty interesting tool and in this case can help make a simulated environment seem more like the real world, the only thing is that this is a static snapshot and there is ongoing churn within the internet.