The case of Nokia Virtual Service Router and the non-unique Chassis MAC Address

So I’m playing with eve-ng and have decided to work on a Layer 2 scenario and a few problems with my emulation environment came up which needed a way forward, which resulted in this rambling tale…

SROS 12.0R6 5 Router Topology

R1, R2 and R3 Will be the MPLS Core with VPLS configured, while R4 and R5 will be Layer 3 CE devices that talk to each other over the VPLS.

The CE Devices are pretty straight forward so we’ll get those up first

R4 is a single-ended configuration with Interface R5 on Port 1/1/1 having the IP 192.168.1.4/27

configure
    system
        name "R4"
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    router 
        interface "R5"
            address 192.168.1.4/27
            port 1/1/1
            no shutdown
        exit
        interface "system"
            no shutdown
        exit
    exit
exit all

R5 is a a little more complex, it has a LAG toward – Interface R4 on LAG-1 with Ports 1/1/1 and 1/1/2 having the IP 192.168.1.5/27

configure
    system
        name "R5"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
            autonegotiate limited
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
            autonegotiate limited
        exit
        no shutdown
    exit
    lag 1                             
        port 1/1/1 
        port 1/1/2 
        lacp active administrative-key 32768 
        no shutdown
    exit
    router 
        interface "R4"
            address 192.168.1.5/27
            port lag-1
            no shutdown
        exit
        interface "system"
            no shutdown
        exit                          
    exit
exit all

Multi-speed Ethernet interfaces when associated with a LAG must have autonegotiate set to limited to control the bundle member speed so they all bundle members operate the same speed

Now to Develop the MPLS Core Configuration on R1, R2 and R3 – this is quite straight forward, we are just going to use OSPF and LDP on the directly connected interfaces:

configure
    system
        name "R1"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        shutdown
        ethernet
        exit
    exit
    router 
        interface "R2"
            address 10.1.2.1/27
            port 1/1/1
            no shutdown
        exit
        interface "R3"
            address 10.1.3.1/27
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.1/32
            no shutdown
        exit
        ospf
            area 0.0.0.0              
                interface "system"
                    no shutdown
                exit
                interface "R2"
                    no shutdown
                exit
                interface "R3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "R2"
                exit
                interface "R3"
                exit
            exit
            targeted-session
            exit                      
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R2"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        shutdown
        ethernet
        exit
    exit
    router 
        interface "R1"
            address 10.1.2.2/27
            port 1/1/1
            no shutdown
        exit
        interface "R3"
            address 10.2.3.2/27
            port 1/1/2
            no shutdown
        exit
        interface "system"
            address 10.10.10.2/32
            no shutdown
        exit
        ospf
            area 0.0.0.0              
                interface "system"
                    no shutdown
                exit
                interface "R1"
                    no shutdown
                exit
                interface "R3"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "R1"
                exit
                interface "R3"
                exit
            exit
            targeted-session
            exit                      
            no shutdown
        exit
    exit
exit all

configure
    system
        name "R3"
    exit
    card 1
        card-type iom3-xp-b
        mda 1
            mda-type m5-1gb-sfp-b
            no shutdown               
        exit
        no shutdown
    exit
    port 1/1/1
        ethernet
        exit
        no shutdown
    exit
    port 1/1/2
        ethernet
        exit
        no shutdown
    exit
    port 1/1/3
        shutdown
        ethernet
        exit
    exit
    router 
        interface "R1"
            address 10.1.3.3/27
            port 1/1/2
            no shutdown
        exit
        interface "R2"
            address 10.2.3.3/27
            port 1/1/3
            no shutdown
        exit
        interface "system"
            address 10.10.10.3/32
            no shutdown
        exit
        ospf
            area 0.0.0.0              
                interface "system"
                    no shutdown
                exit
                interface "R1"
                    no shutdown
                exit
                interface "R2"
                    no shutdown
                exit
            exit
        exit
        ldp
            interface-parameters
                interface "R1"
                exit
                interface "R2"
                exit
            exit
            targeted-session
            exit                      
            no shutdown
        exit
    exit
exit all

The Layer 2 Service that we are going to build is a VPLS and will be using Spoke-SDPs that connected to each adjacent router (an alternate could be to use a full-mesh but I specifically want to test STP operation here)

*A:R1>config>service# info 
----------------------------------------------
        sdp 2 mpls create
            far-end 10.10.10.2
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit
        sdp 3 mpls create
            far-end 10.10.10.3
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit

*A:R2>config>service# info 
----------------------------------------------
        sdp 1 mpls create
            far-end 10.10.10.1
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit
        sdp 3 mpls create
            far-end 10.10.10.3
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit

*A:R3>config>service# info 
----------------------------------------------
        sdp 1 mpls create
            far-end 10.10.10.1
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit
        sdp 2 mpls create
            far-end 10.10.10.2
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit

Verifying the SDPs are up:

A:R1# show service sdp 

============================================================================
Services: Service Destination Points
============================================================================
SdpId  AdmMTU  OprMTU  Far End          Adm  Opr         Del     LSP   Sig
----------------------------------------------------------------------------
2      0       8914    10.10.10.2       Up   Up          MPLS    L     TLDP
3      0       8914    10.10.10.3       Up   Up          MPLS    L     TLDP
----------------------------------------------------------------------------
Number of SDPs : 2
----------------------------------------------------------------------------
Legend: R = RSVP, L = LDP, B = BGP, M = MPLS-TP, n/a = Not Applicable
============================================================================

A:R2# show service sdp 

============================================================================
Services: Service Destination Points
============================================================================
SdpId  AdmMTU  OprMTU  Far End          Adm  Opr         Del     LSP   Sig
----------------------------------------------------------------------------
1      0       8914    10.10.10.1       Up   Up          MPLS    L     TLDP
3      0       8914    10.10.10.3       Up   Up          MPLS    L     TLDP
----------------------------------------------------------------------------
Number of SDPs : 2
----------------------------------------------------------------------------
Legend: R = RSVP, L = LDP, B = BGP, M = MPLS-TP, n/a = Not Applicable
============================================================================

A:R3# show service sdp 

============================================================================
Services: Service Destination Points
============================================================================
SdpId  AdmMTU  OprMTU  Far End          Adm  Opr         Del     LSP   Sig
----------------------------------------------------------------------------
1      0       8914    10.10.10.1       Up   Up          MPLS    L     TLDP
2      0       8914    10.10.10.2       Up   Up          MPLS    L     TLDP
----------------------------------------------------------------------------
Number of SDPs : 2
----------------------------------------------------------------------------
Legend: R = RSVP, L = LDP, B = BGP, M = MPLS-TP, n/a = Not Applicable
============================================================================

With the transport infrastructure in place VPLS 100 without the customer access components can be set up:

*A:R1>config>service>vpls$ pwc 
-------------------------------------------------------------------------------
Present Working Context :
-------------------------------------------------------------------------------
 <root>
  configure 
  service 
  vpls "100" customer 1 create 
-------------------------------------------------------------------------------
A:R1>config>service>vpls$ info 
----------------------------------------------
            stp
                no shutdown
            exit
            spoke-sdp 2:100 create
                no shutdown
            exit
            spoke-sdp 3:100 create
                no shutdown
            exit
            no shutdown

*A:R2>config>service>vpls$ pwc 
-------------------------------------------------------------------------------
Present Working Context :
-------------------------------------------------------------------------------
 <root>
  configure 
  service 
  vpls "100" customer 1 create 
-------------------------------------------------------------------------------
A:R2>config>service>vpls$ info 
----------------------------------------------
            stp
                no shutdown
            exit
            spoke-sdp 1:100 create
                no shutdown
            exit
            spoke-sdp 3:100 create
                no shutdown
            exit
            no shutdown

*A:R3>config>service>vpls$ pwc 
-------------------------------------------------------------------------------
Present Working Context :
-------------------------------------------------------------------------------
 <root>
  configure 
  service 
  vpls "100" customer 1 create 
-------------------------------------------------------------------------------
A:R3>config>service>vpls$ info 
----------------------------------------------
            stp
                no shutdown
            exit
            spoke-sdp 1:100 create
                no shutdown
            exit
            spoke-sdp 2:100 create
                no shutdown
            exit
            no shutdown

Verify that VPLS 100 is up and running:

*A:R1>config>service>*A:R1# show service id 100 base | match Ident post-lines 3 
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sdp:2:100 S(10.10.10.2)                  Spok         0       8914    Up   Up
sdp:3:100 S(10.10.10.3)                  Spok         0       8914    Up   Up

A:R2# show service id 100 base | match Ident post-lines 3 
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sdp:1:100 S(10.10.10.1)                  Spok         0       8914    Up   Up
sdp:3:100 S(10.10.10.3)                  Spok         0       8914    Up   Up

A:R3# show service id 100 base | match Ident post-lines 3 
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sdp:1:100 S(10.10.10.1)                  Spok         0       8914    Up   Up
sdp:2:100 S(10.10.10.2)                  Spok         0       8914    Up   Up

Looks good With 3 routers each connecting to each other using spokes will introduce a bridging loop so we need a loop avoidance mechanism – luckily we enabled STP, so lets see how STP is behaving:

*A:R1# show service id 100 stp                        

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 80:00.da:00:ff:00:00:01  Top. Change Count : 4
Root Bridge        : This Bridge              Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:10:13
Vcp Active Prot.   : N/A                      
Root Port          : N/A                      External RPC      : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
2:100              Up        Designated Forward     2049   True   Pt-pt  Rstp
3:100              Up        Backup     Discard     2050   False  Pt-pt  Rstp
===============================================================================

*A:R2# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 80:00.da:00:ff:00:00:01  Top. Change Count : 3
Root Bridge        : This Bridge              Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:10:47
Vcp Active Prot.   : N/A                      
Root Port          : N/A                      External RPC      : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
1:100              DwnstrmLp Designated Discard     2049   False  Pt-pt  Rstp
3:100              Up        Backup     Discard     2050   False  Pt-pt  Rstp
===============================================================================

*A:R3# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 80:00.da:00:ff:00:00:01  Top. Change Count : 3
Root Bridge        : This Bridge              Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:10:54
Vcp Active Prot.   : N/A                      
Root Port          : N/A                      External RPC      : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
1:100              Up        Designated Forward     2048   False  Pt-pt  Rstp
2:100              Up        Designated Forward     2049   False  Pt-pt  Rstp
===============================================================================

This doesn’t seem right, SDP 1:100 on R2 is saying that the downstream interface is looped and both interfaces are discarding!

If we look at the highlighted lines on each of the router outputs we notice that all Routers in the VPLS have the same Bridge ID, which is definitely a bad thing.

For SROS, the Bridge Id is partly derived from the chassis MAC address:

*A:R1# show chassis detail | match MAC  
  Base MAC address                  : da:00:ff:00:00:01

*A:R2# show chassis detail | match MAC  
  Base MAC address                  : da:00:ff:00:00:01

*A:R3# show chassis detail | match MAC  
  Base MAC address                  : da:00:ff:00:00:01

With real hardware, the Chassis MAC address actually is unique so this problem wont come up – however with the VSRs they’re all the same.

As an asside, the Chassis MAC address is used in a few places besides STP, one is with the SNMP engine id

*A:R1# show chassis detail | match MAC      
  Base MAC address                  : da:00:ff:00:00:01
*A:R1# show system information | match Engine 
SNMP Engine ID         : 0000197f0000da00ff000001
SNMP Engine Boots      : 11

It is possible within the configuration to manually set the Engine ID (I think it would probably be best to do this in production just in case you end up replacing faulty hardware)

With SROS version 14.0R4 a new option for the boot options file (or bof) was introduced which allows the manual setting of the chassis MAC address (followed by a reboot):

*A:R14# bof system-base-mac 00:11:22:33:44:02 
*A:R14# bof save 
Writing BOF to cf3:/bof.cfg ... OK
Completed.
Writing configuration to cf3:\config.cfg
Saving configuration ... OK
Completed.
A:R14# /admin reboot 
Are you sure you want to reboot (y/n)? y

Which is great but this particular set up is using SROS 12.0R6 and that BOF option doesn’t exist an alternate method is required.

For STP we can cast our mind back to remember what the Bridge ID consists of… It’s both the Priority (which by default is 32768) and the Bridge MAC address.

So as a quick and nasty fix, I should just be able to change the STP Priority in VPLS 100 on R1/R2/R3 and resolve the STP problem, it also will allow me to specifically select a root bridge which is probably a good thing to do.
*A:R1# configure service vpls 100 stp priority 4096
*A:R2# configure service vpls 100 stp priority 8192
*A:R3# configure service vpls 100 stp priority 16384
Lets see how things are going now:

*A:R1# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 10:00.da:00:ff:00:00:01  Top. Change Count : 6
Root Bridge        : This Bridge              Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:00:35
Vcp Active Prot.   : N/A                      
Root Port          : N/A                      External RPC      : 0

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
2:100              Up        Designated Forward     2049   False  Pt-pt  Rstp
3:100              Up        Designated Forward     2050   False  Pt-pt  Rstp
===============================================================================

*A:R2# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 20:00.da:00:ff:00:00:01  Top. Change Count : 4
Root Bridge        : 10:00.da:00:ff:00:00:01  Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:01:07
Vcp Active Prot.   : N/A                      
Root Port          : 2049                     External RPC      : 10

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
1:100              Up        Root       Forward     2049   False  Pt-pt  Rstp
3:100              Up        Designated Forward     2050   False  Pt-pt  Rstp
===============================================================================

*A:R3# show service id 100 stp 

===============================================================================
Stp info, Service 100
===============================================================================
Bridge Id          : 40:00.da:00:ff:00:00:01  Top. Change Count : 4
Root Bridge        : 10:00.da:00:ff:00:00:01  Stp Oper State    : Up
Primary Bridge     : N/A                      Topology Change   : Inactive
Mode               : Rstp                     Last Top. Change  : 0d 00:01:52
Vcp Active Prot.   : N/A                      
Root Port          : 2048                     External RPC      : 10

===============================================================================
Stp port info
===============================================================================
Sap/Sdp/PIP Id     Oper-     Port-      Port-       Port-  Oper-  Link-  Active
                   State     Role       State       Num    Edge   Type   Prot.
-------------------------------------------------------------------------------
1:100              Up        Root       Forward     2048   False  Pt-pt  Rstp
2:100              Up        Alternate  Discard     2049   False  Pt-pt  Rstp
===============================================================================

Success, all routers have different bridge IDs and all agree that R1 is the root and only one port is in discarding state.

Now we will create the CE router attachments (Service Access Points) on the Core starting with R3 which is facing R4 – by default Ethernet ports are in network mode, to be able to bind to a service, the port must be mode access (or hybrid)

*A:R3# /configure port 1/1/1     
*A:R3>config>port# shutdown 
*A:R3>config>port# ethernet mode access 
*A:R3>config>port# ethernet encap-type null 
*A:R3>config>port# no shutdown 
*A:R3>config>port# /configure service vpls 100 
*A:R3>config>service>vpls# sap 1/1/1 create 
*A:R3>config>service>vpls>sap$ show service id 100 base

===============================================================================
Service Basic Information
===============================================================================
Service Id        : 100                 Vpn Id            : 0
Service Type      : VPLS                
Name              : (Not Specified)
Description       : (Not Specified)
Customer Id       : 1                   Creation Origin   : manual
Last Status Change: 04/21/2017 13:20:28 
Last Mgmt Change  : 04/21/2017 13:44:59 
Etree Mode        : Disabled            
Admin State       : Up                  Oper State        : Up
MTU               : 1514                Def. Mesh VC Id   : 100
SAP Count         : 1                   SDP Bind Count    : 2
Snd Flush on Fail : Disabled            Host Conn Verify  : Disabled
Propagate MacFlush: Disabled            Per Svc Hashing   : Disabled
Allow IP Intf Bind: Disabled            
Def. Gateway IP   : None                
Def. Gateway MAC  : None                
Temp Flood Time   : Disabled            Temp Flood        : Inactive
Temp Flood Chg Cnt: 0                   
VSD Domain        : <none>            
 
-------------------------------------------------------------------------------
Service Access & Destination Points
-------------------------------------------------------------------------------
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sap:1/1/1                                null         1514    1514    Up   Up
sdp:1:100 S(10.10.10.1)                  Spok         0       8914    Up   Up
sdp:2:100 S(10.10.10.2)                  Spok         0       8914    Up   Up
===============================================================================

Now things are going to get a little more complicated on R1 and R2 as we are going to establish a Multi-Chassis LAG towards R5. R5 is unaware of the MC-LAG, it is just talking LACP to R1 and R2 thinking they are just one system. R1 and R2 require synchronisation between each other to set up the Active-Standby LAG.

We’ll start by creating regular LAG-1 Facing R5 on R1 and R2 with a single port in each:

*A:R1# /configure port 1/1/3 shutdown                          
*A:R1# /configure port 1/1/3 ethernet mode access 
*A:R1# /configure port 1/1/3 ethernet encap-type null 
*A:R1# /configure port 1/1/3 ethernet autonegotiate limited 
*A:R1# /configure port 1/1/3 no shutdown                    
*A:R1# /configure lag 1 
*A:R1>config>lag$ mode access 
*A:R1>config>lag$ lacp active 
*A:R1>config>lag$ port 1/1/3 
*A:R1>config>lag$ no shutdown

*A:R2# /configure port 1/1/3 shutdown                          
*A:R2# /configure port 1/1/3 ethernet mode access 
*A:R2# /configure port 1/1/3 ethernet encap-type null 
*A:R2# /configure port 1/1/3 ethernet autonegotiate limited 
*A:R2# /configure port 1/1/3 no shutdown                    
*A:R2# /configure lag 1 
*A:R2>config>lag$ mode access 
*A:R2>config>lag$ lacp active 
*A:R2>config>lag$ port 1/1/3 
*A:R2>config>lag$ no shutdown

Now to set up MC-LAG we need to set up a multi-chassis peering between R1 and R2 (multi-chassis redundancy supports more than just MC-LAG):

*A:R1>config>lag# /configure redundancy multi-chassis peer 10.10.10.2 create
*A:R1>config>redundancy>multi-chassis>peer# no shutdown

*A:R2>config>lag# /configure redundancy multi-chassis peer 10.10.10.1 create 
*A:R2>config>redundancy>multi-chassis>peer# no shutdown

Then we create the MC-LAG itself, we require the lacp-key, system-id and priority to be the same on each router:

*A:R1>config>redundancy>multi-chassis>peer# mc-lag
*A:R1>config>redundancy>mc>peer>mc-lag#lag 1 lacp-key 2468 remote-lag 1 system-id 00:00:be:ef:ca:fe system-priority 1000 
*A:R1>config>redundancy>mc>peer>mc-lag#no shutdown

*A:R2>config>redundancy>multi-chassis>peer# mc-lag
*A:R2>config>redundancy>mc>peer>mc-lag#lag 1 lacp-key 2468 remote-lag 1 system-id 00:00:be:ef:ca:fe system-priority 1000 
*A:R2>config>redundancy>mc>peer>mc-lag#no shutdown

Now the MC-LAG should be up and running, first we’ll check the peering

*A:R1>config>redundancy>mc>peer>mc-lag# show redundancy multi-chassis all 

===============================================================================
Multi-Chassis Peers
===============================================================================
Peer IP          Peer Admin      Client    Admin        Oper         State
 Src IP           Auth                                               
-------------------------------------------------------------------------------
10.10.10.2       Enabled         MC-Sync:  --           --           --
 10.10.10.1       None           MC-Ring:  --           --           --
                                 MC-Endpt: --           --           --
                                 MC-Lag:   Enabled      Enabled      --
                                 MC-IPsec: --           --           Disabled
===============================================================================

*A:R2>config>redundancy>mc>peer>mc-lag# show redundancy multi-chassis all 

===============================================================================
Multi-Chassis Peers
===============================================================================
Peer IP          Peer Admin      Client    Admin        Oper         State
 Src IP           Auth                                               
-------------------------------------------------------------------------------
10.10.10.1       Enabled         MC-Sync:  --           --           --
 10.10.10.2       None           MC-Ring:  --           --           --
                                 MC-Endpt: --           --           --
                                 MC-Lag:   Enabled      Enabled      --
                                 MC-IPsec: --           --           Disabled
===============================================================================

Looks promising, lets check our LAG status
*A:R1>config>redundancy>mc>peer>mc-lag# show lag 

===============================================================================
Lag Data
===============================================================================
Lag-id         Adm     Opr     Weighted Threshold Up-Count MC Act/Stdby
-------------------------------------------------------------------------------
1              up      down    No       0         0        standby
-------------------------------------------------------------------------------
Total Lag-ids: 1       Single Chassis: 0        MC Act: 0       MC Stdby: 1
===============================================================================

*A:R2>config>redundancy>mc>peer>mc-lag# show lag 

===============================================================================
Lag Data
===============================================================================
Lag-id         Adm     Opr     Weighted Threshold Up-Count MC Act/Stdby
-------------------------------------------------------------------------------
1              up      down    No       0         0        standby
-------------------------------------------------------------------------------
Total Lag-ids: 1       Single Chassis: 0        MC Act: 0       MC Stdby: 1
===============================================================================

Ummm… both of these are showing that they are in Multi-Chassis Standby

It turns out that within the MC-LAG configuration, the Base Chassis MAC needs to be unique too. While we cannot directly change the Base MAC prior to SROS version 14.0R4 there is actually an alternative method available. if we set the out-of-band management ethernet IP address, this will influence the chassis MAC address.

*A:R1>config>lag# show bof 
===============================================================================
BOF (Memory)
===============================================================================
    primary-image    cf3:\timos\both.tim
    primary-config   cf3:\config.cfg
    autonegotiate
    duplex           full
    speed            100
    wait             3
    persist          off
    no li-local-save
    no li-separate
    console-speed    115200
===============================================================================
*A:R1>config>lag# /bof address 192.168.100.1/24 
*A:R1>config>lag# /bof save 
Writing BOF to cf3:/bof.cfg ... OK
Completed.
*A:R1>config>lag# show bof 
===============================================================================
BOF (Memory)
===============================================================================
    primary-image    cf3:\timos\both.tim
    primary-config   cf3:\config.cfg
    address          192.168.100.1/24 active
    autonegotiate
    duplex           full
    speed            100
    wait             3
    persist          off
    no li-local-save
    no li-separate
    console-speed    115200
===============================================================================

Save and reboot
*A:R1>config>lag# /admin save 
Writing configuration to cf3:\config.cfg
Saving configuration ... OK
Completed.
A:R1>config>lag# /admin reboot 
Are you sure you want to reboot (y/n)? y

We’ll do the same thing with R2 but give it a different IP so the MAC Addresses should be different:
*A:R2>config>lag# /bof address 192.168.100.2/24 
*A:R2>config>lag# /bof save 
Writing BOF to cf3:/bof.cfg ... OK
Completed.
*A:R2>config>lag# /admin save 
Writing configuration to cf3:\config.cfg
Saving configuration ... OK
Completed.
A:R2>config>lag# /admin reboot 
Are you sure you want to reboot (y/n)? y 

After the reboot we can compare R1 and R2’s Base MAC Address
A:R1# show chassis detail | match MAC 
  Base MAC address                  : c8:01:ff:00:00:00

A:R2# show chassis detail | match MAC 
  Base MAC address                  : c8:02:ff:00:00:00

Okay they are different now – has it resolved our MC-LAG issue?
A:R1# show lag 1 port 

===============================================================================
Lag Port States
LACP Status: e - Enabled, d - Disabled
===============================================================================
Lag-id Port-id   Adm   Act/Stdby Opr   Primary  Sub-group     Forced  Priority
-------------------------------------------------------------------------------
1(e)   1/1/3     up    active    up    yes      1             -       32768
===============================================================================

A:R2# show lag 1 port 

===============================================================================
Lag Port States
LACP Status: e - Enabled, d - Disabled
===============================================================================
Lag-id Port-id   Adm   Act/Stdby Opr   Primary  Sub-group     Forced  Priority
-------------------------------------------------------------------------------
1(e)   1/1/3     up    standby   down  yes      1             -       32768
===============================================================================

A:R5# show lag 1 port 

===============================================================================
Lag Port States
LACP Status: e - Enabled, d - Disabled
===============================================================================
Lag-id Port-id   Adm   Act/Stdby Opr   Primary  Sub-group     Forced  Priority
-------------------------------------------------------------------------------
1(e)   1/1/1     up    active    up    yes      1             -       32768
       1/1/2     up    active    down           1             -       32768
===============================================================================

Yes R1, R2 and R5 are in alignment, now lets put the LAG into VPLS 100 on R1 and R2
A:R1# /configure service vpls 100 sap lag-1 create
A:R2# /configure service vpls 100 sap lag-1 create
Lets see if R5 can ping R4
A:R5# ping 192.168.1.4 count 1 
PING 192.168.1.4 56 data bytes
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=12.3ms.

---- 192.168.1.4 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 12.3ms, avg = 12.3ms, max = 12.3ms, stddev = 0.000ms

Success!

Lets check the MAC address table in vpls 100 (Forwarding Data Base):

*A:R1>config>service>vpls>sap$ show service id 100 fdb detail 

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
100       50:00:00:07:00:01 sdp:3:100                L/0      04/21/17 14:47:33
100       da:00:ff:00:01:42 sap:lag-1                L/0      04/21/17 14:52:57
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

*A:R2>config>service>vpls>sap$ show service id 100 fdb detail 

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
100       50:00:00:07:00:01 sdp:1:100                L/90     04/21/17 14:53:01
100       da:00:ff:00:01:42 sdp:1:100                L/90     04/21/17 14:45:05
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

*A:R2>config>service>vpls>sap$ show service id 100 fdb detail

===============================================================================
Forwarding Database, Service 100
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
100       50:00:00:07:00:01 sap:1/1/1                L/0      04/21/17 14:52:42
100       da:00:ff:00:01:42 sdp:1:100                L/0      04/21/17 14:44:46
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

Now to check out the MC-LAG resiliency, we’ll start a continuous ping on R5 to R4 and then shutdown port 1/1/3 (LAG-1) on R1
*A:R1>config>service>vpls>sap$ /configure port 1/1/3 shutdown
And Check if R2 LAG 1 Port 1/1/3 goes from standby to active
*A:R2>config>service>vpls>sap$ show lag 1 port 

===============================================================================
Lag Port States
LACP Status: e - Enabled, d - Disabled
===============================================================================
Lag-id Port-id   Adm   Act/Stdby Opr   Primary  Sub-group     Forced  Priority
-------------------------------------------------------------------------------
1(e)   1/1/3     up    active    up    yes      1             -       32768
===============================================================================

We can see the interface has come up and there were a few packets lost but the link recovered – we could speed up the link convergence time but I think the general concept has been demonstrated sucessfully.

The moral of the story here – with Virtual SROS systems, it’s worth ensuring you have a unique chassis MAC address!

Published by

Adam

Adam enjoys playing with and learning about networks and is based in Queensland, Australia.