Configuring 802.1q tunnels over Vlan-based EoMPLS & MUX-UNI

One of the main disadvantages (besides all the usual disputes about L2 vs L3) of 802.1q tunneling is the fact that all the switches the tunnel is passing through, are learning all the MAC addresses passing through it, plus the consumption of other system resources like vlans and spanning-tree instances. So if you have many 802.1q tunnels passing through one or more central switches as bridged L2 traffic, you might end up filling their TCAM space. For point-2-point 802.1q tunnels, an easy solution is to use EoMPLS tunnels on the central routers/switches in order to allow the double-tagged 802.1q traffic to pass over them as transparently as possible.

The following is our test network.



We need to transfer L2 traffic belonging to vlans 100-110 and 200-210 across some switches/routers and at the same time we need to pass IP traffic belonging to vlans 10 and 20 from the 6500s to the 7600s and vice-versa. All of the above using a single link between 6500s and 7600s.

EoMPLS uses IP/MPLS as its underlying technology, so it really doesn't matter what you have between the EoMPLS endpoints, as long as you have IP/MPLS connectivity between them (the ingress and egress PEs are the only two routers in the MPLS backbone with knowledge of the Layer 2 VCs, so only these require EoMPLS functionality). If you want to avoid MPLS and use solely IP, then you you can create a tunnel based on L2TPv3 and transfer Ethernet frames over it.

EoMPLS is one of the AToM (Any Transport over MPLS) transport types. AToM transports Layer 2 packets over a IP/MPLS backbone using a directed LDP session between edge routers for setting up and maintaining connections. Normally forwarding occurs through the use of two level labels that provide switching between the edge routers. The Tunnel label (external) routes the packet over the MPLS backbone to the egress PE at the ingress PE. The VC label is a demuxing label that determines the connection at the tunnel endpoint (the particular egress interface on the egress PE as well as the vlan-id for an ethernet frame). If PE routers are connected back-2-back (like in this example), then there is no need for a tunnel label (the "implicit null" label is used to signal that to the peer).

EoMPLS works by encapsulating ethernet PDUs in MPLS packets and forwarding them across the MPLS network. Each PDU is transported as a single packet. It can work in 2 modes:

1) Port-based, where everything that comes into a port gets tunneled (using a VC type of 5)
2) Vlan-based, where only specific vlans get tunneled (using a VC type of either 4 or 5)

Here i'm using the vlan-based mode, because i want to combine it with MUX-UNI. MUX-UNI is a nice feature on 6500s and 7600s, that provides the ability to partition a physical port in such a way that multiple Layer 2 and Layer 3 services can be created over a single UNI (usually the point where CE connects to the ingress PE). Is works very well on the 6500/7600 LAN cards (i.e. 67xx), but if you are thinking of more advanced things you probably need the ES/ES+ ones (i will make a new post about various ES+ setups soon).

Before configuring EoMPLS, you have to make sure that IP routing in enabled so that the PE routers (in our case the 7600s) can reach each other and MPLS is enabled so that a label switched path (LSP) exists between the PE routers. No need for complexity here, so we use just Vlan 30 in both 7600s for this. CEF is needed too, but it's enabled by default and cannot be disabled on 6500s/7600s.

Configuration

These are the relevant configurations:

6500-1

interface GigabitEthernet2/1
 description ** vlans 100-110 **
 switchport
 switchport access vlan 2001
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
 description ** vlans 200-210 **
 switchport
 switchport access vlan 2002
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface Vlan10
 description ** test **
 ip address 10.10.10.1 255.255.255.0
!
interface GigabitEthernet2/3
 description ** 7600-1 Gi3/1 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10,2001,2002
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable

7600-1

interface GigabitEthernet3/1
 description ** 6500-1 G2/3 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
 encapsulation dot1Q 2001
 xconnect 30.30.30.2 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
 encapsulation dot1Q 2002
 xconnect 30.30.30.2 2002 encapsulation mpls
!
interface GigabitEthernet3/2
 description ** 7600-2 Gi3/2 **
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 mtu 9216
!
interface Vlan10
 description ** test **
 ip address 10.10.10.2 255.255.255.0
!
interface Vlan30
 description ** MPLS **
 ip address 30.30.30.1 255.255.255.0
 mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
 permit 30.30.30.0 0.0.0.255

7600-2

interface GigabitEthernet3/1
 description ** 6500-2 G2/3 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 20
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
 encapsulation dot1Q 2001
 xconnect 30.30.30.1 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
 encapsulation dot1Q 2002
 xconnect 30.30.30.1 2002 encapsulation mpls
!
interface GigabitEthernet3/2
 description ** 7600-1 Gi3/2 **
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 mtu 9216
!
interface Vlan20
 description ** test **
 ip address 20.20.20.2 255.255.255.0
!
interface Vlan30
 description ** MPLS **
 ip address 30.30.30.2 255.255.255.0
 mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
 permit 30.30.30.0 0.0.0.255

6500-2

interface GigabitEthernet2/1
 description ** vlans 100-110 **
 switchport
 switchport access vlan 2001
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
 description ** vlans 200-210 **
 switchport
 switchport access vlan 2002
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface Vlan20
 description ** test **
 ip address 20.20.20.1 255.255.255.0
!
interface GigabitEthernet2/3
 description ** 7600-2 Gi3/1 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 20,2001,2002
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable

Verification

Check the 802.1q tunnel ports on the edges of the network

6500-1#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-1#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)


6500-2#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-2#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)

Check all other ports in the core network. Keep in mind that we terminate locally vlans 10 and 20, while we tunnel vlans 2001 and 2002.

! Connection between 6500-1 Gi2/3 and 7600-1 Gi3/1
6500-1#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10,2001,2002

7600-1#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10

! Connection between 7600-1 Gi3/2 and 7600-2 Gi3/2
7600-1#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

7600-2#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

! Connection between 7600-2 Gi3/1 and 6500-2 Gi2/3
7600-2#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20

6500-2#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20,2001,2002

Check MPLS/LDP functionality (no need to advertise labels for all prefixes)

7600-1#sh mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
Vlan30                 Yes (ldp)     No       No  No     Yes


7600-2#sh mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
Vlan30                 Yes (ldp)     No       No  No     Yes


7600-1#sh mpls ldp nei
    Peer LDP Ident: 30.30.30.2:0; Local LDP Ident 30.30.30.1:0
        TCP connection: 30.30.30.2.47252 - 30.30.30.1.646
        State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
        Up time: 2d18h
        LDP discovery sources:
          Vlan30, Src IP addr: 30.30.30.2
          Targeted Hello 30.30.30.1 -> 30.30.30.2, active, passive
        Addresses bound to peer LDP Ident:
          30.30.30.2


7600-2#sh mpls ldp nei
    Peer LDP Ident: 30.30.30.1:0; Local LDP Ident 30.30.30.2:0
        TCP connection: 30.30.30.1.646 - 30.30.30.2.47252
        State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
        Up time: 2d18h
        LDP discovery sources:
          Vlan30, Src IP addr: 30.30.30.1
          Targeted Hello 30.30.30.2 -> 30.30.30.1, active, passive
        Addresses bound to peer LDP Ident:
          30.30.30.1


7600-1#sh mpls ldp binding det
Advertisement spec:
        Prefix acl = MPLS-LABELS-ACL

  lib entry: 0.0.0.0/0, rev 12, chkpt: none
        local binding:  label: imp-null (owner LDP)
  lib entry: 30.30.30.0/24, rev 13, chkpt: none
        local binding:  label: imp-null (owner LDP)
          Advertised to:
          30.30.30.2:0
        remote binding: lsr: 30.30.30.2:0, label: imp-null
        Advert acl(s): Prefix acl MPLS-LABELS-ACL


 
7600-2#sh mpls ldp bindings det
Advertisement spec:
        Prefix acl = MPLS-LABELS-ACL

  lib entry: 0.0.0.0/0, rev 14, chkpt: none
        local binding:  label: imp-null (owner LDP)
  lib entry: 30.30.30.0/24, rev 15, chkpt: none
        local binding:  label: imp-null (owner LDP)
          Advertised to:
          30.30.30.1:0
        remote binding: lsr: 30.30.30.1:0, label: imp-null
        Advert acl(s): Prefix acl MPLS-LABELS-ACL
 

7600-1#sh mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     No Label      l2ckt(2001)       1710943694212 Gi3/1.2001  point2point
17     No Label      l2ckt(2002)       2053098230805 Gi3/1.2002  point2point


7600-2#sh mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     No Label      l2ckt(2001)       869456241933  Gi3/1.2001  point2point
17     No Label      l2ckt(2002)       1083085136320 Gi3/1.2002  point2point

Check EoMPLS L2 circuits

7600-1#sh mpls l2transport vc

Local intf     Local circuit              Dest address    VC ID      Status
-------------  -------------------------- --------------- ---------- ----------
Gi3/1.2001     Eth VLAN 2001              30.30.30.2      2001       UP
Gi3/1.2002     Eth VLAN 2002              30.30.30.2      2002       UP

7600-2#sh mpls l2transport vc

Local intf     Local circuit              Dest address    VC ID      Status
-------------  -------------------------- --------------- ---------- ----------
Gi3/1.2001     Eth VLAN 2001              30.30.30.1      2001       UP
Gi3/1.2002     Eth VLAN 2002              30.30.30.1      2002       UP


7600-1#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.2, VC ID: 2001, VC status: up
    Output interface: Vl30, imposed label stack {16}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.2
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.2:0 up
    Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
    MPLS VC labels: local 16, remote 16
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 1866599200, send 1608328927
    byte totals:   receive 1706612231959, send 857848421519
    packet drops:  receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.2, VC ID: 2002, VC status: up
    Output interface: Vl30, imposed label stack {17}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.2
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.2:0 up
    Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
    MPLS VC labels: local 17, remote 17
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 2345120706, send 2080672994
    byte totals:   receive 2109127676973, send 1102024312190
    packet drops:  receive 0, send 0


7600-2#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.1, VC ID: 2001, VC status: up
    Output interface: Vl30, imposed label stack {16}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.1
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.1:0 up
    Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
    MPLS VC labels: local 16, remote 16
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 1612416460, send 1870837231
    byte totals:   receive 860082881318, send 1710279266411
    packet drops:  receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.1, VC ID: 2002, VC status: up
    Output interface: Vl30, imposed label stack {17}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.1
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.1:0 up
    Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
    MPLS VC labels: local 17, remote 17
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 2085506054, send 2349856170
    byte totals:   receive 1104792851185, send 2112955222087
    packet drops:  receive 0, send 0


7600-1#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP     ac   Gi3/1.2001 2001(Eth VLAN)    UP mpls 30.30.30.2:2001              UP
UP     ac   Gi3/1.2002 2002(Eth VLAN)    UP mpls 30.30.30.2:2002              UP

7600-2#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP     ac   Gi3/1.2001 2001(Eth VLAN)    UP mpls 30.30.30.1:2001              UP
UP     ac   Gi3/1.2002 2002(Eth VLAN)    UP mpls 30.30.30.1:2002              UP

The term "Eth Vlan" means a VC type 4 is used.

Ports Gi2/1 & Gi2/2 on both the 6500s are using the 802.1q tunnel functionality.
Ports Gi3/1 on both the 7600s are using the MUX-UNI functionality, passing vlans 10,20 as L2 and vlans 2001,2002 as EoMPLS. These are the customer-facing or edge ports.
Ports Gi3/2 on both the 7600s are simple trunk ports allowing all L2 vlans. These are the core-facing ports.

This is what happens to a packet (frame) belonging to vlan 100 and moving from 6500-1 to 6500-2:

1) Packet enters 6500-1 through Gi2/1, having a single 802.1q header with vlan-id 100
2) A second 802.1q header with vlan-id 2001 is added by 6500-1
3) Packet exits 6500-1 through Gi2/3 and enters 7600-1 through Gi3/1, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100

4) A MPLS header with VC label 2001 is added by 7600-1
5) Packet exits 7600-1 through Gi3/2 and enters 7600-2 through Gi3/2, having a MPLS header with VC label 2001 followed by a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
6) The MPLS header with VC label 2001 is removed by 7600-2

7) Packet exits 7600-2 through Gi3/1 and enters 6500-2 through Gi2/3, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
8) The second 802.1q header with vlan-id 2001 is removed by 6500-2
9) Packet exits 6500-2 through Gi2/1, having a single 802.1q header with vlan-id 100


Notes:

1) The EoMPLS tunnel destination route in the routing table (and the CEF table) must be a /32 address to ensure that there is a label-switched path (LSP) from PE to PE. That's another reason why loopbacks are preferred. In the above test there is no need for actual routing on the EoMPLS path, due to directly connected PEs. Also, targeted LDP sessions between the PEs would be needed if the PEs weren't directly connected (since only a single LDP session is required between the peers, one is only created if not already present).

2) EoMPLS based on a PFC (used by LAN cards on 6500/7600) does not perform any L2 lookup to determine if the destination MAC address resides on the local or remote segment and also does not perform any L2 address learning (as traditional LAN bridging does).

3) All GE cards used were WS-X6724-SFP. SUP720s in 6500s were running 12.2(33)SXI1 and RSP720s in 7600s were running 12.2(33)SRD1.

4) Always keep an eye on the MTU. If you don't want to sit down and make arithmetic operations, just use a high MTU that can cover all possible cases. Generally your MTU should be higher than "1500 + Transport Header + AToM Header + (MPLS Labels x 4)", which is 1530 for a usual Vlan-based EoMPLS.

5) Numbers 2001 and 2002 used with the "xconnect" command define the VC id and they don't have to be the same as the vlans.

6) Using the command "sh mpls l2transport hw-capability interface x/x", you can find out what transport modes are supported for an interface, as well as what functionality is supported for core or edge ports.

7) In some 7600 IOS releases, there is bug in the output of "sh mpls l2transport binding" that shows the VC as Ethernet (Type 5), while it's Ether Vlan (Type 4). Just ignore it.

8) The command "no mpls ldp advertise-labels" must be configured together with "mpls ldp advertise-labels...", in order to make filtering of label advertisements work.

Should the crash be an option?

During the previous weekend i had a 7600/RSP720 crashing multiple times in the middle of the night without any apparent reason. Console was showing the following messages:

%Software-forced reload


03:31:06 EET Sat May 9 2009: Unexpected exception to CPU Vector 1500, PC = 0x0A7FA5       
       B0, LR = 0x0A7FA550

-Traceback= A7FA5B0 A7FA550 A558DD8 A40B698 A442928 A442994 8814E18 8815FB8 8816CD4       
      A874C8C A8755BC A82EFCC A82F1C0 A8306C0 A7F05EC A87A0DC

CPU Register Context:
MSR = 0x00029200  CR  = 0x24444024  CTR = 0x0A44DA0C  XER   = 0x00000000
R0  = 0x0A7FA550  R1  = 0x13D4C3E0  R2  = 0xFFFCFFFC  R3    = 0x137005B4
R4  = 0xFFFFFFFE  R5  = 0x00000000  R6  = 0x13D4C3B8  R7    = 0x00029200
R8  = 0x00029200  R9  = 0x00000000  R10 = 0x0D06DBF8  R11   = 0x137005B0
R12 = 0x00013FB4  R13 = 0x04044000  R14 = 0x00000000  R15   = 0x00000000
R16 = 0x0E844B94  R17 = 0x0E8663A8  R18 = 0x0C236BD4  R19   = 0x00000000
R20 = 0x13731364  R21 = 0x79D97210  R22 = 0x13D4C580  R23   = 0x0E851E4C
R24 = 0x0D050000  R25 = 0x00000000  R26 = 0x1373158C  R27   = 0x0000001F
R28 = 0x00000001  R29 = 0x0C420000  R30 = 0x103FEAA0  R31   = 0x00000000

Writing crashinfo to bootdisk:crashinfo_20090509-033106-EET

*** System received a Software forced crash ***
signal= 0x17, code= 0x1500, context= 0xce08b9c
PC = 0x8273dcc, Vector = 0x1500, SP = 0x1b394d08
rommon 2 >

Trying to boot manually from the rommon didn't have any positive result, because after the boot was completed (and you were given about 10-20 secs of CLI access!), the RSP crashed again. Since rommon doesn't provide a way to view crashinfo files, you're stuck with the crashes, trying your best at various guesses.

This was a crash that appeared at a random time and on a 7600 that gets configured once in a month, so by best guess was a faulty or stuck supervisor card. After the card was OIRed by the oncall engineer, it booted fine and i finally had a look at the crashinfos. The crashinfo on the RP didn't show anything useful. But the crashinfo on the SP showed the following:

%ONLINE-SP-6-INITFAIL: Module 6: Failed to synchronize Port asic

CCO describes it like below:

Description
The cause of the crash is that the Pinnacle ASIC failed to synchronize. This is usually caused by a bad contact or a badly seated card.

Workaround
The system recovers without user intervention. If the error message recurs, then reseat the concerned line card or module.

Yeah, sure. The system was stuck to rommon after some crashes:

Active crashed three times, disabling auto-boot and dropping to rommon

If anyone would like to answer, these are my questions:

Why couldn't the RSP print an error message (after 3 crashes) on the console while in IOS and sit there waiting for the user to act? What benefit does this specific crash and return to rommon offer?

I understand that generally a crash/reload is used to force the device recover from a bad situation and possibly avoid a long downturn situation. In my case, the situation was a bad contact or a badly seated card (although there is lot of arguing that can be done here). How was the reload supposed to solve this? Was the reload going to "reseat" the module somehow? Are there any pins moved while reloading? Does rommon offer a better seating?

After all, as it proved out the crash didn't help at all, because the RSP returned to ROM, sitting there and waiting, while it could stay in IOS, which provides much better feedback to the user.

On the other hand, it's the redundancy factor. Which, imho, is the only case that a crash/reload should occur (i'm still referring to the case of a badly seated module). As long as you have a 2nd working RSP, it should take the active role. But the system knows whether you have a redundant setup long before giving you the CLI prompt, so it should be easy to check upon it.

What i'm trying to say is that a crash followed by a reload is not always a panacea in case of a problem. It should be an option used, only when there is no other solution. In case of a badly seated module, crashing and moving into rommon isn't of any help (at least until rommon gets upgraded to something more intelligent).

"aaa accounting system guarantee-first" Pt2

Just an update on the "aaa accounting system guarantee-first" issue. I submitted a PER and as it turned out, BU & developers decided to treat it like a bug and fix it. But not completely.

Having to choose between the following 2 solutions, they chose the easy one; the first.

1) "Console/Aux user should be permitted if authentication is not configured with any AAA Server Method (radius/tacacs)".

2) "Console/Aux/VTY access should be permitted if authentication is not configured with the same AAA method (radius/tacacs) as system accounting".

As you can see the 1st solution is applicable only to con/aux, while the 2nd is applicable to con/aux/vty.
Also the 1st solution mixes tacacs and radius, like they are used for the same type of access, something that is not true for the majority of services (Cisco knows that better that anyone else).

Cisco expects me to be satisfied from the PER result, but i'm not. They had a chance of fixing something completely, but they decided to just patch it temporarily.

Cisco : "this behavior is expected" & Archive Config Logger

The last 2 months i've been busy playing with an ASR1000, finding bugs and "fighting" with the BUs.

Dealing with a lot of issues lately (not necessarily of ASR's), there are a few times that i have come across Cisco's motto "this behavior is expected". One of them was the fact that the very useful functionality of archive config logger (Configuration Logging) is ruined by the inability to filter out per-user configs applied through an aaa server.

The Configuration Change Notification and Logging (Configuration Logging) feature allows the tracking of configuration changes entered on a per-session and per-user basis by implementing a configuration log. The configuration log tracks each configuration command that is applied, who applied the command, the parser return code for the command, and the time the command was applied.

As you all know, a basic configuration like the following will help you keep track of changes in the configuration of a router:

archive
 log config
  logging enable
  hidekeys

In a "normal" router you'll probably see something like the following:

router#sh archive log config all
 idx   sess           user@line      Logged command
    8     3         user@vty0     | no ipv6 nd suppress-ra
    9     3         user@vty0     | ipv6 ospf 100 area 10
   10     4         user@vty0     |no ip sla monitor 99
   11     5         user@vty0     |interface GigabitEthernet0/2.35211201
   12     5         user@vty0     | encapsulation dot1Q 3521 second-dot1q 1201
   13     5         user@vty0     | ip address 10.10.10.129 255.255.255.252

If you happen to have a router that is acting like a bras, accepting ppp sessions and using radius attributes in order to apply per-user ACLs and/or routes, then you'll find out that the above feature is totally useless, because all these per-user entries are stored into the archive logging buffer, overwriting the actual configuration changes you want to watch (when the configuration log is full, the oldest log entry will be removed every time a new entry is added).

router#sh archive log config all
 idx   sess           user@line      Logged command
  177     0        console@console  |permit ip host 1.1.1.1 any 
  178     0        console@console  |deny ip any any 
  179     0        console@console  |deny ip any any 
  180     0        console@console  |ip route 2.2.2.0 255.255.255.240 3.3.3.3 
  181     0        console@console  |ip route 2.2.2.0 255.255.255.240 3.3.3.3 
  182     0        console@console  |ip access-list extended Virtual-Access2.10785#106240381
  183     0        console@console  |deny ip any 10.0.0.0 0.255.255.255
  184     0        console@console  |deny ip any 10.0.0.0 0.255.255.255
  185     0        console@console  |deny ip any 192.168.0.0 0.0.255.255
  186     0        console@console  |deny ip any 192.168.0.0 0.0.255.255
  187     0        console@console  |deny ip any 224.0.0.0 15.255.255.255
  188     0        console@console  |deny ip any 224.0.0.0 15.255.255.255

Note : Please don't ask me about the double entries...

According to Cisco's cli help, only changes to the running config, not the one stored on nvram, are to be logged.

router(config-archive)#log ?
  config  Logging changes to the running configuration

If you issue the "show run" command (which is supposed to display the running configuration), you won't see these per-user config changes. That's because all these per-user configs, after being downloaded from the aaa server, have a flag set that they should not be nvgen'ed. So they are not shown.

In the Cisco IOS software configuration model, the configuration state is maintained in a distributed manner, with each component storing its own configuration state. To retrieve configuration information, the software must poll every component to collect the distributed information. This configuration state retrieval operation is performed by a process known as nonvolatile generation (NVGEN), and it is used by command line interpreter (CLI) commands such as show running-configuration, write memory, and copy system:running-configuration to display or copy the running system configuration. When invoked, NVGEN queries each system component and each instance of interface or other configuration objects. A running configuration file is constructed as NVGEN traverses the system performing these queries.

Also, according to Cisco:

Restrictions for Configuration Change Notification and Logging
•Only complete commands input in a configuration mode are logged.
•Commands that are part of a configuration file applied with the copy command are not logged.

The Configuration Change Notification and Logging feature tracks changes made to the Cisco IOS software running configuration by maintaining a configuration log. This configuration log tracks changes initiated only through the command-line interface (CLI) or HTTP.

Why Cisco chose to include these per-user "dynamic" config changes into the archive logger, making it another thrown-away feature, is still a mystery to me.

There is also the possibility that this kind of continuous logging might be causing a little increase in cpu usage too.

Yep, time for another Cisco-should-have-fixed-it-in-the-first-place PER.