Saturday, May 23, 2009

Configuring 802.1q tunnels over Vlan-based EoMPLS & MUX-UNI

One of the main disadvantages (besides all the usual disputes about L2 vs L3) of 802.1q tunneling is the fact that all the switches the tunnel is passing through, are learning all the MAC addresses passing through it, plus the consumption of other system resources like vlans and spanning-tree instances. So if you have many 802.1q tunnels passing through one or more central switches as bridged L2 traffic, you might end up filling their TCAM space. For point-2-point 802.1q tunnels, an easy solution is to use EoMPLS tunnels on the central routers/switches in order to allow the double-tagged 802.1q traffic to pass over them as transparently as possible.

The following is our test network.



We need to transfer L2 traffic belonging to vlans 100-110 and 200-210 across some switches/routers and at the same time we need to pass IP traffic belonging to vlans 10 and 20 from the 6500s to the 7600s and vice-versa. All of the above using a single link between 6500s and 7600s.

EoMPLS uses IP/MPLS as its underlying technology, so it really doesn't matter what you have between the EoMPLS endpoints, as long as you have IP/MPLS connectivity between them (the ingress and egress PEs are the only two routers in the MPLS backbone with knowledge of the Layer 2 VCs, so only these require EoMPLS functionality). If you want to avoid MPLS and use solely IP, then you you can create a tunnel based on L2TPv3 and transfer Ethernet frames over it.

EoMPLS is one of the AToM (Any Transport over MPLS) transport types. AToM transports Layer 2 packets over a IP/MPLS backbone using a directed LDP session between edge routers for setting up and maintaining connections. Normally forwarding occurs through the use of two level labels that provide switching between the edge routers. The Tunnel label (external) routes the packet over the MPLS backbone to the egress PE at the ingress PE. The VC label is a demuxing label that determines the connection at the tunnel endpoint (the particular egress interface on the egress PE as well as the vlan-id for an ethernet frame). If PE routers are connected back-2-back (like in this example), then there is no need for a tunnel label (the "implicit null" label is used to signal that to the peer).

EoMPLS works by encapsulating ethernet PDUs in MPLS packets and forwarding them across the MPLS network. Each PDU is transported as a single packet. It can work in 2 modes:

1) Port-based, where everything that comes into a port gets tunneled (using a VC type of 5)
2) Vlan-based, where only specific vlans get tunneled (using a VC type of either 4 or 5)

Here i'm using the vlan-based mode, because i want to combine it with MUX-UNI. MUX-UNI is a nice feature on 6500s and 7600s, that provides the ability to partition a physical port in such a way that multiple Layer 2 and Layer 3 services can be created over a single UNI (usually the point where CE connects to the ingress PE). Is works very well on the 6500/7600 LAN cards (i.e. 67xx), but if you are thinking of more advanced things you probably need the ES/ES+ ones (i will make a new post about various ES+ setups soon).

Before configuring EoMPLS, you have to make sure that IP routing in enabled so that the PE routers (in our case the 7600s) can reach each other and MPLS is enabled so that a label switched path (LSP) exists between the PE routers. No need for complexity here, so we use just Vlan 30 in both 7600s for this. CEF is needed too, but it's enabled by default and cannot be disabled on 6500s/7600s.

Configuration

These are the relevant configurations:

6500-1


interface GigabitEthernet2/1
description ** vlans 100-110 **
switchport
switchport access vlan 2001
switchport mode dot1q-tunnel
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
description ** vlans 200-210 **
switchport
switchport access vlan 2002
switchport mode dot1q-tunnel
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface Vlan10
description ** test **
ip address 10.10.10.1 255.255.255.0
!
interface GigabitEthernet2/3
description ** 7600-1 Gi3/1 **
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,2001,2002
switchport mode trunk
switchport nonegotiate
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable


7600-1

interface GigabitEthernet3/1
description ** 6500-1 G2/3 **
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10
switchport mode trunk
switchport nonegotiate
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
encapsulation dot1Q 2001
xconnect 30.30.30.2 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
encapsulation dot1Q 2002
xconnect 30.30.30.2 2002 encapsulation mpls
!
interface GigabitEthernet3/2
description ** 7600-2 Gi3/2 **
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
mtu 9216
!
interface Vlan10
description ** test **
ip address 10.10.10.2 255.255.255.0
!
interface Vlan30
description ** MPLS **
ip address 30.30.30.1 255.255.255.0
mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
permit 30.30.30.0 0.0.0.255


7600-2

interface GigabitEthernet3/1
description ** 6500-2 G2/3 **
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20
switchport mode trunk
switchport nonegotiate
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
encapsulation dot1Q 2001
xconnect 30.30.30.1 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
encapsulation dot1Q 2002
xconnect 30.30.30.1 2002 encapsulation mpls
!
interface GigabitEthernet3/2
description ** 7600-1 Gi3/2 **
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
mtu 9216
!
interface Vlan20
description ** test **
ip address 20.20.20.2 255.255.255.0
!
interface Vlan30
description ** MPLS **
ip address 30.30.30.2 255.255.255.0
mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
permit 30.30.30.0 0.0.0.255


6500-2

interface GigabitEthernet2/1
description ** vlans 100-110 **
switchport
switchport access vlan 2001
switchport mode dot1q-tunnel
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
description ** vlans 200-210 **
switchport
switchport access vlan 2002
switchport mode dot1q-tunnel
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface Vlan20
description ** test **
ip address 20.20.20.1 255.255.255.0
!
interface GigabitEthernet2/3
description ** 7600-2 Gi3/1 **
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20,2001,2002
switchport mode trunk
switchport nonegotiate
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable


Verification

Check the 802.1q tunnel ports on the edges of the network


6500-1#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-1#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)


6500-2#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-2#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)


Check all other ports in the core network. Keep in mind that we terminate locally vlans 10 and 20, while we tunnel vlans 2001 and 2002.


! Connection between 6500-1 Gi2/3 and 7600-1 Gi3/1
6500-1#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10,2001,2002

7600-1#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10

! Connection between 7600-1 Gi3/2 and 7600-2 Gi3/2
7600-1#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

7600-2#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

! Connection between 7600-2 Gi3/1 and 6500-2 Gi2/3
7600-2#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20

6500-2#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20,2001,2002


Check MPLS/LDP functionality (no need to advertise labels for all prefixes)


7600-1#sh mpls interfaces
Interface IP Tunnel BGP Static Operational
Vlan30 Yes (ldp) No No No Yes


7600-2#sh mpls interfaces
Interface IP Tunnel BGP Static Operational
Vlan30 Yes (ldp) No No No Yes


7600-1#sh mpls ldp nei
Peer LDP Ident: 30.30.30.2:0; Local LDP Ident 30.30.30.1:0
TCP connection: 30.30.30.2.47252 - 30.30.30.1.646
State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
Up time: 2d18h
LDP discovery sources:
Vlan30, Src IP addr: 30.30.30.2
Targeted Hello 30.30.30.1 -> 30.30.30.2, active, passive
Addresses bound to peer LDP Ident:
30.30.30.2


7600-2#sh mpls ldp nei
Peer LDP Ident: 30.30.30.1:0; Local LDP Ident 30.30.30.2:0
TCP connection: 30.30.30.1.646 - 30.30.30.2.47252
State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
Up time: 2d18h
LDP discovery sources:
Vlan30, Src IP addr: 30.30.30.1
Targeted Hello 30.30.30.2 -> 30.30.30.1, active, passive
Addresses bound to peer LDP Ident:
30.30.30.1


7600-1#sh mpls ldp binding det
Advertisement spec:
Prefix acl = MPLS-LABELS-ACL

lib entry: 0.0.0.0/0, rev 12, chkpt: none
local binding: label: imp-null (owner LDP)
lib entry: 30.30.30.0/24, rev 13, chkpt: none
local binding: label: imp-null (owner LDP)
Advertised to:
30.30.30.2:0
remote binding: lsr: 30.30.30.2:0, label: imp-null
Advert acl(s): Prefix acl MPLS-LABELS-ACL



7600-2#sh mpls ldp bindings det
Advertisement spec:
Prefix acl = MPLS-LABELS-ACL

lib entry: 0.0.0.0/0, rev 14, chkpt: none
local binding: label: imp-null (owner LDP)
lib entry: 30.30.30.0/24, rev 15, chkpt: none
local binding: label: imp-null (owner LDP)
Advertised to:
30.30.30.1:0
remote binding: lsr: 30.30.30.1:0, label: imp-null
Advert acl(s): Prefix acl MPLS-LABELS-ACL


7600-1#sh mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
16 No Label l2ckt(2001) 1710943694212 Gi3/1.2001 point2point
17 No Label l2ckt(2002) 2053098230805 Gi3/1.2002 point2point


7600-2#sh mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
16 No Label l2ckt(2001) 869456241933 Gi3/1.2001 point2point
17 No Label l2ckt(2002) 1083085136320 Gi3/1.2002 point2point



Check EoMPLS L2 circuits


7600-1#sh mpls l2transport vc

Local intf Local circuit Dest address VC ID Status
------------- -------------------------- --------------- ---------- ----------
Gi3/1.2001 Eth VLAN 2001 30.30.30.2 2001 UP
Gi3/1.2002 Eth VLAN 2002 30.30.30.2 2002 UP

7600-2#sh mpls l2transport vc

Local intf Local circuit Dest address VC ID Status
------------- -------------------------- --------------- ---------- ----------
Gi3/1.2001 Eth VLAN 2001 30.30.30.1 2001 UP
Gi3/1.2002 Eth VLAN 2002 30.30.30.1 2002 UP


7600-1#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
MPLS VC type is Eth VLAN, interworking type is Ethernet
Destination address: 30.30.30.2, VC ID: 2001, VC status: up
Output interface: Vl30, imposed label stack {16}
Preferred path: not configured
Default path: active
Next hop: 30.30.30.2
Create time: 2d23h, last status change time: 2d19h
Signaling protocol: LDP, peer 30.30.30.2:0 up
Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
MPLS VC labels: local 16, remote 16
Group ID: local 0, remote 0
MTU: local 9216, remote 9216
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 1866599200, send 1608328927
byte totals: receive 1706612231959, send 857848421519
packet drops: receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
MPLS VC type is Eth VLAN, interworking type is Ethernet
Destination address: 30.30.30.2, VC ID: 2002, VC status: up
Output interface: Vl30, imposed label stack {17}
Preferred path: not configured
Default path: active
Next hop: 30.30.30.2
Create time: 2d23h, last status change time: 2d19h
Signaling protocol: LDP, peer 30.30.30.2:0 up
Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
MPLS VC labels: local 17, remote 17
Group ID: local 0, remote 0
MTU: local 9216, remote 9216
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 2345120706, send 2080672994
byte totals: receive 2109127676973, send 1102024312190
packet drops: receive 0, send 0


7600-2#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
MPLS VC type is Eth VLAN, interworking type is Ethernet
Destination address: 30.30.30.1, VC ID: 2001, VC status: up
Output interface: Vl30, imposed label stack {16}
Preferred path: not configured
Default path: active
Next hop: 30.30.30.1
Create time: 2d23h, last status change time: 2d19h
Signaling protocol: LDP, peer 30.30.30.1:0 up
Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
MPLS VC labels: local 16, remote 16
Group ID: local 0, remote 0
MTU: local 9216, remote 9216
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 1612416460, send 1870837231
byte totals: receive 860082881318, send 1710279266411
packet drops: receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
MPLS VC type is Eth VLAN, interworking type is Ethernet
Destination address: 30.30.30.1, VC ID: 2002, VC status: up
Output interface: Vl30, imposed label stack {17}
Preferred path: not configured
Default path: active
Next hop: 30.30.30.1
Create time: 2d23h, last status change time: 2d19h
Signaling protocol: LDP, peer 30.30.30.1:0 up
Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
MPLS VC labels: local 17, remote 17
Group ID: local 0, remote 0
MTU: local 9216, remote 9216
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 2085506054, send 2349856170
byte totals: receive 1104792851185, send 2112955222087
packet drops: receive 0, send 0


7600-1#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST Segment 1 S1 Segment 2 S2
------+---------------------------------+--+---------------------------------+--
UP ac Gi3/1.2001 2001(Eth VLAN) UP mpls 30.30.30.2:2001 UP
UP ac Gi3/1.2002 2002(Eth VLAN) UP mpls 30.30.30.2:2002 UP

7600-2#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST Segment 1 S1 Segment 2 S2
------+---------------------------------+--+---------------------------------+--
UP ac Gi3/1.2001 2001(Eth VLAN) UP mpls 30.30.30.1:2001 UP
UP ac Gi3/1.2002 2002(Eth VLAN) UP mpls 30.30.30.1:2002 UP


The term "Eth Vlan" means a VC type 4 is used.

Ports Gi2/1 & Gi2/2 on both the 6500s are using the 802.1q tunnel functionality.
Ports Gi3/1 on both the 7600s are using the MUX-UNI functionality, passing vlans 10,20 as L2 and vlans 2001,2002 as EoMPLS. These are the customer-facing or edge ports.
Ports Gi3/2 on both the 7600s are simple trunk ports allowing all L2 vlans. These are the core-facing ports.

This is what happens to a packet (frame) belonging to vlan 100 and moving from 6500-1 to 6500-2:

1) Packet enters 6500-1 through Gi2/1, having a single 802.1q header with vlan-id 100
2) A second 802.1q header with vlan-id 2001 is added by 6500-1
3) Packet exits 6500-1 through Gi2/3 and enters 7600-1 through Gi3/1, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100

4) A MPLS header with VC label 2001 is added by 7600-1
5) Packet exits 7600-1 through Gi3/2 and enters 7600-2 through Gi3/2, having a MPLS header with VC label 2001 followed by a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
6) The MPLS header with VC label 2001 is removed by 7600-2

7) Packet exits 7600-2 through Gi3/1 and enters 6500-2 through Gi2/3, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
8) The second 802.1q header with vlan-id 2001 is removed by 6500-2
9) Packet exits 6500-2 through Gi2/1, having a single 802.1q header with vlan-id 100


Notes:

1) The EoMPLS tunnel destination route in the routing table (and the CEF table) must be a /32 address to ensure that there is a label-switched path (LSP) from PE to PE. That's another reason why loopbacks are preferred. In the above test there is no need for actual routing on the EoMPLS path, due to directly connected PEs. Also, targeted LDP sessions between the PEs would be needed if the PEs weren't directly connected (since only a single LDP session is required between the peers, one is only created if not already present).

2) EoMPLS based on a PFC (used by LAN cards on 6500/7600) does not perform any L2 lookup to determine if the destination MAC address resides on the local or remote segment and also does not perform any L2 address learning (as traditional LAN bridging does).

3) All GE cards used were WS-X6724-SFP. SUP720s in 6500s were running 12.2(33)SXI1 and RSP720s in 7600s were running 12.2(33)SRD1.

4) Always keep an eye on the MTU. If you don't want to sit down and make arithmetic operations, just use a high MTU that can cover all possible cases. Generally your MTU should be higher than "1500 + Transport Header + AToM Header + (MPLS Labels x 4)", which is 1530 for a usual Vlan-based EoMPLS.

5) Numbers 2001 and 2002 used with the "xconnect" command define the VC id and they don't have to be the same as the vlans.

6) Using the command "sh mpls l2transport hw-capability interface x/x", you can find out what transport modes are supported for an interface, as well as what functionality is supported for core or edge ports.

7) In some 7600 IOS releases, there is bug in the output of "sh mpls l2transport binding" that shows the VC as Ethernet (Type 5), while it's Ether Vlan (Type 4). Just ignore it.

8) The command "no mpls ldp advertise-labels" must be configured together with "mpls ldp advertise-labels...", in order to make filtering of label advertisements work.

Saturday, May 16, 2009

Should the crash be an option?

During the previous weekend i had a 7600/RSP720 crashing multiple times in the middle of the night without any apparent reason. Console was showing the following messages:


%Software-forced reload


03:31:06 EET Sat May 9 2009: Unexpected exception to CPU Vector 1500, PC = 0x0A7FA5
B0, LR = 0x0A7FA550

-Traceback= A7FA5B0 A7FA550 A558DD8 A40B698 A442928 A442994 8814E18 8815FB8 8816CD4
A874C8C A8755BC A82EFCC A82F1C0 A8306C0 A7F05EC A87A0DC

CPU Register Context:
MSR = 0x00029200 CR = 0x24444024 CTR = 0x0A44DA0C XER = 0x00000000
R0 = 0x0A7FA550 R1 = 0x13D4C3E0 R2 = 0xFFFCFFFC R3 = 0x137005B4
R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x13D4C3B8 R7 = 0x00029200
R8 = 0x00029200 R9 = 0x00000000 R10 = 0x0D06DBF8 R11 = 0x137005B0
R12 = 0x00013FB4 R13 = 0x04044000 R14 = 0x00000000 R15 = 0x00000000
R16 = 0x0E844B94 R17 = 0x0E8663A8 R18 = 0x0C236BD4 R19 = 0x00000000
R20 = 0x13731364 R21 = 0x79D97210 R22 = 0x13D4C580 R23 = 0x0E851E4C
R24 = 0x0D050000 R25 = 0x00000000 R26 = 0x1373158C R27 = 0x0000001F
R28 = 0x00000001 R29 = 0x0C420000 R30 = 0x103FEAA0 R31 = 0x00000000

Writing crashinfo to bootdisk:crashinfo_20090509-033106-EET

*** System received a Software forced crash ***
signal= 0x17, code= 0x1500, context= 0xce08b9c
PC = 0x8273dcc, Vector = 0x1500, SP = 0x1b394d08
rommon 2 >


Trying to boot manually from the rommon didn't have any positive result, because after the boot was completed (and you were given about 10-20 secs of CLI access!), the RSP crashed again. Since rommon doesn't provide a way to view crashinfo files, you're stuck with the crashes, trying your best at various guesses.

This was a crash that appeared at a random time and on a 7600 that gets configured once in a month, so by best guess was a faulty or stuck supervisor card. After the card was OIRed by the oncall engineer, it booted fine and i finally had a look at the crashinfos. The crashinfo on the RP didn't show anything useful. But the crashinfo on the SP showed the following:


%ONLINE-SP-6-INITFAIL: Module 6: Failed to synchronize Port asic


CCO describes it like below:

Description
The cause of the crash is that the Pinnacle ASIC failed to synchronize. This is usually caused by a bad contact or a badly seated card.

Workaround
The system recovers without user intervention. If the error message recurs, then reseat the concerned line card or module.


Yeah, sure. The system was stuck to rommon after some crashes:

Active crashed three times, disabling auto-boot and dropping to rommon


If anyone would like to answer, these are my questions:

Why couldn't the RSP print an error message (after 3 crashes) on the console while in IOS and sit there waiting for the user to act? What benefit does this specific crash and return to rommon offer?

I understand that generally a crash/reload is used to force the device recover from a bad situation and possibly avoid a long downturn situation. In my case, the situation was a bad contact or a badly seated card (although there is lot of arguing that can be done here). How was the reload supposed to solve this? Was the reload going to "reseat" the module somehow? Are there any pins moved while reloading? Does rommon offer a better seating?

After all, as it proved out the crash didn't help at all, because the RSP returned to ROM, sitting there and waiting, while it could stay in IOS, which provides much better feedback to the user.

On the other hand, it's the redundancy factor. Which, imho, is the only case that a crash/reload should occur (i'm still referring to the case of a badly seated module). As long as you have a 2nd working RSP, it should take the active role. But the system knows whether you have a redundant setup long before giving you the CLI prompt, so it should be easy to check upon it.

What i'm trying to say is that a crash followed by a reload is not always a panacea in case of a problem. It should be an option used, only when there is no other solution. In case of a badly seated module, crashing and moving into rommon isn't of any help (at least until rommon gets upgraded to something more intelligent).

Friday, May 15, 2009

"aaa accounting system guarantee-first" Pt2

Just an update on the "aaa accounting system guarantee-first" issue. I submitted a PER and as it turned out, BU & developers decided to treat it like a bug and fix it. But not completely.

Having to choose between the following 2 solutions, they chose the easy one; the first.

1) "Console/Aux user should be permitted if authentication is not configured with any AAA Server Method (radius/tacacs)".

2) "Console/Aux/VTY access should be permitted if authentication is not configured with the same AAA method (radius/tacacs) as system accounting".

As you can see the 1st solution is applicable only to con/aux, while the 2nd is applicable to con/aux/vty.
Also the 1st solution mixes tacacs and radius, like they are used for the same type of access, something that is not true for the majority of services (Cisco knows that better that anyone else).

Cisco expects me to be satisfied from the PER result, but i'm not. They had a chance of fixing something completely, but they decided to just patch it temporarily.

Wednesday, May 6, 2009

Cisco : "this behavior is expected" & Archive Config Logger

The last 2 months i've been busy playing with an ASR1000, finding bugs and "fighting" with the BUs.

Dealing with a lot of issues lately (not necessarily of ASR's), there are a few times that i have come across Cisco's motto "this behavior is expected". One of them was the fact that the very useful functionality of archive config logger (Configuration Logging) is ruined by the inability to filter out per-user configs applied through an aaa server.

The Configuration Change Notification and Logging (Configuration Logging) feature allows the tracking of configuration changes entered on a per-session and per-user basis by implementing a configuration log. The configuration log tracks each configuration command that is applied, who applied the command, the parser return code for the command, and the time the command was applied.

As you all know, a basic configuration like the following will help you keep track of changes in the configuration of a router:


archive
log config
logging enable
hidekeys

In a "normal" router you'll probably see something like the following:


router#sh archive log config all
idx sess user@line Logged command
8 3 user@vty0 | no ipv6 nd suppress-ra
9 3 user@vty0 | ipv6 ospf 100 area 10
10 4 user@vty0 |no ip sla monitor 99
11 5 user@vty0 |interface GigabitEthernet0/2.35211201
12 5 user@vty0 | encapsulation dot1Q 3521 second-dot1q 1201
13 5 user@vty0 | ip address 10.10.10.129 255.255.255.252

If you happen to have a router that is acting like a bras, accepting ppp sessions and using radius attributes in order to apply per-user ACLs and/or routes, then you'll find out that the above feature is totally useless, because all these per-user entries are stored into the archive logging buffer, overwriting the actual configuration changes you want to watch (when the configuration log is full, the oldest log entry will be removed every time a new entry is added).


router#sh archive log config all
idx sess user@line Logged command
177 0 console@console |permit ip host 1.1.1.1 any
178 0 console@console |deny ip any any
179 0 console@console |deny ip any any
180 0 console@console |ip route 2.2.2.0 255.255.255.240 3.3.3.3
181 0 console@console |ip route 2.2.2.0 255.255.255.240 3.3.3.3
182 0 console@console |ip access-list extended Virtual-Access2.10785#106240381
183 0 console@console |deny ip any 10.0.0.0 0.255.255.255
184 0 console@console |deny ip any 10.0.0.0 0.255.255.255
185 0 console@console |deny ip any 192.168.0.0 0.0.255.255
186 0 console@console |deny ip any 192.168.0.0 0.0.255.255
187 0 console@console |deny ip any 224.0.0.0 15.255.255.255
188 0 console@console |deny ip any 224.0.0.0 15.255.255.255

Note : Please don't ask me about the double entries...

According to Cisco's cli help, only changes to the running config, not the one stored on nvram, are to be logged.


router(config-archive)#log ?
config Logging changes to the running configuration


If you issue the "show run" command (which is supposed to display the running configuration), you won't see these per-user config changes. That's because all these per-user configs, after being downloaded from the aaa server, have a flag set that they should not be nvgen'ed. So they are not shown.

In the Cisco IOS software configuration model, the configuration state is maintained in a distributed manner, with each component storing its own configuration state. To retrieve configuration information, the software must poll every component to collect the distributed information. This configuration state retrieval operation is performed by a process known as nonvolatile generation (NVGEN), and it is used by command line interpreter (CLI) commands such as show running-configuration, write memory, and copy system:running-configuration to display or copy the running system configuration. When invoked, NVGEN queries each system component and each instance of interface or other configuration objects. A running configuration file is constructed as NVGEN traverses the system performing these queries.

Also, according to Cisco:

Restrictions for Configuration Change Notification and Logging
•Only complete commands input in a configuration mode are logged.
•Commands that are part of a configuration file applied with the copy command are not logged.

The Configuration Change Notification and Logging feature tracks changes made to the Cisco IOS software running configuration by maintaining a configuration log. This configuration log tracks changes initiated only through the command-line interface (CLI) or HTTP.


Why Cisco chose to include these per-user "dynamic" config changes into the archive logger, making it another thrown-away feature, is still a mystery to me.

There is also the possibility that this kind of continuous logging might be causing a little increase in cpu usage too.

Yep, time for another Cisco-should-have-fixed-it-in-the-first-place PER.

 
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Greece License.