EVC : Flexible Service Mapping

After 2 months of intense network re-designing (mixed with a little bit of vacation days) and following my two previous posts about Flexible Frame Matching and Flexible VLAN Tag Rewrite, i finally found some free time to write a post about the third step in configuring service instances using the EVC framework : Flexible Service Mapping.

Probably the biggest advantage of the EVC framework is the ability to support multiple services per physical port. This means that under a single physical port you can have any of the following mixed together :

- 802.1q trunk
- 802.1q tunnel
- Local connect
- Scalable EoMPLS (EoMPLS xconnect)
- Multipoint Bridging (L2 bridging)
- Multipoint Bridging (VPLS, SVI-based EoMPLS)
- L3 termination

Besides all of the above, by using the EVC framework you can combine multiple different services from different physical ports, i.e. when using multipoint bridging (aka bridge-domains), in order to put them into the same virtual circuit.



Local connect is a L2 point-to-point service between two service instances on the same system. The service instances can by under the same port (hair-pinning) or under different ports. In contrast with the traditional L2 bridging, this one doesn't use any MAC learning and it's solely between 2 points. Also Local Connect doesn't require any global VLAN resource.

In order to have the following two service instances connect to each other by a L2 point-to-point service, you need first to remove their difference, which is the outer tag (you can also remove both tags).

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10 second-dot1q 100
  rewrite ingress tag pop 1 symmetric

interface Gi1/2
 service instance 20 ethernet
  encapsulation dot1q 20 second-dot1q 100
  rewrite ingress tag pop 1 symmetric

! EVC-LC-10-20 is just a name for this point-to-point connection
connect EVC-LC-10-20 Gi1/1 10 Gi1/2 20

Note : You can use the same service instance number under different physical ports.

In order to have the following two service instances be connected by Local Connect, you don't need any VLAN tag rewrite, because they both have the same vlans.

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10-20

interface Gi1/2
 service instance 20 ethernet
  encapsulation dot1q 10-20

connect EVC-LC-10-20 Gi1/1 10 Gi1/2 20

In order to have the following two service instances be connected by Local Connect, you can either translate the vlan on one of them, or remove the tags on both of them.

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  rewrite ingress tag translate 1-to-1 dot1q 20 symmetric

interface Gi1/2
 service instance 20 ethernet
  encapsulation dot1q 20

connect EVC-LC-10-20 Gi1/1 10 Gi1/2 20

Scalable EoMPLS or EoMPLS xconnect is a L2 point-to-point service between two service instances on different systems. Like Local Connect it doesn't use any MAC learning and it's solely between 2 points. It also doesn't require any global VLAN resource (this applies to scalable EoMPLS only; for SVI-based EoMPLS check VPLS below).

You can have any VLAN tag rewrite configuration under the service instances, as long as you keep in mind the following :

a) If both sides are EVC based, then you need to have common VLAN tag rewrite configurations on both sides
b) If one side is not EVC based, then depending on whether it's a physical interface or a subinterface, you'll probably need to remove one tag from the EVC side (subinterfaces remove tags by default)

Note : By default, VC type 5 is used for EoMPLS. In case VC type 4 is negotiated and used, an additional tag will be added after the VLAN tag rewrite configuration and before the data gets EoMPLS encapsulated.

7600-1

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  xconnect 1.1.1.2 10 encapsulation mpls

7600-2

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  xconnect 1.1.1.1 10 encapsulation mpls

Note : Have a look at Scalable EoMPLS for additional information.

Multipoint Bridging uses the concept of bridge-domains. Bridge-domain (BD) is like a traditional L2 broadcast domain where MAC-based forwarding is used for communication between participants (i'll try to write a new post with more details about bridge-domains). Bridge-domains use global VLAN resources.

In the following example, three service instances are put into the same bridge-domain by translating the tags where necessary.

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  rewrite ingress tag translate 1-to-1 dot1q 20 symmetric
  bridge-domain 20

interface Gi1/2
 service instance 20 ethernet
  encapsulation dot1q 20
  bridge-domain 20

interface Gi1/3
 service instance 30 ethernet
  encapsulation dot1q 30
  rewrite ingress tag translate 1-to-1 dot1q 20 symmetric
  bridge-domain 20

The bridge-domain ID represents the global VLAN used in the system. Extra care needs to taken in case of L2 trunk/tunnel ports participating in a bridge-domain :

a) L2 trunk ports remove automatically the tag on ingress and add it automatically on egress. Equivalent EVC ports need that to be done manually by using the appropriate rewrite actions.
b) L2 tunnel ports add a new tag on ingress and remove it on egress. Equivalent EVC ports do not need any similar rewrite actions, because by default bridge-domains add a new tag on top of the already existing one.

In the following example two ports (a L2 trunk port and an EVC port) are put into the same bridge-domain (Vlan 20). Tag 10 needs to be removed from the EVC port before it joins bridge-domain 20.

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  rewrite ingress tag pop 1 symmetric
  bridge-domain 20

interface Gi2/1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 20
 switchport mode trunk

In the following example two ports (a L2 tunnel port and an EVC port) are put into the same bridge-domain (Vlan 20). On the EVC port, tag 20 is added on top of tag 10 in order to have the incoming frames join bridge-domain 20.

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  bridge-domain 20

interface Gi2/1
 switchport access vlan 20
 switchport mode dot1q-tunnel

VPLS or SVI-based EoMPLS can be accomplished by configuring xconnect under a SVI. This SVI is the same as the one defined by the bridge-domain ID.

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10-20
  bridge-domain 30

interface Gi1/2
 service instance 10 ethernet
  encapsulation dot1q 10-20
  bridge-domain 30

interface Vlan 30
  xconnect 1.1.1.2 10 encapsulation mpls

By adding "split-horizon" after the bridge-domain ID in both service instances, there can be no L2 communication between them.

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10-20
  bridge-domain 30 split-horizon

interface Gi1/2
 service instance 10 ethernet
  encapsulation dot1q 10-20
  bridge-domain 30 split-horizon

interface Vlan 30
  xconnect 1.1.1.2 10 encapsulation mpls

By adding an additional tag through a rewrite action in both service instances, you can differentiate them, while they are being transfered through the same VC.

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10-20
  rewrite ingress tag push dot1q 21 symmetric
  bridge-domain 30 split-horizon

interface Gi1/2
 service instance 10 ethernet
  encapsulation dot1q 10-20
  rewrite ingress tag push dot1q 22 symmetric
  bridge-domain 30 split-horizon

interface Vlan 30
  xconnect 1.1.1.2 10 encapsulation mpls

SVI-based EoMPLS can be considered like a VPLS, where there is only one VC pointing to one neighbor.

Note : Have a look at SVI-based EoMPLS for additional information.

For L3 termination you have the usual two options : use subinterfaces or use bridge-domains (just like switchports) and SVIs. ES/ES+ and SIP-400 cards support termination of double-tagged traffic too.

Keep in mind the following :

a) you must remove all tags before terminating L3 traffic
b) you must use matching rules based on unique single or double tags (no vlan ranges are supported, although they might be accepted)

This is an example using a bridge-domain and the equivalent SVI:

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  rewrite ingress tag pop 1 symmetric
  bridge-domain 40

interface Gi1/2
 service instance 10 ethernet
  encapsulation dot1q 20 second-dot1q 30
  rewrite ingress tag pop 2 symmetric
  bridge-domain 40

interface Vlan 40
  ip address 1.1.1.1 255.255.255.0

This is an example using subinterfaces:

interface Gi1/1.10
 encapsulation dot1q 10
 ip address 1.1.1.1 255.255.255.0

interface Gi1/1.20
 encapsulation dot1q 20 second-dot1q 30
 ip address 1.1.1.1 255.255.255.0

Note : ES cards have a major limitation : single-tagged vlans configured under a subinterface are global significant. On the other hand, double-tagged vlans are local significant. On the ES+ and SIP-400 cards, both single-tagged and double-tagged vlans are local significant.

EVC : Flexible VLAN Tag Rewrite

Following the previous post about Flexible Frame Matching, this new post describes the second major step in configuring service instances using the EVC framework : Flexible VLAN Tag Rewrite.

Each service instance can change the existing VLAN tag to be a new VLAN tag by adding, removing, or translating one or two VLAN tags. Flexible VLAN tag rewrite includes 3 main operations :

1) pop (remove an existing tag)
2) push (add a new tag)
3) translate (change one or two tags to another one or two tags) - this can be seen as a combination of pop and push operations

Theoretically, any existing combination of one or two VLAN tags can be changed to any new combination of one or two VLAN tags by just using a simple (as soon as you get the idea) line of configuration. Practically, there are some limitations what you'll see below.

These are the relevant CLI options under the service instance (you need first to have configured flexible frame matching for these to appear) :

7600(config-if-srv)#rewrite ingress tag ?
pop        Pop the tag
push       Rewrite Operation of push
translate  Translate Tag

Pop operation

7600(config-if-srv)#rewrite ingress tag pop ?
1  Pop the outermost tag
2  Pop two outermost tags

! remove one tag
7600(config-if-srv)#rewrite ingress tag pop 1 ?
symmetric  Tag egress packets as specified in encapsulation
<cr>

! remove two tags
7600(config-if-srv)#rewrite ingress tag pop 2 ?
symmetric  Tag egress packets as specified in encapsulation
<cr>

Push operation

7600(config-if-srv)#rewrite ingress tag push ?
dot1q  Push dot1q tag

! add one tag
7600(config-if-srv)#rewrite ingress tag push dot1q ?
<1-4094>  VLAN id

7600(config-if-srv)#rewrite ingress tag push dot1q 20 ?
second-dot1q  Push second dot1q tag
symmetric     Tag egress packets as specified in encapsulation
<cr>

! add two tags
7600(config-if-srv)#rewrite ingress tag push dot1q 20 second-dot1q ?
<1-4094>  VLAN id

7600(config-if-srv)#rewrite ingress tag push dot1q 20 second-dot1q 30 ?
symmetric  Tag egress packets as specified in encapsulation
<cr>

Translate operation

7600(config-if-srv)#rewrite ingress tag translate ?
1-to-1  Translate 1-to-1
1-to-2  Translate 1-to-2
2-to-1  Translate 2-to-1
2-to-2  Translate 2-to-2

! remove one tag and add one new tag
7600(config-if-srv)#rewrite ingress tag translate 1-to-1 dot1q 20 ?
symmetric  Tag egress packets as specified in encapsulation
<cr>

! remove one tag and add two new tags
7600(config-if-srv)#rewrite ingress tag translate 1-to-2 dot1q 20 second-dot1q 30 ?
symmetric  Tag egress packets as specified in encapsulation
<cr>

! remove two tags and add one new tag
7600(config-if-srv)#rewrite ingress tag translate 2-to-1 dot1q 20 ?
symmetric  Tag egress packets as specified in encapsulation
<cr>

! remove two tags and add two new tags
7600(config-if-srv)#rewrite ingress tag translate 2-to-2 dot1q 20 second-dot1q 30 ?
symmetric  Tag egress packets as specified in encapsulation
<cr>

Examples

interface GigabitEthernet1/2
!
service instance 10 ethernet
encapsulation dot1q 10
! remove one tag (10) on ingress
! add one tag (10) on egress
rewrite ingress tag pop 1 symmetric
!
service instance 20 ethernet
encapsulation dot1q 10 second-dot1q 20
! remove two tags (10/20) on ingress
! add two tags (10/20) on egress
rewrite ingress tag pop 2 symmetric
!
service instance 30 ethernet
encapsulation dot1q 30
! add one tag (300) on ingress
! remove one tag (300) on egress (if the resulting frame doesn't match tag 30, it's dropped)
rewrite ingress tag push dot1q 300 symmetric
!
service instance 40 ethernet
encapsulation dot1q 40
! add two tags (400/410) on ingress
! remove two tags (400/410) on egress (if the resulting frame doesn't match tag 40, it's dropped)
rewrite ingress tag push dot1q 400 second-dot1q 410 symmetric
!
service instance 50 ethernet
encapsulation dot1q 50 second-dot1q 1-4094
! remove one tag (50) and add one new tag (500) on ingress
! remove one tag (500) and add one new tag (50) on egress
! the inner tags (1-4094) remain unchanged
rewrite ingress tag translate 1-to-1 dot1q 500 symmetric
!
service instance 60 ethernet
encapsulation dot1q 60
! remove one tag (60) and add two new tags (600/610) on ingress
! remove two tags (600/610) and add one new tag (60) on egress
rewrite ingress tag translate 1-to-2 dot1q 600 second-dot1q 610 symmetric
!
service instance 70 ethernet
encapsulation dot1q 70 second-dot1q 100
! remove two tags (70/100) and add one new tag (700) on ingress
! remove one tag (700) and add two new tags (70/100) on egress
rewrite ingress tag translate 2-to-1 dot1q 700 symmetric
!
service instance 80 ethernet
encapsulation dot1q 80 second-dot1q 200
! remove two tags (80/200) and add two new tags (800/810) on ingress
! remove two tags (800/810) and add two new tags (80/200) on egress
rewrite ingress tag translate 2-to-2 dot1q 800 second-dot1q 810 symmetric

There are some important things to keep in mind when configuring Flexible VLAN Tag Rewrite.

1) You have to use the "symmetric" keyword, although the CLI might not give you this impression:

7600(config-if-srv)#rewrite ingress tag pop 1 ?
symmetric  Tag egress packets as specified in encapsulation
<cr>

7600(config-if-srv)#rewrite ingress tag pop 1
Configuration not accepted by the platform
7600(config-if-srv)#rewrite ingress tag pop 1 symmetric
7600(config-if-srv)#

Generally rewrite configurations should always be symmetric. Whatever rewrites are on the ingress direction, you should have the reverse rewrites on the egress direction for the same service instance configuration. So, if you pop the outer VLAN tag on ingress direction, then you need to push the original outer VLAN tag back on the egress direction for that same service instance. All this is done automatically by the system when using the "symmetric" keyword. Have a look at the examples included above and check the comments to see what operations are happening on ingress and egress.

2) Due to the mandatory symmetry, some operations can only be applied to a unique tag matching service instance (so they are not supported for VLAN range configurations) or cannot be applied at all.

i.e.
You cannot translate a range of vlans

7600(config-if-srv)#encapsulation dot1q 10 - 20
7600(config-if-srv)#rewrite ingress tag translate 1-to-1 dot1q 30 symmetric
Encapsulation change is not logically valid.

You cannot pop a range of vlans

7600(config-if-srv)#encapsulation dot1q 10 second-dot1q 20,30
7600(config-if-srv)#rewrite ingress tag pop 2 symmetric
Encapsulation change is not logically valid.

If supposedly you could do the above, how could the opposite be done? i.e. if the system removed the tags from frames matching inner vlans 20,30 on the ingress, how would the system know on which frames to add 20 and on which to add 30 on the egress?

Of course you can push a new vlan over a range of vlans.

7600(config-if-srv)#encapsulation dot1q 10-20
7600(config-if-srv)#rewrite ingress tag push dot1q 30 symmetric

You can only push one or two tags for "encapsulation untagged" and "encapsulation default". No pop or translate operations are supported.

As a rule you can think of "you cannot pop or translate something that is not specifically defined as a single unit". Just imagine what would happen in the opposite direction and everything should become clear.

Keep in mind that some configurations might be accepted, but they won't work.

3) You cannot have more than one VLAN tag rewrite configuration under a single service instance. That means you can have either none or one. If there is no VLAN tag rewrite configuration, the existing VLAN tag(s) will be kept unchanged. If you need more than one, you might want to try to create more service instances using more specific frame matching criteria on each one. The translate operation might also seem useful in such conditions.

4) You need to be extra careful when using Flexible VLAN Tag Rewrite and Bridge Domains. Flooded (broadcast/multicast/unknown unicast) packets will get dropped by the service instances that do not agree on the egress tag. Although all service instances under a common bridge domain will get the flooded frame, there is an internal validation mechanism that checks whether the result of egress rewrite (based on the opposite of ingress rewrite) will allow the flooded frame to pass. The push operations under the examples show this behavior.

5) To have an EVC based port act like a L2 802.1q trunk port, you need to remove the outer tag manually and then put it under a bridge domain. On normal L2 switchports this is done automatically by the system.

So this

interface Gi1/1
 switchport
 switchport mode trunk
 switchport trunk allowed vlan 10

is equivalent to this

interface Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  rewrite ingress tag pop 1 symmetric
  bridge-domain 10

Note: The above examples were done on a 7600 with ES+ cards running 12.2(33)SRB IOS.

EVC : Flexible Frame Matching

EVC stands for Ethernet Virtual Connection and in Cisco's platforms it's used to represent Cisco's software architecture to address Carrier Ethernet Services. In MEF (Metro Ethernet Forum) terminology EVC means "Ethernet Virtual Connection/Circuit", but here EVC represents also the whole Carrier Ethernet software infrastructure developed by Cisco.

EVC has many advantages (which i will try to describe in future posts), one of them being the Flexible Frame Matching. Flexible Frame Matching is a functionality that allows each service instance to match frames with either a unique single vlan, or a list/range of vlans. It can also match single/double tagged frames, untagged frames, or everything else that belongs to the default category.

Flexible Frame Matching is the first major step after configuring a service instance. This is the complete idea:

1) Service Instance definition (create the service instance)
2) Flexible frame matching (configure what frames need to be matched based on vlan match criteria)
3) Flexible VLAN tag rewrite (configure the action to do on the matched frames' vlan tags)
4) Flexible Service Mapping (map the service instance to a service)
5) Extra service features (apply some extra features on the service instance, i.e. QoS)

The middle 3 most important steps can also be described as:

a) Frame matching
b) Frame rewrite
c) Frame forwarding

Example

interface Gi1/1
 ! Service Instance definition
 ! ID is local port significant
service instance 10 ethernet
  ! Flexible frame matching
encapsulation dot1q 10 second-dot1q 20
  ! Flexible VLAN tag rewrite
rewrite ingress tag pop 1 symmetric 
  ! Service Mapping
xconnect 10.10.10.10 100 encapsulation mpls 
  ! Extra service features
service-policy input TEST-INPUT-POLICY

The current EVC implementation supports matching only on vlan tags, but in future we may see matching on other L2 fields too, since the hardware is quite capable.

These are the current supported vlan matching configurations:

• Single tagged frames, where match criteria can be a single vlan, a list/range of vlans, or any vlan (1-4094)

encapsulation dot1q <vlan-id>
encapsulation dot1q <vlan-id>, <vlan-id>
encapsulation dot1q <vlan-id> - <vlan-id>
encapsulation dot1q any

• Double tagged frames, where first VLAN tag can be only single (software limitation), while second VLAN tag can be single, list/range, or any

encapsulation dot1q <vlan-id> second-dot1q <vlan-id>
encapsulation dot1q <vlan-id> second-dot1q <vlan-id>, <vlan-id>
encapsulation dot1q <vlan-id> second-dot1q <vlan-id> - <vlan-id>
encapsulation dot1q <vlan-id> second-dot1q any

• Untagged frames, where all untagged frames are matched

encapsulation untagged

• Default tag frames, where all tagged/untagged frames that are not matched by other more specific service instances are matched

encapsulation default

Examples

interface Gi1/1
!
service instance 10
  ! single tagged frames with a specific tag
encapsulation dot1q 10
!
service instance 20
  ! single tagged frames with multiple tags
encapsulation dot1q 20,22,24,26-28
!
service instance 30
  ! single tagged frames with any tag
encapsulation dot1q any
!
service instance 40
  ! frames with a specific single outer tag and specific single inner tag
encapsulation dot1q 10 second-dot1q 20
!
service instance 50
  ! frames with a specific single outer tag and multiple inner tags
encapsulation dot1q 10 second-dot1q 20,22,24,26-28
!
service instance 60
  ! frames with a specific single outer tag and any inner tag
encapsulation dot1q 10 second-dot1q any
!
service instance 70
  ! frames without a tag
encapsulation untagged
!
service instance 80
  ! frames that do not match under any other service instance
encapsulation default

There are some important things to keep in mind when configuring Flexible Frame Matching.

1) When you have multiple vlan match criteria configured under different service instances of a single physical interface, the most specific is the one that wins (it's like the longest match rule used in the routing table). So the order of service instances under an interface doesn't have the same effect like the classes in MQC. This is because frame matching is done by hardware using the linecard's TCAM table, where each frame matching configuration gets converted to 1 or more TCAM entries (vlan lists/ranges in matching criteria are the most TCAM consuming configurations). The number of 16000 service instances per ES20 module is based on the assumption that each service instance uses a single TCAM entry.

2) When you don't get any match according to the above longest match rule, matching is done according to a looser match algorithm, where a single tag configuration matches all frames that have a common outer tag (regardless of the number of inner tags) and a double tag configuration matches all frames that have common the first 2 tags (regardless of the number of 2+ inner tags; btw, i'm planning of doing a triple-tag test soon).

Example

interface G1/1
service instance 10 ethernet
encapsulation dot1q 10
service instance 20 ethernet
encapsulation dot1q 10 second-dot1q 20
service instance 30 ethernet
encapsulation default

On the above configuration:

10/20 will be matched by service instance 20 (both tags matched)
10/30 will be matched by service instance 10 (outer tag matched)
20/30 will be matched by service instance 30 (no tag matched)

"encapsulation dot1q 10" matches "10", "10/20", "10/30" and so on.
"encapsulation dot1q 10 second-dot1q 20" matches "10/20", "10/20/30", "10/20/40" and so on.

Note: The above examples were done on a 7600 with ES+ cards running 12.2(33)SRB IOS.

Debugging SUP720 booting process

On Tuesday i had an rommon issue with a RSP720 in a 7600. One of the things i tried on a spare SUP720 (i didn't have any spare RSP720) in order to check for possible solutions, was to remove the NVRAM battery for a while and then reinsert it. As it proved out, i found a nice way to "debug" the booting process. Probably config-register changed to a factory default value (different than the usual 0x2102), causing all these messages to be displayed on console.

System Bootstrap, Version 8.4(2) Release
Copyright (c) 1994-2005 by cisco Systems, Inc.

Testing lower main memory - data equals address
Testing lower main memory - checkerboard
Testing lower main memory - inverse checkerboard
Clearing lower memory for cache initialization
Clearing bss
Clearing autoboot state machine
melody_present_reg: 1st read w/ 0xffff
melody_present_reg: 2nd read w/ 0xffff, reversed: 0x0
melody_present_reg: 1st read w/ 0xffff
melody_present_reg: 2nd read w/ 0xffff, reversed: 0x0
Bootdisk adapter not detected, use bootflash instead.

Reading monitor variables from NVRAM
Reset reason for CPU board 0xffff , BaseBoard 0x280ffff, display 0x20000System Reset by Software.

Enabling interrupts
Initializing TLB
Initializing cache
Initializing required TLB entries
Initializing main memory
Sizing NVRAM
Initializing PCMCIA controller
Exiting init
Cat6k-Sup720/SP processor with 1048576 Kbytes of main memory



rommon 1 > boot disk0:c7600s72033-advipservicesk9-mz.122-33.SRD2.bin
Loading image, please wait ...

Stack pointer       : 0x8FFFFF80
monstack            : 0x800FFFC0
monra               : 0xBFC26794
edata : 0x801097A0
magic : 0xFEEDFACE
memsize             : 0x10000000
uncomp_size         : 0x07EC9400
comp_size           : 0x07EC9400
comp_checksum       : 0xEC6D779E
uncomp_checksum     : 0xEC6D779E
CZIP_MEM_BASE       : 0x80000000
_end       : 0x8010CC40

Self extracting the image...
IOS compressed src copy is     : 0x801097B4

czip + IOS tar size is :0x00B2BB2C

czip + IOS Tar image is now :0x801099B4

cptr is now :0x8010CD40
IOS compressed dest copy is     : 0x8010CD40
 [OK]
image_entry :0x80100000
image_entry :0x80100000
__start : 0x80100000
tar_avail_size is now :0x0739D600
Tar image address is             : 0x80C38940
tar_size is           : 0x0739D200
cpu type                   : 0x00000019
uncomp_size                : 0x07EC9400
monstack                   : 0x800FFFC0

image_info.entry_point   = 0x80100000
image_info.section_count = 0x00000005
image_info.monstack      = 0x800FFFC0
image_info.monra         = 0xBFC26794
image_info.param0        = 0x00000002
image_info.param1        = 0x00000000
image_info.param2        = 0x800068D8
image_info.param3        = 0x80100000
image_info.reg_k0        = 0x80C38940
image_info.reg_k1        = 0x0739D200
Section Index = 0x00000000
    source    = 0x8010CE5C
    dest      = 0x80100000
    bytes     = 0x00010000
Section Index = 0x00000001
    source    = 0x8011CE5C
    dest      = 0x80110000
    bytes     = 0x00001940
Section Index = 0x00000002
    source    = 0x8011E79C
    dest      = 0x80111940
    bytes     = 0x00000020
Section Index = 0x00000003
    source    = 0x8011E7BC
    dest      = 0x80111960
    bytes     = 0x00B1A094
Section Index = 0x00000004
    source    = 0x80C38850
    dest      = 0x80C2B9F4
    bytes     = 0x00000000
reg_v0: 0x00000000
reg_k0: 0x80C38940
reg_k1: 0x0739D200
tar_start: 0x00000000
tar_size: 0x00000000
 Tar image address is             : 0x80C38940
tar size is :0x009CD42A
 Tar magic : ustar Tar filename : C2LC memsize             : 0x10000000
Tar gid         : 035231Tar uncomp_size         : 0x009CD42A
Tar mtime         : 11205304144 Tar username          : ciiTar comp_checksum       : 0x00001200
Tar group name    : buildTar prefix    : tar_size  in czip         : 0x0739D200
Stack pointer       : 0x8FFFFF80
monstack            : 0x800FFFC0
monra               : 0xBFC26794
edata : 0x80111960
magic : 0xFEEDFACE
memsize             : 0x10000000
uncomp_size         : 0x0209842C
comp_size           : 0x00B1A07F
comp_checksum       : 0x8FE686D4
uncomp_checksum     : 0xB44EFA0C
Compressed IOS src copy is     : 0x80111974
tar_dest is :0x88C5A600
tar_size is :0x0739D200
Compressed IOS dest copy is     : 0x88140580
Tar src before IOS decompression is     : 0x80C38940
Tar dest before IOS decompression is     : 0x88C5A600
compressed IOS  src is     : 0x88140580
IOS uncompressed dest copy is     : 0x8013D3E0
Self decompressing the image : ####################################################################################
####################################################################################
# [OK]
e_shoff :
0x0209829C
e_flags : 0x10001001
e_phnum :
0x00000001
Source elf_hdr->e_shnum = 0x0000000A
Setting up to copy ELF section 0x00000001
 to image_info section 0x00000000
 sh_name = 0x0000000B
 sh_type = 0x00000001
 sh_flags = 0x00000007
 sh_addr = 0x80100000
 sh_offset = 0x00000060
 sh_size = 0x01CA8000
 sh_link = 0x00000000
 sh_info = 0x00000000
 sh_addralign = 0x00000008
 sh_entsize = 0x00000000
Setting up to copy ELF section 0x00000002
 to image_info section 0x00000001
 sh_name = 0x00000011
 sh_type = 0x00000001
 sh_flags = 0x00000003
 sh_addr = 0x81DA8000
 sh_offset = 0x01CA8060
 sh_size = 0x002A46E0
 sh_link = 0x00000000
 sh_info = 0x00000000
 sh_addralign = 0x00000008
 sh_entsize = 0x00000000
Setting up to copy ELF section 0x00000003
 to image_info section 0x00000002
 sh_name = 0x00000017
 sh_type = 0x00000001
 sh_flags = 0x00000003
 sh_addr = 0x8204C6E0
 sh_offset = 0x01F4C740
 sh_size = 0x00040ECC
 sh_link = 0x00000000
 sh_info = 0x00000000
 sh_addralign = 0x00000004
 sh_entsize = 0x00000000
Setting up to copy ELF section 0x00000004
 to image_info section 0x00000003
 sh_name = 0x00000024
 sh_type = 0x00000001
 sh_flags = 0x10000003
 sh_addr = 0x8208D5AC
 sh_offset = 0x01F8D60C
 sh_size = 0x00004034
 sh_link = 0x00000000
 sh_info = 0x00000000
 sh_addralign = 0x00000010
 sh_entsize = 0x00000000
 sh_type = 0x00000008
 sh_flags = 0x10000003
 sh_addr = 0x820915E0
 sh_offset = 0x01F91640
 sh_size = 0x000009E0
 sh_type = 0x00000008
 sh_flags = 0x00000003
 sh_addr = 0x82091FC0
 sh_offset = 0x01F91640
 sh_size = 0x021EB140
tar file start = 0x8428A070
cpu type                   : 0x00000019
uncomp_size                : 0x0209842C
monstack                   : 0x800FFFC0
image_info.entry_point   = 0x80100000
image_info.section_count = 0x00000005
image_info.monstack      = 0x800FFFC0
image_info.monra         = 0xBFC26794
image_info.param0        = 0x00000002
image_info.param1        = 0x00000000
image_info.param2        = 0x800068D8
image_info.param3        = 0x80100000
image_info.reg_k0        = 0x8428A070
image_info.reg_k1        = 0x0739D200
Section Index = 0x00000000
    source    = 0x8013D440
    dest      = 0x80100000
    bytes     = 0x01CA8000
Section Index = 0x00000001
    source    = 0x81DE5440
    dest      = 0x81DACF70
    bytes     = 0x002A46E0
Section Index = 0x00000002
    source    = 0x82089B20
    dest      = 0x82051650
    bytes     = 0x00040ECC
Section Index = 0x00000003
    source    = 0x820CA9EC
    dest      = 0x8209251C
    bytes     = 0x00004034
Section Index = 0x00000004
    source    = 0x88C5A600
    dest      = 0x8428A070
    bytes     = 0x0739D200

data_size  in czip         : 0x00008000

bss end of IOS is         : 0x84282070

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, c7600s72033_sp Software (c7600s72033_sp-ADVIPSERVICESK9-M), Version 12.2(33)SRD2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 21-May-09 09:55 by prod_rel_team
Image text-base: 0x401012B8, data-base: 0x41DACF70


*Jan  1 00:00:11.983: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Jan  1 00:00:09.523: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor

*Jan  1 00:00:11.983: %OIR-SP-6-CONSOLE: Changing console ownership to route processor



System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2006 by cisco Systems, Inc.
Cat6k-Sup720/RP platform with 1048576 Kbytes of main memory

Download Start
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Download Completed! Booting the image.
Self decompressing the image : ####################################################################################
####################################################################################
####################################################################################
####################################################################################
######## [OK]

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, c7600s72033_rp Software (c7600s72033_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRD2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 21-May-09 09:42 by prod_rel_team
Image text-base: 0x401012B8, data-base: 0x4412DDF0


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco CISCO7609 (R7000) processor (revision 1.2) with 983008K/65536K bytes of memory.
Processor board ID XXX
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
1 Virtual Ethernet interface
76 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.



Press RETURN to get started!

Setting the config-register back to 0x2102 stopped the above messages from being displayed while booting. You have to re-upgrade (or re-enable) the rommon too.

VLAN resource vs VLAN significance

These are two terms that get used quite often when talking about hardware switching platforms (i.e. 6500/7600).

As you all know, VLAN is a virtual LAN defined initially for L2 bridging. One VLAN represents a broadcast domain and packet forwarding within the same VLAN is based on mac-address learning. Another name for this is multipoint bridging (you'll meet that term quite frequently in ES/ES+ setups).

VLANs are also used in the 802.1q ethernet encapsulation method. Services like L2 point-to-point are usually using this functionality. Such services, as EoMPLS xconnect or local-connect (more on that on another post about ES/ES+), do not require a VLAN for bridging, but they might require a vlan as a packet encapsulation method.

Actually you might differentiate the above 2 VLAN types like below:

system VLAN or global VLAN : used for L2 bridging
port VLAN or access VLAN : used for L2 point-to-point services (do not confuse it with the vlan-id used under switchports in access mode)

Global VLAN resource is per system and is limited to 4094 (0 & 4095 are reserved) on 6500s/7600s (you can use "sh platform hardware capacity vlan" to check this).
Access VLAN resource is per linecard and is limited to 16000 on the ES/ES+ cards (probably lower on some SIPs), which equals the number of service instances supported.

L3 services are a different story. Each physical/logical L3 interface requires a global VLAN resource (use "sh vlan internal usage" to find these). An exception on this are the PPP and ISG sessions (supported on SIP-400 cards).

Bridge-domains require also a global VLAN resource each. A bridge-domain, which is used in the ES/ES+ cards, is like a traditional L2 bridging instance, where many physical or logical ports connect to. What is interesting about this, is the ability to have L2 interworking between normal switchports, EVC service instances, ATM/FR PVCs/DLCIs, EoMPLS/VPLS VCs, etc.

Regarding the VLAN significance, there are also 2 VLAN types here:

per system or global significant : when a vlan-id must be unique across the whole system
per port or local significant : when a vlan-id can be the same across different ports

Although it might seem a little bit confusing, VLAN resource is somewhat independent of VLAN significance. i.e. in a case of a L3 subinterface, you always require a global VLAN resource, regardless of the linecard used. But whether you'll have local VLAN significance or not depends on the linecard itself.

SIP-400 and ES/ES+ cards support local VLAN significance, but keep in mind that the "older" ES cards do not support it when using single-tagged subinterfaces.

EoMPLS on 7600 : PFC-based, SVI-based, Scalable

With the "recent" addition of the ES/ES+ cards (although some SIP cards might belong in the same category), 7600 can now support 3 types of EoMPLS configurations . Depending on your needs and the available hardware, you will find at least one between them that can help you implement your network design.

PFC-based EoMPLS
This is the most well-known EoMPLS that is configured under a physical interface or subinterface using the xconnect command. Label imposition/disposition is being done by either the SUP's PFC or the ingress module's DFC, so it's supported on all the usual LAN (i.e. 67xx) cards.

You can use it for point-to-point L2 services between single attachment circuits, where you do not require any local switching or mac-address learning. Each local attachment circuit uses a single EoMPLS pseudowire to connect to a single remote attachment circuit.

Local Router

int Gi1/1
 xconnect 1.1.1.2 10 encapsulation mpls

int Gi1/2.20
 encapsulation dot1q 20
 xconnect 2.2.2.2 20 encapsulation mpls

Remote Router

int Gi1/1
 xconnect 1.1.1.1 10 encapsulation mpls

int Gi1/2.20
 encapsulation dot1q 20
 xconnect 2.2.2.1 20 encapsulation mpls

SVI-based EoMPLS
This EoMPLS is configured under a vlan (SVI) interface using also the xconnect command. Label imposition/disposition is being done by the egress module (where the EoMPLS VC points to), so it must be either a SIP or an ES/ES+ based one. Also there is a limitation that you must have a L3 (sub)interface as the egress interface where the EoMPLS VC will pass through.

You can use it for point-to-point L2 services between multiple attachment circuits, where you also require local switching and mac-address learning. It's like a mini VPLS implementation, where multiple local attachment circuits use a single EoMPLS pseudowire to connect to multiple remote attachment circuits.

Local Router

int Gi1/1
 switchport
 switchport mode access
 switchport access vlan 10

int Gi1/2
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport trunk allow vlan 10

int Vlan 10
 xconnect 1.1.1.2 10 encapsulation mpls

Remote Router

int Gi1/1
 switchport
 switchport mode access
 switchport access vlan 10

int Gi1/2
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport trunk allow vlan 10

int Vlan 10
 xconnect 1.1.1.1 10 encapsulation mpls

Scalable EoMPLS
This new EoMPLS is configured under an EVC service instance using again the xconnect command. Label imposition/disposition is being done by the ingress module where the service instance is configured, so a module with EVC support (i.e. ES/ES+) is required.

You can use it for point-to-point L2 services between single attachment circuits, just like the PFC-based EoMPLS.

Besides that, when dealing with EVCs you can also configure the xconnect under a vlan (SVI) interface (when the service instance is mapped to a bridge-domain) for point-to-point L2 services between multiple attachment circuits, where you can also enable or disable local L2 connectivity using split-horizon.

Both EoMPLS configurations can be used concurrently on the same physical port, with less global resources (i.e. vlans) used in comparison to the first two EoMPLS types. Keep in mind that you still need a SIP or an ES/ES+ egress module for the SVI-based EoMPLS.

Local Router

int Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  xconnect 1.1.1.2 10 encapsulation mpls
 service instance 20 ethernet
  encapsulation dot1q 20
  rewrite ingress tag pop 1 symmetric
  bridge-domain 20

int Vlan 20
 xconnect 2.2.2.2 20 encapsulation mpls

Remote Router

int Gi1/1
 service instance 10 ethernet
  encapsulation dot1q 10
  xconnect 1.1.1.1 10 encapsulation mpls
 service instance 20 ethernet
  encapsulation dot1q 20
  rewrite ingress tag pop 1 symmetric
  bridge-domain 20

int Vlan 20
 xconnect 2.2.2.1 20 encapsulation mpls

Notes:
1) The vlan configured in "encapsulation vlan-id" under the service instance is local significant. The vlan configured in bridge-domain is a global vlan-id and can be different from the first one.

2) Scalable EoMPLS doesn't require global VLAN resources for the EoMPLS xconnect. In PFC-based EoMPLS each (sub)interface requires one global VLAN resource.

3) An attachment circuit (AC) is usually the customer connection to a service provider network (CE-PE). An AC may be a physical or virtual port/circuit and may be any transport technology, i.e. Frame Relay DLCI, ATM PVC, Ethernet VLAN.

Configuring 802.1q tunnels over Vlan-based EoMPLS & MUX-UNI

One of the main disadvantages (besides all the usual disputes about L2 vs L3) of 802.1q tunneling is the fact that all the switches the tunnel is passing through, are learning all the MAC addresses passing through it, plus the consumption of other system resources like vlans and spanning-tree instances. So if you have many 802.1q tunnels passing through one or more central switches as bridged L2 traffic, you might end up filling their TCAM space. For point-2-point 802.1q tunnels, an easy solution is to use EoMPLS tunnels on the central routers/switches in order to allow the double-tagged 802.1q traffic to pass over them as transparently as possible.

The following is our test network.



We need to transfer L2 traffic belonging to vlans 100-110 and 200-210 across some switches/routers and at the same time we need to pass IP traffic belonging to vlans 10 and 20 from the 6500s to the 7600s and vice-versa. All of the above using a single link between 6500s and 7600s.

EoMPLS uses IP/MPLS as its underlying technology, so it really doesn't matter what you have between the EoMPLS endpoints, as long as you have IP/MPLS connectivity between them (the ingress and egress PEs are the only two routers in the MPLS backbone with knowledge of the Layer 2 VCs, so only these require EoMPLS functionality). If you want to avoid MPLS and use solely IP, then you you can create a tunnel based on L2TPv3 and transfer Ethernet frames over it.

EoMPLS is one of the AToM (Any Transport over MPLS) transport types. AToM transports Layer 2 packets over a IP/MPLS backbone using a directed LDP session between edge routers for setting up and maintaining connections. Normally forwarding occurs through the use of two level labels that provide switching between the edge routers. The Tunnel label (external) routes the packet over the MPLS backbone to the egress PE at the ingress PE. The VC label is a demuxing label that determines the connection at the tunnel endpoint (the particular egress interface on the egress PE as well as the vlan-id for an ethernet frame). If PE routers are connected back-2-back (like in this example), then there is no need for a tunnel label (the "implicit null" label is used to signal that to the peer).

EoMPLS works by encapsulating ethernet PDUs in MPLS packets and forwarding them across the MPLS network. Each PDU is transported as a single packet. It can work in 2 modes:

1) Port-based, where everything that comes into a port gets tunneled (using a VC type of 5)
2) Vlan-based, where only specific vlans get tunneled (using a VC type of either 4 or 5)

Here i'm using the vlan-based mode, because i want to combine it with MUX-UNI. MUX-UNI is a nice feature on 6500s and 7600s, that provides the ability to partition a physical port in such a way that multiple Layer 2 and Layer 3 services can be created over a single UNI (usually the point where CE connects to the ingress PE). Is works very well on the 6500/7600 LAN cards (i.e. 67xx), but if you are thinking of more advanced things you probably need the ES/ES+ ones (i will make a new post about various ES+ setups soon).

Before configuring EoMPLS, you have to make sure that IP routing in enabled so that the PE routers (in our case the 7600s) can reach each other and MPLS is enabled so that a label switched path (LSP) exists between the PE routers. No need for complexity here, so we use just Vlan 30 in both 7600s for this. CEF is needed too, but it's enabled by default and cannot be disabled on 6500s/7600s.

Configuration

These are the relevant configurations:

6500-1

interface GigabitEthernet2/1
 description ** vlans 100-110 **
 switchport
 switchport access vlan 2001
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
 description ** vlans 200-210 **
 switchport
 switchport access vlan 2002
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface Vlan10
 description ** test **
 ip address 10.10.10.1 255.255.255.0
!
interface GigabitEthernet2/3
 description ** 7600-1 Gi3/1 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10,2001,2002
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable

7600-1

interface GigabitEthernet3/1
 description ** 6500-1 G2/3 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
 encapsulation dot1Q 2001
 xconnect 30.30.30.2 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
 encapsulation dot1Q 2002
 xconnect 30.30.30.2 2002 encapsulation mpls
!
interface GigabitEthernet3/2
 description ** 7600-2 Gi3/2 **
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 mtu 9216
!
interface Vlan10
 description ** test **
 ip address 10.10.10.2 255.255.255.0
!
interface Vlan30
 description ** MPLS **
 ip address 30.30.30.1 255.255.255.0
 mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
 permit 30.30.30.0 0.0.0.255

7600-2

interface GigabitEthernet3/1
 description ** 6500-2 G2/3 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 20
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
 encapsulation dot1Q 2001
 xconnect 30.30.30.1 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
 encapsulation dot1Q 2002
 xconnect 30.30.30.1 2002 encapsulation mpls
!
interface GigabitEthernet3/2
 description ** 7600-1 Gi3/2 **
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 mtu 9216
!
interface Vlan20
 description ** test **
 ip address 20.20.20.2 255.255.255.0
!
interface Vlan30
 description ** MPLS **
 ip address 30.30.30.2 255.255.255.0
 mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
 permit 30.30.30.0 0.0.0.255

6500-2

interface GigabitEthernet2/1
 description ** vlans 100-110 **
 switchport
 switchport access vlan 2001
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
 description ** vlans 200-210 **
 switchport
 switchport access vlan 2002
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface Vlan20
 description ** test **
 ip address 20.20.20.1 255.255.255.0
!
interface GigabitEthernet2/3
 description ** 7600-2 Gi3/1 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 20,2001,2002
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable

Verification

Check the 802.1q tunnel ports on the edges of the network

6500-1#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-1#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)


6500-2#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-2#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)

Check all other ports in the core network. Keep in mind that we terminate locally vlans 10 and 20, while we tunnel vlans 2001 and 2002.

! Connection between 6500-1 Gi2/3 and 7600-1 Gi3/1
6500-1#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10,2001,2002

7600-1#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10

! Connection between 7600-1 Gi3/2 and 7600-2 Gi3/2
7600-1#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

7600-2#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

! Connection between 7600-2 Gi3/1 and 6500-2 Gi2/3
7600-2#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20

6500-2#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20,2001,2002

Check MPLS/LDP functionality (no need to advertise labels for all prefixes)

7600-1#sh mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
Vlan30                 Yes (ldp)     No       No  No     Yes


7600-2#sh mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
Vlan30                 Yes (ldp)     No       No  No     Yes


7600-1#sh mpls ldp nei
    Peer LDP Ident: 30.30.30.2:0; Local LDP Ident 30.30.30.1:0
        TCP connection: 30.30.30.2.47252 - 30.30.30.1.646
        State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
        Up time: 2d18h
        LDP discovery sources:
          Vlan30, Src IP addr: 30.30.30.2
          Targeted Hello 30.30.30.1 -> 30.30.30.2, active, passive
        Addresses bound to peer LDP Ident:
          30.30.30.2


7600-2#sh mpls ldp nei
    Peer LDP Ident: 30.30.30.1:0; Local LDP Ident 30.30.30.2:0
        TCP connection: 30.30.30.1.646 - 30.30.30.2.47252
        State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
        Up time: 2d18h
        LDP discovery sources:
          Vlan30, Src IP addr: 30.30.30.1
          Targeted Hello 30.30.30.2 -> 30.30.30.1, active, passive
        Addresses bound to peer LDP Ident:
          30.30.30.1


7600-1#sh mpls ldp binding det
Advertisement spec:
        Prefix acl = MPLS-LABELS-ACL

  lib entry: 0.0.0.0/0, rev 12, chkpt: none
        local binding:  label: imp-null (owner LDP)
  lib entry: 30.30.30.0/24, rev 13, chkpt: none
        local binding:  label: imp-null (owner LDP)
          Advertised to:
          30.30.30.2:0
        remote binding: lsr: 30.30.30.2:0, label: imp-null
        Advert acl(s): Prefix acl MPLS-LABELS-ACL


 
7600-2#sh mpls ldp bindings det
Advertisement spec:
        Prefix acl = MPLS-LABELS-ACL

  lib entry: 0.0.0.0/0, rev 14, chkpt: none
        local binding:  label: imp-null (owner LDP)
  lib entry: 30.30.30.0/24, rev 15, chkpt: none
        local binding:  label: imp-null (owner LDP)
          Advertised to:
          30.30.30.1:0
        remote binding: lsr: 30.30.30.1:0, label: imp-null
        Advert acl(s): Prefix acl MPLS-LABELS-ACL
 

7600-1#sh mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     No Label      l2ckt(2001)       1710943694212 Gi3/1.2001  point2point
17     No Label      l2ckt(2002)       2053098230805 Gi3/1.2002  point2point


7600-2#sh mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     No Label      l2ckt(2001)       869456241933  Gi3/1.2001  point2point
17     No Label      l2ckt(2002)       1083085136320 Gi3/1.2002  point2point

Check EoMPLS L2 circuits

7600-1#sh mpls l2transport vc

Local intf     Local circuit              Dest address    VC ID      Status
-------------  -------------------------- --------------- ---------- ----------
Gi3/1.2001     Eth VLAN 2001              30.30.30.2      2001       UP
Gi3/1.2002     Eth VLAN 2002              30.30.30.2      2002       UP

7600-2#sh mpls l2transport vc

Local intf     Local circuit              Dest address    VC ID      Status
-------------  -------------------------- --------------- ---------- ----------
Gi3/1.2001     Eth VLAN 2001              30.30.30.1      2001       UP
Gi3/1.2002     Eth VLAN 2002              30.30.30.1      2002       UP


7600-1#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.2, VC ID: 2001, VC status: up
    Output interface: Vl30, imposed label stack {16}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.2
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.2:0 up
    Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
    MPLS VC labels: local 16, remote 16
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 1866599200, send 1608328927
    byte totals:   receive 1706612231959, send 857848421519
    packet drops:  receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.2, VC ID: 2002, VC status: up
    Output interface: Vl30, imposed label stack {17}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.2
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.2:0 up
    Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
    MPLS VC labels: local 17, remote 17
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 2345120706, send 2080672994
    byte totals:   receive 2109127676973, send 1102024312190
    packet drops:  receive 0, send 0


7600-2#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.1, VC ID: 2001, VC status: up
    Output interface: Vl30, imposed label stack {16}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.1
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.1:0 up
    Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
    MPLS VC labels: local 16, remote 16
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 1612416460, send 1870837231
    byte totals:   receive 860082881318, send 1710279266411
    packet drops:  receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.1, VC ID: 2002, VC status: up
    Output interface: Vl30, imposed label stack {17}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.1
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.1:0 up
    Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
    MPLS VC labels: local 17, remote 17
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 2085506054, send 2349856170
    byte totals:   receive 1104792851185, send 2112955222087
    packet drops:  receive 0, send 0


7600-1#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP     ac   Gi3/1.2001 2001(Eth VLAN)    UP mpls 30.30.30.2:2001              UP
UP     ac   Gi3/1.2002 2002(Eth VLAN)    UP mpls 30.30.30.2:2002              UP

7600-2#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP     ac   Gi3/1.2001 2001(Eth VLAN)    UP mpls 30.30.30.1:2001              UP
UP     ac   Gi3/1.2002 2002(Eth VLAN)    UP mpls 30.30.30.1:2002              UP

The term "Eth Vlan" means a VC type 4 is used.

Ports Gi2/1 & Gi2/2 on both the 6500s are using the 802.1q tunnel functionality.
Ports Gi3/1 on both the 7600s are using the MUX-UNI functionality, passing vlans 10,20 as L2 and vlans 2001,2002 as EoMPLS. These are the customer-facing or edge ports.
Ports Gi3/2 on both the 7600s are simple trunk ports allowing all L2 vlans. These are the core-facing ports.

This is what happens to a packet (frame) belonging to vlan 100 and moving from 6500-1 to 6500-2:

1) Packet enters 6500-1 through Gi2/1, having a single 802.1q header with vlan-id 100
2) A second 802.1q header with vlan-id 2001 is added by 6500-1
3) Packet exits 6500-1 through Gi2/3 and enters 7600-1 through Gi3/1, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100

4) A MPLS header with VC label 2001 is added by 7600-1
5) Packet exits 7600-1 through Gi3/2 and enters 7600-2 through Gi3/2, having a MPLS header with VC label 2001 followed by a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
6) The MPLS header with VC label 2001 is removed by 7600-2

7) Packet exits 7600-2 through Gi3/1 and enters 6500-2 through Gi2/3, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
8) The second 802.1q header with vlan-id 2001 is removed by 6500-2
9) Packet exits 6500-2 through Gi2/1, having a single 802.1q header with vlan-id 100


Notes:

1) The EoMPLS tunnel destination route in the routing table (and the CEF table) must be a /32 address to ensure that there is a label-switched path (LSP) from PE to PE. That's another reason why loopbacks are preferred. In the above test there is no need for actual routing on the EoMPLS path, due to directly connected PEs. Also, targeted LDP sessions between the PEs would be needed if the PEs weren't directly connected (since only a single LDP session is required between the peers, one is only created if not already present).

2) EoMPLS based on a PFC (used by LAN cards on 6500/7600) does not perform any L2 lookup to determine if the destination MAC address resides on the local or remote segment and also does not perform any L2 address learning (as traditional LAN bridging does).

3) All GE cards used were WS-X6724-SFP. SUP720s in 6500s were running 12.2(33)SXI1 and RSP720s in 7600s were running 12.2(33)SRD1.

4) Always keep an eye on the MTU. If you don't want to sit down and make arithmetic operations, just use a high MTU that can cover all possible cases. Generally your MTU should be higher than "1500 + Transport Header + AToM Header + (MPLS Labels x 4)", which is 1530 for a usual Vlan-based EoMPLS.

5) Numbers 2001 and 2002 used with the "xconnect" command define the VC id and they don't have to be the same as the vlans.

6) Using the command "sh mpls l2transport hw-capability interface x/x", you can find out what transport modes are supported for an interface, as well as what functionality is supported for core or edge ports.

7) In some 7600 IOS releases, there is bug in the output of "sh mpls l2transport binding" that shows the VC as Ethernet (Type 5), while it's Ether Vlan (Type 4). Just ignore it.

8) The command "no mpls ldp advertise-labels" must be configured together with "mpls ldp advertise-labels...", in order to make filtering of label advertisements work.

Should the crash be an option?

During the previous weekend i had a 7600/RSP720 crashing multiple times in the middle of the night without any apparent reason. Console was showing the following messages:

%Software-forced reload


03:31:06 EET Sat May 9 2009: Unexpected exception to CPU Vector 1500, PC = 0x0A7FA5       
       B0, LR = 0x0A7FA550

-Traceback= A7FA5B0 A7FA550 A558DD8 A40B698 A442928 A442994 8814E18 8815FB8 8816CD4       
      A874C8C A8755BC A82EFCC A82F1C0 A8306C0 A7F05EC A87A0DC

CPU Register Context:
MSR = 0x00029200  CR  = 0x24444024  CTR = 0x0A44DA0C  XER   = 0x00000000
R0  = 0x0A7FA550  R1  = 0x13D4C3E0  R2  = 0xFFFCFFFC  R3    = 0x137005B4
R4  = 0xFFFFFFFE  R5  = 0x00000000  R6  = 0x13D4C3B8  R7    = 0x00029200
R8  = 0x00029200  R9  = 0x00000000  R10 = 0x0D06DBF8  R11   = 0x137005B0
R12 = 0x00013FB4  R13 = 0x04044000  R14 = 0x00000000  R15   = 0x00000000
R16 = 0x0E844B94  R17 = 0x0E8663A8  R18 = 0x0C236BD4  R19   = 0x00000000
R20 = 0x13731364  R21 = 0x79D97210  R22 = 0x13D4C580  R23   = 0x0E851E4C
R24 = 0x0D050000  R25 = 0x00000000  R26 = 0x1373158C  R27   = 0x0000001F
R28 = 0x00000001  R29 = 0x0C420000  R30 = 0x103FEAA0  R31   = 0x00000000

Writing crashinfo to bootdisk:crashinfo_20090509-033106-EET

*** System received a Software forced crash ***
signal= 0x17, code= 0x1500, context= 0xce08b9c
PC = 0x8273dcc, Vector = 0x1500, SP = 0x1b394d08
rommon 2 >

Trying to boot manually from the rommon didn't have any positive result, because after the boot was completed (and you were given about 10-20 secs of CLI access!), the RSP crashed again. Since rommon doesn't provide a way to view crashinfo files, you're stuck with the crashes, trying your best at various guesses.

This was a crash that appeared at a random time and on a 7600 that gets configured once in a month, so by best guess was a faulty or stuck supervisor card. After the card was OIRed by the oncall engineer, it booted fine and i finally had a look at the crashinfos. The crashinfo on the RP didn't show anything useful. But the crashinfo on the SP showed the following:

%ONLINE-SP-6-INITFAIL: Module 6: Failed to synchronize Port asic

CCO describes it like below:

Description
The cause of the crash is that the Pinnacle ASIC failed to synchronize. This is usually caused by a bad contact or a badly seated card.

Workaround
The system recovers without user intervention. If the error message recurs, then reseat the concerned line card or module.

Yeah, sure. The system was stuck to rommon after some crashes:

Active crashed three times, disabling auto-boot and dropping to rommon

If anyone would like to answer, these are my questions:

Why couldn't the RSP print an error message (after 3 crashes) on the console while in IOS and sit there waiting for the user to act? What benefit does this specific crash and return to rommon offer?

I understand that generally a crash/reload is used to force the device recover from a bad situation and possibly avoid a long downturn situation. In my case, the situation was a bad contact or a badly seated card (although there is lot of arguing that can be done here). How was the reload supposed to solve this? Was the reload going to "reseat" the module somehow? Are there any pins moved while reloading? Does rommon offer a better seating?

After all, as it proved out the crash didn't help at all, because the RSP returned to ROM, sitting there and waiting, while it could stay in IOS, which provides much better feedback to the user.

On the other hand, it's the redundancy factor. Which, imho, is the only case that a crash/reload should occur (i'm still referring to the case of a badly seated module). As long as you have a 2nd working RSP, it should take the active role. But the system knows whether you have a redundant setup long before giving you the CLI prompt, so it should be easy to check upon it.

What i'm trying to say is that a crash followed by a reload is not always a panacea in case of a problem. It should be an option used, only when there is no other solution. In case of a badly seated module, crashing and moving into rommon isn't of any help (at least until rommon gets upgraded to something more intelligent).

"aaa accounting system guarantee-first" Pt2

Just an update on the "aaa accounting system guarantee-first" issue. I submitted a PER and as it turned out, BU & developers decided to treat it like a bug and fix it. But not completely.

Having to choose between the following 2 solutions, they chose the easy one; the first.

1) "Console/Aux user should be permitted if authentication is not configured with any AAA Server Method (radius/tacacs)".

2) "Console/Aux/VTY access should be permitted if authentication is not configured with the same AAA method (radius/tacacs) as system accounting".

As you can see the 1st solution is applicable only to con/aux, while the 2nd is applicable to con/aux/vty.
Also the 1st solution mixes tacacs and radius, like they are used for the same type of access, something that is not true for the majority of services (Cisco knows that better that anyone else).

Cisco expects me to be satisfied from the PER result, but i'm not. They had a chance of fixing something completely, but they decided to just patch it temporarily.

Cisco : "this behavior is expected" & Archive Config Logger

The last 2 months i've been busy playing with an ASR1000, finding bugs and "fighting" with the BUs.

Dealing with a lot of issues lately (not necessarily of ASR's), there are a few times that i have come across Cisco's motto "this behavior is expected". One of them was the fact that the very useful functionality of archive config logger (Configuration Logging) is ruined by the inability to filter out per-user configs applied through an aaa server.

The Configuration Change Notification and Logging (Configuration Logging) feature allows the tracking of configuration changes entered on a per-session and per-user basis by implementing a configuration log. The configuration log tracks each configuration command that is applied, who applied the command, the parser return code for the command, and the time the command was applied.

As you all know, a basic configuration like the following will help you keep track of changes in the configuration of a router:

archive
 log config
  logging enable
  hidekeys

In a "normal" router you'll probably see something like the following:

router#sh archive log config all
 idx   sess           user@line      Logged command
    8     3         user@vty0     | no ipv6 nd suppress-ra
    9     3         user@vty0     | ipv6 ospf 100 area 10
   10     4         user@vty0     |no ip sla monitor 99
   11     5         user@vty0     |interface GigabitEthernet0/2.35211201
   12     5         user@vty0     | encapsulation dot1Q 3521 second-dot1q 1201
   13     5         user@vty0     | ip address 10.10.10.129 255.255.255.252

If you happen to have a router that is acting like a bras, accepting ppp sessions and using radius attributes in order to apply per-user ACLs and/or routes, then you'll find out that the above feature is totally useless, because all these per-user entries are stored into the archive logging buffer, overwriting the actual configuration changes you want to watch (when the configuration log is full, the oldest log entry will be removed every time a new entry is added).

router#sh archive log config all
 idx   sess           user@line      Logged command
  177     0        console@console  |permit ip host 1.1.1.1 any 
  178     0        console@console  |deny ip any any 
  179     0        console@console  |deny ip any any 
  180     0        console@console  |ip route 2.2.2.0 255.255.255.240 3.3.3.3 
  181     0        console@console  |ip route 2.2.2.0 255.255.255.240 3.3.3.3 
  182     0        console@console  |ip access-list extended Virtual-Access2.10785#106240381
  183     0        console@console  |deny ip any 10.0.0.0 0.255.255.255
  184     0        console@console  |deny ip any 10.0.0.0 0.255.255.255
  185     0        console@console  |deny ip any 192.168.0.0 0.0.255.255
  186     0        console@console  |deny ip any 192.168.0.0 0.0.255.255
  187     0        console@console  |deny ip any 224.0.0.0 15.255.255.255
  188     0        console@console  |deny ip any 224.0.0.0 15.255.255.255

Note : Please don't ask me about the double entries...

According to Cisco's cli help, only changes to the running config, not the one stored on nvram, are to be logged.

router(config-archive)#log ?
  config  Logging changes to the running configuration

If you issue the "show run" command (which is supposed to display the running configuration), you won't see these per-user config changes. That's because all these per-user configs, after being downloaded from the aaa server, have a flag set that they should not be nvgen'ed. So they are not shown.

In the Cisco IOS software configuration model, the configuration state is maintained in a distributed manner, with each component storing its own configuration state. To retrieve configuration information, the software must poll every component to collect the distributed information. This configuration state retrieval operation is performed by a process known as nonvolatile generation (NVGEN), and it is used by command line interpreter (CLI) commands such as show running-configuration, write memory, and copy system:running-configuration to display or copy the running system configuration. When invoked, NVGEN queries each system component and each instance of interface or other configuration objects. A running configuration file is constructed as NVGEN traverses the system performing these queries.

Also, according to Cisco:

Restrictions for Configuration Change Notification and Logging
•Only complete commands input in a configuration mode are logged.
•Commands that are part of a configuration file applied with the copy command are not logged.

The Configuration Change Notification and Logging feature tracks changes made to the Cisco IOS software running configuration by maintaining a configuration log. This configuration log tracks changes initiated only through the command-line interface (CLI) or HTTP.

Why Cisco chose to include these per-user "dynamic" config changes into the archive logger, making it another thrown-away feature, is still a mystery to me.

There is also the possibility that this kind of continuous logging might be causing a little increase in cpu usage too.

Yep, time for another Cisco-should-have-fixed-it-in-the-first-place PER.

Convert Cisco dumps into Wireshark's pcap format

Lately i've been experimenting a lot with the hidden "dump" option of "debug ip packet", trying to decode various protocols. Yesterday i wrote a post about decoding the tacacs communication, something that required a little bit of awkward text editing.

Tonight i created the following perl program in order to make the whole process easier.

#!/opt/perl/bin/perl

# ciscodump2text v0.1
#
# Convert Cisco hex dump format (captured through the "debug ip packet dump" command)
# to a special text format that can then be fed into text2pcap 
# so a pcap file for Wireshark can be created at the end
#
# Copyright (C) 2009 Tassos (http://ccie-in-3-months.blogspot.com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

@packets = ();
$junk_line = 1;
$i = 0;

while (<>) {
        $line = $_;

        if ( $line =~ /^[0-9A-F]{8}: / ) {

                $hex_part = substr($line, 10, 36);
                $hex_part =~ s/\s//g;

                if ( $junk_line ) {
                        $i++;
                        $packets[$i] = "";
                        $junk_line = 0;
                };

                $packets[$i] .= $hex_part;
        } else {
                $junk_line = 1;
        }
}

for ($i = 1; $i <= @packets; $i++) {

        if ( exists $packets[$i] ) {

                for ( $j = 0; $j < length($packets[$i]); $j += 2 ) {
                        if ( $j == 0 )  {
                                printf "# Packet $i\n%08X", $j/2;
                        } elsif ( $j % 32 == 0 ) {
                                printf " #\n%08X", $j/2;
                        }
                        print " ".substr($packets[$i], $j, 2);
                }

                print " #\n";
        }
}

print "\n";

You just give it the cisco dump as input (only lines starting with an 8-char hex number are processed, so you don't have to worry about other lines) and it produces a text file that can be fed into text2pcap. Then you give text2pcap the new text file as input and it produces a pcap file as output, which can be opened with Wireshark.

i.e. we want to convert the following log produced by "debug ip packet dump" to a text file compatible with the "text2pcap" program.

*Apr 18 18:19:51.887: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:19:51.887: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:19:51.887: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 80, sending
*Apr 18 18:19:51.887:     TCP src=20596, dst=49, seq=1187787226, ack=3815757335, win=4128 ACK
082D17D0:                   45000050 678B0000          E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._
*Apr 18 18:19:51.887: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 80, sending full packet
*Apr 18 18:19:51.887:     TCP src=20596, dst=49, seq=1187787226, ack=3815757335, win=4128 ACK
082D17C0:                       0200 4C4F4F50            ..LOOP
082D17D0: CA000CB8 00060800 45000050 678B0000  J..8....E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._
*Apr 18 18:19:52.103: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:19:52.107: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:19:52.107: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending
*Apr 18 18:19:52.111:     TCP src=20596, dst=49, seq=1187787266, ack=3815757363, win=4100 ACK
080014D0:                   45000028 678C0000          E..(g...
080014E0: FF062C1D 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
080014F0: 46CC3202 E36FD633 50101004 F4920000  FL2.coV3P...t...
08001500:
*Apr 18 18:19:52.127: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending full packet
*Apr 18 18:19:52.131:     TCP src=20596, dst=49, seq=1187787266, ack=3815757363, win=4100 ACK
080014C0:                       0200 4C4F4F50            ..LOOP
080014D0: CA000CB8 00060800 45000028 678C0000  J..8....E..(g...
080014E0: FF062C1D 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
080014F0: 46CC3202 E36FD633 50101004 F4920000  FL2.coV3P...t...
08001500:

First we save it in a ascii/text file (i.e. "test.dump") and we give it as input to the "ciscodump2text" program.

tassos$ ciscodump2text test.dump > test.text

The created "test.text" file contains the following :

# Packet 1
00000000 45 00 00 50 67 8B 00 00 FF 06 2B F6 0A 0A 0A 09 #
00000010 0A 0A 0A 0A 50 74 00 31 46 CC 31 DA E3 6F D6 17 #
00000020 50 10 10 20 9F 96 00 00 C0 01 01 00 99 64 8E 6A #
00000030 00 00 00 1C B8 14 5E C8 A8 B3 C2 B9 3E C1 2A 1F #
00000040 AA 40 DE 66 D7 42 9C 89 0B F9 11 F3 C7 24 9F 5F #
# Packet 2
00000000 02 00 4C 4F 4F 50 CA 00 0C B8 00 06 08 00 45 00 #
00000010 00 50 67 8B 00 00 FF 06 2B F6 0A 0A 0A 09 0A 0A #
00000020 0A 0A 50 74 00 31 46 CC 31 DA E3 6F D6 17 50 10 #
00000030 10 20 9F 96 00 00 C0 01 01 00 99 64 8E 6A 00 00 #
00000040 00 1C B8 14 5E C8 A8 B3 C2 B9 3E C1 2A 1F AA 40 #
00000050 DE 66 D7 42 9C 89 0B F9 11 F3 C7 24 9F 5F #
# Packet 3
00000000 45 00 00 28 67 8C 00 00 FF 06 2C 1D 0A 0A 0A 09 #
00000010 0A 0A 0A 0A 50 74 00 31 46 CC 32 02 E3 6F D6 33 #
00000020 50 10 10 04 F4 92 00 00 #
# Packet 4
00000000 02 00 4C 4F 4F 50 CA 00 0C B8 00 06 08 00 45 00 #
00000010 00 28 67 8C 00 00 FF 06 2C 1D 0A 0A 0A 09 0A 0A #
00000020 0A 0A 50 74 00 31 46 CC 32 02 E3 6F D6 33 50 10 #
00000030 10 04 F4 92 00 00 #

Then, we convert this new text file into a pcap file by using the "text2pcap" program, which is included with Wireshark.

tassos$ text2pcap.exe" -d test.text test.pcap
Input from: test.text
Output to: test.pcap
Start new packet
Start new packet
Wrote packet of 80 bytes at 0
Start new packet
Wrote packet of 94 bytes at 80
Start new packet
Wrote packet of 40 bytes at 174
Wrote packet of 54 bytes at 214

-------------------------
Read 4 potential packets, wrote 4 packets

The resulting pcap file "test.pcap" can be opened for further processing with Wireshark.

That way you can very easily create pcap files of almost everything happening on your router.

Notes : The "dump" option is supported since 12.0 IOS. Latest IOS includes EPC, which makes the whole capture & convert-to-pcap process much easier.

PS : I'm not a very good perl programmer (although i have written a lot of custom perl scripts for my job), so someone playing with perl for years will probably produce a more compact "perlish" code. I just tried to interpret my logic into perl code.