Saturday, May 23, 2009

Configuring 802.1q tunnels over Vlan-based EoMPLS & MUX-UNI

One of the main disadvantages (besides all the usual disputes about L2 vs L3) of 802.1q tunneling is the fact that all the switches the tunnel is passing through, are learning all the MAC addresses passing through it, plus the consumption of other system resources like vlans and spanning-tree instances. So if you have many 802.1q tunnels passing through one or more central switches as bridged L2 traffic, you might end up filling their TCAM space. For point-2-point 802.1q tunnels, an easy solution is to use EoMPLS tunnels on the central routers/switches in order to allow the double-tagged 802.1q traffic to pass over them as transparently as possible.

The following is our test network.



We need to transfer L2 traffic belonging to vlans 100-110 and 200-210 across some switches/routers and at the same time we need to pass IP traffic belonging to vlans 10 and 20 from the 6500s to the 7600s and vice-versa. All of the above using a single link between 6500s and 7600s.

EoMPLS uses IP/MPLS as its underlying technology, so it really doesn't matter what you have between the EoMPLS endpoints, as long as you have IP/MPLS connectivity between them (the ingress and egress PEs are the only two routers in the MPLS backbone with knowledge of the Layer 2 VCs, so only these require EoMPLS functionality). If you want to avoid MPLS and use solely IP, then you you can create a tunnel based on L2TPv3 and transfer Ethernet frames over it.

EoMPLS is one of the AToM (Any Transport over MPLS) transport types. AToM transports Layer 2 packets over a IP/MPLS backbone using a directed LDP session between edge routers for setting up and maintaining connections. Normally forwarding occurs through the use of two level labels that provide switching between the edge routers. The Tunnel label (external) routes the packet over the MPLS backbone to the egress PE at the ingress PE. The VC label is a demuxing label that determines the connection at the tunnel endpoint (the particular egress interface on the egress PE as well as the vlan-id for an ethernet frame). If PE routers are connected back-2-back (like in this example), then there is no need for a tunnel label (the "implicit null" label is used to signal that to the peer).

EoMPLS works by encapsulating ethernet PDUs in MPLS packets and forwarding them across the MPLS network. Each PDU is transported as a single packet. It can work in 2 modes:

1) Port-based, where everything that comes into a port gets tunneled (using a VC type of 5)
2) Vlan-based, where only specific vlans get tunneled (using a VC type of either 4 or 5)

Here i'm using the vlan-based mode, because i want to combine it with MUX-UNI. MUX-UNI is a nice feature on 6500s and 7600s, that provides the ability to partition a physical port in such a way that multiple Layer 2 and Layer 3 services can be created over a single UNI (usually the point where CE connects to the ingress PE). Is works very well on the 6500/7600 LAN cards (i.e. 67xx), but if you are thinking of more advanced things you probably need the ES/ES+ ones (i will make a new post about various ES+ setups soon).

Before configuring EoMPLS, you have to make sure that IP routing in enabled so that the PE routers (in our case the 7600s) can reach each other and MPLS is enabled so that a label switched path (LSP) exists between the PE routers. No need for complexity here, so we use just Vlan 30 in both 7600s for this. CEF is needed too, but it's enabled by default and cannot be disabled on 6500s/7600s.

Configuration

These are the relevant configurations:

6500-1


interface GigabitEthernet2/1
description ** vlans 100-110 **
switchport
switchport access vlan 2001
switchport mode dot1q-tunnel
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
description ** vlans 200-210 **
switchport
switchport access vlan 2002
switchport mode dot1q-tunnel
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface Vlan10
description ** test **
ip address 10.10.10.1 255.255.255.0
!
interface GigabitEthernet2/3
description ** 7600-1 Gi3/1 **
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,2001,2002
switchport mode trunk
switchport nonegotiate
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable


7600-1

interface GigabitEthernet3/1
description ** 6500-1 G2/3 **
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10
switchport mode trunk
switchport nonegotiate
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
encapsulation dot1Q 2001
xconnect 30.30.30.2 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
encapsulation dot1Q 2002
xconnect 30.30.30.2 2002 encapsulation mpls
!
interface GigabitEthernet3/2
description ** 7600-2 Gi3/2 **
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
mtu 9216
!
interface Vlan10
description ** test **
ip address 10.10.10.2 255.255.255.0
!
interface Vlan30
description ** MPLS **
ip address 30.30.30.1 255.255.255.0
mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
permit 30.30.30.0 0.0.0.255


7600-2

interface GigabitEthernet3/1
description ** 6500-2 G2/3 **
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20
switchport mode trunk
switchport nonegotiate
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
encapsulation dot1Q 2001
xconnect 30.30.30.1 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
encapsulation dot1Q 2002
xconnect 30.30.30.1 2002 encapsulation mpls
!
interface GigabitEthernet3/2
description ** 7600-1 Gi3/2 **
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
mtu 9216
!
interface Vlan20
description ** test **
ip address 20.20.20.2 255.255.255.0
!
interface Vlan30
description ** MPLS **
ip address 30.30.30.2 255.255.255.0
mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
permit 30.30.30.0 0.0.0.255


6500-2

interface GigabitEthernet2/1
description ** vlans 100-110 **
switchport
switchport access vlan 2001
switchport mode dot1q-tunnel
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
description ** vlans 200-210 **
switchport
switchport access vlan 2002
switchport mode dot1q-tunnel
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable
!
interface Vlan20
description ** test **
ip address 20.20.20.1 255.255.255.0
!
interface GigabitEthernet2/3
description ** 7600-2 Gi3/1 **
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20,2001,2002
switchport mode trunk
switchport nonegotiate
mtu 9216
spanning-tree portfast trunk
spanning-tree bpdufilter enable


Verification

Check the 802.1q tunnel ports on the edges of the network


6500-1#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-1#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)


6500-2#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-2#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)


Check all other ports in the core network. Keep in mind that we terminate locally vlans 10 and 20, while we tunnel vlans 2001 and 2002.


! Connection between 6500-1 Gi2/3 and 7600-1 Gi3/1
6500-1#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10,2001,2002

7600-1#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10

! Connection between 7600-1 Gi3/2 and 7600-2 Gi3/2
7600-1#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

7600-2#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

! Connection between 7600-2 Gi3/1 and 6500-2 Gi2/3
7600-2#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20

6500-2#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20,2001,2002


Check MPLS/LDP functionality (no need to advertise labels for all prefixes)


7600-1#sh mpls interfaces
Interface IP Tunnel BGP Static Operational
Vlan30 Yes (ldp) No No No Yes


7600-2#sh mpls interfaces
Interface IP Tunnel BGP Static Operational
Vlan30 Yes (ldp) No No No Yes


7600-1#sh mpls ldp nei
Peer LDP Ident: 30.30.30.2:0; Local LDP Ident 30.30.30.1:0
TCP connection: 30.30.30.2.47252 - 30.30.30.1.646
State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
Up time: 2d18h
LDP discovery sources:
Vlan30, Src IP addr: 30.30.30.2
Targeted Hello 30.30.30.1 -> 30.30.30.2, active, passive
Addresses bound to peer LDP Ident:
30.30.30.2


7600-2#sh mpls ldp nei
Peer LDP Ident: 30.30.30.1:0; Local LDP Ident 30.30.30.2:0
TCP connection: 30.30.30.1.646 - 30.30.30.2.47252
State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
Up time: 2d18h
LDP discovery sources:
Vlan30, Src IP addr: 30.30.30.1
Targeted Hello 30.30.30.2 -> 30.30.30.1, active, passive
Addresses bound to peer LDP Ident:
30.30.30.1


7600-1#sh mpls ldp binding det
Advertisement spec:
Prefix acl = MPLS-LABELS-ACL

lib entry: 0.0.0.0/0, rev 12, chkpt: none
local binding: label: imp-null (owner LDP)
lib entry: 30.30.30.0/24, rev 13, chkpt: none
local binding: label: imp-null (owner LDP)
Advertised to:
30.30.30.2:0
remote binding: lsr: 30.30.30.2:0, label: imp-null
Advert acl(s): Prefix acl MPLS-LABELS-ACL



7600-2#sh mpls ldp bindings det
Advertisement spec:
Prefix acl = MPLS-LABELS-ACL

lib entry: 0.0.0.0/0, rev 14, chkpt: none
local binding: label: imp-null (owner LDP)
lib entry: 30.30.30.0/24, rev 15, chkpt: none
local binding: label: imp-null (owner LDP)
Advertised to:
30.30.30.1:0
remote binding: lsr: 30.30.30.1:0, label: imp-null
Advert acl(s): Prefix acl MPLS-LABELS-ACL


7600-1#sh mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
16 No Label l2ckt(2001) 1710943694212 Gi3/1.2001 point2point
17 No Label l2ckt(2002) 2053098230805 Gi3/1.2002 point2point


7600-2#sh mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
16 No Label l2ckt(2001) 869456241933 Gi3/1.2001 point2point
17 No Label l2ckt(2002) 1083085136320 Gi3/1.2002 point2point



Check EoMPLS L2 circuits


7600-1#sh mpls l2transport vc

Local intf Local circuit Dest address VC ID Status
------------- -------------------------- --------------- ---------- ----------
Gi3/1.2001 Eth VLAN 2001 30.30.30.2 2001 UP
Gi3/1.2002 Eth VLAN 2002 30.30.30.2 2002 UP

7600-2#sh mpls l2transport vc

Local intf Local circuit Dest address VC ID Status
------------- -------------------------- --------------- ---------- ----------
Gi3/1.2001 Eth VLAN 2001 30.30.30.1 2001 UP
Gi3/1.2002 Eth VLAN 2002 30.30.30.1 2002 UP


7600-1#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
MPLS VC type is Eth VLAN, interworking type is Ethernet
Destination address: 30.30.30.2, VC ID: 2001, VC status: up
Output interface: Vl30, imposed label stack {16}
Preferred path: not configured
Default path: active
Next hop: 30.30.30.2
Create time: 2d23h, last status change time: 2d19h
Signaling protocol: LDP, peer 30.30.30.2:0 up
Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
MPLS VC labels: local 16, remote 16
Group ID: local 0, remote 0
MTU: local 9216, remote 9216
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 1866599200, send 1608328927
byte totals: receive 1706612231959, send 857848421519
packet drops: receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
MPLS VC type is Eth VLAN, interworking type is Ethernet
Destination address: 30.30.30.2, VC ID: 2002, VC status: up
Output interface: Vl30, imposed label stack {17}
Preferred path: not configured
Default path: active
Next hop: 30.30.30.2
Create time: 2d23h, last status change time: 2d19h
Signaling protocol: LDP, peer 30.30.30.2:0 up
Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
MPLS VC labels: local 17, remote 17
Group ID: local 0, remote 0
MTU: local 9216, remote 9216
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 2345120706, send 2080672994
byte totals: receive 2109127676973, send 1102024312190
packet drops: receive 0, send 0


7600-2#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
MPLS VC type is Eth VLAN, interworking type is Ethernet
Destination address: 30.30.30.1, VC ID: 2001, VC status: up
Output interface: Vl30, imposed label stack {16}
Preferred path: not configured
Default path: active
Next hop: 30.30.30.1
Create time: 2d23h, last status change time: 2d19h
Signaling protocol: LDP, peer 30.30.30.1:0 up
Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
MPLS VC labels: local 16, remote 16
Group ID: local 0, remote 0
MTU: local 9216, remote 9216
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 1612416460, send 1870837231
byte totals: receive 860082881318, send 1710279266411
packet drops: receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
MPLS VC type is Eth VLAN, interworking type is Ethernet
Destination address: 30.30.30.1, VC ID: 2002, VC status: up
Output interface: Vl30, imposed label stack {17}
Preferred path: not configured
Default path: active
Next hop: 30.30.30.1
Create time: 2d23h, last status change time: 2d19h
Signaling protocol: LDP, peer 30.30.30.1:0 up
Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
MPLS VC labels: local 17, remote 17
Group ID: local 0, remote 0
MTU: local 9216, remote 9216
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 2085506054, send 2349856170
byte totals: receive 1104792851185, send 2112955222087
packet drops: receive 0, send 0


7600-1#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST Segment 1 S1 Segment 2 S2
------+---------------------------------+--+---------------------------------+--
UP ac Gi3/1.2001 2001(Eth VLAN) UP mpls 30.30.30.2:2001 UP
UP ac Gi3/1.2002 2002(Eth VLAN) UP mpls 30.30.30.2:2002 UP

7600-2#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST Segment 1 S1 Segment 2 S2
------+---------------------------------+--+---------------------------------+--
UP ac Gi3/1.2001 2001(Eth VLAN) UP mpls 30.30.30.1:2001 UP
UP ac Gi3/1.2002 2002(Eth VLAN) UP mpls 30.30.30.1:2002 UP


The term "Eth Vlan" means a VC type 4 is used.

Ports Gi2/1 & Gi2/2 on both the 6500s are using the 802.1q tunnel functionality.
Ports Gi3/1 on both the 7600s are using the MUX-UNI functionality, passing vlans 10,20 as L2 and vlans 2001,2002 as EoMPLS. These are the customer-facing or edge ports.
Ports Gi3/2 on both the 7600s are simple trunk ports allowing all L2 vlans. These are the core-facing ports.

This is what happens to a packet (frame) belonging to vlan 100 and moving from 6500-1 to 6500-2:

1) Packet enters 6500-1 through Gi2/1, having a single 802.1q header with vlan-id 100
2) A second 802.1q header with vlan-id 2001 is added by 6500-1
3) Packet exits 6500-1 through Gi2/3 and enters 7600-1 through Gi3/1, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100

4) A MPLS header with VC label 2001 is added by 7600-1
5) Packet exits 7600-1 through Gi3/2 and enters 7600-2 through Gi3/2, having a MPLS header with VC label 2001 followed by a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
6) The MPLS header with VC label 2001 is removed by 7600-2

7) Packet exits 7600-2 through Gi3/1 and enters 6500-2 through Gi2/3, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
8) The second 802.1q header with vlan-id 2001 is removed by 6500-2
9) Packet exits 6500-2 through Gi2/1, having a single 802.1q header with vlan-id 100


Notes:

1) The EoMPLS tunnel destination route in the routing table (and the CEF table) must be a /32 address to ensure that there is a label-switched path (LSP) from PE to PE. That's another reason why loopbacks are preferred. In the above test there is no need for actual routing on the EoMPLS path, due to directly connected PEs. Also, targeted LDP sessions between the PEs would be needed if the PEs weren't directly connected (since only a single LDP session is required between the peers, one is only created if not already present).

2) EoMPLS based on a PFC (used by LAN cards on 6500/7600) does not perform any L2 lookup to determine if the destination MAC address resides on the local or remote segment and also does not perform any L2 address learning (as traditional LAN bridging does).

3) All GE cards used were WS-X6724-SFP. SUP720s in 6500s were running 12.2(33)SXI1 and RSP720s in 7600s were running 12.2(33)SRD1.

4) Always keep an eye on the MTU. If you don't want to sit down and make arithmetic operations, just use a high MTU that can cover all possible cases. Generally your MTU should be higher than "1500 + Transport Header + AToM Header + (MPLS Labels x 4)", which is 1530 for a usual Vlan-based EoMPLS.

5) Numbers 2001 and 2002 used with the "xconnect" command define the VC id and they don't have to be the same as the vlans.

6) Using the command "sh mpls l2transport hw-capability interface x/x", you can find out what transport modes are supported for an interface, as well as what functionality is supported for core or edge ports.

7) In some 7600 IOS releases, there is bug in the output of "sh mpls l2transport binding" that shows the VC as Ethernet (Type 5), while it's Ether Vlan (Type 4). Just ignore it.

8) The command "no mpls ldp advertise-labels" must be configured together with "mpls ldp advertise-labels...", in order to make filtering of label advertisements work.

7 comments:

  1. Hi!

    I'm wondering how to provide EoMPLS on Cisco Catalyst 6500
    switch.

    You are writing about MUX-UNI feature but current Cisco
    IOS software, wchich my platform is already running, that is
    12.2(18)SXD6 doesn't support EoMPLS in VLAN mode.

    I think I should upgrade the software to the latest 12.2SX release,
    taht is 12.2(33)SXH.

    I'm also wondering if upgrading the softare will be enaugh
    to run EoMPLS.

    I mean if need to put in the
    Switch some additional stuff like Supervisor Engine to make
    EoMPLS possible.

    Please meke me sure to understand differnces between two EoMPLS modes.


    Does Port-based EoMPLS supports VLAN traffic to pass through it,
    causu You write that everything that comes is tunneld?

    I'm trying to plan MPLS in the core of campus network.

    Maybe for the first comment it will be enaugh, cause I have many questions
    about that stuff.

    I'm at the beginning of my journey to computer networking.

    I'm grateful for Your answers and open for comments and ideas.

    Best Regards,
    Agata Czekalska
    Technical University of Lodz

    ReplyDelete
  2. If you have SUP720, then it's better to upgrade to the latest SXI. SXD supports port-based EoMPLS only, so for the MUX-UNI functionality you need at least the SXH release.

    In port-based EoMPLS, everything that comes into a port of a switch, goes out of a port on another switch.
    In vlan-based EoMPLS, only specific vlans are being forwarded from one point to another point.

    ReplyDelete
  3. Hello
    I notice that on your port configurations e.g on the 7600-1 the port is both a do1q trunk and also has subinterfaces at the same time is this only supported on specific hardware ?
    I currently using 7600 ES+ 20xGE SFP but it gives an error when trying to create subinterfaces on a port configured as a switchport.

    Any Advice on this ?

    ReplyDelete
  4. This is called MUX-UNI and it's supported on LAN cards. On ES+ you should use the EVC infrastructure, which provides much better flexibility.

    ReplyDelete
  5. Hi, great article, but did you make a typo with your VC identifiers?

    xconnect 30.30.30.1 2001 encapsulation mpls

    I think you meant to say 3001 there? You talk about VC id 3001 in the packet (frame) walkthrough at the end, and in the diagram.

    Many thanks once again.

    ReplyDelete
  6. You're right; there was a typo on the diagram and on the packet walk-through (2001 is the correct VC; i prefer to use common vc and vlan numbers).
    Now it must be ok.

    ReplyDelete
  7. hi - I know its an old post but came across yr blog as I am looking into something similar - I need to pass l2tpv3 over an ISP link that is using QnQ over an MPLS core as its backbone

    ReplyDelete

 
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Greece License.