Configuring 802.1q tunnels over Vlan-based EoMPLS & MUX-UNI

One of the main disadvantages (besides all the usual disputes about L2 vs L3) of 802.1q tunneling is the fact that all the switches the tunnel is passing through, are learning all the MAC addresses passing through it, plus the consumption of other system resources like vlans and spanning-tree instances. So if you have many 802.1q tunnels passing through one or more central switches as bridged L2 traffic, you might end up filling their TCAM space. For point-2-point 802.1q tunnels, an easy solution is to use EoMPLS tunnels on the central routers/switches in order to allow the double-tagged 802.1q traffic to pass over them as transparently as possible.

The following is our test network.



We need to transfer L2 traffic belonging to vlans 100-110 and 200-210 across some switches/routers and at the same time we need to pass IP traffic belonging to vlans 10 and 20 from the 6500s to the 7600s and vice-versa. All of the above using a single link between 6500s and 7600s.

EoMPLS uses IP/MPLS as its underlying technology, so it really doesn't matter what you have between the EoMPLS endpoints, as long as you have IP/MPLS connectivity between them (the ingress and egress PEs are the only two routers in the MPLS backbone with knowledge of the Layer 2 VCs, so only these require EoMPLS functionality). If you want to avoid MPLS and use solely IP, then you you can create a tunnel based on L2TPv3 and transfer Ethernet frames over it.

EoMPLS is one of the AToM (Any Transport over MPLS) transport types. AToM transports Layer 2 packets over a IP/MPLS backbone using a directed LDP session between edge routers for setting up and maintaining connections. Normally forwarding occurs through the use of two level labels that provide switching between the edge routers. The Tunnel label (external) routes the packet over the MPLS backbone to the egress PE at the ingress PE. The VC label is a demuxing label that determines the connection at the tunnel endpoint (the particular egress interface on the egress PE as well as the vlan-id for an ethernet frame). If PE routers are connected back-2-back (like in this example), then there is no need for a tunnel label (the "implicit null" label is used to signal that to the peer).

EoMPLS works by encapsulating ethernet PDUs in MPLS packets and forwarding them across the MPLS network. Each PDU is transported as a single packet. It can work in 2 modes:

1) Port-based, where everything that comes into a port gets tunneled (using a VC type of 5)
2) Vlan-based, where only specific vlans get tunneled (using a VC type of either 4 or 5)

Here i'm using the vlan-based mode, because i want to combine it with MUX-UNI. MUX-UNI is a nice feature on 6500s and 7600s, that provides the ability to partition a physical port in such a way that multiple Layer 2 and Layer 3 services can be created over a single UNI (usually the point where CE connects to the ingress PE). Is works very well on the 6500/7600 LAN cards (i.e. 67xx), but if you are thinking of more advanced things you probably need the ES/ES+ ones (i will make a new post about various ES+ setups soon).

Before configuring EoMPLS, you have to make sure that IP routing in enabled so that the PE routers (in our case the 7600s) can reach each other and MPLS is enabled so that a label switched path (LSP) exists between the PE routers. No need for complexity here, so we use just Vlan 30 in both 7600s for this. CEF is needed too, but it's enabled by default and cannot be disabled on 6500s/7600s.

Configuration

These are the relevant configurations:

6500-1

interface GigabitEthernet2/1
 description ** vlans 100-110 **
 switchport
 switchport access vlan 2001
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
 description ** vlans 200-210 **
 switchport
 switchport access vlan 2002
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface Vlan10
 description ** test **
 ip address 10.10.10.1 255.255.255.0
!
interface GigabitEthernet2/3
 description ** 7600-1 Gi3/1 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10,2001,2002
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable

7600-1

interface GigabitEthernet3/1
 description ** 6500-1 G2/3 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
 encapsulation dot1Q 2001
 xconnect 30.30.30.2 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
 encapsulation dot1Q 2002
 xconnect 30.30.30.2 2002 encapsulation mpls
!
interface GigabitEthernet3/2
 description ** 7600-2 Gi3/2 **
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 mtu 9216
!
interface Vlan10
 description ** test **
 ip address 10.10.10.2 255.255.255.0
!
interface Vlan30
 description ** MPLS **
 ip address 30.30.30.1 255.255.255.0
 mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
 permit 30.30.30.0 0.0.0.255

7600-2

interface GigabitEthernet3/1
 description ** 6500-2 G2/3 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 20
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet3/1.2001
 encapsulation dot1Q 2001
 xconnect 30.30.30.1 2001 encapsulation mpls
!
interface GigabitEthernet3/1.2002
 encapsulation dot1Q 2002
 xconnect 30.30.30.1 2002 encapsulation mpls
!
interface GigabitEthernet3/2
 description ** 7600-1 Gi3/2 **
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 mtu 9216
!
interface Vlan20
 description ** test **
 ip address 20.20.20.2 255.255.255.0
!
interface Vlan30
 description ** MPLS **
 ip address 30.30.30.2 255.255.255.0
 mpls ip
!
mpls ldp holdtime 30
no mpls ldp advertise-labels
mpls ldp advertise-labels for MPLS-LABELS-ACL
mpls label protocol ldp
mpls ldp router-id Vlan30 force
!
ip access-list standard MPLS-LABELS-ACL
 permit 30.30.30.0 0.0.0.255

6500-2

interface GigabitEthernet2/1
 description ** vlans 100-110 **
 switchport
 switchport access vlan 2001
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/2
 description ** vlans 200-210 **
 switchport
 switchport access vlan 2002
 switchport mode dot1q-tunnel
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface Vlan20
 description ** test **
 ip address 20.20.20.1 255.255.255.0
!
interface GigabitEthernet2/3
 description ** 7600-2 Gi3/1 **
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 20,2001,2002
 switchport mode trunk
 switchport nonegotiate
 mtu 9216
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable

Verification

Check the 802.1q tunnel ports on the edges of the network

6500-1#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-1#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)


6500-2#sh int gi2/1 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2001 (VLAN2001)

6500-2#sh int gi2/2 switchport | i Administrative Mode|Access Mode
Administrative Mode: tunnel
Access Mode VLAN: 2002 (VLAN2002)

Check all other ports in the core network. Keep in mind that we terminate locally vlans 10 and 20, while we tunnel vlans 2001 and 2002.

! Connection between 6500-1 Gi2/3 and 7600-1 Gi3/1
6500-1#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10,2001,2002

7600-1#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 10

! Connection between 7600-1 Gi3/2 and 7600-2 Gi3/2
7600-1#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

7600-2#sh int gi3/2 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: ALL

! Connection between 7600-2 Gi3/1 and 6500-2 Gi2/3
7600-2#sh int gi3/1 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20

6500-2#sh int gi2/3 switchport | i Administrative Mode|Trunking VLANs
Administrative Mode: trunk
Trunking VLANs Enabled: 20,2001,2002

Check MPLS/LDP functionality (no need to advertise labels for all prefixes)

7600-1#sh mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
Vlan30                 Yes (ldp)     No       No  No     Yes


7600-2#sh mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
Vlan30                 Yes (ldp)     No       No  No     Yes


7600-1#sh mpls ldp nei
    Peer LDP Ident: 30.30.30.2:0; Local LDP Ident 30.30.30.1:0
        TCP connection: 30.30.30.2.47252 - 30.30.30.1.646
        State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
        Up time: 2d18h
        LDP discovery sources:
          Vlan30, Src IP addr: 30.30.30.2
          Targeted Hello 30.30.30.1 -> 30.30.30.2, active, passive
        Addresses bound to peer LDP Ident:
          30.30.30.2


7600-2#sh mpls ldp nei
    Peer LDP Ident: 30.30.30.1:0; Local LDP Ident 30.30.30.2:0
        TCP connection: 30.30.30.1.646 - 30.30.30.2.47252
        State: Oper; Msgs sent/rcvd: 27555/27555; Downstream
        Up time: 2d18h
        LDP discovery sources:
          Vlan30, Src IP addr: 30.30.30.1
          Targeted Hello 30.30.30.2 -> 30.30.30.1, active, passive
        Addresses bound to peer LDP Ident:
          30.30.30.1


7600-1#sh mpls ldp binding det
Advertisement spec:
        Prefix acl = MPLS-LABELS-ACL

  lib entry: 0.0.0.0/0, rev 12, chkpt: none
        local binding:  label: imp-null (owner LDP)
  lib entry: 30.30.30.0/24, rev 13, chkpt: none
        local binding:  label: imp-null (owner LDP)
          Advertised to:
          30.30.30.2:0
        remote binding: lsr: 30.30.30.2:0, label: imp-null
        Advert acl(s): Prefix acl MPLS-LABELS-ACL


 
7600-2#sh mpls ldp bindings det
Advertisement spec:
        Prefix acl = MPLS-LABELS-ACL

  lib entry: 0.0.0.0/0, rev 14, chkpt: none
        local binding:  label: imp-null (owner LDP)
  lib entry: 30.30.30.0/24, rev 15, chkpt: none
        local binding:  label: imp-null (owner LDP)
          Advertised to:
          30.30.30.1:0
        remote binding: lsr: 30.30.30.1:0, label: imp-null
        Advert acl(s): Prefix acl MPLS-LABELS-ACL
 

7600-1#sh mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     No Label      l2ckt(2001)       1710943694212 Gi3/1.2001  point2point
17     No Label      l2ckt(2002)       2053098230805 Gi3/1.2002  point2point


7600-2#sh mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     No Label      l2ckt(2001)       869456241933  Gi3/1.2001  point2point
17     No Label      l2ckt(2002)       1083085136320 Gi3/1.2002  point2point

Check EoMPLS L2 circuits

7600-1#sh mpls l2transport vc

Local intf     Local circuit              Dest address    VC ID      Status
-------------  -------------------------- --------------- ---------- ----------
Gi3/1.2001     Eth VLAN 2001              30.30.30.2      2001       UP
Gi3/1.2002     Eth VLAN 2002              30.30.30.2      2002       UP

7600-2#sh mpls l2transport vc

Local intf     Local circuit              Dest address    VC ID      Status
-------------  -------------------------- --------------- ---------- ----------
Gi3/1.2001     Eth VLAN 2001              30.30.30.1      2001       UP
Gi3/1.2002     Eth VLAN 2002              30.30.30.1      2002       UP


7600-1#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.2, VC ID: 2001, VC status: up
    Output interface: Vl30, imposed label stack {16}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.2
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.2:0 up
    Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
    MPLS VC labels: local 16, remote 16
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 1866599200, send 1608328927
    byte totals:   receive 1706612231959, send 857848421519
    packet drops:  receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.2, VC ID: 2002, VC status: up
    Output interface: Vl30, imposed label stack {17}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.2
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.2:0 up
    Targeted Hello: 30.30.30.1(LDP Id) -> 30.30.30.2
    MPLS VC labels: local 17, remote 17
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 2345120706, send 2080672994
    byte totals:   receive 2109127676973, send 1102024312190
    packet drops:  receive 0, send 0


7600-2#sh mpls l2transport vc det
Local interface: Gi3/1.2001 up, line protocol up, Eth VLAN 2001 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.1, VC ID: 2001, VC status: up
    Output interface: Vl30, imposed label stack {16}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.1
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.1:0 up
    Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
    MPLS VC labels: local 16, remote 16
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 1612416460, send 1870837231
    byte totals:   receive 860082881318, send 1710279266411
    packet drops:  receive 0, send 0

Local interface: Gi3/1.2002 up, line protocol up, Eth VLAN 2002 up
  MPLS VC type is Eth VLAN, interworking type is Ethernet
  Destination address: 30.30.30.1, VC ID: 2002, VC status: up
    Output interface: Vl30, imposed label stack {17}
    Preferred path: not configured
    Default path: active
    Next hop: 30.30.30.1
  Create time: 2d23h, last status change time: 2d19h
  Signaling protocol: LDP, peer 30.30.30.1:0 up
    Targeted Hello: 30.30.30.2(LDP Id) -> 30.30.30.1
    MPLS VC labels: local 17, remote 17
    Group ID: local 0, remote 0
    MTU: local 9216, remote 9216
    Remote interface description:
  Sequencing: receive disabled, send disabled
  VC statistics:
    packet totals: receive 2085506054, send 2349856170
    byte totals:   receive 1104792851185, send 2112955222087
    packet drops:  receive 0, send 0


7600-1#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP     ac   Gi3/1.2001 2001(Eth VLAN)    UP mpls 30.30.30.2:2001              UP
UP     ac   Gi3/1.2002 2002(Eth VLAN)    UP mpls 30.30.30.2:2002              UP

7600-2#sh xconnect all
Legend: XC ST=Xconnect State, S1=Segment1 State, S2=Segment2 State
UP=Up, DN=Down, AD=Admin Down, IA=Inactive, NH=No Hardware
XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP     ac   Gi3/1.2001 2001(Eth VLAN)    UP mpls 30.30.30.1:2001              UP
UP     ac   Gi3/1.2002 2002(Eth VLAN)    UP mpls 30.30.30.1:2002              UP

The term "Eth Vlan" means a VC type 4 is used.

Ports Gi2/1 & Gi2/2 on both the 6500s are using the 802.1q tunnel functionality.
Ports Gi3/1 on both the 7600s are using the MUX-UNI functionality, passing vlans 10,20 as L2 and vlans 2001,2002 as EoMPLS. These are the customer-facing or edge ports.
Ports Gi3/2 on both the 7600s are simple trunk ports allowing all L2 vlans. These are the core-facing ports.

This is what happens to a packet (frame) belonging to vlan 100 and moving from 6500-1 to 6500-2:

1) Packet enters 6500-1 through Gi2/1, having a single 802.1q header with vlan-id 100
2) A second 802.1q header with vlan-id 2001 is added by 6500-1
3) Packet exits 6500-1 through Gi2/3 and enters 7600-1 through Gi3/1, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100

4) A MPLS header with VC label 2001 is added by 7600-1
5) Packet exits 7600-1 through Gi3/2 and enters 7600-2 through Gi3/2, having a MPLS header with VC label 2001 followed by a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
6) The MPLS header with VC label 2001 is removed by 7600-2

7) Packet exits 7600-2 through Gi3/1 and enters 6500-2 through Gi2/3, having a double 802.1q header with outer vlan-id 2001 and inner vlan-id 100
8) The second 802.1q header with vlan-id 2001 is removed by 6500-2
9) Packet exits 6500-2 through Gi2/1, having a single 802.1q header with vlan-id 100


Notes:

1) The EoMPLS tunnel destination route in the routing table (and the CEF table) must be a /32 address to ensure that there is a label-switched path (LSP) from PE to PE. That's another reason why loopbacks are preferred. In the above test there is no need for actual routing on the EoMPLS path, due to directly connected PEs. Also, targeted LDP sessions between the PEs would be needed if the PEs weren't directly connected (since only a single LDP session is required between the peers, one is only created if not already present).

2) EoMPLS based on a PFC (used by LAN cards on 6500/7600) does not perform any L2 lookup to determine if the destination MAC address resides on the local or remote segment and also does not perform any L2 address learning (as traditional LAN bridging does).

3) All GE cards used were WS-X6724-SFP. SUP720s in 6500s were running 12.2(33)SXI1 and RSP720s in 7600s were running 12.2(33)SRD1.

4) Always keep an eye on the MTU. If you don't want to sit down and make arithmetic operations, just use a high MTU that can cover all possible cases. Generally your MTU should be higher than "1500 + Transport Header + AToM Header + (MPLS Labels x 4)", which is 1530 for a usual Vlan-based EoMPLS.

5) Numbers 2001 and 2002 used with the "xconnect" command define the VC id and they don't have to be the same as the vlans.

6) Using the command "sh mpls l2transport hw-capability interface x/x", you can find out what transport modes are supported for an interface, as well as what functionality is supported for core or edge ports.

7) In some 7600 IOS releases, there is bug in the output of "sh mpls l2transport binding" that shows the VC as Ethernet (Type 5), while it's Ether Vlan (Type 4). Just ignore it.

8) The command "no mpls ldp advertise-labels" must be configured together with "mpls ldp advertise-labels...", in order to make filtering of label advertisements work.