Solving routing asymmetry due to HSRP by using EEM & EOT

This is a topic that comes up at least once in a year at the c-nsp list (and everywhere else i believe), but i still cannot understand why Cisco doesn't provide a solution about it.

1) Introducing the problem

The scenario is simple. You have 2 routers running HSRP on a LAN serving various hosts. These 2 routers have a third -common- router as an uplink and all three of them are running a routing protocol. Since HSRP enabled interfaces are always (regardless of their state) included in the connected routes table, the uplink router will always know 2 paths to reach the hosts. You can change the metrics of connected/static routes when redistributing them into the routing protocol so you can give preference only to the active HSRP router's path, but this is something static. If the HSRP changes state, the routes will continue to follow the initial path (which leads to routing asymmetry...or is it asymmetric routing?), unless somehow you change the routes too.
Asymmetric routing can cause communication problems when the packets pass through different "stateful inspection enabled" (Reflexive ACLs, CBAC, older Firewalls, etc.) paths, or unicast flood problems when the mac (on L2 devices) and arp (on L3 devices) timers do not match.

2) Finding solutions

The previous week i decided to experiment with EEM (Embedded Event Manager) & EOT (Enhanced Object Tracking) in order to change dynamically the paths of HSRP related routes. This is more a hack than a solution. But it should work fine for most simple cases.

There are mainly 2 kinds of routes that need to be changed : connected routes and static routes. Connected routes are the most difficult to handle, so the only easy way i could come up with was to change their redistribution metric by changing the relevant configuration under the routing protocol (you can also use prefix-lists/route-maps with redistribution, if you want to be more selective). On the other hand, static routes can depend on the state of a tracked object, so they are much easier to handle.

Note : Although Cisco has the IP Event Dampening code ready (which removes connected/static routes from dampened interfaces), it doesn't seem they are really interested in using it for removing a connected route from an interface when it's in the HSRP standby state. I guess a good reason is that IP Event Dampening doesn't support subinterfaces and it also blocks HSRP, so it probably needs to be redesigned.

These are the steps needed to accomplish our solution:

2.1) Creating tracked objects for static routes

In latest IOS (where EEM 2.2 is supported), EOT is integrated with EEM so you can have EEM events based on tracked object states ("event track X state Y") or EEM actions for manipulating tracked object states ("action X track set/read Y"). Also a new type of tracking object—a stub object—is created. The stub object can be manipulated easily using the existing CLI commands that already allow tracked objects to be manipulated. That way we can change its state through EEM.

First we have to make sure that all our static routes that point to HSRP interfaces depend on tracked objects. We're using the stub-object of EOT in order to accomplish this.

track 1 stub-object
 default-state up
!
ip route 4.4.4.4 255.255.255.255 10.10.10.3 track 1

R1#sh track
Track 1
  Stub-object
  State is Up
    2 changes, last change 01:38:36, by Undefined
  Tracked by:
    STATIC-IP-ROUTING 0

2.2) Detecting HSRP states using EEM

Then we're using EEM in order to detect HSRP state changes on both routers; one event for the Active state and another event for every other state. There are probably other and more elegant ways, but i have tested only the following two:

Checking syslog logs for HSRP messages:

event manager applet HSRP-SYSLOG-EVENT-UP
 event syslog pattern "FastEthernet0/0 Grp 1 state .* -> Active"
!
event manager applet HSRP-SYSLOG-EVENT-DOWN
 event syslog pattern "FastEthernet0/0 Grp 1 state Active -> .*"

Checking HSRP states through snmp

event manager applet HSRP-SNMP-EVENT-UP
 event snmp oid "1.3.6.1.4.1.9.9.106.1.2.1.1.15.1.1" get-type exact entry-op eq entry-val "6" entry-type value exit-op ne exit-val "6" exit-type value poll-interval 1
event manager applet HSRP-SNMP-EVENT-DOWN
 event snmp oid "1.3.6.1.4.1.9.9.106.1.2.1.1.15.1.1" get-type exact entry-op ne entry-val "6" entry-type value exit-op eq exit-val "6" exit-type value poll-interval 1

"1.3.6.1.4.1.9.9.106.1.2.1.1.15.1.1" is an OID containing the following 3 parts:

1.3.6.1.4.1.9.9.106.1.2.1.1.15 = cHsrpGrpStandbyState
1 = interface index (you can easily find it by using "show snmp mib ifmib ifindex" or snmpwalk)
1 = HSRP group number (the one you use as "standby X" under the interface)

The result returned by snmp is a number representing the current HSRP state of this group on this interface:

6 = Active
5 = Standby
4 = Speak
3 = Listen
2 = Learn
1 = Init

Don't forget to enable the snmp agent ("snmp-server enable traps X") and the snmp ifindex persistence ("snmp-server ifindex persist"), if you're going to use the snmp method.

Note : In my example, i'll be using the syslog based detection, because it's a lot easier to watch/debug. Nevertheless, i believe the snmp based detection is much better (although maybe harder to implement), because it doesn't rely on logging (which for various reasons could be disabled/rate-limited; try using severity level 4 for your logs!) and it can "fix" the routing behavior at any time (polling every 1") vs the syslog which "fixes" the routing behavior only once (while log is caught at a specific time).


2.3) Acting on HSRP state changes using EEM

At the end we have to make some actions based on our events. For static routes we just change the tracked object's state. For connected routes we have to change their redistribution metric under the routing protocol.

Changing the state of the tracked object:

event manager applet HSRP-SYSLOG-EVENT-UP
 action 2.1 track set 1 state up
 action 2.3 track read 1

event manager applet HSRP-SYSLOG-EVENT-DOWN
 action 2.1 track set 1 state down
 action 2.3 track read 1

Changing the redistribution metric of connected routes

event manager applet HSRP-SYSLOG-EVENT-UP
 action 3.1 cli command "enable"
 action 3.3 cli command "conf t"
 action 3.5 cli command "router ospf 100"
 action 3.7 cli command "redistribute connected metric 100 subnets"

event manager applet HSRP-SYSLOG-EVENT-DOWN
 action 3.1 cli command "enable"
 action 3.3 cli command "conf t"
 action 3.5 cli command "router ospf 100"
 action 3.7 cli command "redistribute connected metric 200 subnets"

3) Testing the solutions

3.1) The test network

Ok, time to try it on a test network.



For the CORE->HOST direction, we want to have (on the CORE router) the 4.4.4.4/32 route (L0 of HOST) to point always to the active HSRP router (R1 or R2). The HOST->CORE direction is handled by HSRP, so we don't need to worry about it.

We're using OSPF as our routing protocol, but i suppose it could be anything else.

3.2) The router configurations

These are the interesting configuration parts of all participant routers:

CORE

ip host HOST-F0-0 10.10.10.3
ip host R2-S1-0 23.23.23.2
ip host R1-S1-0 13.13.13.1
!
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
!
interface Serial1/1
 description ** R1 **
 ip address 13.13.13.3 255.255.255.0
!
interface Serial1/2
 description ** R2 **
 ip address 23.23.23.3 255.255.255.0
!
router ospf 100
 passive-interface Loopback0
 network 3.3.3.3 0.0.0.0 area 0
 network 13.13.13.0 0.0.0.255 area 0
 network 23.23.23.0 0.0.0.255 area 0
!

R1

track 1 stub-object
 default-state up
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
interface FastEthernet0/0
 ip address 10.10.10.1 255.255.255.0
 standby version 2
 standby 1 ip 10.10.10.50
 standby 1 priority 101
 standby 1 preempt
!
interface Serial1/0
 description ** CORE **
 ip address 13.13.13.1 255.255.255.0
!
router ospf 100
 redistribute connected metric 100 subnets
 redistribute static subnets
 passive-interface Loopback0
 network 1.1.1.1 0.0.0.0 area 0
 network 13.13.13.0 0.0.0.255 area 0
!
ip route 4.4.4.4 255.255.255.255 10.10.10.3 track 1
!
event manager applet HSRP-SYSLOG-EVENT-UP
 event syslog pattern "FastEthernet0/0 Grp 1 state .* -> Active"
 action 1.1 syslog msg "HSRP : Active"
 action 2.1 track set 1 state up
 action 2.3 track read 1
 action 3.1 cli command "enable"
 action 3.3 cli command "conf t"
 action 3.5 cli command "router ospf 100"
 action 3.7 cli command "redistribute connected metric 100 subnets"
!
event manager applet HSRP-SYSLOG-EVENT-DOWN
 event syslog pattern "FastEthernet0/0 Grp 1 state Active -> .*"
 action 1.1 syslog msg "HSRP : NOT Active"
 action 2.1 track set 1 state down
 action 2.3 track read 1
 action 3.1 cli command "enable"
 action 3.3 cli command "conf t"
 action 3.5 cli command "router ospf 100"
 action 3.7 cli command "redistribute connected metric 200 subnets"
!

R2

track 1 stub-object
 default-state down ! this is the default and not shown
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
interface FastEthernet0/0
 ip address 10.10.10.2 255.255.255.0
 standby version 2
 standby 1 ip 10.10.10.50
 standby 1 priority 99
 standby 1 preempt
!
interface Serial1/0
 description ** CORE **
 ip address 23.23.23.2 255.255.255.0
!
router ospf 100
 redistribute connected metric 200 subnets
 redistribute static subnets
 passive-interface Loopback0
 network 2.2.2.2 0.0.0.0 area 0
 network 23.23.23.0 0.0.0.255 area 0
!
ip route 4.4.4.4 255.255.255.255 10.10.10.3 track 1
!
event manager applet HSRP-SYSLOG-EVENT-UP
 event syslog pattern "FastEthernet0/0 Grp 1 state .* -> Active"
 action 1.1 syslog msg "HSRP : Active"
 action 2.1 track set 1 state up
 action 2.3 track read 1
 action 3.1 cli command "enable"
 action 3.3 cli command "conf t"
 action 3.5 cli command "router ospf 100"
 action 3.7 cli command "redistribute connected metric 100 subnets"
!
event manager applet HSRP-SYSLOG-EVENT-DOWN
 event syslog pattern "FastEthernet0/0 Grp 1 state Active -> .*"
 action 1.1 syslog msg "HSRP : NOT Active"
 action 2.1 track set 1 state down
 action 2.3 track read 1
 action 3.1 cli command "enable"
 action 3.3 cli command "conf t"
 action 3.5 cli command "router ospf 100"
 action 3.7 cli command "redistribute connected metric 200 subnets"

HOST

ip host R1-F0-0 10.10.10.1
ip host R2-F0-0 10.10.10.2
ip host CORE-S1-1 13.13.13.3
ip host CORE-S1-2 23.23.23.3
!
interface Loopback0
 ip address 4.4.4.4 255.255.255.255
!
interface FastEthernet0/0
 ip address 10.10.10.3 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 10.10.10.50

As you can see, the EEM configurations of R1 & R2 are the same. The only thing that's different is the initial state of the tracked object and the initial redistribution metric.

3.3) R1 Active - R2 Standby

By default, R1 is in the Active HSRP state and R2 is in the Standby state.

R1#sh standby br
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    101 P Active  local           10.10.10.2      10.10.10.50


R2#sh standby br
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    99  P Standby 10.10.10.1      local           10.10.10.50

The 4.4.4.4/32 static route is active only on R1, due to our tracked object's state :

R1#sh ip route track-table
 ip route 4.4.4.4 255.255.255.255 10.10.10.3 track 1 state is [up]

R1#sh track 1
Track 1
  Stub-object
  State is Up
    6 changes, last change 00:00:54, by EEM
  Tracked by:
    STATIC-IP-ROUTING 0

R1#sh ip route static
     4.0.0.0/32 is subnetted, 1 subnets
S       4.4.4.4 [1/0] via 10.10.10.3


R2#sh ip route track-table
 ip route 4.4.4.4 255.255.255.255 10.10.10.3 track 1 state is [down]

R2#sh track 1
Track 1
  Stub-object
  State is Down
    5 changes, last change 00:00:57, by EEM
  Tracked by:
    STATIC-IP-ROUTING 0

R2#sh ip route static

The 10.10.10.0/24 connected route is inserted twice (by R1 and R2) into the ospf database, but only the one with the lowest metric (from R1) is used by the CORE's RIB.

CORE#sh ip ospf data ext 10.10.10.0 | i Advert|Metric:
  Advertising Router: 1.1.1.1
        Metric: 100
  Advertising Router: 2.2.2.2
        Metric: 200

Traffic between CORE and HOST should (according to our initial setup) pass through R1 in both directions. CORE is using OSPF to decide, HOST is using static routing + HSRP.

CORE#sh ip route 4.4.4.4
Routing entry for 4.4.4.4/32
  Known via "ospf 100", distance 110, metric 20, type extern 2, forward metric 64
  Last update from 13.13.13.1 on Serial1/1, 02:28:05 ago
  Routing Descriptor Blocks:
  * 13.13.13.1, from 1.1.1.1, 02:28:05 ago, via Serial1/1
      Route metric is 20, traffic share count is 1

CORE#sh ip route 10.10.10.3
Routing entry for 10.10.10.0/24
  Known via "ospf 100", distance 110, metric 100, type extern 2, forward metric 64
  Last update from 13.13.13.1 on Serial1/1, 02:28:25 ago
  Routing Descriptor Blocks:
  * 13.13.13.1, from 1.1.1.1, 02:28:25 ago, via Serial1/1
      Route metric is 100, traffic share count is 1

CORE => HOST

CORE#tr 4.4.4.4 so l0

Type escape sequence to abort.
Tracing the route to 4.4.4.4

  1 R1-S1-0 (13.13.13.1) 32 msec 36 msec 8 msec
  2 HOST-F0-0 (10.10.10.3) 44 msec *  64 msec
CORE#

HOST => CORE

HOST#tr 3.3.3.3 so l0

Type escape sequence to abort.
Tracing the route to 3.3.3.3

  1 R1-F0-0 (10.10.10.1) 48 msec 60 msec 12 msec
  2 CORE-S1-1 (13.13.13.3) 56 msec *  76 msec
HOST#

3.4) R1 Standby - R2 Active

Now let's make the R2 Active by increasing its HSRP priority (we have preempt enabled). As you can see from the logs below, EEM and tracking worked fine.

R2

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#interface FastEthernet0/0
R2(config-if)#standby 1 priority 103
R2(config-if)#^Z
R2#

*Nov 22 15:00:46.239: %HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Standby -> Active
*Nov 22 15:00:46.407: %SYS-5-CONFIG_I: Configured from console by console
*Nov 22 15:00:46.447: %HA_EM-6-LOG: HSRP-SYSLOG-EVENT-UP: HSRP : Active
*Nov 22 15:00:46.447: %TRACKING-5-STATE: 1    Down->Up
*Nov 22 15:00:47.167: %SYS-5-CONFIG_I: Configured from console by vty0

R1

*Nov 22 15:00:45.875: %HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Active -> Speak
*Nov 22 15:00:46.055: %HA_EM-6-LOG: HSRP-SYSLOG-EVENT-DOWN: HSRP : NOT Active
*Nov 22 15:00:46.063: %TRACKING-5-STATE: 1    Up->Down
*Nov 22 15:00:46.607: %SYS-5-CONFIG_I: Configured from console by vty0
*Nov 22 15:00:57.839: %HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Speak -> Standby

Now R2 is in the Active HSRP state and R1 is in the Standby state.

R1#sh standby br
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    101 P Standby 10.10.10.2      local           10.10.10.50


R2#sh standby br
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/0       1    103 P Active  local           10.10.10.1      10.10.10.50

The 4.4.4.4/32 static route is active only on R2, due to our tracked object's new state :

R1#sh ip route track-table
 ip route 4.4.4.4 255.255.255.255 10.10.10.3 track 1 state is [down]

R1#sh track 1
Track 1
  Stub-object
  State is Down
    3 changes, last change 00:00:06, by EEM
  Tracked by:
    STATIC-IP-ROUTING 0

R1#sh ip route static


R2#sh ip route track-table
 ip route 4.4.4.4 255.255.255.255 10.10.10.3 track 1 state is [up]

R2#sh track 1
Track 1
  Stub-object
  State is Up
    2 changes, last change 00:00:06, by EEM
  Tracked by:
    STATIC-IP-ROUTING 0

R2#sh ip route static
     4.0.0.0/32 is subnetted, 1 subnets
S       4.4.4.4 [1/0] via 10.10.10.3

The 10.10.10.0/24 connected route is again inserted twice (by R1 and R2) into the ospf database, but this time the one from R2 is used by the CORE's RIB.

CORE#sh ip ospf data ext 10.10.10.0 | i Advert|Metric:
  Advertising Router: 1.1.1.1
        Metric: 200
  Advertising Router: 2.2.2.2
        Metric: 100

This time, traffic should pass through R2 in both directions.

CORE#sh ip route 4.4.4.4
Routing entry for 4.4.4.4/32
  Known via "ospf 100", distance 110, metric 20, type extern 2, forward metric 64
  Last update from 23.23.23.2 on Serial1/2, 00:02:09 ago
  Routing Descriptor Blocks:
  * 23.23.23.2, from 2.2.2.2, 00:02:09 ago, via Serial1/2
      Route metric is 20, traffic share count is 1

CORE#sh ip route 10.10.10.3
Routing entry for 10.10.10.0/24
  Known via "ospf 100", distance 110, metric 100, type extern 2, forward metric 64
  Last update from 23.23.23.2 on Serial1/2, 00:02:19 ago
  Routing Descriptor Blocks:
  * 23.23.23.2, from 2.2.2.2, 00:02:19 ago, via Serial1/2
      Route metric is 100, traffic share count is 1

CORE => HOST

CORE#tr 4.4.4.4 so l0

Type escape sequence to abort.
Tracing the route to 4.4.4.4

  1 R2-S1-0 (23.23.23.2) 36 msec 64 msec 20 msec
  2 HOST-F0-0 (10.10.10.3) 44 msec *  44 msec
CORE#

HOST => CORE

HOST#tr 3.3.3.3 so l0

Type escape sequence to abort.
Tracing the route to 3.3.3.3

  1 R2-F0-0 (10.10.10.2) 80 msec 28 msec 12 msec
  2 CORE-S1-2 (23.23.23.3) 60 msec *  84 msec
HOST#

3.5) R1 Active - R2 Standby (again)

Ok, let's return to the initial state again (R1 Active, R2 Standby).

R2

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#interface FastEthernet0/0
R2(config-if)#standby 1 priority 99
R2(config-if)#^Z
R2#
*Nov 22 16:10:40.538: %SYS-5-CONFIG_I: Configured from console by console
*Nov 22 16:10:41.070: %HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Active -> Speak
*Nov 22 16:10:41.282: %HA_EM-6-LOG: HSRP-SYSLOG-EVENT-DOWN: HSRP : NOT Active
*Nov 22 16:10:41.282: %TRACKING-5-STATE: 1    Up->Down
*Nov 22 16:10:41.810: %SYS-5-CONFIG_I: Configured from console by vty0
*Nov 22 16:10:51.918: %HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Speak -> Standby

R1

*Nov 22 16:10:40.602: %HSRP-5-STATECHANGE: FastEthernet0/0 Grp 1 state Standby -> Active
*Nov 22 16:10:40.754: %HA_EM-6-LOG: HSRP-SYSLOG-EVENT-UP: HSRP : Active
*Nov 22 16:10:40.754: %TRACKING-5-STATE: 1    Down->Up
*Nov 22 16:10:41.270: %SYS-5-CONFIG_I: Configured from console by vty0

Traffic in both directions should go through R1 again.

CORE => HOST

CORE#tr 4.4.4.4 so l0

Type escape sequence to abort.
Tracing the route to 4.4.4.4

  1 R1-S1-0 (13.13.13.1) 24 msec 72 msec 12 msec
  2 HOST-F0-0 (10.10.10.3) 44 msec *  80 msec
CORE#

HOST => CORE

HOST#tr 3.3.3.3 so l0

Type escape sequence to abort.
Tracing the route to 3.3.3.3

  1 R1-F0-0 (10.10.10.1) 40 msec 32 msec 20 msec
  2 CORE-S1-1 (13.13.13.3) 32 msec *  96 msec
HOST#

4) Comments

4.1) If you're making some kind of load-sharing using MHSRP or you're using many HSRP groups on many interfaces, then the whole thing gets much more complicated because you end up having a lot of extra EEM configuration. And you don't want to begin troubleshooting it!

4.2) Hosts directly connected to either router (R1 or R2) will always prefer the local connected route, so you cannot avoid the asymmetry there.

4.3) You might get into confusion when EEM changes the configuration. I don't know if adding a "wr mem" as the last action command would help in such a situation (although i don't want to imagine what would happen if there were many consecutive HSRP state changes).

4.4) EEM 2.2 is needed for support of Enhanced Object Tracking. According to CCO, it should be supported in 12.4(2)T, 12.2(31)SB3, 12.2(33)SRB and above. My test was done on 7200s running 12.4(20)T.

4.5) In IOS 12.4(11)T and above, BFD comes with support for HSRP. According to CCO this is particular useful in architectures where a single interface hosts a large number of HSRP groups, due to BFD's low CPU memory consumption and processing overhead. If you want to use it, just enable BFD ("bfd interval ...") under the HSRP interfaces and you're done.

I hope Cisco will soon provide the "suppress-connected-route-on-standby-interfaces" functionality, as other vendors already do for their FHRPs. It's a so much needed feature. Also the HSRP state detections can be made much easier if Cisco implements an EEM event detector that watches the state of a redundancy group. After all, isn't that the reason "standby X name XXX" got implemented in the first place?

PS: Can someone tell me why Cisco insists on using the "standby" keyword instead of "hsrp"?