EEM 3.0 – Variable Logic Pt #2

Continuing from my previous "EEM 3.0 Variable Logic Part #1" article, here is Part #2. You'll notice that although EEM begun as a customized approach to event detection and recovery, it has turned into a very powerful framework that not only has the ability to monitor events and act accordingly, but it can also take complex informational or corrective actions when the monitored events occur. Of course there is still TCL that can be used for all kinds of scripting magic, but it's nice to have something completely IOS-configured.

1) Checking traceroute hops automatically
Below you'll find an applet that checks the output of a traceroute everyday at 17:07 and if a specific hop is found, then the ospf process is cleared.

Actions used : "set", "if/else/end", "regexp".

event manager applet CHECK-TRACEROUTE-HOPS-APPLET
 ! the applet is run daily at 17:07 with a max execution time of 60"
 ! cron-entry uses the known unix syntax
 event timer cron name Daily-17:07 cron-entry "07 17 * * *" maxrun 60
 action 1.1   set trace_dest "3.3.3.3"
 action 1.3   set hop_found "23.23.23.3"
 action 1.5   syslog msg "Searching for hop $hop_found in traceroute to $trace_dest."
 action 2.1   cli command "trace $trace_dest" pattern "#"
 ! we match a regexp of a pattern (which includes the hop we're looking for) on the traceroute result
 action 2.3   regexp "(.*) (23\.23\.23\.3) (.*)" "$_cli_result" _match _sub1 _sub2
 ! we check the regexp result (1=found, 0=not found)
 action 2.5   if $_regexp_result eq 1
 action 2.5.1  syslog msg "Hop $hop_found found in traceroute to $trace_dest."
 action 2.5.3  cli command "enable"
 action 2.5.5  cli command "clear ip ospf process" pattern "[no]"
 action 2.5.7  cli command "yes" pattern "#"
 action 2.5.9  syslog msg "OSPF process cleared."
 action 2.7   else
 action 2.7.1  syslog msg "Hop $hop_found not found in traceroute to $trace_dest."
 action 2.9   end

You can use something like your provider's web server for traceroute destination. If traceroute takes too long to complete, you can increase the applet execution time (default is 20") like below:

 event timer cron name Daily-17:07 cron-entry "07 17 * * *" maxrun 120

When the applet is executed automatically, the following is displayed on screen (having "term mon" enabled) :

Feb 21 17:07:00.031: %HA_EM-6-LOG: CHECK-TRACEROUTE-HOPS-APPLET: Searching for hop 23.23.23.3 in traceroute to 3.3.3.3.
Feb 21 17:07:03.303: %HA_EM-6-LOG: CHECK-TRACEROUTE-HOPS-APPLET: Hop 23.23.23.3 found in traceroute to 3.3.3.3.
Feb 21 17:07:03: %OSPF-5-ADJCHG: Process 100, Nbr 2.2.2.2 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
Feb 21 17:07:03.847: %HA_EM-6-LOG: CHECK-TRACEROUTE-HOPS-APPLET: OSPF process cleared.
Feb 21 17:07:04: %OSPF-5-ADJCHG: Process 100, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done

2) Checking traceroute hops automatically until a specific hop is not found
You can also change the cli commands executed; i.e. in a broadband connection, you can have the dialer interface cleared, until you get connected (or not connected) to a specific router.

Actions used : "set", "decrement", "wait", "while/end", "if/else/end", "regexp".

event manager applet CHECK-DIALER-TRACEROUTE-HOPS-APPLET
 event timer cron name Daily-21:40 cron-entry "40 21 * * *" maxrun 60
 action 1.1 set trace_dest "3.3.3.3"
 action 1.3 set hop_to_find "23.23.23.3"
 action 1.9 syslog msg "Searching for hop $hop_to_find in traceroute to $trace_dest."
 ! we do a traceroute 3 times only, in order to avoid having the link continuously flapping
 action 2.1 set trace_tries "3"
 action 2.9 while $trace_tries gt 0
 action 3.1  cli command "trace $trace_dest" pattern "#"
 action 3.3  regexp "(.*) (23\.23\.23\.3) (.*)" "$_cli_result"
 action 4.1  if $_regexp_result eq 1
 action 4.3   syslog msg "Hop $hop_to_find found in traceroute to $trace_dest."
 action 4.4   cli command "enable"
 action 4.5   cli command "clear int dialer 1"
 action 4.6   syslog msg "Interface dialer 1 cleared."
 action 4.7   decrement trace_tries 1
 action 4.8   syslog msg "$trace_tries tries left."
 action 4.9   wait 10
 action 5.1  else
 action 5.3   syslog msg "Hop $hop_to_find not found in traceroute to $trace_dest."
 action 5.5   syslog msg "Keeping connection."
 action 5.7   set trace_tries "0"
 action 5.9  end
 action 6.1 end

When the applet is executed automatically, the following is displayed on screen :

As long as (max 3 tries) the router we connect is the one we want to avoid, we drop the connection.

Feb 21 21:46:00.042: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Searching for hop 23.23.23.3 in traceroute to 3.3.3.3.
R1#
Feb 21 21:46:03.338: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Hop 23.23.23.3 found in traceroute to 3.3.3.3.
Feb 21 21:46:03.410: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Interface dialer 1 cleared.
Feb 21 21:46:03.414: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: 2 tries left.
R1#
Feb 21 21:46:16.774: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Hop 23.23.23.3 found in traceroute to 3.3.3.3.
Feb 21 21:46:16.822: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Interface dialer 1 cleared.
Feb 21 21:46:16.834: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: 1 tries left.
R1#
Feb 21 21:46:30.098: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Hop 23.23.23.3 found in traceroute to 3.3.3.3.
Feb 21 21:46:30.142: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Interface dialer 1 cleared.
Feb 21 21:46:30.154: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: 0 tries left.

When we get connected to a router different from the one we want to avoid, we keep the connection.

Feb 21 21:48:00.042: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Searching for hop 23.23.23.3 in traceroute to 3.3.3.3.
Feb 21 21:48:03.318: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Hop 23.23.23.3 not found in traceroute to 3.3.3.3.
Feb 21 21:48:03.322: %HA_EM-6-LOG: CHECK-DIALER-TRACEROUTE-HOPS-APPLET: Keeping connection.

3) Checking traceroute hops manually providing arguments
If you want to modify the initial traceroute applet, so you can run it manually (passing the ip addresses as arguments), you can use the following code. I have added a little bit of the new cli-interactivity too, by using the new "puts" action.

Actions used : "set", "if/else/end", "regexp", "puts".

event manager applet MANUAL-CHECK-TRACEROUTE-HOPS-APPLET
 ! we make the applet synchronous so we can get the output on our local tty
 event none sync yes maxrun 60
 ! we check the number of arguments
 action 1.1 if $_none_argc eq 2
 ! we use the 1st argument as the traceroute destination address
 action 2.1  set trace_dest "$_none_arg1"
 ! we use the 2nd argument as the hop we're looking for
 action 2.3  set hop_found "$_none_arg2"
 action 2.5  puts "Searching for hop $hop_found in traceroute to $trace_dest."
 action 3.1  cli command "trace $trace_dest" pattern "#"
 action 3.3  regexp "(.*) (23\.23\.23\.3) (.*)" "$_cli_result" _match _sub1 _sub2
 action 4.1  if $_regexp_result eq 1
 action 4.3   puts "Hop $hop_found found in traceroute to $trace_dest."
 action 4.5   cli command "enable"
 action 4.6   cli command "clear ip ospf process" pattern "[no]"
 action 4.7   cli command "yes" pattern "#"
 action 4.9   puts "OSPF process cleared."
 action 5.1  else
 action 5.3   puts "Hop $hop_found not found in traceroute to $trace_dest."
 action 5.5  end
 action 6.1 else
 action 6.3  puts "Wrong number of arguments supplied."
 action 6.5  puts "Please use : applet <dest> <hop>"
 action 6.9 end

When the applet is executed manually, the following is displayed on screen :

R1#event manager run MANUAL-CHECK-TRACEROUTE-HOPS-APPLET 3.3.3.3
Wrong number of arguments supplied.
Please use : applet <dest> <hop>

R1#event manager run MANUAL-CHECK-TRACEROUTE-HOPS-APPLET
Wrong number of arguments supplied.
Please use : applet <dest> <hop>

R1#event manager run MANUAL-CHECK-TRACEROUTE-HOPS-APPLET 2.2.2.2 23.23.23.3
Searching for hop 23.23.23.3 in traceroute to 2.2.2.2.
Hop 23.23.23.3 not found in traceroute to 2.2.2.2.

R1#event manager run MANUAL-CHECK-TRACEROUTE-HOPS-APPLET 3.3.3.3 23.23.23.3
Searching for hop 23.23.23.3 in traceroute to 3.3.3.3.
Hop 23.23.23.3 found in traceroute to 3.3.3.3.
OSPF process cleared.

Feb 21 17:26:12: %OSPF-5-ADJCHG: Process 100, Nbr 2.2.2.2 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
Feb 21 17:26:18: %OSPF-5-ADJCHG: Process 100, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done

Notes
a) In all TRACEROUTE applets, i couldn't find a way to include a variable into the regexp (this is why the 2nd argument is not actually used). I someone could shed some light, i would be very grateful.

b) There is an action "string replace" which could be used to replace "." with "\." in the regexp ("." means a single character in regular expressions, so we need to put a "\" in front of it, if we want to treat it like the actual dot character), but it supports only string replacement based on character positions.

c) The "_match _sub1 _sub2" arguments in the regexp action are not actually needed in our case, because we're just searching into a string (the arguments are needed if you want to get the results of the match too). We could use the "match string" action instead of regexp (see below), but it doesn't seem to work. Documentation is probably wrong too, because the syntax description is different from the example.

 action 3.3  string match "$hop_found" "$_cli_result"
 action 4.1  if $_string_result eq 1

d) The "puts" command displays the output on your local console (tty), when the applet is run as synchronous. This means that the system logger (i.e. syslog) is bypassed. If you change the "sync yes" to "sync no", then you get the output on syslog.

e) The timestamp of syslog output produced by the applets should use the "service timestamps log" format, but it doesn't.

f) I hope EEM 3.0 will soon get included in other releases too, like the SX/SR ones.