Embedded Packet Capture - How to make routers more active

How many times where you in a need to get a remote packet capture and either ERSPAN wasn't available or the network architecture couldn't help you accomplish it easily?
How many times were you wondering what packets are being dropped in a interface or are being punted to the CPU?
Can you think of an easy way to capture traffic with source and destination on the same router?

Using EPC you can solve -quite easily- all of these issues.

Cisco IOS Embedded Packet Capture (EPC) (available in latest 12.4(20)T; Cisco says 12.2SR too, but SRC on 7200 and SRB on 7600 do not support it) is a powerful troubleshooting and tracing tool which allows network administrators to capture data packets flowing through, to, and from, a Cisco router.

It's actually a onboard packet capture facility that facilitates troubleshooting, the gathering of information on packet format, application analysis, and security. The devices become active participants in the management and operation of the network.

Generally Cisco seem to be investing a lot on this kind of active management; We see EEM get expanded release by release and new Embedded XXX features are coming often into surface on each new release. Long gone are the days of dumb passive routers; the future needs active devices (maybe "I, Robot" was sponsored by Cisco?)

Ok, back to reality...EPC now...

EPC can be used in troubleshooting scenarios where it is helpful to see the actual data being sent through, from, or to the network device. It simplifies operations by allowing the devices to become active participants in the management and operation of the network.

To capture packet data and analyze it, the following tasks need to be performed:

1) Define a capture buffer on a device
2) Define a capture point on a device
3) Associate some capture points with a capture buffer
4) Start (& stop if needed) the capture point
5) View the buffer data either locally or remotely in pcap format


1) Defining a capture buffer on a device

The Capture Buffer is where the packet data will be contained. Capture Buffers are user-named and you can define how the buffer handles the data going into it.

The available configuration offers the following options:

You can specify the size (why only 512KB?) and type of buffer: linear or circular.

• A linear buffer will stop capturing automatically when full.
• A circular buffer will continue to capture packet data (overwriting old data with newer as it fills up).

R1#monitor capture buffer EPC-BUFFER-1 ?
  circular  Circular Buffer
  ...
  linear    Linear Buffer(Default)
  ...
  size      Packet Dump buffer size (in Kbytes)


R1#monitor capture buffer EPC-BUFFER-1 size ?
  <1-512>  Buffer size in Kbytes : 512K or less (default is 256K)

The maximum number of bytes to capture per packet can be limited to save space. But why only 1024?

R1#monitor capture buffer EPC-BUFFER-1 ?
  ...
  max-size  Maximum size of element in the buffer (in bytes)
  ...


R1#monitor capture buffer EPC-BUFFER-1 max-size ?
  <68-1024>  Element size in bytes : 1024 bytes or less (default is 68 bytes)

Rate limiting can also be enabled to:

• Specify a max capture rate (in packets per second).
• Capture every "nth" packet.

Or an automatic limit criteria can be defined to:

• Stop the capture after a specified time interval.
• Stop the capture after capturing a given number of packets.

R1#monitor capture buffer EPC-BUFFER-1 limit ?
  allow-nth-pak    Allow every nth packet through
  duration         Duration of capture
  packet-count     Limit total Number of packets captured
  packets-per-sec  Limit number of packets copied per sec

Filters can also be set for packets being stored in a buffer via ACLs.

R1#monitor capture buffer EPC-BUFFER-1 ?
  ...
  filter    Configure filters

R1#monitor capture buffer EPC-BUFFER-1 filter ?
  access-list  Set access list

Two types of data are stored in a capture buffer: Meta Data and Packet Data.

Meta Data (which helps in filtering too) contains:

• A timestamp of when it is added to a buffer.
• Direction, egress or ingress.
• The switch path it captured.
• Encapsulation type corresponding to input/output interface to allow the decoding of L2.
• Offset to network_start, to facilitate the decoding of L3, if complete L2 decoders are unavailable.
• L3 protocol ID, to facilitate the decoding of L3, if complete L2 decoders are unavailable.

R1#sh monitor capture buffer EPC-BUFFER-1 filter ?
  direction         Filter output based on direction
  input-interface   Filters packet on an input interface
  l3protocol        Filter packets with specific L3 protocol
  output-interface  Filters packet on an output interface
  pak-size          Filter output based on packet size
  time              Filter packets from a specific clock time/date

The packet data starts from datagram-start and copies a minimum of the per packet capture size or datagram-size.

In our example we'll use a circular capture buffer (so it won't stop automatically), 512 Kbytes long ("size"), set to include up to 1024 bytes per packet ("max-size"). We'll call the buffer EPC-BUFFER-1.

R1#monitor capture buffer EPC-BUFFER-1 size 512 max-size 1024 circular

R1#sh monitor capture buffer all parameters
Capture buffer EPC-BUFFER-1 (circular buffer)
Buffer Size : 524288 bytes, Max Element Size : 1024 bytes, Packets : 0
Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
Associated Capture Points:
Configuration:
monitor capture buffer EPC-BUFFER-1 size 512 max-size 1024 circular

Note: The "monitor capture" commands are on exec-level, so they are not stored in the config/nvram. So, it's a nice addition that you can see them using the appropriate show commands.


2) Defining a capture point on a device

The Capture Point is a traffic transit point where the packet capture takes place. It is also identified by a user-defined name.

The following protocols are available as capture points:

• IPv4
• IPv6

And each one of them can be combined with the following forwarding methods:

• CEF
• process switching

R1#monitor capture point ?
  ...
  ip            IPv4
  ipv6          IPv6
  ...

R1#monitor capture point ip ?
  cef               IPv4 CEF
  process-switched  Process switched packets

R1#monitor capture point ipv6 ?
  cef               IPv6 CEF
  process-switched  Process switched packets

Capture points can be interface specific or include all interfaces.

R1#monitor capture point ip cef EPC-POINT-1 ?
  ...
  Async              Async interface
  BVI                Bridge-Group Virtual Interface
  Dialer             Dialer interface
  FastEthernet       FastEthernet IEEE 802.3
  Group-Async        Async Group interface
  Loopback           Loopback interface
  all                All interfaces
  ...

They can also include specific kinds of traffic like :

• Punt
• Drop
• From-us (locally generated packets) - This applies only to process switched traffic

R1#monitor capture point ip cef EPC-POINT-1 ?
  ...
  drop               Drop on any interface
  punt               Punt on any interface
  ...

R1#monitor capture point ip process-switched EPC-POINT-2 ?
  ...
  from-us  Packets originating locally
  ...

Capture points can also be specific to traffic direction:

• In (meaning capture at ingress)
• Out (meaning capture at egress)
• Both

R1#monitor capture point ip cef EPC-POINT-1 S1/0 ?
  both  capture ingress and egress
  in    capture on ingress
  out   capture on egress

R1#monitor capture point ip process-switched EPC-POINT-2 ?
  both     Inbound and outbound and packets
  ...
  in       Inbound packets
  out      Outbound packets

Note: I couldn't find the combination of process-switched traffic and an interface. Also Cisco provides an example of a combination of CEF switched traffic and local originated. Strange...

We'll use 2 capture points : One for IPv4 process switched traffic originated from the router itself and another one for IPv4 CEF switched traffic passing through S1/0 in both directions.

R1#monitor capture point ip process-switched EPC-POINT-1 from-us
R1#
*Jul 16 01:37:14.571: %BUFCAP-6-CREATE: Capture Point EPC-POINT-1 created.

R1#monitor capture point ip cef EPC-POINT-2 S1/0 both
R1#
*Jul 16 01:36:06.027: %BUFCAP-6-CREATE: Capture Point EPC-POINT-2 created.

R1#sh monitor capture point all
Status Information for Capture Point EPC-POINT-2
IPv4 CEF
Switch Path: IPv4 CEF            , Capture Buffer: None
Status : Inactive

Configuration:
monitor capture point ip cef EPC-POINT-2 Serial1/0 both

Status Information for Capture Point EPC-POINT-1
IPv4 Process
Switch Path: IPv4 Process        , Capture Buffer: None
Status : Inactive

Configuration:
monitor capture point ip process-switched EPC-POINT-1 from-us

3) Associating some capture points with a capture buffer

Association (and disassociation) are actions that can be performed in order to combine a capture point with a capture buffer. A capture point can only be associated with one capture buffer (an ACL filter can also be applied). A capture buffer can be associated with many capture points. So, a buffer can collect data from many points but a point can send data to only one buffer.

R1#monitor capture point ?
  associate     Associate capture point with capture buffer
  disassociate  Dis-associate capture point from capture buffer

Multiple packet capture points may be active on a given interface simultaneusly; i.e. BGP packets can be captured into one capture buffer and OSPF packets into another. You just have to use the filter/acl option while creating each capture buffer.

We'll associate our 2 capture points with our single capture buffer..

R1#monitor capture point associate EPC-POINT-1 EPC-BUFFER-1

R1#monitor capture point associate EPC-POINT-2 EPC-BUFFER-1

R1#sh monitor capture point all
Status Information for Capture Point EPC-POINT-2
IPv4 CEF
Switch Path: IPv4 CEF            , Capture Buffer: EPC-BUFFER-1
Status : Inactive

Configuration:
monitor capture point ip cef EPC-POINT-2 Serial1/0 both

Status Information for Capture Point EPC-POINT-1
IPv4 Process
Switch Path: IPv4 Process        , Capture Buffer: EPC-BUFFER-1
Status : Inactive

Configuration:
monitor capture point ip process-switched EPC-POINT-1 from-us

R1#sh monitor capture buffer all parameters
Capture buffer EPC-BUFFER-1 (circular buffer)
Buffer Size : 524288 bytes, Max Element Size : 1024 bytes, Packets : 0
Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
Associated Capture Points:
Name : EPC-POINT-1, Status : Inactive
Name : EPC-POINT-2, Status : Inactive
Configuration:
monitor capture buffer EPC-BUFFER-1 size 512 max-size 1024 circular
monitor capture point associate EPC-POINT-1 EPC-BUFFER-1
monitor capture point associate EPC-POINT-2 EPC-BUFFER-1

4) Starting (& stopping if needed) the capture

The creation of capture points doesn't automatically start them too. You have to do it manually for all capture points that you need to enable.

R1#monitor capture point ?
  ...
  start         Enable Capture Point
  stop          Disable Capture Point

R1#monitor capture point start ?
  WORD  Name of the Capture Point
  all   All Capture Points

R1#monitor capture point stop ?
  WORD  Name of the Capture Point
  all   All Capture Points

So everything is ready for us to start capturing. Just note that in all of our previous outputs, status was inactive. After staring them, it turns to active.

R1#monitor capture point start EPC-POINT-1
R1#
*Jul 16 01:55:13.475: %BUFCAP-6-ENABLE: Capture Point EPC-POINT-1 enabled.
R1#monitor capture point start EPC-POINT-2
R1#
*Jul 16 01:55:21.503: %BUFCAP-6-ENABLE: Capture Point EPC-POINT-2 enabl

R1#sh monitor capture point all
Status Information for Capture Point EPC-POINT-2
IPv4 CEF
Switch Path: IPv4 CEF            , Capture Buffer: EPC-BUFFER-1
Status : Active

Configuration:
monitor capture point ip cef EPC-POINT-2 Serial1/0 both

Status Information for Capture Point EPC-POINT-1
IPv4 Process
Switch Path: IPv4 Process        , Capture Buffer: EPC-BUFFER-1
Status : Active

Configuration:
monitor capture point ip process-switched EPC-POINT-1 from-us

R1#sh monitor capture buffer all parameters
Capture buffer EPC-BUFFER-1 (circular buffer)
Buffer Size : 524288 bytes, Max Element Size : 1024 bytes, Packets : 0
Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
Associated Capture Points:
Name : EPC-POINT-1, Status : Active
Name : EPC-POINT-2, Status : Active
Configuration:
monitor capture buffer EPC-BUFFER-1 size 512 max-size 1024 circular
monitor capture point associate EPC-POINT-1 EPC-BUFFER-1
monitor capture point associate EPC-POINT-2 EPC-BUFFER-1
R1#

5) Viewing the packet data either locally or remotely in pcap format

You can display the buffer contents in either an ASCII-like format (not very useful) or in HEX format (by using the "dumb" keyword at the end).

Below, you can see a ping from 9.9.9.2 (R2) to 9.9.9.1 (R1) passing through S1/0.

R2#ping 9.9.9.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/40/60 ms

Let's check the buffer contents in R1:

R1#sh monitor capture buffer EPC-BUFFER-1
02:03:02.579 UTC Jul 16 2008 : IPv4 CEF Turbo  : Se1/0 None

02:03:02.579 UTC Jul 16 2008 : IPv4 LES CEF    : Se1/0 None

02:03:02.583 UTC Jul 16 2008 : IPv4 Process    : None Se1/0

02:03:02.663 UTC Jul 16 2008 : IPv4 CEF Turbo  : Se1/0 None

02:03:02.663 UTC Jul 16 2008 : IPv4 LES CEF    : Se1/0 None

02:03:02.663 UTC Jul 16 2008 : IPv4 Process    : None Se1/0

02:03:02.683 UTC Jul 16 2008 : IPv4 CEF Turbo  : Se1/0 None

02:03:02.683 UTC Jul 16 2008 : IPv4 LES CEF    : Se1/0 None

02:03:02.683 UTC Jul 16 2008 : IPv4 Process    : None Se1/0

02:03:02.719 UTC Jul 16 2008 : IPv4 CEF Turbo  : Se1/0 None

02:03:02.719 UTC Jul 16 2008 : IPv4 LES CEF    : Se1/0 None

02:03:02.719 UTC Jul 16 2008 : IPv4 Process    : None Se1/0

02:03:02.735 UTC Jul 16 2008 : IPv4 CEF Turbo  : Se1/0 None

02:03:02.735 UTC Jul 16 2008 : IPv4 LES CEF    : Se1/0 None

02:03:02.735 UTC Jul 16 2008 : IPv4 Process    : None Se1/0
...

R1#sh monitor capture buffer EPC-BUFFER-1 dump
02:03:02.579 UTC Jul 16 2008 : IPv4 CEF Turbo  : Se1/0 None

6771C8A0: 0F000800 45000064 00000000 FE019884  ....E..d....~...
6771C8B0: 09090902 09090901 08008E62 00000000  ...........b....
6771C8C0: 00000000 0003EFE4 ABCDABCD ABCDABCD  ......od+M+M+M+M
6771C8D0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
6771C8E0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
6771C8F0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
6771C900: ABCDABCD ABCDABCD 00                 +M+M+M+M.

02:03:02.579 UTC Jul 16 2008 : IPv4 LES CEF    : Se1/0 None

6771C8A0: 0F000800 45000064 00000000 FE019884  ....E..d....~...
6771C8B0: 09090902 09090901 08008E62 00000000  ...........b....
6771C8C0: 00000000 0003EFE4 ABCDABCD ABCDABCD  ......od+M+M+M+M
6771C8D0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
6771C8E0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
6771C8F0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
6771C900: ABCDABCD ABCDABCD 00                 +M+M+M+M.
...

Analyzing the first part we get the following for the header :

0F000800 45000064 00000000 FE019884

0F00 = Cisco HDLC, Address: Unicast (0x0f)
0800 = Protocol: IP (0x0800)
45   = 0100 0101 = IP v4
00   = TOS/DSCP 0x00
0064 = Total Length: 100
01   = Protocol: ICMP (0x01)

09090902 09090901 08008E62 00000000

09090902 = Source 9.9.9.2
09090901 = Destination 9.9.9.1
08       = Type: 8 (Echo (ping) request)
00       = Code: 0

Now, if we send 36 bytes ICMP packets, we can see the difference:

R2#ping 9.9.9.1 size 36

Type escape sequence to abort.
Sending 5, 36-byte ICMP Echos to 9.9.9.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/32/48 ms

We check again the buffer contents:

R1#sh monitor capture buffer EPC-BUFFER-1 dump
02:30:47.419 UTC Jul 16 2008 : IPv4 Process    : None Se1/0

6771C8A0: 0F000800 45000024 000A0000 FF0197BA  ....E..$.......:
6771C8B0: 09090901 09090902 0000A8C4 00020000  ..........(D....
6771C8C0: 00000000 001D571C 00                 ......W..

02:30:47.491 UTC Jul 16 2008 : IPv4 Process    : None Se1/0

6771C8A0: 0F000800 45000024 000B0000 FF0197B9  ....E..$.......9
6771C8B0: 09090901 09090902 0000A88F 00020001  ..........(.....
6771C8C0: 00000000 001D5750 00                 ......WP.

And analyze again the header:

0F000800 45000024 000A0000 FF0197BA

0F00 = Cisco HDLC, Address: Unicast (0x0f)
0800 = Protocol: IP (0x0800)
45   = 0100 0101 = IP v4
00   = TOS/DSCP 0x00
0024 = Total Length: 36
01   = Protocol: ICMP (0x01)

You can also export the buffer contents to an external location in the pcap format. Then you can use Wireshark to examine them more thoroughly.

R1#monitor capture buffer EPC-BUFFER-1 export ?
  ftp:    Location to dump buffer
  http:   Location to dump buffer
  https:  Location to dump buffer
  pram:   Location to dump buffer
  rcp:    Location to dump buffer
  scp:    Location to dump buffer
  tftp:   Location to dump buffer

PS: Btw, you can find a lot of sample pcap files in Jeremy's excellent site, PacketLife.


Generally, EPC provides the following :

• Ability to capture IPv4 and IPv6 packets in the Cisco Express Forwarding path
• A flexible method for specifying the capture buffer size and type
• EXEC-level commands to start and stop the capture
• Show commands to display packet contents on the device
• Facility to export the Packet Capture in PCAP format suitable for analysis using an external tool such as Wireshark
• Extensible infrastructure for enabling packet capture points


Another hooray for Cisco!!!

There seem to be some inconsistencies on this release of EPC plus some without-obvious-reason limitations, but i hope newer releases will improve its functionality (an extra option showing the basic fields of the packet header in understandable format would be a welcome addition). We needed it and we welcome it!.