Thursday, March 19, 2009

Decoding telnet passwords using a hidden debug

While messing with packet capture, i found an interesting feature, well hidden inside IOS.

Almost all of you should probably know the command "debug ip packet detail" which displays details about process switched (received/generated/forwarded) packets. There is a hidden option "dump" that provides extra information including raw packet data in hexadecimal and ASCII forms. According to Cisco, "the dump keyword is not fully supported and should be used only in collaboration with Cisco Technical Support. Because of the risk of using significant CPU utilization, the dump keyword is hidden from the user and cannot be seen using the "?" prompt."

What makes nasty-interesting this command is that you are able to get the credentials of everyone logging into the router through a unencrypted method, i.e. telnet (packets destined to the router are always process switched) and then impersonate him/her. Some other debug methods (i.e. aaa auth, tacacs, radius) hide the password field in their output.

First of all, you have to create an acl that matches the source of the victim and the destination port of telnet. This is not mandatory, but it'll help you keep logs to a minimum.


access-list 111 permit tcp host x.x.x.x any eq telnet

If you don't know the source ip, you can simply use "any".

access-list 111 permit tcp any any eq telnet

Then you have to enable detailed ip packet debugging for the above acl and wait for the victim to appear.

router#debug ip packet detail 111 ?
<cr>

router#debug ip packet detail 111 dump
IP packet debugging is on (detailed) (dump) for access list 111

After the victim has successfully logged in, you can use the following regexp in order to get the useful logs for decoding his/her credentials :

router#sh log | i ^.+: .+ .+ ....0000 [0-9|A-F]+ .+
07B131A0: 00000000 60022238 919B0000 020405B4 ....`."8.......4
07B15D60: 2932EDFC 50182238 35A80000 FFFD03FF )2m|P."85(...}..
07B3F660: 2932EE08 50182238 95610000 FFFD01FF )2n.P."8.a...}..
07B09CE0: 2932EE63 50182238 D4D40000 FFFA1800 )2ncP."8TT...z..
07B1BDA0: 2932EE66 50182238 1B740000 76 )2nfP."8.t..v
07AFD3A0: 2932EE67 50182238 28720000 69 )2ngP."8(r..i
07B203A0: 2932EE68 50182238 2E700000 63 )2nhP."8.p..c
07B0FD20: 2932EE69 50182238 1D6E0000 74 )2niP."8.n..t
07B1FAE0: 2932EE6A 50182238 286C0000 69 )2njP."8(l..i
07B19AA0: 2932EE6B 50182238 246A0000 6D )2nkP."8$j..m
07B27560: 2932EE6C 50182238 84670000 0D00 )2nlP."8.g....
07B263E0: 2932EE78 50182238 285A0000 69 )2nxP."8(Z..i
07B1C660: 2932EE78 50182238 50590000 41 )2nxP."8PY..A
07B02260: 2932EE78 50182238 44580000 4D )2nxP."8DX..M
07B2A9E0: 2932EE78 50182238 1D570000 74 )2nxP."8.W..t
07B22F60: 2932EE78 50182238 29560000 68 )2nxP."8)V..h
07B26CA0: 2932EE78 50182238 2C550000 65 )2nxP."8,U..e
07B2F8A0: 2932EE78 50182238 3F540000 52 )2nxP."8?T..R
07B312E0: 2932EE78 50182238 22530000 6F )2nxP."8"S..o
07B2B2A0: 2932EE78 50182238 22520000 6F )2nxP."8"R..o
07B33EA0: 2932EE78 50182238 22510000 6F )2nxP."8"Q..o
07AFB960: 2932EE78 50182238 89500000 08 )2nxP."8.P...
07B20C60: 2932EE78 50182238 894F0000 08 )2nxP."8.O...
07B2BB60: 2932EE78 50182238 894E0000 08 )2nxP."8.N...
07B34760: 2932EE78 50182238 424D0000 4F )2nxP."8BM..O
07B226A0: 2932EE78 50182238 424C0000 4F )2nxP."8BL..O
07B3DC20: 2932EE78 50182238 3D4B0000 54 )2nxP."8=K..T
07B4E2A0: 2932EE78 50182238 84490000 0D00 )2nxP."8.I....

You're mostly interested in the last hex column (telnet data is at the end of the packet) and you can safely ignore everything that looks abnormal (like FF). Looking at the ascii table, you can see that hex 0D translates to CR (CarriageReturn) and hex 08 translates to BS (BackSpace). So the ascii character stream of the last hex column is :

v
i
c
t
i
m
CR (CarriageReturn) - "Enter" is pressed
i
A
M
t
h
e
R
o
o
o
BS (BackSpace) - 1st "o" is deleted
BS (BackSpace) - 2nd "o" is deleted
BS (BackSpace) - 3rd "o" is deleted
O
O
O
T
CR (CarriageReturn) - "Enter" is pressed

which translates to username "victim" and password "iAMtheROOT".

Keep in mind that you might see more characters in the same line, if the input is keyed in quickly by the user. If you see 4 chars, then you'll have to check manually for the existence of the rest of the data. You're always looking for 0D00 in order to find the end of each input.

router#sh log | i ^.+: .+ .+ ....0000 [0-9|A-F]+ .+
07B1D7E0: 05DE7766 50182238 219F0000 FFFD03FF .^wfP."8!....}..
07B079E0: 05DE7772 50182238 81580000 FFFD01FF .^wrP."8.X...}..
07B16EE0: 05DE77C7 50182238 BCF80000 FFFA1800 .^wGP."8<x...z..
07B14320: 05DE77CD 50182238 59730000 FFFB24 .^wMP."8Ys...{$
07B25B20: 05DE77D0 50182238 3A1B0000 76696374 .^wPP."8:...vict
07B588E0: 05DE77D6 50182238 705E0000 0D00 .^wVP."8p^....
07B45F60: 05DE77E2 50182238 BC890000 69414D74 .^wbP."8<...iAMt
07B2E720: 05DE77E2 50182238 70460000 0D00 .^wbP."8pF....

router#sh log | b 07B25B20
07B25B20: 05DE77D0 50182238 3A1B0000 76696374 .^wPP."8:...vict
07B25B30: 696D im

router#sh log | b 07B45F60
07B45F60: 05DE77E2 50182238 BC890000 69414D74 .^wbP."8<...iAMt
07B45F70: 6865524F 4F54 heROOT


Notes :

1. Probably a tcl script could decode the above debug outputs much easier/better.

2. You can also include other ips (i.e. aaa server) in your acl in order to watch -probably- easier (i.e. look for Username/Password strings) the whole login process, because in the packet data you'll find everything the user types.

3. The beginning of a packet may start at different locations in the dump output depending on the specific router, interface type, and packet header processing that may have occurred before the output is displayed.

4. The length of the displayed packet information may exceed the actual packet length and include additional padding bytes that do not belong to the actual IP packet.


So, next time you decide to use telnet to login into a router, you'd better think twice before doing so. Besides sniffing on the wire, someone can do sniffing on the router too.

Tuesday, March 10, 2009

Using EEM 3.0 to get the top 3 processes

As you may already know, IOS > 12.3(4)T provides an easy way to get notified through syslog and snmp, when the cpu load of your router exceeds a specified threshold for a specific duration.

This is accomplished with the following global configuration commands :


process cpu threshold type total rising 40 interval 30
snmp-server enable traps cpu threshold

The output produced, when the threshold is exceeded, looks like the following :

%SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 42%/19%, Top 3 processes(Pid/Util): 89/38%, 138/2%, 28/0%

%SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 3%/19%.

It sure looks quite informative, since the top 3 process IDs are included with details about their cpu usage. You can make it even more informative by using EEM 3.0, so you can have the processes names too (plus other details). Here is a sample applet that uses regexp to get the 3 PIDs and then passes them to the "sh proc cpu" command using another regexp :

event manager applet SHOW-TOP3-PROCS-APPLET
event syslog pattern "^.*CPURISINGTHRESHOLD.*Top 3 processes.*"
action 1.1 regexp "^.*CPURISINGTHRESHOLD.*Top 3 processes.*:[ ]+([0-9]+)/[0-9]+%, ([0-9]+)/[0-9]+%, ([0-9]+)/[0-9]+%$" "$_syslog_msg" _match _sub1 _sub2 _sub3
action 2.1 if $_regexp_result eq 1
action 3.1 cli command "sh proc cpu sort | inc (CPU utilization|Runtime|^[ ]*($_sub1|$_sub2|$_sub3)_)"
action 3.3 syslog msg "$_cli_result"
action 4.1 else
action 5.1 syslog msg "No Match. Please re-check your regexp code."
action 6.1 end

I have colored the 3 variables (=PIDs) so you can easily see their usage in each regexp.

When an applet is triggered by an event (like the syslog in our example), you cannot run it manually :

R3#event manager run SHOW-TOP3-PROCS-APPLET
EEM policy SHOW-TOP3-PROCS-APPLET not registered with event none Event Detector

You have to use "event none", if you want to do so.

But, when the trigger event is a syslog message, you can create the appropriate message manually by using the "send log" command. That way you can test your applet without waiting for the syslog message to appear.

R3#send log %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 26%/40%, Top 3 processes(Pid/Util): 89/21%, 138/3%, 43/0%

R3#
%SYS-2-LOGMSG: Message from 0(): %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 26%/40%, Top 3 processes(Pid/Util): 89/21%, 138/3%, 43/0%
R3#
%HA_EM-6-LOG: SHOW-TOP3-PROCS-APPLET: CPU utilization for five seconds: 8%/0%; one minute: 5%; five minutes: 6%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
138 126668 1493020 84 5.19% 3.35% 2.86% 0 HQF Shaper Backg
89 122052 8421 14493 0.95% 0.60% 1.80% 0 Exec
43 20368 6154 3309 0.15% 0.18% 0.28% 0 Per-Second Jobs
R3>
R3#

Of course, the cpu load numbers between your message and the cpu load output do not agree in such a case, because the current cpu load is different from the one you sent through your syslog message.

So, let's create a little bit of real load, in order to make the applet run automatically too:

R3# show tech
R3#
%SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 42%/19%, Top 3 processes(Pid/Util): 89/38%, 138/2%, 28/0%
%HA_EM-6-LOG: SHOW-TOP3-PROCS-APPLET: CPU utilization for five seconds: 42%/19%; one minute: 12%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
89 127716 8452 15110 38.00% 8.16% 2.59% 0 Exec
138 133132 1555879 85 2.95% 2.58% 3.02% 0 HQF Shaper Backg
28 8344 1922 4341 0.23% 0.10% 0.11% 0 HC Counter Timer
R3>
R3#

As you can see, PIDs 89 (Exec), 138 (HQF Shaper Backg), 28 (HC Counter Timer) are the top 3 in this case.

Please keep in mind that EEM 3.0 requires latest 12.4(22)T.

Monday, March 9, 2009

DHCP Snooping & the TCAM ACL table

While playing with DHCP snooping on a ME-3400 switch (that shouldn't be different from a 3560/3750), i found out the following interesting information.

When you enable DHCP snooping, 3 special acl entries are created dynamically into the TCAM ACL table. These are defined as L3 Input and you can check them out by using the following command :


switch#sh platform tcam table acl | i 43
58 40_00000000_00000000-00_40000043_00430000 0000001F L3 Input
60 40_00000000_00000000-00_40000044_00430000 0000001F L3 Input
62 40_00000000_00000000-00_40000043_00440000 0000001F L3 Input


43 in hex is 67 in dec, which is the udp port a bootp/dhcp server listen to.
44 in hex is 68 in dec, which is the udp port a bootp/dhcp client sends from.

You can also see more details about these tcam entries by specifying the index number and adding the detail keyword :


switch#sh platform tcam table acl index 58 detail

=============================================================================
ACL Cam Table (#entries: 8192, startIndex: 13696)

Index ACL CAM Table ACL
-----------------------------------------------------------------------------
mask-> F0_00000000_00000000-00_C000FFFF_FFFF0000
58 40_00000000_00000000-00_40000043_00430000 0000001F

l3CamInputAclDescriptor Value Mask
-----------------------------------------------------------------------------
lookupType: 4 F
cos: 0 0
inputAclType: 0 0
l3Destination: 00.00.00.00 00.00.00.00
l3Source: 00.00.00.00 00.00.00.00
inputAclLabel: 0 0
l4Select: 1 3
l3DontFragment: 0 0
l3MoreFragments: 0 0
l3SmallOffset: 0 0
l3NotFirstFragment: 0 0
l3ReservedFlag: 0 0
l2Bcast: 0 0
l3Dscp: 0 0
l3TosReserved: 0 0
l4Destination: 43 FFFF
l4Source: 43 FFFF
l4Map: 0 0

AclDescriptor
-----------------------------------------------------------------------------
aclStatisticsIndex: 0
aclLogIndex: 0
denyRoute: 0
denyBridge: 0
spanDest0En: 0
spanDest1En: 0
redirectIndex: 1F


switch#sh platform tcam table acl index 60 detail

=============================================================================
ACL Cam Table (#entries: 8192, startIndex: 13696)

Index ACL CAM Table ACL
-----------------------------------------------------------------------------
mask-> F0_00000000_00000000-00_C000FFFF_FFFF0000
60 40_00000000_00000000-00_40000044_00430000 0000001F

l3CamInputAclDescriptor Value Mask
-----------------------------------------------------------------------------
lookupType: 4 F
cos: 0 0
inputAclType: 0 0
l3Destination: 00.00.00.00 00.00.00.00
l3Source: 00.00.00.00 00.00.00.00
inputAclLabel: 0 0
l4Select: 1 3
l3DontFragment: 0 0
l3MoreFragments: 0 0
l3SmallOffset: 0 0
l3NotFirstFragment: 0 0
l3ReservedFlag: 0 0
l2Bcast: 0 0
l3Dscp: 0 0
l3TosReserved: 0 0
l4Destination: 44 FFFF
l4Source: 43 FFFF
l4Map: 0 0

AclDescriptor
-----------------------------------------------------------------------------
aclStatisticsIndex: 0
aclLogIndex: 0
denyRoute: 0
denyBridge: 0
spanDest0En: 0
spanDest1En: 0
redirectIndex: 1F


switch#sh platform tcam table acl index 62 detail

=============================================================================
ACL Cam Table (#entries: 8192, startIndex: 13696)

Index ACL CAM Table ACL
-----------------------------------------------------------------------------
mask-> F0_00000000_00000000-00_C000FFFF_FFFF0000
62 40_00000000_00000000-00_40000043_00440000 0000001F

l3CamInputAclDescriptor Value Mask
-----------------------------------------------------------------------------
lookupType: 4 F
cos: 0 0
inputAclType: 0 0
l3Destination: 00.00.00.00 00.00.00.00
l3Source: 00.00.00.00 00.00.00.00
inputAclLabel: 0 0
l4Select: 1 3
l3DontFragment: 0 0
l3MoreFragments: 0 0
l3SmallOffset: 0 0
l3NotFirstFragment: 0 0
l3ReservedFlag: 0 0
l2Bcast: 0 0
l3Dscp: 0 0
l3TosReserved: 0 0
l4Destination: 43 FFFF
l4Source: 44 FFFF
l4Map: 0 0

AclDescriptor
-----------------------------------------------------------------------------
aclStatisticsIndex: 0
aclLogIndex: 0
denyRoute: 0
denyBridge: 0
spanDest0En: 0
spanDest1En: 0
redirectIndex: 1F


All the above values are in hex and with a little bit of experimenting (and guessing), you'll find that :

l3Destination is the destination ip address
l3Source is the source ip address
l4Select is the L4 protocol (1=udp, 2=tcp)
l4Destination is the destination port
l4Source is the source port

Mask is used to define the (don't)care bits, which are also used in ACL construction.
Since mask 3 is used for the l4Select value, i guess we do care only about the last 2 bits (11 in bin = 3 in hex) of this field. Mask FFFF, which is used for the src/dst ports, means that we care about all bits (values 0-65535).

So the above 3 tcam entries get actually translated to:


udp any eq 67 any eq 67
udp any eq 67 any eq 68
udp any eq 68 any eq 67

According to the DHCP snooping operation, the 1st (which seems a little bit strange) and 2nd entry should be somehow used on the trusted ports (DHCP servers), while the 3rd entry should be used on the untrusted ports (DHCP clients).

What i haven't figured out yet, is how these special acl entries are actually used. Imho, it looks like an acl used for matching traffic, like the one you use under class maps in QoS. Also, IP Source Guard and Dynamic ARP Inspection might use these as well.

Tuesday, March 3, 2009

Would you mind waiting for 2-3 mins in a console?

Details about bug CSCed45578 :

Console locked for 2 mins after booting with system accounting

Symptom:

When an IOS system is configured with "aaa accounting system", then users may
be prevented from starting an exec session on the console or terminal lines for
as much as three minutes, after the system reloads.

Conditions:


This delay occurs when the RADIUS and/or TACACS+ accounting server(s) are not
reachable at boot time.

Workaround:


Configure the following hidden global command:

router(config)#no aaa accounting system guarantee-first


Imagine this now on a router using many different authentication methods :


aaa authentication login default group tacacs+
aaa authentication login CONSOLE-ACCESS local
aaa authentication ppp default group radius
!
aaa accounting exec default start-stop group tacacs+
aaa accounting network default start-stop group radius
aaa accounting system default start-stop group radius
!
line con 0
login authentication CONSOLE-ACCESS

Terminal clients are authenticated through tacacs and their accounting is sent to tacacs too. These are probably the engineers trying to do their job in the router.

Console clients are authenticated using the local usernames ("enable" could be used too) but their accounting is sent to tacacs. These are probably the high-level engineers looking to do a somewhat more important job in the router.

PPP (i.e. adsl) customers are authenticated through radius and their accounting is sent to radius too.

System accounting (before reload and after reload/power-on) is also sent to radius, so ppp users from this router left in the database can be erased after the forementioned events.


What do you think will happen immediately after a reload, if the radius server is not reachable, but the router has ip connectivity?


1) ppp customers will be unable to connect (radius auth)
2) engineers will be unable to connect (tacacs auth)
3) high-level engineers will be unable to connect (local auth)

If you answered 1 (like i believed until some months ago), then you're 1/3 correct.
NOBODY will be able to connect to this router, regardless of the access method used.
This will last for 2-3 minutes in the best case (retransmit and timeout intervals of the aaa server do not seem to have any effect), but i have seen it lasting for 7-8 minutes in some cases (i guess before bug CSCed45578 was fixed).

Well, tell me honestly; would you mind waiting for 2-3 mins in a console, while trying to troubleshoot an issue? But let's suppose you can wait (some mins are nothing in comparison to eternity). What if there is a crash or an event happening during this small time and you need to further troubleshoot the issue? EEM can help in some circumstances, but it's not a panacea.

Unfortunately you have to live with it! And according to Cisco this behavior is expected.

But, there is a workaround! No need to worry. "no aaa accounting system guarantee-first", which was a hidden command in older releases, will fix the above issue, so anyone not affected by the non-working aaa server can have access to the router as soon as it boots up.

And when you though that finally a solution has been found, you discover that this command is causing another headache. And here is the complete story...


When "aaa accounting system default start-stop group radius" is enabled, router will sent an Accounting-On packet to the radius server sometime when it boots up. It will also try to sent an Accounting-Off, but that is not guaranteed (i.e.: after a crash or lost connection).

When the radius server receives the Accounting-Off packet, it assumes that all users disconnected from the router. For the same reason, when it receives the Accounting-On packet it assumes that the router just booted up, so the users were disconnected some time ago and now there should be nobody connected.

If you have configured:

aaa accounting system default start-stop group radius
aaa accounting system guarantee-first ! enabled by default => not shown

then router guarantees that the first aaa packet sent will be the Accounting-On. So, no login (in whatever form) is allowed, until a response is received from the radius server (or whatever system accounting method you have used), or the 2-3 mins waiting period passes. This way you're sure that the Accounting-On packet will be the first one to reach the radius.

If you have configured:

aaa accounting system default start-stop group radius
no aaa accounting system guarantee-first

then some users might log in before the router sends the Accounting-On packet. As a consequence, there is a possibility that some accounting start packets will be sent before the Accounting-On packet and/or some others might be lost. For most radius server implementations it is expected that all the previous user entries will be cleared from the database, when the Accounting-On packet is received. So, in your database, you'll see the above logged-in users as not connected.

I understand the motivation behind Cisco's implementation, but i think it lacks flexibility. In my case i don't care what will happen to the accounting records produced by the con/vty exec sessions during the bootup time. I prefer to lose them (i might even not create them in the first place), instead of postponing my access to the router.

A solution that comes to my mind right now would be to add this first-packet-guarantee specifically on each authentication method, like below:

aaa authentication login default group tacacs+
aaa authentication login CONSOLE-ACCESS enable
aaa authentication ppp default group radius guarantee-system-acct-first
aaa authentication ppp AAA-RADIUS group radius guarantee-system-acct-first

or disable this first-packet-guarantee per aaa method :

aaa authentication login default group tacacs+ no-guarantee-system-acct-first
aaa authentication login CONSOLE-ACCESS enable no-guarantee-system-acct-first
aaa authentication ppp default group radius
aaa authentication ppp AAA-RADIUS group radius

or enable/disable it per aaa server group, something like :

aaa group server radius RADIUS-GROUP
server 1.1.1.1 auth-port 1645 acct-port 1646
server 2.2.2.2 auth-port 1645 acct-port 1646
guarantee-system-acct-first


Before an aaa method or server group with the "guarantee-system-acct-first" parameter is invoked, the router should make sure that the "Accounting-On" packet has been sent and a response has been received. All other aaa methods should be invoked without any restrictions.

This way, engineers will be able to access the router without any waiting (through tacacs/local auth), but ppp customers will not be allowed to login, until the radius server has responded to the Accounting-On packet or the waiting time has passed.

Cisco recommends this to be handled through a PER (Product Enhancement Request) without of course being able to guarantee (have they tried using the above command? :-p) its solution.

I, personally, have negative experience from 3 PERs until now and i don't see the reason to make them 4. How about you? Have you ever submitted a PER to Cisco and what was the result? Please spare a min and complete the poll to the right.

Update 9 Mar 2009 : Bug CSCsy20392 has been opened, for everyone interested in following it. It has been opened as enhancement (to change the default behaviour of accounting guarantee-first command) and hopefully a PER will be attached to it.

 
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Greece License.