Would you mind waiting for 2-3 mins in a console?

Details about bug CSCed45578 :

Console locked for 2 mins after booting with system accounting

Symptom:

When an IOS system is configured with "aaa accounting system", then users may
be prevented from starting an exec session on the console or terminal lines for
as much as three minutes, after the system reloads.

Conditions:

This delay occurs when the RADIUS and/or TACACS+ accounting server(s) are not
reachable at boot time.

Workaround:

Configure the following hidden global command:

router(config)#no aaa accounting system guarantee-first


Imagine this now on a router using many different authentication methods :

aaa authentication login default group tacacs+ 
aaa authentication login CONSOLE-ACCESS local
aaa authentication ppp default group radius
!
aaa accounting exec default start-stop group tacacs+
aaa accounting network default start-stop group radius
aaa accounting system default start-stop group radius
!
line con 0
 login authentication CONSOLE-ACCESS

Terminal clients are authenticated through tacacs and their accounting is sent to tacacs too. These are probably the engineers trying to do their job in the router.

Console clients are authenticated using the local usernames ("enable" could be used too) but their accounting is sent to tacacs. These are probably the high-level engineers looking to do a somewhat more important job in the router.

PPP (i.e. adsl) customers are authenticated through radius and their accounting is sent to radius too.

System accounting (before reload and after reload/power-on) is also sent to radius, so ppp users from this router left in the database can be erased after the forementioned events.


What do you think will happen immediately after a reload, if the radius server is not reachable, but the router has ip connectivity?

1) ppp customers will be unable to connect (radius auth)
2) engineers will be unable to connect (tacacs auth)
3) high-level engineers will be unable to connect (local auth)

If you answered 1 (like i believed until some months ago), then you're 1/3 correct.
NOBODY will be able to connect to this router, regardless of the access method used.
This will last for 2-3 minutes in the best case (retransmit and timeout intervals of the aaa server do not seem to have any effect), but i have seen it lasting for 7-8 minutes in some cases (i guess before bug CSCed45578 was fixed).

Well, tell me honestly; would you mind waiting for 2-3 mins in a console, while trying to troubleshoot an issue? But let's suppose you can wait (some mins are nothing in comparison to eternity). What if there is a crash or an event happening during this small time and you need to further troubleshoot the issue? EEM can help in some circumstances, but it's not a panacea.

Unfortunately you have to live with it! And according to Cisco this behavior is expected.

But, there is a workaround! No need to worry. "no aaa accounting system guarantee-first", which was a hidden command in older releases, will fix the above issue, so anyone not affected by the non-working aaa server can have access to the router as soon as it boots up.

And when you though that finally a solution has been found, you discover that this command is causing another headache. And here is the complete story...


When "aaa accounting system default start-stop group radius" is enabled, router will sent an Accounting-On packet to the radius server sometime when it boots up. It will also try to sent an Accounting-Off, but that is not guaranteed (i.e.: after a crash or lost connection).

When the radius server receives the Accounting-Off packet, it assumes that all users disconnected from the router. For the same reason, when it receives the Accounting-On packet it assumes that the router just booted up, so the users were disconnected some time ago and now there should be nobody connected.

If you have configured:

aaa accounting system default start-stop group radius
aaa accounting system guarantee-first ! enabled by default => not shown

then router guarantees that the first aaa packet sent will be the Accounting-On. So, no login (in whatever form) is allowed, until a response is received from the radius server (or whatever system accounting method you have used), or the 2-3 mins waiting period passes. This way you're sure that the Accounting-On packet will be the first one to reach the radius.

If you have configured:

aaa accounting system default start-stop group radius
no aaa accounting system guarantee-first

then some users might log in before the router sends the Accounting-On packet. As a consequence, there is a possibility that some accounting start packets will be sent before the Accounting-On packet and/or some others might be lost. For most radius server implementations it is expected that all the previous user entries will be cleared from the database, when the Accounting-On packet is received. So, in your database, you'll see the above logged-in users as not connected.

I understand the motivation behind Cisco's implementation, but i think it lacks flexibility. In my case i don't care what will happen to the accounting records produced by the con/vty exec sessions during the bootup time. I prefer to lose them (i might even not create them in the first place), instead of postponing my access to the router.

A solution that comes to my mind right now would be to add this first-packet-guarantee specifically on each authentication method, like below:

aaa authentication login default group tacacs+
aaa authentication login CONSOLE-ACCESS enable 
aaa authentication ppp default group radius guarantee-system-acct-first
aaa authentication ppp AAA-RADIUS group radius guarantee-system-acct-first

or disable this first-packet-guarantee per aaa method :

aaa authentication login default group tacacs+ no-guarantee-system-acct-first
aaa authentication login CONSOLE-ACCESS enable no-guarantee-system-acct-first
aaa authentication ppp default group radius
aaa authentication ppp AAA-RADIUS group radius

or enable/disable it per aaa server group, something like :

aaa group server radius RADIUS-GROUP
 server 1.1.1.1 auth-port 1645 acct-port 1646
 server 2.2.2.2 auth-port 1645 acct-port 1646
 guarantee-system-acct-first

Before an aaa method or server group with the "guarantee-system-acct-first" parameter is invoked, the router should make sure that the "Accounting-On" packet has been sent and a response has been received. All other aaa methods should be invoked without any restrictions.

This way, engineers will be able to access the router without any waiting (through tacacs/local auth), but ppp customers will not be allowed to login, until the radius server has responded to the Accounting-On packet or the waiting time has passed.

Cisco recommends this to be handled through a PER (Product Enhancement Request) without of course being able to guarantee (have they tried using the above command? :-p) its solution.

I, personally, have negative experience from 3 PERs until now and i don't see the reason to make them 4. How about you? Have you ever submitted a PER to Cisco and what was the result? Please spare a min and complete the poll to the right.

Update 9 Mar 2009 : Bug CSCsy20392 has been opened, for everyone interested in following it. It has been opened as enhancement (to change the default behaviour of accounting guarantee-first command) and hopefully a PER will be attached to it.