Cisco : "this behavior is expected" & Archive Config Logger

The last 2 months i've been busy playing with an ASR1000, finding bugs and "fighting" with the BUs.

Dealing with a lot of issues lately (not necessarily of ASR's), there are a few times that i have come across Cisco's motto "this behavior is expected". One of them was the fact that the very useful functionality of archive config logger (Configuration Logging) is ruined by the inability to filter out per-user configs applied through an aaa server.

The Configuration Change Notification and Logging (Configuration Logging) feature allows the tracking of configuration changes entered on a per-session and per-user basis by implementing a configuration log. The configuration log tracks each configuration command that is applied, who applied the command, the parser return code for the command, and the time the command was applied.

As you all know, a basic configuration like the following will help you keep track of changes in the configuration of a router:

archive
 log config
  logging enable
  hidekeys

In a "normal" router you'll probably see something like the following:

router#sh archive log config all
 idx   sess           user@line      Logged command
    8     3         user@vty0     | no ipv6 nd suppress-ra
    9     3         user@vty0     | ipv6 ospf 100 area 10
   10     4         user@vty0     |no ip sla monitor 99
   11     5         user@vty0     |interface GigabitEthernet0/2.35211201
   12     5         user@vty0     | encapsulation dot1Q 3521 second-dot1q 1201
   13     5         user@vty0     | ip address 10.10.10.129 255.255.255.252

If you happen to have a router that is acting like a bras, accepting ppp sessions and using radius attributes in order to apply per-user ACLs and/or routes, then you'll find out that the above feature is totally useless, because all these per-user entries are stored into the archive logging buffer, overwriting the actual configuration changes you want to watch (when the configuration log is full, the oldest log entry will be removed every time a new entry is added).

router#sh archive log config all
 idx   sess           user@line      Logged command
  177     0        console@console  |permit ip host 1.1.1.1 any 
  178     0        console@console  |deny ip any any 
  179     0        console@console  |deny ip any any 
  180     0        console@console  |ip route 2.2.2.0 255.255.255.240 3.3.3.3 
  181     0        console@console  |ip route 2.2.2.0 255.255.255.240 3.3.3.3 
  182     0        console@console  |ip access-list extended Virtual-Access2.10785#106240381
  183     0        console@console  |deny ip any 10.0.0.0 0.255.255.255
  184     0        console@console  |deny ip any 10.0.0.0 0.255.255.255
  185     0        console@console  |deny ip any 192.168.0.0 0.0.255.255
  186     0        console@console  |deny ip any 192.168.0.0 0.0.255.255
  187     0        console@console  |deny ip any 224.0.0.0 15.255.255.255
  188     0        console@console  |deny ip any 224.0.0.0 15.255.255.255

Note : Please don't ask me about the double entries...

According to Cisco's cli help, only changes to the running config, not the one stored on nvram, are to be logged.

router(config-archive)#log ?
  config  Logging changes to the running configuration

If you issue the "show run" command (which is supposed to display the running configuration), you won't see these per-user config changes. That's because all these per-user configs, after being downloaded from the aaa server, have a flag set that they should not be nvgen'ed. So they are not shown.

In the Cisco IOS software configuration model, the configuration state is maintained in a distributed manner, with each component storing its own configuration state. To retrieve configuration information, the software must poll every component to collect the distributed information. This configuration state retrieval operation is performed by a process known as nonvolatile generation (NVGEN), and it is used by command line interpreter (CLI) commands such as show running-configuration, write memory, and copy system:running-configuration to display or copy the running system configuration. When invoked, NVGEN queries each system component and each instance of interface or other configuration objects. A running configuration file is constructed as NVGEN traverses the system performing these queries.

Also, according to Cisco:

Restrictions for Configuration Change Notification and Logging
•Only complete commands input in a configuration mode are logged.
•Commands that are part of a configuration file applied with the copy command are not logged.

The Configuration Change Notification and Logging feature tracks changes made to the Cisco IOS software running configuration by maintaining a configuration log. This configuration log tracks changes initiated only through the command-line interface (CLI) or HTTP.

Why Cisco chose to include these per-user "dynamic" config changes into the archive logger, making it another thrown-away feature, is still a mystery to me.

There is also the possibility that this kind of continuous logging might be causing a little increase in cpu usage too.

Yep, time for another Cisco-should-have-fixed-it-in-the-first-place PER.