Upgrading ROMMONs in 6500s/7600s with SUP720

The Supervisor Engine 720 (or the Supervisor Engine 720-10GE) ROMMONs consists of two modules:
- A resident module that is not changed during the upgrade procedure. This is the Gold region one.
- An upgradeable module that is updated during the upgrade procedure. This is the only module that you have to download from cisco.

There are three regions (including the Gold region) where versions of the ROMMON image can be stored. The ROMMON software upgrade feature allows you to have two upgraded ROMMON images (one in region F1, the second in region F2) in addition to the Gold ROMMON stored on the one-time programmable (OTP) EPROM section of the ROMMON. You can use the "upgrade rom-monitor slot X preference" command to select which ROMMON will be the preferred ROMMON the next time that the system is booted. You can change the preference as often as you like, but keep in mind that the changes do not take effect until you reset the system.

You can also disqualify a specific region of ROMMON and use the other region, or go back to using the Gold ROMMON stored in the OTP EPROM section, by using the "upgrade rom-monitor slot X invalidate" command.

If you're referring to the SUP720 module, you'll need to include "sp" or "rp" after the module number:

6509#upgrade rom-monitor slot 6 ?
  rp  upgrade the rommon of Router Processor
  sp  upgrade the rommon of Switch Processor

There are 3 main ROMMONs you can upgrade on this platform.

SUP720 SP ROMMON
This is the Supervisor Switch Processor ROMMON. You'll find the release notes here (latest version is 8.5(3)). You can check the running version using one of the following methods:

Check the logs while booting:

Jun 13 20:00:10: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor

System Bootstrap, Version 8.5(2)
Copyright (c) 1994-2007 by cisco Systems, Inc.
Cat6k-Sup720/SP processor with 1048576 Kbytes of main memory

Use "sh ver" on SP:

6509#remote command switch sh ver | i ROM
ROM: System Bootstrap, Version 8.5(2)
System returned to ROM by reload at 20:00:10 EET Fri Jun 13 2008

Use "sh mod" :

6509#sh mod 6
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
 6    2  Supervisor Engine 720 (Active)         WS-SUP720-3BXL     XXXXXXXXXXX

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
 6  XXXX.XXXX.XXXX to XXXX.XXXX.XXXX   5.4   8.5(2)       12.2(33)SXH2 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
 6  Policy Feature Card 3       WS-F6K-PFC3BXL     XXXXXXXXXXX  1.8    Ok
 6  MSFC3 Daughterboard         WS-SUP720          XXXXXXXXXXX  3.0    Ok

Mod  Online Diag Status
---- -------------------
 6  Pass

MSFC3 RP ROMMON
This is the MSFC3 Route Processor ROMMON. You'll find the release notes here (latest version is 12.2(17r)SX5). You can check the running version using one of the following methods:

Check the logs while booting:

00:00:06: %OIR-6-CONSOLE: Changing console ownership to route processor

System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2006 by cisco Systems, Inc.
Cat6k-Sup720/RP platform with 1048576 Kbytes of main memory

Use "sh ver"

6509#sh ver | i ROM
ROM: System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)
System returned to ROM by reload at 19:16:07 EET Fri Jun 13 2008 (SP by reload)

CEF720 modules ROMMON
This is the CEF720 ROMMON for modules like WS-X6748-GE-TX, WS-X6724-SFP, WS-X6748-SFP, WS-X6704-10GE . You'll find the release notes here (latest version is 12.2(18r)S1). You can check the running version using one of the following methods:

Use "sh ver" on module:

6509#remote command module 1 sh ver | i ROM
ROM: System Bootstrap, Version 12.2(18r)S1, RELEASE SOFTWARE (fc1)
System returned to ROM by power-on

Use "sh mod":

6509#sh mod 1
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       XXXXXXXXXXX

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  XXXX.XXXX.XXXX to XXXX.XXXX.XXXX   3.1   12.2(18r)S1  12.2(33)SXH2 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  1  Distributed Forwarding Card WS-F6700-DFC3BXL   XXXXXXXXXXX  5.3    Ok

Mod  Online Diag Status
---- -------------------
  1  Pass

You can also find the version of all ROMMONs by using the command "sh mod version" and checking the "Fw:" line of each module:

6509#sh mod version
Mod  Port Model              Serial #    Versions
---- ---- ------------------ ----------- -------------------------------------
 1   24  WS-X6724-SFP       XXXXXXXXXXX Hw : 3.1
                                        Fw : 12.2(18r)S1
                                        Sw : 12.2(33)SXH2a
                                        Sw1: 8.7(0.22)H2A1
         WS-F6700-DFC3BXL   XXXXXXXXXXX Hw : 5.3
 2   48  WS-X6748-GE-TX     XXXXXXXXXXX Hw : 2.6
                                        Fw : 12.2(18r)S1
                                        Sw : 12.2(33)SXH2a
                                        Sw1: 8.7(0.22)H2A1
         WS-F6700-DFC3BXL   XXXXXXXXXXX Hw : 5.3
 6    2  WS-SUP720-3BXL     XXXXXXXXXXX Hw : 5.4
                                        Fw : 8.5(2)
                                        Sw : 12.2(33)SXH2a
                                        Sw1: 8.7(0.22)H2A1
         WS-SUP720          XXXXXXXXXXX Hw : 3.0
                                        Fw : 12.2(17r)SX5
                                        Sw : 12.2(33)SXH2a
         WS-F6K-PFC3BXL     XXXXXXXXXXX Hw : 1.8
 7    4  WS-X6704-10GE      XXXXXXXXXXX Hw : 2.6
                                        Fw : 12.2(18r)S1
                                        Sw : 12.2(33)SXH2a
                                        Sw1: 8.7(0.22)H2A1
         WS-F6700-CFC       XXXXXXXXXXX Hw : 4.0

Upgrading the ROMMON

You can upgrade all ROMMONs like every other file (i.e. IOS). You just need to have the ROMMON file somewhere accessible; i prefer putting all ROMMONs in bootflash.

6509#dir bootflash:
Directory of bootflash:/

   1  -rwx    12949924  Jun 13 2008 19:02:15 +03:00  s72033-boot-mz.122-33.SXH2a.bin
   2  -rw-      649832  Jun 13 2008 19:03:06 +03:00  c2lc-rm2.srec.122-18r.S1
   4  -rw-      736475  Jun 13 2008 19:44:10 +03:00  c6ksup720-rm2.8-5-2.srec
   5  -rw-      669827  Jun 13 2008 19:58:51 +03:00  c6msfc3-rm2.srec.122-17r.SX5

65536000 bytes total (49792824 bytes free)

c6ksup720-rm2.8-5-2.srec refers to the SUP720 SP ROMMON
c6msfc3-rm2.srec.122-17r.SX5 refers to the MSFC3 RP ROMMON
c2lc-rm2.srec.122-18r.S1 refers to the CEF720 modules ROMMON


NOTE : Always have "term mon" turned on when upgrading, so you can watch the various messages of the upgrade process.

The following is usually the default output, when you haven't upgraded the ROMMON.
The ROMMON from the Gold region is running and regions F1 and F2 do not have valid ROMMONs.

6509#sh rom-monitor slot 1
Region F1: INVALID
Region F2: INVALID
Currently running ROMMON from S (Gold) region

Ok, let's upgrade the ROMMON of module 1 now:

6509#upgrade rom-monitor slot 1 file bootflash:c2lc-rm2.srec.122-18r.S1
Copying bootflash:c2lc-rm2.srec.122-18r.S1 onto bootflash of dfc#1
Copy in progress...CCCCCCCCCC
Jun 13 14:24:29.109: DFC1: ROMMON image upgrade in progress
Jun 13 14:24:29.109: DFC1: ROMMON current update region unknown = 0
Jun 13 14:24:29.125: DFC1: Erasing flash
Jun 13 14:24:30.793: DFC1: Programming flash
Jun 13 14:24:32.285: DFC1: Verifying new image
Jun 13 14:24:32.489: DFC1: ROMMON image upgrade complete
 The card must be reset for this to take effect

This is the output after you have upgraded the ROMMON (in region F1). The ROMMON from the Gold region is still running, but Region F1 has a ROMMON that will be used after you reset the module.

6509#sh rom-monitor slot 1
Region F1: FIRST_RUN, preferred
Region F2: INVALID
Currently running ROMMON from S (Gold) region

We reset the module

6509#hw-module module 1 reset

This is the output after you have upgraded the ROMMON (in region F1) and have reseted the module. The ROMMON from the F1 region is now running.

6509#sh rom-monitor slot 1
Region F1: APPROVED, preferred
Region F2: INVALID
Currently running ROMMON from F1 region

You can also check the logs while the various modules are powered on.

Module in slot 1 is powered on:

00:00:23: %SYS-DFC1-5-RESTART: System restarted --
Cisco IOS Software, c6lc2 Software (c6lc2-SP-M), Version 12.2(33)SXH2a, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Fri 25-Apr-08 08:07 by prod_rel_team
Jun 13 16:17:33.300: DFC1: Currently running ROMMON from F1 region
Jun 13 16:17:33.300: DFC1:  ROMMON upgrade successful

If you upgrade again the ROMMON of this module, region F2 will be used automatically:

6509#sh rom-monitor slot 1
Region F1: APPROVED
Region F2: FIRST_RUN, preferred
Currently running ROMMON from F1 region

PS: You can download (as long as you have the appropriate access) all the ROMMONs from here.

Last Update : 06-Mar-2009

What is the longest time you had a Cisco SR open?

Yesterday i was having a review of all our Cisco cases (open and closed) and i felt disappointed.

I realized that there is one platform that has the most cases (should i say above 70?) during the last 4 years and some of them have lasted for over a year!

What does it mean for me?

• The platform has a difficult (and probably not optimal) architecture
• The platform doesn't have enough engineers to deal with its software
• I'm doing a thorough check of everything on this platform (shouldn't I?)

Regarding the time it takes Cisco to solve many of our cases, here are some possible reasons:

• The tac engineer cannot handle/understand the case (so it gets escalated or moved to another engineer)
• The Developer Team needs to be contacted
  
• A lab reproduction needs to be done
• The case is not considered critical
• The case cannot be replicated easily
• I insist on leaving cases open until the fix gets execute committed in a specific release (past experience made me act this way)
• My company hasn't bought many pieces of this platform (so it's not getting the appropriate attention)

On the other hand, there are some platforms that get immediate attention and Cisco is willing to create a custom engineering image for you, if you're in urgent need.

So, what is the longest time you had a Cisco SR open? Please take some time and vote in the poll to the right.

Also something else that i'm interested in:

1. Have you ever issued a PER (Product Enhancement Request)? If yes, was it actually done and how much time did it took? Did you use any informal way?

2. Cisco says that CCIEs get a better service from Cisco TAC in their cases. Have you seen that?


Update 1 Aug 2008: I have attached the results from the poll. Although the sample is very small, as it seems Cisco is actually slow when solving some of its cases. I guess it has to do with the usual business impact. If you are DT or BT, you're not probably going to wait so much.

Security best practices = How to enable vulnerabilities ?

Today Cisco published a new security advisory regarding SNMPv3 (CVSS Base Score 10).

Successful exploitation of these vulnerabilities could result in the disclosure of sensitive information on a device or allow an attacker to make configuration changes to a vulnerable device that is based on the SNMP configuration.

A few days ago it was SSH's turn (CVSS Base Score 7.8).

Successful exploitation of these vulnerabilities may result in a spurious memory access or, in certain cases, reload the device potentially resulting in a DoS condition.

The usage of both of these protocols is recommended by Cisco in its best practices docs instead of SNMPv2/v1 and telnet respectively.

In an effort to prevent information disclosure or unauthorized access to the data that is transmitted between the administrator and the device, transport input ssh should be used instead of clear-text protocols, such as Telnet and rlogin.
...
SNMPv3 provides secure access to devices by authenticating and optionally encrypting packets over the network. Where supported, SNMPv3 can be used in order to add another layer of security when deploying SNMP.

So what's the catch?

Do people have to meet new vulnerabilities, while trying to make their networks more secure?