Get webhook notifications whenever Network & Infrastructure creates an incident, updates an incident, resolves an incident or changes a component status.
We need to update the n5548 couple that manages these 2 networks.
A bug en 6.0(2)N2(2) caused some chassis ports to go down and stay down.
Currently 3 FEX have been single homed by the bug. There's no impact on production but the 3 FEXs are no longer redundant.
To prepare for the ISSU update (hitless), we're going to reboot the Nexus module 2 which has its ports down.
Hopefully this will be enough to get the ports back up. If not, we'll need to fully reload the chassis.
Afterwards, we will update to 7.0.5.N1a.
The intervention will begin in a few minutes.
Update(s):
Date: 2015-06-23 12:10:16 UTC Wow!
Update now complete - 3 hours ...
Date: 2015-06-23 12:08:53 UTC Reload okay. I will remount the FEX on this Nexus to reload the 1st one.
Date: 2015-06-23 12:08:27 UTC # reload power-cycle
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y
Date: 2015-06-23 11:49:21 UTC Performing supervisor state verification.
SUCCESS
Supervisor non-disruptive upgrade successful.
Install has been successful.
Date: 2015-06-23 11:49:13 UTC Module 1: Refreshing compact flash and upgrading bios/loader/bootrom/power-seq.
Warning: please do not remove or power off the module at this time.
Note: Power-seq upgrade needs a power-cycle to take into effect.
Use command 'reload power-cycle'
Note: Micro-controller upgrade needs a power-cycle to take into effect.
On success of micro-controller upgrade, SWITCH OFF THE POWER to the system and then, power it up.
Need to do a hard reset or power cycle reload to update the power sequencer.
Date: 2015-06-23 11:48:35 UTC Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
2 yes non-disruptive rolling
3 yes non-disruptive rolling
Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 6.0(2)N2(2) 7.0(5)N1(1a) yes
1 kickstart 6.0(2)N2(2) 7.0(5)N1(1a) yes
1 bios v3.6.0(05/09/2012) v3.6.0(05/09/2012) no
1 power-seq v1.0 v3.0 yes
1 SFP-uC v1.0.0.0 v1.0.0.0 no
2 power-seq v1.0 v1.0 no
3 power-seq v2.0 v2.0 no
1 microcontroller v1.2.0.1 v1.2.0.1 no
Do you want to continue with the installation (y/n)? [n] y
Notifying services about the upgrade.
[####################] 100% -- SUCCESS
Setting boot variables.
[# ] 0%
Date: 2015-06-23 11:48:23 UTC The install is okay on the 1st n5
Module 117: Non-disruptive upgrading.
SUCCESS
Install has been successful.
Software
BIOS: version 3.6.0
loader: version N/A
kickstart: version 7.0(5)N1(1a)
system: version 7.0(5)N1(1a)
Running the ISSU on the 2nd (the fexs are still not linked up - it shouldn't take long)
# install all kickstart bootflash:n5000-uk9-kickstart.7.0.5.N1.1a.bin system bootflash:n5000-uk9.7.0.5.N1.1a.bin
Date: 2015-06-23 11:47:19 UTC the last FEX in progress
Module 115: Non-disruptive upgrading.
SUCCESS
Module 116: Non-disruptive upgrading.
SUCCESS
Module 117: Non-disruptive upgrading.
Date: 2015-06-23 11:47:04 UTC Module 111: Non-disruptive upgrading.
SUCCESS
Module 112: Non-disruptive upgrading.
SUCCESS
Module 113: Non-disruptive upgrading.
Date: 2015-06-23 11:46:57 UTC Pre-loading modules.
SUCCESS
Module 100: Non-disruptive upgrading.
SUCCESS
Module 101: Non-disruptive upgrading.
SUCCESS
Module 102: Non-disruptive upgrading.
SUCCESS
Module 103: Non-disruptive upgrading.
Date: 2015-06-23 11:46:43 UTC finally the 1st FEX is starting to update
Continuing with installation process, please wait.
The login will be disabled until the installation is completed.
Performing supervisor state verification.
SUCCESS
Supervisor non-disruptive upgrade successful.
Pre-loading modules.
SUCCESS
Module 100: Non-disruptive upgrading.
Date: 2015-06-23 11:46:18 UTC yeah!
let's go
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
2 yes non-disruptive rolling
3 yes non-disruptive rolling
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
111 yes non-disruptive rolling
112 yes non-disruptive rolling
113 yes non-disruptive rolling
114 yes non-disruptive rolling
115 yes non-disruptive rolling
116 yes non-disruptive rolling
117 yes non-disruptive rolling
Notifying services about the upgrade.
[####################] 100% -- SUCCESS
Setting boot variables.
[# ] 0%
Date: 2015-06-23 11:45:39 UTC still problems...
# install all kickstart bootflash:n5000-uk9-kickstart.7.0.5.N1.1a.bin system bootflash:n5000-uk9.7.0.5.N1.1a.bin
Pre-upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold).
# wr
[########################################] 100%
Copy complete, now saving to disk (please wait)...
# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y
Date: 2015-06-23 11:41:14 UTC Cutting the FEX 2 port-channel in 2 to reload.
Date: 2015-06-23 11:40:31 UTC next step ... the microcontroller version is out-of-date
Software
BIOS: version 3.6.0
loader: version N/A
kickstart: version 6.0(2)N2(2)
system: version 6.0(2)N2(2)
Power Sequencer Firmware:
Module 1: version v1.0
Module 2: version v1.0
Module 3: version v2.0
Microcontroller Firmware: version v1.2.0.1
SFP uC: Module 1: v0.0.0.0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ça devrait être en v1.0
QSFP uC: Module not detected
BIOS compile time: 05/09/2012
kickstart image file is: bootflash:///n5000-uk9-kickstart.6.0.2.N2.2.bin
kickstart compile time: 10/4/2013 12:00:00 [10/04/2013 21:10:23]
system image file is: bootflash:///n5000-uk9.6.0.2.N2.2.bin
system compile time: 10/4/2013 12:00:00 [10/04/2013 23:29:49]
I need to reload the chassis ... the traffic will be redirected by the 2nd n5
Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 547f.ee1a.8fa8 to 547f.ee1a.8fc7 FOC153255P4
2 002a.6abc.b690 to 002a.6abc.b69f FOC18161R48
3 0000.0000.0000 to 0000.0000.000f FOC15321UJM
(config)#
module up
(config)# no poweroff module 2
(config)#
Date: 2015-06-23 11:35:50 UTC Some ports are in compatibility check failed for speed.
The n5 A is not seeing the same speed as B in certain instances. The traffic is blackholed (another bug fixed in 7.0.5.n1a)
# sh vpc | i fail
101378 Eth100/1/3 up failed Compatibility check failed -
101379 Eth100/1/4 up failed Compatibility check failed -
101380 Eth100/1/5 up failed Compatibility check failed -
101381 Eth100/1/6 up failed Compatibility check failed -
101386 Eth100/1/11 up failed Compatibility check failed -
101396 Eth100/1/21 up failed Compatibility check failed -
101399 Eth100/1/24 up failed Compatibility check failed -
101410 Eth100/1/35 up failed Compatibility check failed -
101411 Eth100/1/36 up failed Compatibility check failed -
101417 Eth100/1/42 up failed Compatibility check failed -
101420 Eth100/1/45 up failed Compatibility check failed -
102405 Eth101/1/6 up failed Compatibility check failed -
102419 Eth101/1/20 up failed Compatibility check failed -
102432 Eth101/1/33 up failed Compatibility check failed -
102436 Eth101/1/37 up failed Compatibility check failed -
102440 Eth101/1/41 up failed Compatibility check failed -
102445 Eth101/1/46 up failed Compatibility check failed -
102447 Eth101/1/48 up failed Compatibility check failed -
103430 Eth102/1/7 up failed Compatibility check failed -
103433 Eth102/1/10 up failed Compatibility check failed -
103442 Eth102/1/19 up failed Compatibility check failed -
103449 Eth102/1/26 up failed Compatibility check failed -
103451 Eth102/1/28 up failed Compatibility check failed -
103455 Eth102/1/32 up failed Compatibility check failed -
103462 Eth102/1/39 up failed Compatibility check failed -
104450 Eth103/1/3 up failed Compatibility check failed -
104455 Eth103/1/8 up failed Compatibility check failed -
104456 Eth103/1/9 up failed Compatibility check failed -
104460 Eth103/1/13 up failed Compatibility check failed -
104478 Eth103/1/31 up failed Compatibility check failed -
104481 Eth103/1/34 up failed Compatibility check failed -
1044 Eth103/1/35 up failed Compatibility check failed -
104484 Eth103/1/37 up failed Compatibility check failed -
104488 Eth103/1/41 up failed Compatibility check failed -
105473 Eth104/1/2 up failed Compatibility check failed -
105479 Eth104/1/8 up failed Compatibility check failed -
105486 Eth104/1/15 up failed Compatibility check failed -
105499 Eth104/1/28 up failed Compatibility check failed -
105501 Eth104/1/30 up failed Compatibility check failed -
105507 Eth104/1/36 up failed Compatibility check failed -
we performed a shut/no shut on the impacted ports to fix them