OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#15537 — gra1-30ab-n56 - nxos update
Scheduled Maintenance Report for Network & Infrastructure
Completed
Hello,

Today we will update a couple of n56 to the last nxos version which fixed many bugs related to fex 2348 and improves the stability generally.
The update will be done at the night from 26th to 27th November 2015 starting from 10:30 PM.

No impact is expected, the upgrade will be done ISSU (hitless way)

The concerned IP blocks are:
149.202.74.0/24
149.202.75.0/24
149.202.76.0/24
149.202.77.0/24

Update(s):

Date: 2015-11-26 23:59:42 UTC
The fex is kill dead.. we will replace it by a spare.

Date: 2015-11-26 23:59:20 UTC
All fex are up except fex108, we rebooted it HARD, it doesn't come back (on any of the nexus).

Date: 2015-11-26 23:58:18 UTC
We will UP the fex on A.

Date: 2015-11-26 23:57:34 UTC
A has crashed with the same way as B during ISSU
gra1-30a-n56# sh system reset-reason
----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) ---
1) At 435386 usecs after Fri Nov 27 00:16:15 2015
Reason: Reset triggered due to HA policy of Reset
Service: ipqosmgr hap reset
Version: 7.1(3)N1(1)


Date: 2015-11-26 23:56:56 UTC
All fex are online

gra1-30b-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Online N2K-C2348TQ-10GE FOC1926R07E
101 fex101 Online N2K-C2248TP-E-1GE FOX1901G31J
102 fex102 Online N2K-C2348TQ-10GE FOC1930R418
103 fex103 Online N2K-C2348TQ-10GE FOC1930R3YG
104 fex104 Online N2K-C2348TQ-10GE FOC1930R52E
105 fex105 Online N2K-C2348TQ-10GE FOC1930R41N
106 fex106 Online N2K-C2348TQ-10GE FOC1928R1JJ
107 fex107 Online N2K-C2232TM-E-10GE SSI191802JF
109 fex109 Online N2K-C2348TQ-10GE FOC1930R3ZW
110 fex110 Online N2K-C2348TQ-10GE FOC1930R13B
111 fex111 Online N2K-C2248TP-E-1GE FOX1923GMSH
112 fex112 Online N2K-C2348TQ-10GE FOC1930R52W
113 fex113 Online N2K-C2348TQ-10GE FOC1930R3Z5
114 fex114 Online N2K-C2348TQ-10GE FOC1930R15T
115 fex115 Online N2K-C2232TM-E-10GE SSI1918027M
116 fex116 Online N2K-C2232TM-E-10GE SSI1915014D
117 fex117 Online N2K-C2348TQ-10GE FOC1930R16N
118 fex118 Online N2K-C2348TQ-10GE FOC1930R17V
119 fex119 Online N2K-C2348TQ-10GE FOC1930R41B
120 fex120 Online N2K-C2348TQ-10GE FOC1930R3ZY


We will update gra1-30a, port-channels are suspended on A to the fex.
Servers are UP via B only.

Date: 2015-11-26 23:55:36 UTC
ra1-30b-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Online N2K-C2348TQ-10GE FOC1926R07E
101 fex101 Online N2K-C2248TP-E-1GE FOX1901G31J
102 fex102 Online N2K-C2348TQ-10GE FOC1930R418
103 fex103 Online N2K-C2348TQ-10GE FOC1930R3YG
104 fex104 Online N2K-C2348TQ-10GE FOC1930R52E
105 fex105 Online N2K-C2348TQ-10GE FOC1930R41N
106 fex106 Online N2K-C2348TQ-10GE FOC1928R1JJ
107 fex107 Offline N2K-C2232TM-E-10GE SSI191802JF
109 fex109 Image Download N2K-C2348TQ-10GE FOC1930R3ZW
110 fex110 Online N2K-C2348TQ-10GE FOC1930R13B
111 fex111 Online N2K-C2248TP-E-1GE FOX1923GMSH
112 fex112 Offline N2K-C2348TQ-10GE FOC1930R52W
113 fex113 Offline N2K-C2348TQ-10GE FOC1930R3Z5
114 fex114 Online N2K-C2348TQ-10GE FOC1930R15T
115 fex115 Online N2K-C2232TM-E-10GE SSI1918027M
116 fex116 Online N2K-C2232TM-E-10GE SSI1915014D
117 fex117 Online N2K-C2348TQ-10GE FOC1930R16N
118 fex118 Offline N2K-C2348TQ-10GE FOC1930R17V
119 fex119 Online N2K-C2348TQ-10GE FOC1930R41B
120 fex120 Online N2K-C2348TQ-10GE FOC1930R3ZY

Date: 2015-11-26 23:55:28 UTC
The first fex are coming back

gra1-30b-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Offline N2K-C2348TQ-10GE FOC1926R07E
101 fex101 Image Download N2K-C2248TP-E-1GE FOX1901G31J
102 fex102 Offline N2K-C2348TQ-10GE FOC1930R418
103 fex103 Online N2K-C2348TQ-10GE FOC1930R3YG
104 fex104 Online N2K-C2348TQ-10GE FOC1930R52E
105 fex105 Offline N2K-C2348TQ-10GE FOC1930R41N
106 fex106 Offline N2K-C2348TQ-10GE FOC1928R1JJ
107 fex107 Image Download N2K-C2232TM-E-10GE SSI191802JF
109 fex109 Offline N2K-C2348TQ-10GE FOC1930R3ZW
110 fex110 Image Download N2K-C2348TQ-10GE FOC1930R13B
111 fex111 Image Download N2K-C2248TP-E-1GE FOX1923GMSH
112 fex112 Image Download N2K-C2348TQ-10GE FOC1930R52W
113 fex113 Image Download N2K-C2348TQ-10GE FOC1930R3Z5
114 fex114 Image Download N2K-C2348TQ-10GE FOC1930R15T
115 fex115 Offline N2K-C2232TM-E-10GE SSI1918027M
116 fex116 Image Download N2K-C2232TM-E-10GE SSI1915014D
117 fex117 Offline N2K-C2348TQ-10GE FOC1930R16N
118 fex118 Image Download N2K-C2348TQ-10GE FOC1930R17V
119 fex119 Offline N2K-C2348TQ-10GE FOC1930R41B
120 fex120 Image Download N2K-C2348TQ-10GE FOC1930R3ZY

Date: 2015-11-26 23:55:04 UTC
A was reloaded, fex didn't come back.

gra1-30b-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Image Download N2K-C2348TQ-10GE FOC1926R07E
101 fex101 Connected N2K-C2248TP-E-1GE FOX1901G31J
102 fex102 Connected N2K-C2348TQ-10GE FOC1930R418
103 fex103 Image Download N2K-C2348TQ-10GE FOC1930R3YG
104 fex104 Image Download N2K-C2348TQ-10GE FOC1930R52E
105 fex105 Image Download N2K-C2348TQ-10GE FOC1930R41N
106 fex106 Image Download N2K-C2348TQ-10GE FOC1928R1JJ
109 fex109 Connected N2K-C2348TQ-10GE FOC1930R3ZW
111 fex111 Connected N2K-C2248TP-E-1GE FOX1923GMSH
112 fex112 Connected N2K-C2348TQ-10GE FOC1930R52W
114 fex114 Connected N2K-C2348TQ-10GE FOC1930R15T
117 fex117 Connected N2K-C2348TQ-10GE FOC1930R16N
118 fex118 AA Version Mismatch N2K-C2348TQ-10GE FOC1930R17V
119 fex119 Connected N2K-C2348TQ-10GE FOC1930R41B
--- -------- Connected N2K-C2348TQ-10GE FOC1930R3Z5
--- -------- Connected N2K-C2348TQ-10GE FOC1930R13B
--- -------- Connected N2K-C2348TQ-10GE FOC1930R3ZY
--- -------- Connected N2K-C2232TM-E-10GE SSI1915014D
--- -------- Connected N2K-C2232TM-E-10GE SSI1918027M
--- -------- Connected N2K-C2232TM-E-10GE SSI191802JF

Each FEX est went (or will go) to AA-Version Mismatch and will be updated to 7.1.3

Date: 2015-11-26 23:53:55 UTC
New crash case HA policy of Reset: Service: ipqosmgr hap reset

ra1-30b-n56# sh system reset-reason
----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) ---
1) At 288287 usecs after Thu Nov 26 23:17:02 2015
Reason: Reset triggered due to HA policy of Reset
Service: ipqosmgr hap reset
Version: 7.1(3)N1(1)

Date: 2015-11-26 23:53:17 UTC
hum.. it didn't go as expected
During the reboot of B, fex went offline on A

gra1-30a-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Offline N2K-C2348TQ-10GE FOC1926R07E
101 fex101 Online N2K-C2248TP-E-1GE FOX1901G31J
102 fex102 Offline N2K-C2348TQ-10GE FOC1930R418
103 fex103 Offline N2K-C2348TQ-10GE FOC1930R3YG
104 fex104 Offline N2K-C2348TQ-10GE FOC1930R52E
105 fex105 Offline N2K-C2348TQ-10GE FOC1930R41N
106 fex106 Offline Sequence N2K-C2348TQ-10GE FOC1928R1JJ
107 fex107 Online N2K-C2232TM-E-10GE SSI191802JF
108 fex108 Offline N2K-C2232TM-E-10GE SSI190301PS
109 fex109 Offline Sequence N2K-C2348TQ-10GE FOC1930R3ZW
110 fex110 Offline Sequence N2K-C2348TQ-10GE FOC1930R13B
111 fex111 Online N2K-C2248TP-E-1GE FOX1923GMSH
112 fex112 Offline Sequence N2K-C2348TQ-10GE FOC1930R52W
113 fex113 Offline Sequence N2K-C2348TQ-10GE FOC1930R3Z5
114 fex114 Offline Sequence N2K-C2348TQ-10GE FOC1930R15T
115 fex115 Online N2K-C2232TM-E-10GE SSI1918027M
116 fex116 Offline Sequence N2K-C2232TM-E-10GE SSI1915014D
117 fex117 Offline Sequence N2K-C2348TQ-10GE FOC1930R16N
118 fex118 Offline Sequence N2K-C2348TQ-10GE FOC1930R17V
119 fex119 Offline Sequence N2K-C2348TQ-10GE FOC1930R41B
120 fex120 Offline Sequence N2K-C2348TQ-10GE FOC1930R3ZY

Date: 2015-11-26 23:51:56 UTC
Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.

Performing runtime checks.
[####################] 100% -- SUCCESS

Notifying services about the upgrade.
[####################] 100% -- SUCCESS

Setting boot variables.
[####################] 100% -- SUCCESS

Performing configuration copy.
[####################] 100% -- SUCCESS

Module 1: Refreshing compact flash and upgrading bios/loader/bootrom/power-seq.
Warning: please do not remove or power off the module at this time.
Note: Power-seq upgrade needs a power-cycle to take into effect.
On success of power-seq upgrade, SWITCH OFF THE POWER to the system and then, power it up.
Note: Micro-controller upgrade needs a power-cycle to take into effect.
On success of micro-controller upgrade, SWITCH OFF THE POWER to the system and then, power it up.
[####################] 100% -- SUCCESS

Upgrade can no longer be aborted, any failure will result in a disruptive upgrade.

Requesting Line Cards to stop communication.
[####################] 100% -- SUCCESS

Requesting Sup Apps to stop communication.
[####################] 100% -- SUCCESS

Freeing memory in the file system.
[####################] 100% -- SUCCESS

Loading images into memory.
[####################] 100% -- SUCCESS

Saving supervisor runtime state.

Date: 2015-11-26 23:51:38 UTC
We started the intervention.

Date: 2015-11-26 23:51:16 UTC
Just Rolling to the new version

gra1-30a-n56# sh install all impact kickstart n6000-uk9-kickstart.7.1.3.N1.1.bin system n6000-uk9.7.1.3.N1.1.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.1.3.N1.1.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.1.3.N1.1.bin for boot variable \"system\".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Extracting \"system\" version from image bootflash:/n6000-uk9.7.1.3.N1.1.bin.
[####################] 100% -- SUCCESS

Extracting \"kickstart\" version from image bootflash:/n6000-uk9-kickstart.7.1.3.N1.1.bin.
[####################] 100% -- SUCCESS

Extracting \"bios\" version from image bootflash:/n6000-uk9.7.1.3.N1.1.bin.
[####################] 100% -- SUCCESS

Extracting \"fex4\" version from image bootflash:/n6000-uk9.7.1.3.N1.1.bin.
[####################] 100% -- SUCCESS

Extracting \"fexth\" version from image bootflash:/n6000-uk9.7.1.3.N1.1.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS



Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
2 yes non-disruptive rolling
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
111 yes non-disruptive rolling
112 yes non-disruptive rolling
113 yes non-disruptive rolling
114 yes non-disruptive rolling
115 yes non-disruptive rolling
116 yes non-disruptive rolling
117 yes non-disruptive rolling
118 yes non-disruptive rolling
119 yes non-disruptive rolling
120 yes non-disruptive rolling



Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 7.1(2)N1(1) 7.1(3)N1(1) yes
1 kickstart 7.1(2)N1(1) 7.1(3)N1(1) yes
1 bios v2.1.1(07/10/2014) v2.1.2(07/16/2014) yes
1 power-seq v4.0 v4.0 no
1 fabric-power-seq v4.0 v4.0 no
2 power-seq v4.0 v4.0 no
100 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
101 fexth 7.1(2)N1(1) 7.1(3)N1(1) yes
102 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
103 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
104 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
105 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
106 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
107 fexth 7.1(2)N1(1) 7.1(3)N1(1) yes
108 fexth 7.1(2)N1(1) 7.1(3)N1(1) yes
109 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
110 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
111 fexth 7.1(2)N1(1) 7.1(3)N1(1) yes
112 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
113 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
114 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
115 fexth 7.1(2)N1(1) 7.1(3)N1(1) yes
116 fexth 7.1(2)N1(1) 7.1(3)N1(1) yes
117 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
118 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
119 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
120 fex4 7.1(2)N1(1) 7.1(3)N1(1) yes
1 microcontroller v0.0.0.15 v0.0.0.15 no
Posted Nov 26, 2015 - 23:21 UTC