OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#16837 — sbg2-20a/b-n56
Scheduled Maintenance Report for Network & Infrastructure
Completed
We will update this Nexus pair on March 3rd as of 11pm.

The update is expected to be disruptive for the Nexux pair, because the pair is only constituted by fex2348, on which we have to upgrade the PHY (firmware managing interfaces) fex2348 implying a fex reload (2-3min of down).
If we need to reload all fex, it is better on disruptive ISSU, which is in charge of preloading the new image in the fex, and reloading them one by one.

Update(s):

Date: 2016-03-04 03:46:32 UTC
20A is up

nous continuons de reload les fexs
sbg2-20b-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Online N2K-C2348TQ-10GE FOC1930R15H
101 fex101 Online N2K-C2348TQ-10GE FOC1930R3YY
102 fex102 Online N2K-C2348TQ-10GE FOC1939R0Z5
103 fex103 Online N2K-C2348TQ-10GE FOC1941R0PN
104 fex104 Offline N2K-C2348TQ-10GE FOC1939R1ZA
105 fex105 Offline N2K-C2348TQ-10GE FOC1939R0Z3
106 fex106 Offline N2K-C2348TQ-10GE FOC1935R0GT
107 fex107 Offline N2K-C2348TQ-10GE FOC1935R2LL
108 fex108 Offline N2K-C2348TQ-10GE FOC1941R0NQ
109 fex109 Offline Sequence N2K-C2348TQ-10GE FOC1941R0H0
110 fex110 Offline Sequence N2K-C2348TQ-10GE FOC1944R0D9
112 fex112 Offline Sequence N2K-C2348TQ-10GE FOC1944R2NP
113 fex113 Online N2K-C2348TQ-10GE FOC1944R03X
114 fex114 Online N2K-C2348TQ-10GE FOC1944R2FC
115 fex115 Online N2K-C2348TQ-10GE FOC1944R0BW
116 fex116 Online N2K-C2348TQ-10GE FOC1944R05G
117 fex117 Online N2K-C2348TQ-10GE FOC1944R04J
118 fex118 Online N2K-C2348TQ-10GE FOC1944R0B0
119 fex119 Online N2K-C2348TQ-10GE FOC1941R0GB
120 fex120 Online N2K-C2348TQ-10GE FOC1944R0AQ

Date: 2016-03-04 03:46:15 UTC
We reloaded the fex to update PHY

bg2-20b-n56#
sbg2-20b-n56# reload fex 101
WARNING: This command will reboot FEX 101
Do you want to continue? (y/n) [n] y
sbg2-20b-n56#
sbg2-20b-n56#
sbg2-20b-n56#
sbg2-20b-n56#
sbg2-20b-n56#
sbg2-20b-n56#
sbg2-20b-n56#
sbg2-20b-n56# reload fex 102
WARNING: This command will reboot FEX 102
Do you want to continue? (y/n) [n] y
sbg2-20b-n56#

Date: 2016-03-04 03:45:48 UTC
reload of A is in progress

sbg2-20a-n56# wr
[########################################] 100%d
Copy complete, now saving to disk (please wait)...
sbg2-20a-n56# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y
[ 6873.695570] Shutdown Ports..
[ 6873.729872] writing reset reason 9,
[ 6873.774594] Resetting board

(c) Copyright 2014, Cisco Systems.
N5K-C5624Q BIOS v.1.0.8, Wed 10/29/2014


Booting kickstart image: bootflash:/n6000-uk9-kickstart.7.1.3.N1.2a.bin

...............................................................................
....................................Image verification OK

Booting kernel
INIT: devmemfd:0x7fab4dd8

Date: 2016-03-04 03:45:15 UTC
20A finalized their ISSU without problems, but B crashed later with no apparent reason.
Traffic was still forwarded by 20A until some point, we started losing IPs,it came back then we had a total blow up.

We have updgraded 20B... fex are attached above it.

We collect traces / logs then reload 20A


Date: 2016-03-04 03:42:17 UTC
Module 108: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 109: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 110: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 112: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 113: Non-disruptive upgrading.
[# ] 0%


Date: 2016-03-04 03:42:09 UTC
2016-03-04T02:02:55+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 01:02:55.262 CET: %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-107 On-line
2016-03-04T02:02:59+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 01:02:59.270 CET: %PFMA-2-FEX_STATUS: Fex 107 is online

Date: 2016-03-04 03:42:00 UTC
The sw has reloaded

il precharge les images sur les fex
2016-03-04T01:43:33+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:43:31.333 CET: %SATCTRL-FEX101-2-SATCTRL_IMAGE: FEX101 Image update in progress.
2016-03-04T01:43:43+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:43:41.343 CET: %SATCTRL-FEX102-2-SATCTRL_IMAGE: FEX102 Image update in progress.
2016-03-04T01:43:53+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:43:51.353 CET: %SATCTRL-FEX103-2-SATCTRL_IMAGE: FEX103 Image update in progress.
2016-03-04T01:44:04+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:44:02.363 CET: %SATCTRL-FEX104-2-SATCTRL_IMAGE: FEX104 Image update in progress.
2016-03-04T01:44:12+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:44:12.050 CET: %SATCTRL-FEX105-2-SATCTRL_IMAGE: FEX105 Image update in progress.
2016-03-04T01:44:22+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:44:22.060 CET: %SATCTRL-FEX106-2-SATCTRL_IMAGE: FEX106 Image update in progress.
2016-03-04T01:44:32+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:44:32.070 CET: %SATCTRL-FEX107-2-SATCTRL_IMAGE: FEX107 Image update in progress.
2016-03-04T01:44:42+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:44:42.080 CET: %SATCTRL-FEX108-2-SATCTRL_IMAGE: FEX108 Image update in progress.
2016-03-04T01:44:52+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:44:52.107 CET: %SATCTRL-FEX109-2-SATCTRL_IMAGE: FEX109 Image update in progress.
2016-03-04T01:45:02+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:45:02.107 CET: %SATCTRL-FEX110-2-SATCTRL_IMAGE: FEX110 Image update in progress.
2016-03-04T01:45:12+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:45:12.107 CET: %SATCTRL-FEX112-2-SATCTRL_IMAGE: FEX112 Image update in progress.
2016-03-04T01:45:22+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:45:22.107 CET: %SATCTRL-FEX113-2-SATCTRL_IMAGE: FEX113 Image update in progress.
2016-03-04T01:45:34+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:45:34.107 CET: %SATCTRL-FEX114-2-SATCTRL_IMAGE: FEX114 Image update in progress.
2016-03-04T01:45:42+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:45:42.107 CET: %SATCTRL-FEX115-2-SATCTRL_IMAGE: FEX115 Image update in progress.
2016-03-04T01:45:52+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:45:52.107 CET: %SATCTRL-FEX116-2-SATCTRL_IMAGE: FEX116 Image update in progress.
2016-03-04T01:46:02+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:46:02.109 CET: %SATCTRL-FEX117-2-SATCTRL_IMAGE: FEX117 Image update in progress.
2016-03-04T01:46:12+02:00 10.21.151.55 sbg2-20a-n56.fr.eu ba [local7.crit] === : 2016 Mar 4 00:46:12.119 CET: %SATCTRL-FEX118-2-SATCTRL_IMAGE: FEX118 Image update in progress.

Date: 2016-03-04 03:41:41 UTC
Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.

Date: 2016-03-04 03:41:26 UTC
Here we go:

sbg2-20a-n56# install all system n6000-uk9.7.1.3.N1.2a.bin kickstart n6000-uk9-kickstart.7.1.3.N1.2a.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.1.3.N1.2a.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.1.3.N1.2a.bin for boot variable \"system\".

Date: 2016-03-04 03:41:02 UTC
We will start maintenance in a few minutes.
We decided to make non-disruptive + reload the fex manually.
Posted Mar 01, 2016 - 18:55 UTC