OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#5096 — Updating of the pCC
Scheduled Maintenance Report for Network & Infrastructure
Completed
We will update the software on the network equipment
of the pCC.

This will allow us to test the good (or the bad)
functioning of the network. We need to have still some
adjustments to add.

We will start with the core of the network.

Update(s):

Date: 2011-02-07 14:50:45 UTC
We are going to test another method on the remaining double-N5.

Date: 2011-02-03 10:31:24 UTC
We have the method:
as each host is connected to two networks A and B. and each network is
in a double-attach. We interrupt port A from the server and we update
the double attach A. First all, the 1st N5 A then when it is up, the 2nd
N5 A. once the first double-N5 is updated, the port is reactive.
and we can do the same operation on the port B. Thus, there is no
interruption in service pCC.



Date: 2011-02-02 17:46:46 UTC
Shit.

The 2nd pair N5 was updated with lots of breakdowns. The N5 with FEX does not work now.

Well, we'll do it differently with the 3rd pair. we cut all ports on the 2 N5. Then we will update them.

Date: 2011-02-02 15:44:02 UTC
UP.

Not terrible.

Basically, the Nexus 5000 with the FEX are not ISSU in dual attach.
The N5 which is updated with the latest version can not update
the FEX with this version because the other N5 has the precedent
version.

We'll try an update with a single attach. For this we
will cut one of two sides and do the update of the remaining running
side. We'll see if it is ISSU.

Date: 2011-02-02 15:39:31 UTC
We did not finish the boot of FEX. The update does not function.
We will update the B side.

2011 Feb 2 15:55:50 pcc-10a-n5 %VPC-2-VPC_ISSU_START: Peer vPC switch ISSU start, locking configuration
2011 Feb 2 15:59:25 pcc-10a-n5 %VPC-2-VPC_ISSU_END: Peer vPC switch ISSU end, unlocking configuration
2011 Feb 2 15:59:31 pcc-10a-n5 %VPC-2-PEER_KEEP_ALIVE_RECV_FAIL: In domain 156, VPC peer keep-alive receive has failed
2011 Feb 2 16:06:17 pcc-10a-n5 %SATCTRL-FEX109-2-SATCTRL: FEX-109 Module 1: Cold boot
2011 Feb 2 16:06:32 pcc-10a-n5 %PFMA-2-FEX_STATUS: Fex 109 is online
2011 Feb 2 16:06:32 pcc-10a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-109 On-line
2011 Feb 2 16:06:36 pcc-10a-n5 %PFMA-2-FEX_STATUS: Fex 109 is online
2011 Feb 2 16:05:45 pcc-10a-n5 %SATCTRL-FEX106-2-SATCTRL: FEX-106 Module 1: Cold boot
2011 Feb 2 16:07:01 pcc-10a-n5 %PFMA-2-FEX_STATUS: Fex 106 is online
2011 Feb 2 16:07:01 pcc-10a-n5 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-106 On-line
2011 Feb 2 16:07:03 pcc-10a-n5 %PFMA-2-FEX_STATUS: Fex 106 is online
[...]


Ports interrupted.

The servers have switched to the second network card with an interruption of 60 seconds.


Date: 2011-02-02 13:41:03 UTC
2011 Feb 2 13:00:05 pcc-10b-n5 %VPC-2-VPC_ISSU_START: Peer vPC switch ISSU start, locking configuration
2011 Feb 2 13:02:49 pcc-10b-n5 %SATCTRL-FEX100-2-SATCTRL_IMAGE: FEX100 Image update in progress.
2011 Feb 2 13:02:59 pcc-10b-n5 %SATCTRL-FEX101-2-SATCTRL_IMAGE: FEX101 Image update in progress.
2011 Feb 2 13:03:09 pcc-10b-n5 %SATCTRL-FEX102-2-SATCTRL_IMAGE: FEX102 Image update in progress.
2011 Feb 2 13:03:18 pcc-10b-n5 %VPC-2-VPC_ISSU_END: Peer vPC switch ISSU end, unlocking configuration
Feb 2 13:04:51 pcc-10b-n5.routers.ovh.net : 2011 Feb 02 13:04:28 CET: %SATCTRL-FEX100-2-SATCTRL_IMAGE: FEX100 Image update failed [/isan/plugin_img/fexth.bin]: File transfer error
Feb 2 13:05:13 pcc-10b-n5.routers.ovh.net : 2011 Feb 02 13:04:50 CET: %SATCTRL-FEX102-2-SATCTRL_IMAGE: FEX102 Image update failed [/isan/plugin_img/fexth.bin]: File transfer error
Feb 2 13:05:21 pcc-10b-n5.routers.ovh.net : 2011 Feb 02 13:04:58 CET: %SATCTRL-FEX101-2-SATCTRL_IMAGE: FEX101 Image update failed [/isan/plugin_img/fexth.bin]: File transfer error

Date: 2011-02-02 13:40:48 UTC
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
111 yes non-disruptive rolling



Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------- ---------------------- ---------------------- ------------
1 system 5.0(2)N1(1) 5.0(2)N2(1) yes
1 kickstart 5.0(2)N1(1) 5.0(2)N2(1) yes
1 bios v1.3.0(09/08/09) v1.3.0(09/08/09) no
100 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
101 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
102 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
103 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
104 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
105 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
106 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
107 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
108 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
109 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
110 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
111 fexth 5.0(2)N1(1) 5.0(2)N2(1) yes
1 power-seq v1.2 v1.2 no

Date: 2011-02-02 12:03:58 UTC
Done.

We will pitch in the racks:
A Side

Date: 2011-02-02 12:02:34 UTC
The A side of the core was finished.

We will restart the B side of the core.

Date: 2011-02-02 12:01:50 UTC
Mod Sw Hw
--- -------------- ------
1 5.1(1) 1.0
2 5.1(1) 1.0
5 5.1(1) 1.8
6 5.1(2) 1.8

\"Switching over onto standby\".

64 bytes from pcc-1a-n7.routers.ovh.net (178.33.123.249): icmp_seq=50 ttl=250 time=4.28 ms
64 bytes from pcc-1a-n7.routers.ovh.net (178.33.123.249): icmp_seq=51 ttl=250 time=4.27 ms
64 bytes from pcc-1a-n7.routers.ovh.net (178.33.123.249): icmp_seq=52 ttl=250 time=4.25 ms

64 bytes from pcc-1a-n7.routers.ovh.net (178.33.123.249): icmp_seq=55 ttl=250 time=1008 ms
64 bytes from pcc-1a-n7.routers.ovh.net (178.33.123.249): icmp_seq=56 ttl=250 time=475 ms
64 bytes from pcc-1a-n7.routers.ovh.net (178.33.123.249): icmp_seq=57 ttl=250 time=4.38 ms

et pas de coupure dans le service

Mod Sw Hw
--- -------------- ------
1 5.1(2) 1.0
2 5.1(2) 1.0
5 5.1(2) 1.8
6 5.1(2) 1.8

Date: 2011-02-02 12:01:38 UTC
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive rolling
2 yes non-disruptive rolling
5 yes non-disruptive reset
6 yes non-disruptive reset



Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
1 lc-d-n7k 5.1(1) 5.1(2) yes
1 lc-d-n7k 5.1(1) 5.1(2) yes
1 bios v1.10.13(03/15/10): v1.10.13(03/15/10) v1.10.14(04/02/10) yes
2 lc-d-n7k 5.1(1) 5.1(2) yes
2 lc-d-n7k 5.1(1) 5.1(2) yes
2 bios v1.10.13(03/15/10): v1.10.13(03/15/10) v1.10.14(04/02/10) yes
5 system 5.1(1) 5.1(2) yes
5 kickstart 5.1(1) 5.1(2) yes
5 bios v3.22.0(02/20/10): v3.22.0(02/20/10) v3.22.0(02/20/10) no
5 cmp 5.1(1) 5.1(1) no
5 cmp-bios 02.01.05 02.01.05 no
6 system 5.1(1) 5.1(2) yes
6 kickstart 5.1(1) 5.1(2) yes
6 bios v3.22.0(02/20/10): v3.22.0(02/20/10) v3.22.0(02/20/10) no
6 cmp 5.1(1) 5.1(1) no
6 cmp-bios 02.01.05 02.01.05 no

Date: 2011-02-02 12:01:25 UTC
The A side is ongoing...
Posted Feb 02, 2011 - 12:00 UTC