rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_green
icon_orange
icon_red
icon_green
icon_red
icon_red
icon_green
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#12679 — FS#17518 — bhs-1a/b-n7

Attached to Project— Network
Maintenance
Beauharnois
Planned
100%
To integrate the latest functional and security patches, we will update this pair of routers on Tuesday, April 19th as of 7am CEST.

The update will be carried out on ISSU.
Comment by OVH - Thursday, 21 April 2016, 18:04PM

We are starting the procedure, operations will be suspended during maintenance.


Comment by OVH - Thursday, 21 April 2016, 18:05PM

We will start by bhs-1a-n7-pcc (vPC role: secondary).


Comment by OVH - Thursday, 21 April 2016, 18:05PM

We will start by isolating switch A.


Comment by OVH - Thursday, 21 April 2016, 18:06PM

Switch A is updated, we are prodding.


Comment by OVH - Thursday, 21 April 2016, 18:06PM

We are isolating switch B.


Comment by OVH - Thursday, 21 April 2016, 18:07PM

A flap occurred during the isolation of switch B, an interruption of a few moments occurred.
Some VMs can be set in read-only.


Comment by OVH - Thursday, 21 April 2016, 18:08PM

Switch B is updated, we are restarting it.


Comment by OVH - Thursday, 21 April 2016, 18:13PM

During isolation of bhs-1b-n7 to prepare the update process, an error occurred.
The two devices are redundant and functionning on virtual PortChannel (vPC). This technology allows two devices to work together and form a single logical entity for all equipment connected to it.
This morning, bhs-1a-n7 was "vPC secondary" and bhs-1b-n7 "vPC primary". We first performed the update on bhs-1a-n7. When we wanted to update bhs-1b-n7, we first cut all links: uplinks (to the OVH network) and downlinks (on switches which are connected hosts and storage). The next step was to cut the links operating vPC: the vPC peer link (which allows to synchronize the states and the traffic that can cross in some cases) and the vPC peer keepalive link (link dedicated to the monitoring of switches between them: messages "hello").
During this stage, the order of execution of the procedure was reversed: the peer link has been disabled before the peer keepalive. Under these conditions, the vPC secondary switch cut all of its ports to avoid a Dual Active scenario. We immediately reactivated the port, but those few seconds outage impacted cutting off access to public paw and communication between storage and hosts which can cause the passage of VMs in read-only.