rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_green
icon_orange
icon_red
icon_green
icon_red
icon_red
icon_green
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#5654 — FS#9573 — N5 - EG/MG

Attached to Project— Network
Incident
Whole Network
CLOSED
100%
Following on from http://status.ovh.co.uk/?do=details&id=5647, we have some instability on the FEXs of the Nexus 5000s.

The Fex disappears and is no longer attached to the switch.

We plan to downgrade the version in order to stabilize the situation.
Date:  Sunday, 27 October 2013, 17:08PM
Reason for closing:  Done
Comment by OVH - Wednesday, 23 October 2013, 15:37PM

We're going to reboot the switches of networks 46.105.106.0/24 and 46.105.107.0/24, one after the other.


Comment by OVH - Wednesday, 23 October 2013, 15:37PM

Done.


Comment by OVH - Wednesday, 23 October 2013, 16:06PM

We're going to reboot a switch of the networks 46.105.108.0/24 and 46.105.109.0/24.


Comment by OVH - Wednesday, 23 October 2013, 17:47PM

done.
We're going to reboot a switch of the networks176.31.228.0/24 et 176.31.229.0/24.


Comment by OVH - Wednesday, 23 October 2013, 17:47PM

done.
We're going to reboot a switch of the networks 176.31.226.0/24 et 176.31.227.0/24.


Comment by OVH - Wednesday, 23 October 2013, 17:52PM

The pair of switches managing 176.31.226 and 227 is unstable again. We're going to move forward the replacement operation that was planned for this evening and do it immediately.
Each of the switches will be shut down and the FEXs (where connected to the servers) will be migrated onto a spare with the same config.


Comment by OVH - Wednesday, 23 October 2013, 18:44PM

The 1st switch of the pair is being replaced. 2 FEXs are offline.


Comment by OVH - Wednesday, 23 October 2013, 18:55PM

We have replaced the 1st n5 on the networks 176.31.226 and 227, all FEXs are up. We are replacing the second.


Comment by OVH - Wednesday, 23 October 2013, 23:59PM

Switches of networks 176.31.226 and 227 were replaced.

We will intervene again tonight starting from 11pm on switches of the following networks in order to make them stable:

* downgrade:
176.31.228
176.31.229
176.31.230
176.31.231
46.105.108
46.105.109

We will also perform an upgrade cold swap on switches of these networks:

176.31.224
176.31.225

Traffic cut-off for 10-15min is expected.


Comment by OVH - Thursday, 24 October 2013, 00:00AM

We will start the intervention on the following network switches:

176.31.230
176.31.231
46.105.108
46.105.109


Comment by OVH - Thursday, 24 October 2013, 00:02AM

Switches of the networks 46.105.108/109 and 176.31.228/229 are stable now, and we didn't notice more issues. We will let them work on the current version.

176.31.230/231 downgrade in progress.

176.31.224/225 upgrade cold swap.


Comment by OVH - Thursday, 24 October 2013, 00:07AM

The mac of fex100 of switch managing the network 176.31.230.0/24 aren't seen anymore. we will restart them electrically.


Comment by OVH - Thursday, 24 October 2013, 00:26AM

Even after the reboot the mac are not recognized.
We are replacing the fex.


Comment by OVH - Thursday, 24 October 2013, 00:27AM

176.31.230/231: downgrade completed but we have a fault on fex100, we will replace them.
176.31.224/225: we noticed again the instability on the new version, we will get back to the old version


Comment by OVH - Thursday, 24 October 2013, 02:12AM

176.31.230/231: we have issues to set the spare fex online
176.31.224/225: switches stable following the downgrade


Comment by OVH - Thursday, 24 October 2013, 02:13AM

176.31.230/231: the second spare was finally the good one and all ports are up.

The situation is back to normal on all switches.