OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#9573 — N5 - EG/MG
Incident
Report for Network & Infrastructure
Resolved
Following on from http://status.ovh.co.uk/?do=details&id=5647, we have some instability on the FEXs of the Nexus 5000s.
The Fex disappears and is no longer attached to the switch.
We plan to downgrade the version in order to stabilize the situation.
Update(s):
Date: 2013-10-24 00:13:13 UTC
176.31.230/231: the second spare was finally the good one and all ports are up.
The situation is back to normal on all switches.
Date: 2013-10-24 00:12:05 UTC
176.31.230/231: we have issues to set the spare fex online
176.31.224/225: switches stable following the downgrade
Date: 2013-10-23 22:27:59 UTC
176.31.230/231: downgrade completed but we have a fault on fex100, we will replace them.
176.31.224/225: we noticed again the instability on the new version, we will get back to the old version
Date: 2013-10-23 22:26:28 UTC
Even after the reboot the mac are not recognized.
We are replacing the fex.
Date: 2013-10-23 22:07:22 UTC
The mac of fex100 of switch managing the network 176.31.230.0/24 aren't seen anymore. we will restart them electrically.
Date: 2013-10-23 22:02:41 UTC
Switches of the networks 46.105.108/109 and 176.31.228/229 are stable now, and we didn't notice more issues. We will let them work on the current version.
176.31.230/231 downgrade in progress.
176.31.224/225 upgrade cold swap.
Date: 2013-10-23 22:00:23 UTC
We will start the intervention on the following network switches:
176.31.230
176.31.231
46.105.108
46.105.109
Date: 2013-10-23 21:59:19 UTC
Switches of networks 176.31.226 and 227 were replaced.
We will intervene again tonight starting from 11pm on switches of the following networks in order to make them stable:
* downgrade:
176.31.228
176.31.229
176.31.230
176.31.231
46.105.108
46.105.109
We will also perform an upgrade cold swap on switches of these networks:
176.31.224
176.31.225
Traffic cut-off for 10-15min is expected.
Date: 2013-10-23 16:55:20 UTC
We have replaced the 1st n5 on the networks 176.31.226 and 227, all FEXs are up. We are replacing the second.
Date: 2013-10-23 16:44:39 UTC
The 1st switch of the pair is being replaced. 2 FEXs are offline.
Date: 2013-10-23 15:52:27 UTC
The pair of switches managing 176.31.226 and 227 is unstable again. We're going to move forward the replacement operation that was planned for this evening and do it immediately.
Each of the switches will be shut down and the FEXs (where connected to the servers) will be migrated onto a spare with the same config.
Date: 2013-10-23 15:47:49 UTC
done.
We're going to reboot a switch of the networks 176.31.226.0/24 et 176.31.227.0/24.
Date: 2013-10-23 15:47:09 UTC
done.
We're going to reboot a switch of the networks176.31.228.0/24 et 176.31.229.0/24.
Date: 2013-10-23 14:06:41 UTC
We're going to reboot a switch of the networks 46.105.108.0/24 and 46.105.109.0/24.
Date: 2013-10-23 13:37:24 UTC
Done.
Date: 2013-10-23 13:37:16 UTC
We're going to reboot the switches of networks 46.105.106.0/24 and 46.105.107.0/24, one after the other.
The Fex disappears and is no longer attached to the switch.
We plan to downgrade the version in order to stabilize the situation.
Update(s):
Date: 2013-10-24 00:13:13 UTC
176.31.230/231: the second spare was finally the good one and all ports are up.
The situation is back to normal on all switches.
Date: 2013-10-24 00:12:05 UTC
176.31.230/231: we have issues to set the spare fex online
176.31.224/225: switches stable following the downgrade
Date: 2013-10-23 22:27:59 UTC
176.31.230/231: downgrade completed but we have a fault on fex100, we will replace them.
176.31.224/225: we noticed again the instability on the new version, we will get back to the old version
Date: 2013-10-23 22:26:28 UTC
Even after the reboot the mac are not recognized.
We are replacing the fex.
Date: 2013-10-23 22:07:22 UTC
The mac of fex100 of switch managing the network 176.31.230.0/24 aren't seen anymore. we will restart them electrically.
Date: 2013-10-23 22:02:41 UTC
Switches of the networks 46.105.108/109 and 176.31.228/229 are stable now, and we didn't notice more issues. We will let them work on the current version.
176.31.230/231 downgrade in progress.
176.31.224/225 upgrade cold swap.
Date: 2013-10-23 22:00:23 UTC
We will start the intervention on the following network switches:
176.31.230
176.31.231
46.105.108
46.105.109
Date: 2013-10-23 21:59:19 UTC
Switches of networks 176.31.226 and 227 were replaced.
We will intervene again tonight starting from 11pm on switches of the following networks in order to make them stable:
* downgrade:
176.31.228
176.31.229
176.31.230
176.31.231
46.105.108
46.105.109
We will also perform an upgrade cold swap on switches of these networks:
176.31.224
176.31.225
Traffic cut-off for 10-15min is expected.
Date: 2013-10-23 16:55:20 UTC
We have replaced the 1st n5 on the networks 176.31.226 and 227, all FEXs are up. We are replacing the second.
Date: 2013-10-23 16:44:39 UTC
The 1st switch of the pair is being replaced. 2 FEXs are offline.
Date: 2013-10-23 15:52:27 UTC
The pair of switches managing 176.31.226 and 227 is unstable again. We're going to move forward the replacement operation that was planned for this evening and do it immediately.
Each of the switches will be shut down and the FEXs (where connected to the servers) will be migrated onto a spare with the same config.
Date: 2013-10-23 15:47:49 UTC
done.
We're going to reboot a switch of the networks 176.31.226.0/24 et 176.31.227.0/24.
Date: 2013-10-23 15:47:09 UTC
done.
We're going to reboot a switch of the networks176.31.228.0/24 et 176.31.229.0/24.
Date: 2013-10-23 14:06:41 UTC
We're going to reboot a switch of the networks 46.105.108.0/24 and 46.105.109.0/24.
Date: 2013-10-23 13:37:24 UTC
Done.
Date: 2013-10-23 13:37:16 UTC
We're going to reboot the switches of networks 46.105.106.0/24 and 46.105.107.0/24, one after the other.