Get webhook notifications whenever Network & Infrastructure creates an incident, updates an incident, resolves an incident or changes a component status.
We have problems in room 23 (23xxx racks)
Our teams are intervening.
Update(s):
Date: 2013-03-13 10:21:30 UTC 20 people worked for 16 hours to fix all servers in rooms 21/22/23/24. Everything is up.
The RBX2 roof has VVVVV shape like everywhere in Northern France. With the vertical wind, about 10T of snow accumulated in part of the roof. The weight of the snow caused a slight deformation of the roof and the creation some infiltrations in the first floor of RBX2. Rooms 21-24 are on the ground floor, with hot air blown from the ground floor to the first floor we have avoided the worst but two racks were still affected and tens of servers here and there in the 4 rooms where there are 4x 5500 servers.
We intervened on 250 servers in total and replaced about 180 CM. This took some time: check each server, one by one, check the component to replace then restart the server.
Congratulations to the team who managed this exceptional failure!
The roof will be reinforced / changed as soon as it is warm enough.
Date: 2013-03-12 16:28:57 UTC Work is progressing, but it's slow :(