OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#12102 — Brussels 's optic transmission node - 26/11
Scheduled Maintenance Report for Network & Infrastructure
Completed
Tonight We are going to work on the Brussels POP. Due to a reorganization of this building, we'll move the entire optical transmission node dwdm Infinera in another room.

This node is on routes Roubaix <> Amsterdam and Roubaix <> Frankfurt (and soon Gravelines <> Frankfurt). All of these routes are protected, there is not any impact to service expected for these routes.The only traffic that will be impacted will be dropped traffic locally at Brussels. The traffic will be rerouted to BNIX for the duration of the work (thus changing route but not interrupting service). The voice with ibnterco Belgacom will be unavailable and we apologize.

Work will start at 9pm. The impact on Brussels traffic will start from approximately 10pm.

Update(s):

Date: 2014-11-27 14:04:24 UTC
Finally, we had a battle against time and the alignment of the latest fiber to Frankfurt lasted almost 3 hours due to wiring problems between the building and the neighboring level3 data center through which the fibers must pass through. The managers of the various buildings and the fiber provider have therefore worked to check each patch panel throughly.
The clock stopped at 3pm for the onsite intervention by our ground team. Congratulations to them.

The optical node is now operating normally. The 40G capa is missing between Roubaix and Strasbourg now all is operational.

Date: 2014-11-27 09:37:07 UTC
We've fixed the problem with the DLM card.

The cable issue at the level of the fiber pair to Frankfort has been identifies. We will aligned here in a few minutes.

Date: 2014-11-27 09:33:54 UTC
We have reestablished IP traffic on BNIX.

Date: 2014-11-27 09:32:48 UTC
All the capa to Brussels aer now operational. Only the line to Frankfurt is down after having problems with the migration of fiber pair on location.

The optical node is ready, and is we are lighting the fiber but we suspect one DLM (line module) to Frankfurt is defective. We are working with Infinera to confirm this hypothesis. If this is the case, it must be replaced.

Because of this work, we still have 2x20G down between Roubaix and Strasbourg (since these are circuits that pass through Frankfurt). The remaining links are congested and thus internal traffic Roubaix <> Strasbourg is currently degraded (packet loss).

The local team has been working hard and nonstop since 12am (not including preparation time). We are sending relief from Roubaix in case the resolution of remaining issues require an extended stay at the site.


Date: 2014-11-27 09:06:31 UTC
Status at 8:36:

- Reconnection of 10G circuits Roubaix <> Brussels IN PROGRESS -> 20G out of 40G cappa are UP. The router and all IP equipment for Brussels are online again.

Date: 2014-11-27 09:06:23 UTC
Status at 8;24:

- reconnection of the second optics to Roubaix DONE
- Start at least one of the 2 transponder cards serving the Brussels POP DONE
- Turn up the line DONE
- Reconnect 10G circuits Roubaix <> Brussels DOING

Date: 2014-11-27 09:06:15 UTC
Chassis # 3 is being started. Its move has taken longer than expected. In a few minutes, the cards will start to boot. The remaining steps in order to restore service are:
- Reconnect the second optical line to Roubaix
- Start at least one of the 2 cards transponders serving the Brussels POP
- Replace the line
- Reconnect 10G circuits Roubaix <> Brussels

The time to complete these steps depends a lot on any issues encountered but we believe we can restore service in Brussels in 1 hour if there are no new major problem. We have already gone far beyond the maintenance window scheduled and we apologize. Our on-site team is doing everything possible to complete the maintenance as quickly as possible. We will continue to keep you informed of the progress of the work.

Date: 2014-11-27 07:08:54 UTC
We have issue there. Work is progressing but not as fast as expected because of various problems: power supply to the new locations, mechanical problems with windows, cleaning of fibers longer than expected (each connector should be linked to fiberscope,kind of electronic microscope, to validate the quality).

We have 2 chassis on 3 active and 2 lines of 4 up (to Amsterdam and Roubaix on coherent 100G). The big issue is behind us and we think we can mount the chassis #3 and the 2nd line to Roubaix by about 1h/1h30. This will be the moment when we can put online the Brussels router.

Date: 2014-11-27 00:51:16 UTC
Optical lines on Brussels just been disconnected. IPs equipment in Brussels are totally isolated.Moving the main chassis Infinera began.

Date: 2014-11-27 00:47:13 UTC
We will reroute BNIX traffic to other paths.

Date: 2014-11-27 00:46:32 UTC
Our team is on site and prepares the intervention with those of Interoute and Level3. We of course are doing everything possible to reduce downtime to the minimum time.

We currently have approximately 1 hour of delay on the planned schedule. The last preparations are in progress.
Posted Nov 26, 2014 - 15:52 UTC