rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_green
icon_orange
icon_red
icon_green
icon_red
icon_red
icon_green
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#9603 — FS#13460 — mtl <> bhs

Attached to Project— Network
Maintenance
Backbone US
CLOSED
100%
We currently have 10 links between Montreal and Beauharnois down.

Traffic is going through Newark <> BHS. We are in contact with our provider. We think it may be due to a fiber cut.

Date:  Thursday, 23 July 2015, 11:46AM
Reason for closing:  Done
Comment by OVH - Thursday, 28 May 2015, 03:30AM

It's an incident with the cable but not a clear cut. Several fibers are affected but not the whole cable. The problem could be caused by a rodent.
Since there are available pairs, the repair of the cable is not expected immediately. The maintainer will transfer the affected links to other pairs in good health. It's expected therefore a recovery of the links BHS <> Montreal in a few hours. No accurate ETA can be provided in the current situation.


Comment by OVH - Thursday, 28 May 2015, 04:30AM

The technicians are on site and looking for the damaged fiber.


Comment by OVH - Thursday, 28 May 2015, 06:37AM

Ovh network need an urgent repair, a 30 minute service interruption is necessary during this time the service will be degraded. (no ETA on when) We are working closely with our technicians on the ground to resolve this issue.


Comment by OVH - Thursday, 28 May 2015, 11:07AM

Hello,

There is currently a very delicate situation at our datacentre in BHS. We have 4 optical fiber cables that connect the datacenter to the world.

1) BHS to Montreal via the North. The link is up and we have 3 pairs of optical fiber cables: to Montreal, New York (600G capacity) and to level 3 (which provides 200G capacity to New York)
2) 2) BHS to Montreal via the South. The link is not yet up. Construction of the connection is delayed in the Indian reserve. We will have 3 pairs of optical fiber cables by July 2015
3) BHS to New York via the South. This link is not yet up either. It takes a particularly long time to get approval to perform work on the Canada/USA border: 3 years. The go-live date: September 2015
4) A backup of the MTL/BHS backup is UP, but with low capacity (10G).

Last night, a car hit one of the poles along the BHS/MTL via the North route, damaging the optical fiber cable: 2 of the 3 pairs of cables are down. The 3rd one, to Level 3, is still UP.
We’ve identified where the incident happened and the maintenance teams are currently working to feed a new 96 FO cable at 300m. This will take 12 hours.

Around 7-8am local time (1-2pm GMT+2), we will begin maintenance on the cable: we will cut off the optical fiber cables that are still up to weld them on to the new cable. To reduce the welding time, we have 2 teams on site at both ends of the cable. We estimate that it will take 30 minutes to cut off and remount our tube of optical fiber cables, in which we have 3 pairs of optical fiber cables. During this intervention, the datacenter will remain connected, but only via the backup of the backup, and therefore the connection will be seriously downgraded.

We will post more information before the work begins.

Best,
Octave Klaba
CTO


Comment by OVH - Thursday, 28 May 2015, 11:13AM

We've started to install the new cable. The welders are about to insert a new optical fiber cable into the connection box. We will then prepare the cable. We estimate that the cut-off will happen around 5:00AM (11.00 GMT+2)


Comment by OVH - Thursday, 28 May 2015, 12:53PM

The fibers have been cut.


Comment by OVH - Thursday, 28 May 2015, 14:01PM

Work is continuing. We're welding, we're welding ...


Comment by OVH - Thursday, 28 May 2015, 14:02PM

Work should be complete in a few minutes.


Comment by OVH - Thursday, 28 May 2015, 14:03PM

During the work, the connection seriously downgraded :( We're getting packet loss

Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. p19-55-6k.fr.eu 0.5% 392 2.1 19.7 0.2 279.1 51.0
2. p19-2-6k.fr.eu 0.0% 392 1.8 14.4 0.2 219.2 38.7
3. th2-g1-a9.fr.eu 0.0% 392 1.0 1.0 0.6 34.0 1.8
4. rbx-g1-a9.fr.eu 0.0% 392 4.5 4.9 4.2 11.3 0.9
5. ldn-1-6.uk.eu 99.7% 392 104.6 104.6 104.6 104.6 0.0
6. nwk-1-6k.nj.us 0.0% 392 76.7 76.7 76.5 83.9 0.6
7. mtl-1-6k.qc.ca 0.0% 392 84.0 93.5 83.8 265.8 31.1
8. bhs-g1-6k.qc.ca 7.2% 392 138.9 139.8 138.6 155.6 1.4
9. bhs-3a-a9.qc.ca 4.9% 391 140.7 141.0 139.0 163.1 2.3
10. bhs.proof.ovh.net 5.4% 391 141.8 141.7 141.0 150.5 1.1


Comment by OVH - Thursday, 28 May 2015, 14:32PM

The welding is bad quality. We need to redo it. We will cut everything off, we will breathe and start again.

Welding is in progress. One of the fibers in our tube is up. We're receiving signal from BHS to MTL. We're waiting to receive it in the other direction.


Comment by OVH - Thursday, 28 May 2015, 14:33PM

Our BHS/MTL 6x100G link is up.


Comment by OVH - Thursday, 28 May 2015, 14:33PM

MTL/BHS 1x100G is UP.


Comment by OVH - Thursday, 28 May 2015, 14:34PM

MTL/NWK 20x10G Level3 is up


Comment by OVH - Thursday, 28 May 2015, 14:57PM

The 3 pairs of optical fiber cables are up. Huge service interruptions lasted 1 hour 40 - longer than we expected. We will continue to weld other tubes, but we're no longer impacted.

We're sorry for the incident. We're continuing to work on fiber 2) and 3) which will enable us to fully secure the BHS - Montreal connection, and the BHS - New York connection.