OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#10528 — bhs < > Nwk
Incident Report for Network & Infrastructure
Resolved
The Newark < > Beauharnois fibre is down.

We will investigate.

Update(s):

Date: 2014-04-02 07:46:31 UTC
The fibre provider has completed all splicing. The cables have been buried and their crews are hands-off at this time. The Transport NOC confirms that alarms remain clear and the Technical Service Center has confirmed restoration with the affected customers. The fibre cut was caused by road construction. The fibre was damaged during the installation of a new post during repairs to a highway guard rail along a bridge. The provider confirmed that this was a permanent restoration.

Date: 2014-04-02 07:44:50 UTC
The Transport NOC confirms that all alarms are clear at this time and the final system restored at approximately 17:17 GMT. The Technical Service Center continues to confirm restoration with customers. The Transport NOC will continue to monitor alarms while awaiting confirmation that the fibre provider is hands-off.

Date: 2014-04-02 07:43:54 UTC
The Transport NOC sees the majority of alarms clear with all affected systems up at this time. The Technical Service Center has begun to confirm service restoration with customers.


Date: 2014-04-02 07:40:05 UTC
The fibre provider is completing splicing on one end and has two buffer tubes remaining to splice on the other end. The Transport NOC verified that additional alarms have cleared and services are being restored.


Date: 2014-04-02 07:39:15 UTC
The fibre provider continues with restoration efforts at this time. The Transport NOC is monitoring alarms and confirms some systems have been restored.


Date: 2014-04-02 07:38:46 UTC
The fibre provider has completed splicing eight of the ten buffer tubes on one side and has completed splicing of five buffer tubes on the other end. Once all the splicing has been completed on one side, those resources will assist with splicing on the other end for a faster restoration. No estimated restoration time has been provided.

Date: 2014-04-02 07:38:20 UTC
The cable has been repaired and the service back up. We are redeploying the peering and transit on New York/Newark/Ashburn and Miami.


Date: 2014-04-02 07:37:41 UTC
The field technician reports that the provider has completed the splicing of three buffer tubes on both sides and is beginning splicing on the fourth tube at this time. The fibre provider has advised they are not able to provide an estimated restoration time, but may as work progresses further. The Level 3 field technician remains on site to obtain updates and observe the progress of the repair efforts.

Date: 2014-04-01 16:01:57 UTC
The Transport NOC reports that splicing is now underway on both ends of the cable. No estimated restoration time has been provided as of yet. The Level 3 field technician on site is working to obtain an estimated restoration time.

Date: 2014-04-01 15:15:57 UTC
The Transport NOC has advised that the additional damaged cable has been fully repaired and the restoration cable for the 120-count fiber has been set in the conduit. The Transport NOC is currently attempting to obtain an estimated time of restoral.

Date: 2014-04-01 12:18:38 UTC
The splice crews have begun prepping the 120-count fibre for splicing.

Date: 2014-04-01 10:08:16 UTC
The Transport NOC advised that the splicing of the additional damaged cable continues. In an attempt to complete repairs quickly, four splice crews are on site working on the damaged fibres. An estimated restoration time is not currently available.

Date: 2014-04-01 08:45:40 UTC
The Transport NOC advised that the excavation crew has arrived to the site and efforts to excavate the damaged fibres are underway. An estimated time of restoral is not available at this time.

Date: 2014-04-01 07:57:44 UTC
The Transport NOC advised that the fibre provider continues to await the arrival of the excavation crew. An estimated time of arrival is not available at this time.

Date: 2014-04-01 07:51:28 UTC
The Transport NOC advised that the provider has engaged an excavation crew, but an estimated time of arrival is not available at this time. Upon completion of the excavation, the splice crew will complete the necessary fibre repairs.

Date: 2014-04-01 07:51:23 UTC
The provider's splice crews continue assessing the damage.


Date: 2014-04-01 07:51:13 UTC
The provider has confirmed that their splice crew has arrived to the fault location, and they are currently assessing the situation. Once the extent of the damage is known, an appropriate repair strategy will be developed for implementation, and an estimated restoral time can be provided.


Date: 2014-04-01 07:51:04 UTC
The provider has reported that the fault has been located, and was determined to have been caused by road construction. It appears that the 120-count fibre was damaged by a new post being installed during the repairs to a highway guardrail. Splice crews remain en route to the area; an estimated arrival time has not yet been provided.


Date: 2014-04-01 07:50:53 UTC
The Transport NOC has confirmed that the Level 3 field technician has arrived to the area. Efforts to isolate the location of the damage continue.



Date: 2014-04-01 07:50:34 UTC
The provider has advised that the fault has been isolated to a fibre section along a bridge. Splice crews have been engaged to assist with locating the physical damage, and are currently en route to the area. An estimated arrival time is not yet available.



Date: 2014-04-01 07:50:16 UTC
The provider's field technicians continue working to locate the fault.


Date: 2014-04-01 07:47:26 UTC
The fibre provider has reported that their field technicians are currently on site at two adjacent locations, and are working to isolate the exact break point.



Date: 2014-04-01 07:47:00 UTC
The fibre provider has confirmed a fibre cut on their portion of the span between Suffern and Woodbury, NY. At this time, the fibre provider as well as Level 3 are dispatching field technicians to assist with isolating the exact location of the break. The Level 3 field technician has provided an estimated arrival time of approximately two hours, and an estimated arrival time for the fibre provider's field technician will be provided once it has been made available.

Date: 2014-04-01 07:44:58 UTC
The Transport NOC has advised that the issue is believed
to have been isolated to a fibre provider's network.
Efforts to contact the fibre provider are currently ongoing.

Date: 2014-04-01 07:44:43 UTC
Level3 GFS field technician shot the fibre from the site,
and a fault has been isolated approximately 60 kilometers out.
Outside Plant Engineers have been contacted, and investigations
are ongoing to determine whether the fault is on a fibre
provider's portion of the span.

Date: 2014-04-01 07:43:57 UTC
We have redirected the traffic from:
- Miami,FL
- Atlanta,GA
- Ashburn,VA
- Newark,NJ
- New York,NYC
on the West Coast of North America.
We have activated the PNI with Comcast.

Date: 2014-04-01 07:43:13 UTC
The cut in the fibre is at the New York exit towards Albanie
and it's affecting multiple providers and services
- Level3 NWK / BHS (10x10G)
- Level3 NWK / MTL (2x10G)
- Hibernia NWK / MTL (2x10G)
- Telia NWK / CHI (6x10G)
- Telia NWK / PAL (2x10G)
- Level3 Paris / New York / BHS (4x10G)

This is normal for the NWK / MTL-BHS link, being the
Newark/Albanie/Montréal route. However the Newark/Chicago by Albanie route
will be changed to a more direct route in order to avoid
access issues to the DC when there's a problem with NWK/MTL.

Date: 2014-04-01 07:38:31 UTC
Technicians are on site.

Date: 2014-04-01 07:38:20 UTC
The provider has reported a cut in the fibre at New York.

Date: 2014-04-01 07:37:39 UTC
The traffic is flowing via Montreal. We will contact our provider.
Posted Apr 01, 2014 - 07:37 UTC