OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#20238 — CHI <> PAL
Incident Report for Network & Infrastructure
Resolved
We lost 10x10G between Chicago and Palto Alto. Our provider confirm a fiber cut.


Update(s):

Date: 2016-09-08 12:53:54 UTC
All is back to normal.

Date: 2016-09-08 11:38:54 UTC
We reactivating the traffic on EQUINIX and local-pref on ANY2


Date: 2016-09-08 11:24:44 UTC
*** CASCADED EXTERNAL NOTES 08-Sep-2016 10:48:48 GMT From CASE: 11432200 - Event
Field Services have completed splicing, and alarms have cleared. The Technical Service Center is currently confirming restoral with customers. If further issues are experienced, please contact the Technical Service Center at 1-877-4-LEVEL3.

Date: 2016-09-08 03:05:35 UTC
09/08/2016 1:52 AM GMT


*** CASCADED EXTERNAL NOTES 08-Sep-2016 01:50:37 GMT From CASE: 11432200 - Event
A partial cut has been confirmed, but the services that are not completely cut are severely degraded. As such, it is necessary to fully cut the cable and repair the damage to restore services to their optimum level. All crews are expected to arrive on site to commence repairs by 04:30 GMT. The next update will be provided at that time, or if new information becomes available.


Date: 2016-09-08 01:58:06 UTC
We shutdown the EQUINIX peering to save some bandwidth on west coast.

Date: 2016-09-08 01:39:48 UTC
*** CASCADED EXTERNAL NOTES 08-Sep-2016 01:26:50 GMT From CASE: 11432200 - Event
Field Services remains en route the testing site in Oasis, but an updated estimated time of arrival is unavailable.

Date: 2016-09-07 22:12:53 UTC
09/07/2016 21:32:52 GMT - The Transport NOC suspects that a fiber issue between Oasis, NV and Elko, NV is impacting unprotected services. Field Services is engaged to dispatch to the site, with an estimated time of arrival of 01:00 GMT on September 8, 2016 due to the remote location of the fault. Once on site, Field Services will perform fiber testing to isolate the fault. The next update will be provided upon arrival, or if new information becomes available. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.

Sep 7, 2016 10:02 PM GMT

09/07/2016 21:56:36 GMT - It has been confirmed that the damage was caused by construction and an approximate location has been identified. Field Services remains en route for further isolation testing, with an estimated time of arrival of 01:00 GMT on September 8, 2016. Repair crews are also engaged to commence repairs upon isolation of the fault. The next update will be provided upon arrival, or if new information becomes available.
Posted Sep 07, 2016 - 22:12 UTC