OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#10458 — Level3 routing
Incident
Report for Network & Infrastructure
Resolved
Certain destinations are reachable with difficulty from some OVH IPs.
We are investigating.
Update(s):
Date: 2014-03-21 15:44:36 UTC
The sessions have been restored. According to the timing of events, the configuration issue was there from the start, but did not have an impact until the end of yesterday when there was a cut in the fibre on the Level3 network. The cut caused this from one of the intercos and thus a change in the routing which updated the config issue.
Everything has now been corrected on Level3's side.
Date: 2014-03-21 15:40:59 UTC
Level3 has identified the source of the problem - a bad configuration on their side. We have remounted the interfaces and we are about to restore the BGP sessions on our routers in Roubaix (\"nitro\" infrastructure).
Date: 2014-03-20 23:47:00 UTC
We have isolated the issue in 2 BGP sessions with Level3 forwarded to routers in Roubaix via our dwdm network. We have cut them and recovered the normal routing with Level3 in Paris and on the range 37.187.0.0/16. No prefixes were announced on these sessions that we're only using for outgoing traffic and therefore we do not explain the reason why at this level, the traffic to 37.187.0.0 was blackholed. We will continue troubleshooting with Level3 in order to understand the exact origin of the issue.
Date: 2014-03-20 21:06:09 UTC
We identified an issue with Level3 in Paris who is impacting network 37.187.0.0/16. The traffic seem to be dropped by Level3 equipment in Paris. We have shutdown the BGP session with L3 in Paris. As emergency plan, we forced traffic going trough Londons but had to split the announcement of 37.187.0.0/16 in 2 /17. We are working with Level3 to find a permanent solution.
We are investigating.
Update(s):
Date: 2014-03-21 15:44:36 UTC
The sessions have been restored. According to the timing of events, the configuration issue was there from the start, but did not have an impact until the end of yesterday when there was a cut in the fibre on the Level3 network. The cut caused this from one of the intercos and thus a change in the routing which updated the config issue.
Everything has now been corrected on Level3's side.
Date: 2014-03-21 15:40:59 UTC
Level3 has identified the source of the problem - a bad configuration on their side. We have remounted the interfaces and we are about to restore the BGP sessions on our routers in Roubaix (\"nitro\" infrastructure).
Date: 2014-03-20 23:47:00 UTC
We have isolated the issue in 2 BGP sessions with Level3 forwarded to routers in Roubaix via our dwdm network. We have cut them and recovered the normal routing with Level3 in Paris and on the range 37.187.0.0/16. No prefixes were announced on these sessions that we're only using for outgoing traffic and therefore we do not explain the reason why at this level, the traffic to 37.187.0.0 was blackholed. We will continue troubleshooting with Level3 in order to understand the exact origin of the issue.
Date: 2014-03-20 21:06:09 UTC
We identified an issue with Level3 in Paris who is impacting network 37.187.0.0/16. The traffic seem to be dropped by Level3 equipment in Paris. We have shutdown the BGP session with L3 in Paris. As emergency plan, we forced traffic going trough Londons but had to split the announcement of 37.187.0.0/16 in 2 /17. We are working with Level3 to find a permanent solution.