OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#16568 — BGP leak
Incident Report for Network & Infrastructure
Resolved
We are receiving a major route-leak from AS31500.

Currently shutting every session we have with this peer (it started on the AMS-IX).

Update(s):

Date: 2016-02-11 18:04:19 UTC
Hello,

We just had a a routing problem which impacted 75% of our traffic. The origin of the problem
has been fixed and we are finally about to manage the incident.

The origin of the problem comes from a peering point DECIX in Frankfurt where one of the networks AS31500
announced through BGP \"all Internet\". This is a handling error which happens quite often and we have protection mechanisms
to filter this kind of human errors. Except that mecanismo was not implemented on the connection of that specific network.
We forgot about it, in the ocasión of recent router migration in Europe.
We are currently investigating the exacta moment in which this error slipped in our configurations
in our configurations.
As a result, 75% of our traffic has been drown in by the network through Frankfurt and that caused a failure.
We have cut the BGP session with the AS and thus stabilized routing.

We are very sorry for this failure. We are coding the script to check out BGP configurations on all our routers
and avoid this kind of error.

Sincerely,
Octave.

Date: 2016-02-11 16:40:23 UTC
Routers mil-1-6k, mil-5-6k and mad-5-6k have been impacted by the TCAM exception. We are currently working to isolate and reboot them.

Date: 2016-02-11 16:31:46 UTC
This leak has impacted some of our routers (6k) which could pass in TCAM exception.

We visited the park to identify and take the required corrective actions.

Date: 2016-02-11 16:23:30 UTC
RP/0/RSP0/CPU0:ams-1-a9#show route

B 1.0.4.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.0.5.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.0.6.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.0.7.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.0.38.0/24 [20/0] via 80.249.209.157, 00:11:55
B 1.0.64.0/18 [200/1] via 178.32.135.253, 2w1d
B 1.0.128.0/17 [20/0] via 80.249.209.157, 00:12:14
B 1.0.128.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.0.128.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.0.128.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.129.0/24 [20/0] via 80.249.209.157, 00:11:55
B 1.0.131.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.139.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.142.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.160.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.0.160.0/21 [20/0] via 80.249.209.157, 00:10:38
B 1.0.192.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.0.192.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.0.192.0/21 [20/0] via 80.249.209.157, 00:10:38
B 1.0.208.0/22 [20/0] via 80.249.209.157, 00:10:38
B 1.0.212.0/23 [20/0] via 80.249.209.157, 00:10:38
B 1.0.214.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.224.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.0.240.0/20 [20/0] via 80.249.209.157, 00:10:38
B 1.1.1.0/24 [20/0] via 80.249.209.157, 00:11:20
B 1.1.8.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.1.20.0/24 [20/0] via 80.249.209.157, 00:11:19
B 1.1.32.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.1.64.0/19 [20/0] via 80.249.209.157, 00:12:31
B 1.1.114.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.115.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.116.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.117.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.118.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.119.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.120.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.121.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.122.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.123.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.124.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.125.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.126.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.127.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.128.0/17 [20/0] via 80.249.209.157, 00:12:14
B 1.1.128.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.1.128.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.1.128.0/21 [20/0] via 80.249.209.157, 00:11:20
B 1.1.136.0/22 [20/0] via 80.249.209.157, 00:10:38
B 1.1.141.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.142.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.144.0/20 [20/0] via 80.249.209.157, 00:11:20
B 1.1.160.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.1.160.0/20 [20/0] via 80.249.209.157, 00:10:38
B 1.1.181.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.184.0/22 [20/0] via 80.249.209.157, 00:10:38
B 1.1.190.0/23 [20/0] via 80.249.209.157, 00:10:38
B 1.1.192.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.1.192.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.1.192.0/20 [20/0] via 80.249.209.157, 00:11:20
B 1.1.208.0/21 [20/0] via 80.249.209.157, 00:11:20
B 1.1.216.0/22 [20/0] via 80.249.209.157, 00:10:38
B 1.1.224.0/19 [20/0] via 80.249.209.157, 00:11:20
B 1.1.224.0/21 [20/0] via 80.249.209.157, 00:11:20
B 1.1.232.0/21 [20/0] via 80.249.209.157, 00:11:20
B 1.1.240.0/21 [20/0] via 80.249.209.157, 00:10:38
B 1.1.248.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.249.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.254.0/23 [20/0] via 80.249.209.157, 00:10:38
B 1.2.1.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.2.4.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.2.11.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.2.128.0/17 [20/0] via 80.249.209.157, 00:12:14
B 1.2.128.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.2.128.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.2.160.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.2.160.0/21 [20/0] via 80.249.209.157, 00:10:38
B 1.2.168.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.2.173.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.2.178.0/24 [20/0] via 80.249.209.157, 00:10:38
Posted Feb 11, 2016 - 16:18 UTC