rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_green
icon_orange
icon_red
icon_green
icon_red
icon_red
icon_green
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#12169 — FS#16568 — BGP leak

Attached to Project— Network
Incident
Whole Network
In progress
0%
We are receiving a major route-leak from AS31500.

Currently shutting every session we have with this peer (it started on the AMS-IX).
Comment by OVH - Thursday, 11 February 2016, 17:23PM

RP/0/RSP0/CPU0:ams-1-a9#show route

B 1.0.4.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.0.5.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.0.6.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.0.7.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.0.38.0/24 [20/0] via 80.249.209.157, 00:11:55
B 1.0.64.0/18 [200/1] via 178.32.135.253, 2w1d
B 1.0.128.0/17 [20/0] via 80.249.209.157, 00:12:14
B 1.0.128.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.0.128.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.0.128.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.129.0/24 [20/0] via 80.249.209.157, 00:11:55
B 1.0.131.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.139.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.142.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.160.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.0.160.0/21 [20/0] via 80.249.209.157, 00:10:38
B 1.0.192.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.0.192.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.0.192.0/21 [20/0] via 80.249.209.157, 00:10:38
B 1.0.208.0/22 [20/0] via 80.249.209.157, 00:10:38
B 1.0.212.0/23 [20/0] via 80.249.209.157, 00:10:38
B 1.0.214.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.0.224.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.0.240.0/20 [20/0] via 80.249.209.157, 00:10:38
B 1.1.1.0/24 [20/0] via 80.249.209.157, 00:11:20
B 1.1.8.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.1.20.0/24 [20/0] via 80.249.209.157, 00:11:19
B 1.1.32.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.1.64.0/19 [20/0] via 80.249.209.157, 00:12:31
B 1.1.114.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.115.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.116.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.117.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.118.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.119.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.120.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.121.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.122.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.123.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.124.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.125.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.126.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.127.0/24 [20/0] via 80.249.209.157, 00:12:31
B 1.1.128.0/17 [20/0] via 80.249.209.157, 00:12:14
B 1.1.128.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.1.128.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.1.128.0/21 [20/0] via 80.249.209.157, 00:11:20
B 1.1.136.0/22 [20/0] via 80.249.209.157, 00:10:38
B 1.1.141.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.142.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.144.0/20 [20/0] via 80.249.209.157, 00:11:20
B 1.1.160.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.1.160.0/20 [20/0] via 80.249.209.157, 00:10:38
B 1.1.181.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.184.0/22 [20/0] via 80.249.209.157, 00:10:38
B 1.1.190.0/23 [20/0] via 80.249.209.157, 00:10:38
B 1.1.192.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.1.192.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.1.192.0/20 [20/0] via 80.249.209.157, 00:11:20
B 1.1.208.0/21 [20/0] via 80.249.209.157, 00:11:20
B 1.1.216.0/22 [20/0] via 80.249.209.157, 00:10:38
B 1.1.224.0/19 [20/0] via 80.249.209.157, 00:11:20
B 1.1.224.0/21 [20/0] via 80.249.209.157, 00:11:20
B 1.1.232.0/21 [20/0] via 80.249.209.157, 00:11:20
B 1.1.240.0/21 [20/0] via 80.249.209.157, 00:10:38
B 1.1.248.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.249.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.1.254.0/23 [20/0] via 80.249.209.157, 00:10:38
B 1.2.1.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.2.4.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.2.11.0/24 [20/0] via 80.249.209.157, 00:12:14
B 1.2.128.0/17 [20/0] via 80.249.209.157, 00:12:14
B 1.2.128.0/18 [20/0] via 80.249.209.157, 00:12:14
B 1.2.128.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.2.160.0/19 [20/0] via 80.249.209.157, 00:12:14
B 1.2.160.0/21 [20/0] via 80.249.209.157, 00:10:38
B 1.2.168.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.2.173.0/24 [20/0] via 80.249.209.157, 00:10:38
B 1.2.178.0/24 [20/0] via 80.249.209.157, 00:10:38


Comment by OVH - Thursday, 11 February 2016, 17:31PM

This leak has impacted some of our routers (6k) which could pass in TCAM exception.

We visited the park to identify and take the required corrective actions.


Comment by OVH - Thursday, 11 February 2016, 17:40PM

Routers mil-1-6k, mil-5-6k and mad-5-6k have been impacted by the TCAM exception. We are currently working to isolate and reboot them.


Comment by OVH - Thursday, 11 February 2016, 19:04PM

Hello,

We just had a a routing problem which impacted 75% of our traffic. The origin of the problem
has been fixed and we are finally about to manage the incident.

The origin of the problem comes from a peering point DECIX in Frankfurt where one of the networks AS31500
announced through BGP "all Internet". This is a handling error which happens quite often and we have protection mechanisms
to filter this kind of human errors. Except that mecanismo was not implemented on the connection of that specific network.
We forgot about it, in the ocasión of recent router migration in Europe.
We are currently investigating the exacta moment in which this error slipped in our configurations
in our configurations.
As a result, 75% of our traffic has been drown in by the network through Frankfurt and that caused a failure.
We have cut the BGP session with the AS and thus stabilized routing.

We are very sorry for this failure. We are coding the script to check out BGP configurations on all our routers
and avoid this kind of error.

Sincerely,
Octave.