OVHcloud Private Cloud Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#8501 — rbx-s6-6k rbx-s5-6k
Incident
Report for Hosted Private Cloud
Resolved
Our monitoring systems have detected network losses on the Private Cloud routers rbx-s6-6k and rbx-s5-6k.
We are investigating.
Update(s):
Date: 2013-04-19 15:01:35 UTC
There is now some loss on the card 11 interfaces,
we are replacing it.
Date: 2013-04-19 14:04:06 UTC
the router is no longer routing in OSPF.
the conf is ok. changing the SUP. same.
thinking.
thinking.
going to eat.
emptying the conf and reapplying a minimal conf. same.
thinking.
emptying the conf again and applying an ultra minimalist conf. it's working again. OSPF is working!
reapplying the whole conf. it works.
cutting the OSPF. no longer working.
shit.
looking for the difference between minimalist and ultra minimaliste conf.
but what's this command???
mls rate-limit unicast cef receive 10000 100
it's not from our side.
shit.
the router added it when it switched over to CEF software.
without warning. and as before rebooting the router
saved the conf.
shit.
no mls rate-limit unicast cef receive 10000 100
wr me
rebooting for procedure's sake.
Date: 2013-04-19 12:09:38 UTC
When an SUP is changed, the conf. application is not completely done.
It must be rebooted twice.
rbx-s6-6k#sh mls cef maximum-routes
Reload scheduled for 19:30:11 GMT Thu Jun 19 2003 (in 4 minutes and 22 seconds)
Reload reason: FIB Protocol Allocation mismatchFIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 + MPLS - 512k (default)
IPv6 + IP Multicast - 256k (default)
Date: 2013-04-19 10:17:31 UTC
We will reboot the router and replay the configuration.
Date: 2013-04-19 10:14:52 UTC
The max-route commands were clearly not taken
into account when the router was updated.
rbx-s6-6k#sh mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 + MPLS - 512k (default)
IPv6 + IP Multicast - 256k (default)
Date: 2013-04-19 10:11:01 UTC
We cannot reproduce the problem. This's normal as there is no problem. We're a long way from saturating the CEF...
Apr 19 10:34:18 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.12 Up
Apr 19 10:34:18 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.13 Up
Apr 19 10:34:18 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.11 Up
Apr 19 10:34:18 GMT: %BGP-5-ADJCHANGE: neighbor 2001:41D0::1024 Up
Date: 2013-04-19 10:07:40 UTC
Apr 19 11:00:46 rbx-s6-6k.fr.eu 50595: Apr 19 10:00:19 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.12 Up
Apr 19 11:01:47 rbx-s6-6k.fr.eu 50606: Apr 19 10:01:22 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.13 Up
Apr 19 11:03:32 rbx-s6-6k.fr.eu 50614: Apr 19 10:03:07 GMT: %MLSCEF-SP-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:03:32 rbx-s6-6k.fr.eu 50615: Apr 19 10:03:07 GMT: %MLSCEF-DFC12-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:03:32 rbx-s6-6k.fr.eu 50616: Apr 19 10:03:07 GMT: %MLSCEF-DFC11-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:03:32 rbx-s6-6k.fr.eu 50617: Apr 19 10:03:08 GMT: %MLSCEF-DFC10-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:03:34 rbx-s6-6k.fr.eu 50620: Apr 19 10:03:09 GMT: %MLSCEF-DFC9-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:09:27 rbx-s6-6k.fr.eu 50629: Apr 19 10:09:02 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.13 Down Admin. shutdown
Apr 19 11:11:04 rbx-s6-6k.fr.eu 51583: Apr 19 10:10:36 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.12 Down Admin. shutdown
Date: 2013-04-19 10:06:57 UTC
Following the BGP session restart and all the RRs, the rbx-s6 router has passed though the routing
software. We have isolated it.
We are investigating.
Update(s):
Date: 2013-04-19 15:01:35 UTC
There is now some loss on the card 11 interfaces,
we are replacing it.
Date: 2013-04-19 14:04:06 UTC
the router is no longer routing in OSPF.
the conf is ok. changing the SUP. same.
thinking.
thinking.
going to eat.
emptying the conf and reapplying a minimal conf. same.
thinking.
emptying the conf again and applying an ultra minimalist conf. it's working again. OSPF is working!
reapplying the whole conf. it works.
cutting the OSPF. no longer working.
shit.
looking for the difference between minimalist and ultra minimaliste conf.
but what's this command???
mls rate-limit unicast cef receive 10000 100
it's not from our side.
shit.
the router added it when it switched over to CEF software.
without warning. and as before rebooting the router
saved the conf.
shit.
no mls rate-limit unicast cef receive 10000 100
wr me
rebooting for procedure's sake.
Date: 2013-04-19 12:09:38 UTC
When an SUP is changed, the conf. application is not completely done.
It must be rebooted twice.
rbx-s6-6k#sh mls cef maximum-routes
Reload scheduled for 19:30:11 GMT Thu Jun 19 2003 (in 4 minutes and 22 seconds)
Reload reason: FIB Protocol Allocation mismatchFIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 + MPLS - 512k (default)
IPv6 + IP Multicast - 256k (default)
Date: 2013-04-19 10:17:31 UTC
We will reboot the router and replay the configuration.
Date: 2013-04-19 10:14:52 UTC
The max-route commands were clearly not taken
into account when the router was updated.
rbx-s6-6k#sh mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 + MPLS - 512k (default)
IPv6 + IP Multicast - 256k (default)
Date: 2013-04-19 10:11:01 UTC
We cannot reproduce the problem. This's normal as there is no problem. We're a long way from saturating the CEF...
Apr 19 10:34:18 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.12 Up
Apr 19 10:34:18 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.13 Up
Apr 19 10:34:18 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.11 Up
Apr 19 10:34:18 GMT: %BGP-5-ADJCHANGE: neighbor 2001:41D0::1024 Up
Date: 2013-04-19 10:07:40 UTC
Apr 19 11:00:46 rbx-s6-6k.fr.eu 50595: Apr 19 10:00:19 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.12 Up
Apr 19 11:01:47 rbx-s6-6k.fr.eu 50606: Apr 19 10:01:22 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.13 Up
Apr 19 11:03:32 rbx-s6-6k.fr.eu 50614: Apr 19 10:03:07 GMT: %MLSCEF-SP-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:03:32 rbx-s6-6k.fr.eu 50615: Apr 19 10:03:07 GMT: %MLSCEF-DFC12-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:03:32 rbx-s6-6k.fr.eu 50616: Apr 19 10:03:07 GMT: %MLSCEF-DFC11-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:03:32 rbx-s6-6k.fr.eu 50617: Apr 19 10:03:08 GMT: %MLSCEF-DFC10-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:03:34 rbx-s6-6k.fr.eu 50620: Apr 19 10:03:09 GMT: %MLSCEF-DFC9-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.
Apr 19 11:09:27 rbx-s6-6k.fr.eu 50629: Apr 19 10:09:02 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.13 Down Admin. shutdown
Apr 19 11:11:04 rbx-s6-6k.fr.eu 51583: Apr 19 10:10:36 GMT: %BGP-5-ADJCHANGE: neighbor 94.23.122.12 Down Admin. shutdown
Date: 2013-04-19 10:06:57 UTC
Following the BGP session restart and all the RRs, the rbx-s6 router has passed though the routing
software. We have isolated it.