Get webhook notifications whenever Network & Infrastructure creates an incident, updates an incident, resolves an incident or changes a component status.
Total Critical Notifications Received: 0
Total Non-critical Notifications Received: 0
Bestpaths Deleted After Last Walk: 0
Bestpaths Changed After Last Walk: 0
Nexthop register:
Sync calls: 2, last sync call: 00:00:05
Async calls: 0, last async call: 00:00:00
Nexthop unregister:
Async calls: 1, last async call: 00:00:22
Nexthop batch finish:
Calls: 37097, last finish call: 00:00:04
Nexthop flush timer:
Times started: 66463, last time flush timer started: 00:00:05
RIB update: 0 rib update runs, last update: 00:00:00
0 prefixes installed, 0 modified, 0 removed
RP/0/RSP0/CPU0:rbx-g1-a9#show controller np struct 6 summary location 0/0/cpu0
Thu Nov 11 02:39:21.139 UTC
Node: 0/0/CPU0:
----------------------------------------------------------------
NP: 0 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 1 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 2 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 3 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 4 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 5 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 6 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 7 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0
Thu Nov 11 02:41:36.395 UTC
CEF resource availability summary state: GREEN
CEF will work normally
No. of times HW caused oor: 3
CEF entered oor at : Nov 11 01:28:50.907
CEF came out of oor at : Nov 11 02:36:21.139
ipv4 shared memory resource:
CurrMode GREEN, CurrAvail 773712739 bytes, MaxAvail 872878080 bytes
ipv6 shared memory resource:
CurrMode GREEN, CurrAvail 773712739 bytes, MaxAvail 872878080 bytes
mpls shared memory resource:
CurrMode GREEN, CurrAvail 773712739 bytes, MaxAvail 872878080 bytes
common shared memory resource:
CurrMode GREEN, CurrAvail 773712739 bytes, MaxAvail 872878080 bytes
DATA_TYPE_TABLE_SET hardware resource: GREEN
DATA_TYPE_TABLE hardware resource: GREEN
DATA_TYPE_IDB hardware resource: GREEN
DATA_TYPE_IDB_EXT hardware resource: GREEN
DATA_TYPE_LEAF hardware resource: GREEN
DATA_TYPE_LOADINFO hardware resource: GREEN
DATA_TYPE_PATH_LIST hardware resource: GREEN
DATA_TYPE_NHINFO hardware resource: GREEN
DATA_TYPE_LABEL_INFO hardware resource: GREEN
DATA_TYPE_FRR_NHINFO hardware resource: GREEN
DATA_TYPE_ECD hardware resource: GREEN
DATA_TYPE_RECURSIVE_NH hardware resource: GREEN
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: GREEN
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: GREEN
DATA_TYPE_ECD_TRACKER hardware resource: GREEN
DATA_TYPE_ECD_V2 hardware resource: GREEN
DATA_TYPE_ATTRIBUTE hardware resource: GREEN
DATA_TYPE_LSPA hardware resource: GREEN
DATA_TYPE_LDI_LW hardware resource: GREEN
DATA_TYPE_LDSH_ARRAY hardware resource: GREEN
DATA_TYPE_TE_TUN_INFO hardware resource: GREEN
DATA_TYPE_DUMMY hardware resource: GREEN
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: GREEN
DATA_TYPE_TABLE_UNRESOLVED hardware resource: GREEN
DATA_TYPE_MOL hardware resource: GREEN
DATA_TYPE_MPI hardware resource: GREEN
DATA_TYPE_SUBS_INFO hardware resource: GREEN
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: GREEN
Date: 2010-11-11 10:59:30 UTC We have just changed the configuration of the route reflector
by forcing \"next-hop-self\" towards the RF
http://status.ovh.net/?do=details&id=782
Date: 2010-11-11 10:57:15 UTC Apparently the problem is due to the fact that we have a BGP configuration
which pushes the traffic towards the ip of the server directly. This way, with
the IP failover, we have many nexthop in the BGP tables.
Too much for the ASR 9000
RP/0/RSP0/CPU0:rbx-g1-a9#show bgp nexthops statistics
Wed Nov 10 16:47:00.089 UTC
Total Nexthop Processing
Time Spent: 9.428 secs
Maximum Nexthop Processing
Received: 01:46:32
Bestpaths Deleted: 0
Bestpaths Changed: 0
Time Spent: 2.029 secs
Last Notification Processing
Received: 00:03:28
Time Spent: 1.324 secs
Total Critical Notifications Received: 6256
Total Non-critical Notifications Received: 33749
Bestpaths Deleted After Last Walk: 0
Bestpaths Changed After Last Walk: 0
Nexthop register:
Sync calls: 30834, last sync call: 00:00:38
Async calls: 3, last async call: 03:13:23
Nexthop unregister:
Async calls: 989, last async call: 00:00:51
Nexthop batch finish:
Calls: 31856, last finish call: 00:00:37
Nexthop flush timer:
Times started: 31828, last time flush timer started: 00:00:38
RIB update: 0 rib update runs, last update: 00:00:00
0 prefixes installed, 0 modified, 0 removed
RP/0/RSP0/CPU0:rbx-g1-a9#show controller np struct 6 summary location 0/0/cpu0
Wed Nov 10 19:04:51.590 UTC
Node: 0/0/CPU0:
----------------------------------------------------------------
NP: 0 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 1 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 2 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 3 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 4 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 5 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 6 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 7 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
Date: 2010-11-11 10:52:21 UTC found :) apparently the ASR cannot hold more than 380/390K routes.
As soon as we surpass the limit, it starts dropping the routes in the CEF
weird ... the TAC was informed ... we will see how they would respond.
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0
Tue Nov 9 23:18:52.874 UTC
CEF resource availability summary state: GREEN
CEF will work normally
No. of times HW caused oor: 7
CEF entered oor at : Nov 9 23:09:37.462
CEF came out of oor at : Nov 9 23:13:12.535
ipv4 shared memory resource:
CurrMode GREEN, CurrAvail 783738455 bytes, MaxAvail 872878080 bytes
ipv6 shared memory resource:
CurrMode GREEN, CurrAvail 783738455 bytes, MaxAvail 872878080 bytes
mpls shared memory resource:
CurrMode GREEN, CurrAvail 783738455 bytes, MaxAvail 872878080 bytes
common shared memory resource:
CurrMode GREEN, CurrAvail 783738455 bytes, MaxAvail 872878080 bytes
DATA_TYPE_TABLE_SET hardware resource: GREEN
DATA_TYPE_TABLE hardware resource: GREEN
DATA_TYPE_IDB hardware resource: GREEN
DATA_TYPE_IDB_EXT hardware resource: GREEN
DATA_TYPE_LEAF hardware resource: GREEN
DATA_TYPE_LOADINFO hardware resource: GREEN
DATA_TYPE_PATH_LIST hardware resource: GREEN
DATA_TYPE_NHINFO hardware resource: GREEN
DATA_TYPE_LABEL_INFO hardware resource: GREEN
DATA_TYPE_FRR_NHINFO hardware resource: GREEN
DATA_TYPE_ECD hardware resource: GREEN
DATA_TYPE_RECURSIVE_NH hardware resource: GREEN
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: GREEN
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: GREEN
DATA_TYPE_ECD_TRACKER hardware resource: GREEN
DATA_TYPE_ECD_V2 hardware resource: GREEN
DATA_TYPE_ATTRIBUTE hardware resource: GREEN
DATA_TYPE_LSPA hardware resource: GREEN
DATA_TYPE_LDI_LW hardware resource: GREEN
DATA_TYPE_LDSH_ARRAY hardware resource: GREEN
DATA_TYPE_TE_TUN_INFO hardware resource: GREEN
DATA_TYPE_DUMMY hardware resource: GREEN
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: GREEN
DATA_TYPE_TABLE_UNRESOLVED hardware resource: GREEN
DATA_TYPE_MOL hardware resource: GREEN
DATA_TYPE_MPI hardware resource: GREEN
DATA_TYPE_SUBS_INFO hardware resource: GREEN
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:18:58.974 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum
Tue Nov 9 23:19:01.674 UTC
BGP router identifier 94.23.2.19, local AS number 16276
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000
BGP main routing table version 5644641
BGP NSR Initial initsync version 366203 (Reached)
BGP scan interval 60 secs
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:07.989 UTC
94.23.2.13 0 16276 512257 884 5643206 0 0 00:01:50 366061
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:10.588 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:13.438 UTC
94.23.2.13 0 16276 512257 884 5644641 0 0 00:01:55 366061
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:17.482 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:19.353 UTC
94.23.2.13 0 16276 513527 884 5644641 0 0 00:02:01 369044
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:22.170 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:24.326 UTC
94.23.2.13 0 16276 513527 885 5644641 0 0 00:02:06 369044
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:27.012 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:34.647 UTC
94.23.2.13 0 16276 513527 885 5647628 0 0 00:02:16 369044
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:52.753 UTC
94.23.2.13 0 16276 514756 885 5649167 0 0 00:02:34 370583
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:54.493 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:20:14.067 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:20:22.080 UTC
94.23.2.13 0 16276 517393 886 5650818 0 0 00:03:05 374009
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:20:27.528 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:21:08.738 UTC
DATA_TYPE_TABLE_SET hardware resource: YELLOW
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:21:10.284 UTC
94.23.2.13 0 16276 528959 886 5662076 0 0 00:03:52 412381
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:21:15.491 UTC
DATA_TYPE_TABLE_SET hardware resource: YELLOW
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:21:17.509 UTC
94.23.2.13 0 16276 532198 886 5662076 0 0 00:03:59 420119
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum
Tue Nov 9 23:21:21.154 UTC
BGP router identifier 94.23.2.19, local AS number 16276
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000
BGP main routing table version 5702662
BGP NSR Initial initsync version 366203 (Reached)
BGP scan interval 60 secs
Date: 2010-11-07 11:11:52 UTC We have opened a TAC in order to understand what is wrong.
Apparently, all of the links towards the distant POP (Bruxelles
and London for the moment) do not work well. All that is
internal at the datacentre is functional.
We have also another problem on the CEF:
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail
Sun Nov 7 10:32:18.941 UTC
CEF resource availability summary state: GREEN
CEF will work normally
ipv4 shared memory resource:
CurrMode GREEN, CurrAvail 961508351 bytes, MaxAvail 1064480768 bytes
ipv6 shared memory resource:
CurrMode GREEN, CurrAvail 961508351 bytes, MaxAvail 1064480768 bytes
mpls shared memory resource:
CurrMode GREEN, CurrAvail 961508351 bytes, MaxAvail 1064480768 bytes
common shared memory resource:
CurrMode GREEN, CurrAvail 961508351 bytes, MaxAvail 1064480768 bytes
DATA_TYPE_TABLE_SET hardware resource: GREEN
DATA_TYPE_TABLE hardware resource: GREEN
DATA_TYPE_IDB hardware resource: GREEN
DATA_TYPE_IDB_EXT hardware resource: GREEN
DATA_TYPE_LEAF hardware resource: GREEN
DATA_TYPE_LOADINFO hardware resource: GREEN
DATA_TYPE_PATH_LIST hardware resource: GREEN
DATA_TYPE_NHINFO hardware resource: GREEN
DATA_TYPE_LABEL_INFO hardware resource: GREEN
DATA_TYPE_FRR_NHINFO hardware resource: GREEN
DATA_TYPE_ECD hardware resource: GREEN
DATA_TYPE_RECURSIVE_NH hardware resource: GREEN
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: GREEN
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: GREEN
DATA_TYPE_ECD_TRACKER hardware resource: GREEN
DATA_TYPE_ECD_V2 hardware resource: GREEN
DATA_TYPE_ATTRIBUTE hardware resource: GREEN
DATA_TYPE_LSPA hardware resource: GREEN
DATA_TYPE_LDI_LW hardware resource: GREEN
DATA_TYPE_LDSH_ARRAY hardware resource: GREEN
DATA_TYPE_TE_TUN_INFO hardware resource: GREEN
DATA_TYPE_DUMMY hardware resource: GREEN
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: GREEN
DATA_TYPE_TABLE_UNRESOLVED hardware resource: GREEN
DATA_TYPE_MOL hardware resource: GREEN
DATA_TYPE_MPI hardware resource: GREEN
DATA_TYPE_SUBS_INFO hardware resource: GREEN
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9#
Date: 2010-11-05 19:08:20 UTC Weird. This morning OSPF is mounted between ASR and LDN-1 but this has provoked
the crash on certain destinations to LDN and BRU.
This afternoon, we have done several tests and it turns out that OSPF does not
mount between LDN-1 and ASR even though it was working normally with 10 other
routers at Roubaix.
In brief, we must update the IOS of LDN (it's expected this night), then we will
see if it mounts well. Moreover, we must upgrade the IOS of all RF because
the RF do not announce anything neither.
That's a real mess in bricks and reinforced. The interoperability of the equipments
within one manufacturer. If we had to make a cisco work with a Jinuper they would have told
me \"It's Cisco Fault\", \"It's Juniper fault\". We have only Cisco .... but it seems
that the 4 equipments are developped by different teams that do not work together :
6K, ASR 1000, ASR 9000, Nexus 7000.
... as though they were 4 manufacturers ...
Date: 2010-11-05 13:00:47 UTC %ROUTING-BGP-3-INITSYNC_FAILED [chars]
Explanation The number of prefixes synced by active doesn't match
the expected count.
Recommended Action Copy the error message exactly as it appears on
the console or in the system log, call your Cisco technical support
representative and provide the representative with the gathered
information.
Date: 2010-11-05 13:00:17 UTC RP/0/RSP1/CPU0:Nov 5 12:45:05 UTC: bgp[1046]:
%ROUTING-BGP-3-INITSYNC_FAILED : Nbr: VRF: default AFI: IPv4 Unicast
Pfx received: 83392 expected: 71163 - will retry once more