Get webhook notifications whenever Network & Infrastructure creates an incident, updates an incident, resolves an incident or changes a component status.
In the recent days we have strange messages:
Sep 8 08:42:54 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B6823C 40B671A4 40B672DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4
It's totally bizarre because the router is a failover
failover that does not route the traffic for now.
Update(s):
Date: 2011-09-08 10:32:56 UTC it's good, no more problem.
Date: 2011-09-08 10:32:24 UTC and the CPU is okey now. so it's really
the backup on the disk0 which blocked all the router!?
Sep 8 10:13:54 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:14:31 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:15:03 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:15:39 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:16:12 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:16:45 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:17:31 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:18:06 GMT: %IPC-SP-5-WATERMARK: 1696 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:18:38 GMT: %IPC-SP-5-WATERMARK: 1687 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:19:14 GMT: %IPC-SP-5-WATERMARK: 1684 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:19:45 GMT: %IPC-SP-5-WATERMARK: 3391 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:20:22 GMT: %IPC-SP-5-WATERMARK: 3397 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:20:51 GMT: %IPC-SP-5-WATERMARK: 1697 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:21:14 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (2/2),process = RFSS worker process.
-Traceback= 40B8A0E4 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278 40D61EF4 40D628CC 40D615AC
Sep 8 10:21:29 GMT: %IPC-SP-5-WATERMARK: 3393 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:22:00 GMT: %IPC-SP-5-WATERMARK: 3399 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:22:39 GMT: %IPC-SP-5-WATERMARK: 1700 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:23:11 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:23:41 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B673B8 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 10:23:57 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:24:41 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:25:17 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:25:48 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:26:24 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:27:04 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:27:35 GMT: %IPC-SP-5-WATERMARK: 1701 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:28:07 GMT: %IPC-SP-5-WATERMARK: 3394 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:28:12 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B68864 40B69890 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 10:28:15 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card removed from disk0
Sep 8 10:28:23 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card inserted into disk0
Sep 8 10:28:23 GMT: %DIAG-SP-6-TEST_RUNNING: Module 5: Running TestCFRW{ID=41} ...
Sep 8 10:28:25 GMT: %DIAG-SP-6-TEST_OK: Module 5: TestCFRW{ID=41} has completed successfully
Sep 8 10:28:41 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:29:12 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:29:42 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:30:13 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:30:43 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:31:01 GMT: %SYS-SP-3-CONFIG_NV_ERR: Nonvolatile storage write error; configuration failed
Sep 8 10:31:16 GMT: %IPC-SP-5-WATERMARK: 2887 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
vss-1b-6k#
Date: 2011-09-08 10:30:49 UTC vss-1b-6k# sh run
Unable to get configuration. Try again later.
vss-1b-6k#wr me
Building configuration...
Compressed configuration from 2398126 bytes to 471700 bytes
[OK]
Date: 2011-09-08 09:33:46 UTC Sep 8 09:52:55 GMT: %IPC-SP-5-WATERMARK: 3396 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:53:33 GMT: %IPC-SP-5-WATERMARK: 1696 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:54:10 GMT: %IPC-SP-5-WATERMARK: 1695 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:54:50 GMT: %IPC-SP-5-WATERMARK: 3393 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:55:33 GMT: %FM-4-FLOWMASK_REDUCED: Features configured on interface Port-channel5 have conflicting flowmask requirements, some features may work in software
Sep 8 09:55:22 GMT: %IPC-SP-5-WATERMARK: 1688 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:55:52 GMT: %IPC-SP-5-WATERMARK: 3396 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:56:21 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (4/4),process = RFSS worker process.
-Traceback= 40B67178 40B672DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0BC 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4
Sep 8 09:56:23 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:56:53 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:57:25 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:57:53 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B673DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 09:57:59 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:58:43 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:59:15 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:59:50 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:00:26 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:00:56 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Date: 2011-09-08 09:33:40 UTC We activate the netflow and we disable it.
vss-1b-6k(config)#mls flow ip interface-full
vss-1b-6k(config)#mls flow ipv6 interface-full
vss-1b-6k(config)#no mls flow ipv6 interface-full
% Reset for ipv6 protocol to the default flow mask type: none
vss-1b-6k(config)#no mls flow ip interface-full
vss-1b-6k(config)#
Date: 2011-09-08 09:32:23 UTC We haven't find the origin of the problem.
ah, again this morning:
Sep 8 09:28:37 GMT: %IPC-SP-5-WATERMARK: 1690 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:29:15 GMT: %IPC-SP-5-WATERMARK: 3411 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:29:48 GMT: %IPC-SP-5-WATERMARK: 3402 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:30:34 GMT: %IPC-SP-5-WATERMARK: 1694 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:31:11 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:31:43 GMT: %IPC-SP-5-WATERMARK: 3408 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:32:17 GMT: %IPC-SP-5-WATERMARK: 1697 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:33:03 GMT: %IPC-SP-5-WATERMARK: 1699 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:33:32 GMT: %IPC-SP-5-WATERMARK: 1694 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:34:03 GMT: %IPC-SP-5-WATERMARK: 3410 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
we have had this problem and it is on the netflow tables.
Sep 8 09:49:29 GMT: %IPC-SP-5-WATERMARK: 1732 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000