OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#8381 — rbx-s4-6k
Scheduled Maintenance Report for Network & Infrastructure
Completed
The router has rebooted.

1y51w: IPC: Cannot find original message for ACK HDR: 8ED28C8 src:
10019, dst: 2150000, index: 0, seq: 55503, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A8E0, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8FC7FC8 src:
10019, dst: 2150000, index: 0, seq: 55508, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A91F, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8F310C8 src:
10019, dst: 2150000, index: 0, seq: 55513, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A955, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8F70EC8 src:
10007, dst: 2150000, index: 1, seq: 65492, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A984, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8F6D6C8 src:
10019, dst: 2150000, index: 0, seq: 55518, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A98B, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8EE55C8 src:
10007, dst: 2150000, index: 1, seq: 65497, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A9CC, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8F738C8 src:
10019, dst: 2150000, index: 0, seq: 55523, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824AA00, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8EF3CC8 src:
10007, dst: 2150000, index: 1, seq: 65502, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824AA1D, lo: 0
Apr 5 10:18:27 GMT: %C6K_PLATFORM-2-PEER_RESET: RP is being reset by
the SP
*** System received a Software forced crash ***
signal= 0x17, code= 0x24, context= 0x46635334
PC = 0x42da3e6c, SP = 0x44944f68, RA = 0x413e9a14
Cause Reg = 0x00003820, Status Reg = 0x34008002
rommon 1 >
No warm reboot Storage
*** System received an unknown failure ***

Update(s):

Date: 2013-04-05 13:24:22 UTC
The router has rebooted. Traffic go to rbx-s3-6k. We had a downtime of 33 seconds.

Date: 2013-04-05 13:21:18 UTC
The router is stable.

result:
1 4x10G card dead
1 sup720 card dead

Sorry for the inconvenience caused. We had 4 dowtime of 30 seconds.



Date: 2013-04-05 13:20:01 UTC
We'll put the traffic back.

Date: 2013-04-05 13:19:30 UTC
The router seems to be stable. We reload it one last time then we put in production.

Date: 2013-04-05 13:18:43 UTC
All ports are still cut. We reload it.

Date: 2013-04-05 13:18:13 UTC
1: ok
2: ok
3: ok
4: ok
6: ok
7: ok
8: ok
9: ok

Date: 2013-04-05 13:17:41 UTC
9: ok

Date: 2013-04-05 13:17:08 UTC
1: ok

Date: 2013-04-05 13:16:51 UTC
Apr 5 11:40:34 GMT: %SYS-SP-5-RELOAD: Reload requested by Delayed
Reload. Reload Reason: FIB Protocol Allocation mismatch.
Apr 5 11:40:35 GMT: %OIR-SP-6-CONSOLE: Changing console ownership to
switch processor

Date: 2013-04-05 13:16:16 UTC
We stop all card before it crashes again.

Stable.

We will activate 8 cards one by one to see why one make the router crash.

Date: 2013-04-05 13:15:04 UTC
Apr 5 11:40:34 GMT: %SYS-SP-5-RELOAD: Reload requested by Delayed
Reload. Reload Reason: FIB Protocol Allocation mismatch.
Apr 5 11:40:35 GMT: %OIR-SP-6-CONSOLE: Changing console ownership to
switch processor

Date: 2013-04-05 13:14:40 UTC
Boot problem with the new image. File problem of corrupted configuration.

We change it.

Boot problem of the new image.
We change the image to the least recent one.

It's booting. The configuration file has been loaded.

Date: 2013-04-05 11:08:11 UTC
The router crashed again. We change the IOS version.

rommon 2 > reset

Date: 2013-04-05 11:07:41 UTC
replaced

Date: 2013-04-05 11:07:22 UTC
One of the cards is dead.

1y51w: IPC: Cannot find original message for ACK HDR: 8ED28C8 src:
10019, dst: 2150000, index: 0, seq: 55503, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A8E0, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8FC7FC8 src:
10019, dst: 2150000, index: 0, seq: 55508, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A91F, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8F310C8 src:
10019, dst: 2150000, index: 0, seq: 55513, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A955, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8F70EC8 src:
10007, dst: 2150000, index: 1, seq: 65492, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A984, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8F6D6C8 src:
10019, dst: 2150000, index: 0, seq: 55518, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A98B, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8EE55C8 src:
10007, dst: 2150000, index: 1, seq: 65497, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824A9CC, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8F738C8 src:
10019, dst: 2150000, index: 0, seq: 55523, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824AA00, lo: 0
1y51w: IPC: Cannot find original message for ACK HDR: 8EF3CC8 src:
10007, dst: 2150000, index: 1, seq: 65502, sz: 20, type: 11, flags: 2,
ext_flags: 0x0, hi: 8824AA1D, lo: 0
Apr 5 10:18:27 GMT: %C6K_PLATFORM-2-PEER_RESET: RP is being reset by
the SP
*** System received a Software forced crash ***
signal= 0x17, code= 0x24, context= 0x46635334
PC = 0x42da3e6c, SP = 0x44944f68, RA = 0x413e9a14
Cause Reg = 0x00003820, Status Reg = 0x34008002
rommon 1 >
No warm reboot Storage
*** System received an unknown failure ***
Posted Apr 05, 2013 - 11:06 UTC