OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#4461 — vss-2-6k
Incident Report for Network & Infrastructure
Resolved
regarding the task on vss-2-6k,
http://status.ovh.net/?do=details&id=363

We are going to change its configuration. Router should be restarted. It will take between 15 to 30 minutes, the time that all services are back.


Update(s):

Date: 2010-08-11 01:34:53 UTC
we are running the vss-2 to SXI4 and vss-1 to SXI3

Date: 2010-08-11 01:34:21 UTC
However, if you enter all sections ARP (ip / mac) we remain in the configuration of vss-2 (without the vss)

vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 346496 199544 1736 9.27% 10.86% 7.00% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 347292 199971 1736 12.55% 11.00% 7.10% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 348988 201194 1734 8.71% 11.00% 7.29% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 349636 201672 1733 6.07% 10.60% 7.27% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 349824 201835 1733 6.07% 10.60% 7.27% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 352136 203472 1730 5.67% 9.90% 7.39% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 352236 203582 1730 5.67% 9.90% 7.39% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 352644 203863 1729 4.95% 9.22% 7.33% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 353312 204440 1728 6.31% 8.99% 7.31% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 353424 204536 1727 6.31% 8.99% 7.31% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 354572 206110 1720 4.00% 7.54% 7.11% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 354652 206216 1719 4.39% 7.29% 7.06% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 354716 206331 1719 4.39% 7.29% 7.06% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 355460 207418 1713 6.07% 7.35% 7.08% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 355620 207676 1712 3.91% 7.08% 7.03% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 355780 207846 1711 4.39% 6.86% 6.99% 0 ARP Input

We running to 7%.

The same thing on vss-1 is not changing.


Date: 2010-08-11 01:30:55 UTC
At the same time

vss-1 is in vss configuration on 2 chassis

vss-1-6k#sh mac address-table count
MAC Entries for all vlans :
Dynamic Address Count: 15335
Static Address (User-defined) Count: 203
Total MAC Addresses In Use: 15538
Total MAC Addresses Available: 98304

vss-1-6k#sh proc cpu sorted 5se | i \\ ARP Inpu
11 2323435524 945282146 2457 14.13% 9.71% 9.00% 0 ARP Input
vss-1-6k#sh proc cpu sorted 5se | i \\ ARP Inpu
11 2323435736 945282250 2457 5.67% 9.39% 8.95% 0 ARP Input
vss-1-6k#sh proc cpu sorted 5se | i \\ ARP Inpu
11 2323435912 945282352 2457 5.67% 9.39% 8.95% 0 ARP Input
vss-1-6k#sh proc cpu sorted 5se | i \\ ARP Inpu
11 2323436364 945282523 2457 6.23% 9.14% 8.90% 0 ARP Input
vss-1-6k#sh proc cpu sorted 5se | i \\ ARP Inpu
11 2323439488 945283857 2457 8.15% 8.44% 8.75% 0 ARP Input
vss-1-6k#sh proc cpu sorted 5se | i \\ ARP Inpu
11 2323439592 945283919 2457 5.83% 8.23% 8.70% 0 ARP Input
vss-1-6k#sh proc cpu sorted 5se | i \\ ARP Inpu
11 2323455304 945290219 2457 12.20% 8.58% 8.76% 0 ARP Input

We are running at the average of 8.7% of CPU

and vss-2 (which is no longer in vss configuration):

vss-2-6k#sh mac address-table count
MAC Entries for all vlans :
Dynamic Address Count: 13852
Static Address (User-defined) Count: 219
Total MAC Addresses In Use: 14071
Total MAC Addresses Available: 98304

vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 357252 209768 1703 4.00% 5.60% 6.65% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 358652 211589 1695 3.51% 4.92% 6.37% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 358972 212102 1692 3.51% 4.69% 6.28% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 359112 212350 1691 3.27% 4.58% 6.22% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 366596 220474 1662 8.23% 4.50% 5.51% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 366708 220685 1661 4.23% 4.47% 5.48% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 366808 220868 1660 3.67% 4.41% 5.45% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 366924 221085 1659 3.67% 4.41% 5.45% 0 ARP Input
vss-2-6k#sh proc cpu sorted 5se | i \\ ARP Input
11 369844 225498 1640 3.19% 3.69% 4.98% 0 ARP Input
vss-2-6k#

We running to less than 5% of CPU and that is always decreasing.



Date: 2010-08-11 01:26:00 UTC
The consequence: BGP works better without the VSS. it does not consume CPU and does not monopolize it.

At the level of ARP, the CPU load is much lower than what we had before. In addition, if we remove couples ARP in hardware configuration, the CPU is less loaded. Suddenly we have tested this configuration on the vss-1 is still configuration vss and it does not at all matches.

http://status.ovh.co.uk/?do=details&id=383

Date: 2010-08-11 01:20:06 UTC
4637 packets transmitted, 1388 received, 70% packet loss, time 4678932ms
oles@ping:~$ echo \"(4637-1388)/60\" | bc -l
54.15000000000000000000

We did it in 54 minutes. not bad.


Date: 2010-08-11 01:19:25 UTC
Renaming the interfaces is accomplished. The uplink to the berries are raised and traffic resumed.

Date: 2010-08-11 01:18:14 UTC
the boot is done.
The establishment of the configuration has not been done. We had to reedit the configuration several times. In vss configuration port accepts up to 512 channels. In configurations without vss is clamped at 256. We had to change again the whole ports configuration.

card 6 is dead. We are replacing it.

Aug 11 01:37:06 20g.vss-2-6k.routers.ovh.net 8887: Aug 10 23:36:48.434:
%SYS-DFC6-5-RESTART: System restarted --

Aug 11 01:37:18 20g.vss-2-6k.routers.ovh.net 8888: Aug 11 00:36:54 GMT:
%DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...

Aug 11 01:37:24 20G.ldn-1-6k.routers.ovh.net 38635: Aug 11 00:37:06 GMT:
%BGP-4-MAXPFX: No. of prefix received from 198.32.176.20 (afi 0) reaches
15289, max 20000
Aug 11 01:37:32 20g.vss-2-6k.routers.ovh.net 8889: Aug 11 00:37:14 GMT:
%PM_SCP-SP-1-LCP_FW_ERR: System resetting module 6 to recover from
error: Linecard received system exception. Errcode =
Aug 11 01:37:32 20g.vss-2-6k.routers.ovh.net 8890: Aug 11 00:37:14 GMT:
%OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled 'Off (Module
Reset due to exception or user request)'
Aug 11 01:37:32 20g.vss-2-6k.routers.ovh.net 8891: .Aug 11 00:37:14 GMT:
%XDR-6-XDRIPCNOTIFY: Message not sent to slot 6/0 (6) because of IPC
error queue flush. Disabling linecard. (Expected during linecard OIR or
system reloads)

Date: 2010-08-11 01:15:15 UTC
The chassis #1 standalone restart did not work as expected. In that configuration, it is not possible to use numbers of PortChannels> 256 which was the case of uplinks to switches in bays. We are renaming these Po to use numbers
Parallel to this problem, the card # 6 does not reboot properly, probably due to a hard-bp. We replace it by the card # 6 of chassis # 2 which is now offline.

Date: 2010-08-11 01:09:59 UTC
5 4 3 2 1 ... go
Posted Aug 11, 2010 - 01:09 UTC