OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#10813 — vRack 1.5
Incident
Report for Network & Infrastructure
Resolved
There is an incident on the infrastructure.
Update(s):
Date: 2014-05-21 04:11:36 UTC
On one of the switchs, we have perofrmed the ussual maintenance operations (adding new hosts).It Seems that the new FEX was drawn incorrectly and visibly looped. this does not explain why the full main switch is down.
2014 May 20 17:24:54 rbx2b-101-n6 %LLDP-FEX105-5-SERVER_ADDED: Server with Chassis ID dca5.f430.9dcc Port ID Eth105/1/11 management address 10.1.0.2 discovered on local port Eth105/1/21 in vlan 101 with enabled capability Bridge Router
2014 May 20 17:24:54 rbx2b-101-n6 %LLDP-FEX105-5-SERVER_ADDED: Server with Chassis ID dca5.f430.9dd6 Port ID Eth105/1/21 management address 10.1.0.2 discovered on local port Eth105/1/11 in vlan 101 with enabled capability Bridge Router
2014 May 20 17:24:55 rbx2b-101-n6 %FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac 000a.f394.2000 among ports Eth105/1/21 and Eth105/1/11 vlan 101 - Disabling dynamic learn notifications for 180 seconds
2014 May 20 17:27:55 rbx2b-101-n6 %FWM-2-STM_LEARNING_RE_ENABLE: Re enabling dynamic learning on all interfaces
2014 May 20 17:27:47 rbx2b-101-n6 %SYSMGR-FEX104-5-MODULE_ONLINE: System Manager has received notification of local module becoming online.
2014 May 20 17:27:47 rbx2b-101-n6 %SATCTRL-FEX104-2-SATCTRL: FEX-104 Module 1: Cold boot
2014 May 20 17:27:56 rbx2b-101-n6 %FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac 90e2.ba61.13f8 among ports Eth105/1/11 and Eth105/1/21 vlan 101 - Disabling dynamic learn notifications for 180 seconds
2014 May 20 17:28:07 rbx2b-101-n6 %LLDP-FEX104-5-SERVER_ADDED: Server with Chassis ID c471.fe0e.c404 Port ID Eth104/1/3 management address 10.1.0.2 discovered on local port Eth104/1/11 in vlan 1 with enabled capability Bridge Router
2014 May 20 17:28:07 rbx2b-101-n6 %LLDP-FEX104-5-SERVER_ADDED: Server with Chassis ID c471.fe0e.c40c Port ID Eth104/1/11 management address 10.1.0.2 discovered on local port Eth104/1/3 in vlan 1 with enabled capability Bridge Router
Date: 2014-05-20 16:07:11 UTC
the vPC link is down between 2 main switches Nexus 7000 it's
down for no reason.
2014 May 20 17:33:24 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel4096: Ethernet1/32 is down
2014 May 20 17:33:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel4096: first operational port changed from Ethernet1/32 to Ethernet2/32
2014 May 20 17:33:27 pcc-2b-n7 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethern2014 May 20 17:35:07 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel4096: Ethernet2/32 is down
et1/32 is down (Initializing)
2014 May 20 17:33:42 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:34:58 pcc-2b-n7 %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going down, suspending all vPCs on secondary
2014 May 20 17:35:07 pcc-2b-n7 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel4096: first operational port changed from Ethernet2/32 to none
2014 May 20 17:35:07 pcc-2b-n7 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet2/32 is down (Initializing)
2014 May 20 17:35:07 pcc-2b-n7 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel4096 is down (No operational members)
2014 May 20 17:35:18 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:35:21 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:35:47 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:35:55 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:36:17 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:36:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:36:46 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:36:55 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:37:17 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:37:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:37:47 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:37:55 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:38:17 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:38:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:38:47 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:38:55 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:39:17 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:39:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:39:47 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:39:51 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel4096: Ethernet1/32 is up
2014 May 20 17:39:51 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel4096: Ethernet2/32 is up
2014 May 20 17:39:51 pcc-2b-n7 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel4096: first operational port changed from none to Ethernet1/32
2014 May 20 17:39:52 pcc-2b-n7 %ETHPORT-5-IF_UP: Interface port-channel4096 is up in mode trunk
2014 May 20 17:39:52 pcc-2b-n7 %ETHPORT-5-IF_UP: Interface Ethernet1/32 is up in mode trunk
Date: 2014-05-20 11:16:46 UTC
We have isolated a faulty uplink between the vRack 1.5 and PCC infrastructures in Roubaix. No impact on the traffic is currently being detected.
Date: 2014-05-20 11:15:15 UTC
We suspect that there is a hardware issue on one of the PCC N7 interfaces.
Update(s):
Date: 2014-05-21 04:11:36 UTC
On one of the switchs, we have perofrmed the ussual maintenance operations (adding new hosts).It Seems that the new FEX was drawn incorrectly and visibly looped. this does not explain why the full main switch is down.
2014 May 20 17:24:54 rbx2b-101-n6 %LLDP-FEX105-5-SERVER_ADDED: Server with Chassis ID dca5.f430.9dcc Port ID Eth105/1/11 management address 10.1.0.2 discovered on local port Eth105/1/21 in vlan 101 with enabled capability Bridge Router
2014 May 20 17:24:54 rbx2b-101-n6 %LLDP-FEX105-5-SERVER_ADDED: Server with Chassis ID dca5.f430.9dd6 Port ID Eth105/1/21 management address 10.1.0.2 discovered on local port Eth105/1/11 in vlan 101 with enabled capability Bridge Router
2014 May 20 17:24:55 rbx2b-101-n6 %FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac 000a.f394.2000 among ports Eth105/1/21 and Eth105/1/11 vlan 101 - Disabling dynamic learn notifications for 180 seconds
2014 May 20 17:27:55 rbx2b-101-n6 %FWM-2-STM_LEARNING_RE_ENABLE: Re enabling dynamic learning on all interfaces
2014 May 20 17:27:47 rbx2b-101-n6 %SYSMGR-FEX104-5-MODULE_ONLINE: System Manager has received notification of local module becoming online.
2014 May 20 17:27:47 rbx2b-101-n6 %SATCTRL-FEX104-2-SATCTRL: FEX-104 Module 1: Cold boot
2014 May 20 17:27:56 rbx2b-101-n6 %FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac 90e2.ba61.13f8 among ports Eth105/1/11 and Eth105/1/21 vlan 101 - Disabling dynamic learn notifications for 180 seconds
2014 May 20 17:28:07 rbx2b-101-n6 %LLDP-FEX104-5-SERVER_ADDED: Server with Chassis ID c471.fe0e.c404 Port ID Eth104/1/3 management address 10.1.0.2 discovered on local port Eth104/1/11 in vlan 1 with enabled capability Bridge Router
2014 May 20 17:28:07 rbx2b-101-n6 %LLDP-FEX104-5-SERVER_ADDED: Server with Chassis ID c471.fe0e.c40c Port ID Eth104/1/11 management address 10.1.0.2 discovered on local port Eth104/1/3 in vlan 1 with enabled capability Bridge Router
Date: 2014-05-20 16:07:11 UTC
the vPC link is down between 2 main switches Nexus 7000 it's
down for no reason.
2014 May 20 17:33:24 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel4096: Ethernet1/32 is down
2014 May 20 17:33:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel4096: first operational port changed from Ethernet1/32 to Ethernet2/32
2014 May 20 17:33:27 pcc-2b-n7 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethern2014 May 20 17:35:07 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel4096: Ethernet2/32 is down
et1/32 is down (Initializing)
2014 May 20 17:33:42 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:34:58 pcc-2b-n7 %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going down, suspending all vPCs on secondary
2014 May 20 17:35:07 pcc-2b-n7 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel4096: first operational port changed from Ethernet2/32 to none
2014 May 20 17:35:07 pcc-2b-n7 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet2/32 is down (Initializing)
2014 May 20 17:35:07 pcc-2b-n7 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel4096 is down (No operational members)
2014 May 20 17:35:18 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:35:21 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:35:47 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:35:55 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:36:17 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:36:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:36:46 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:36:55 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:37:17 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:37:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:37:47 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:37:55 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:38:17 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:38:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:38:47 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:38:55 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:39:17 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:39:25 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/32: Ethernet2/32 is suspended
2014 May 20 17:39:47 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/32: Ethernet1/32 is suspended
2014 May 20 17:39:51 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel4096: Ethernet1/32 is up
2014 May 20 17:39:51 pcc-2b-n7 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel4096: Ethernet2/32 is up
2014 May 20 17:39:51 pcc-2b-n7 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel4096: first operational port changed from none to Ethernet1/32
2014 May 20 17:39:52 pcc-2b-n7 %ETHPORT-5-IF_UP: Interface port-channel4096 is up in mode trunk
2014 May 20 17:39:52 pcc-2b-n7 %ETHPORT-5-IF_UP: Interface Ethernet1/32 is up in mode trunk
Date: 2014-05-20 11:16:46 UTC
We have isolated a faulty uplink between the vRack 1.5 and PCC infrastructures in Roubaix. No impact on the traffic is currently being detected.
Date: 2014-05-20 11:15:15 UTC
We suspect that there is a hardware issue on one of the PCC N7 interfaces.