rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_green
icon_orange
icon_red
icon_green
icon_red
icon_red
icon_green
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#5632 — FS#9551 — pcc-1a/b-n7

Attached to Project— Dedicated Cloud
Maintenance
Backend / Core
CLOSED
100%
We're going to upgrade the supervisor cards of the 2 Nexus 7000 by passing a SUP-1 to a SUP-2.

This upgrade will enable routing on the Nexus.

The maintenance works will begin at 00:00 (CEST) This tuesday October 21st.
Date:  Thursday, 24 October 2013, 17:22PM
Reason for closing:  Done
Comment by OVH - Tuesday, 22 October 2013, 01:24AM

We are starting the maintenance.

We are switching the traffic on pcc-1b-n7 in order to upgrade the pcc-1a-n7.


Comment by OVH - Tuesday, 22 October 2013, 01:25AM

The traffic is switched.

We are suspending N7 to perform the upgrade.


Comment by OVH - Tuesday, 22 October 2013, 01:26AM

The N7 is switched off.

We are switching the cards. The traffic is flowing via pcc-1b-n7.


Comment by OVH - Tuesday, 22 October 2013, 01:26AM

Cards were replaced. We are starting.


Comment by OVH - Tuesday, 22 October 2013, 01:26AM

The N7 has started. We are re-applying the setting.


Comment by OVH - Tuesday, 22 October 2013, 01:27AM

The N7 is performing.

Some ports are down. We are looking up for the reason before performing the other switch.


Comment by OVH - Tuesday, 22 October 2013, 02:39AM

We have switched the optics and the ports are up.

We are switching the traffic on pcc-1a-n7 in order to upgrade pcc-1b-n7.


Comment by OVH - Tuesday, 22 October 2013, 02:39AM

Done

We are replacing the cards.


Comment by OVH - Tuesday, 22 October 2013, 02:40AM

The N7 is switched off.

We are switching the cards. The traffic is flowing via pcc-1a-n7.


Comment by OVH - Tuesday, 22 October 2013, 02:41AM

The N7 has restarted. We are re-applying the setting.


Comment by OVH - Tuesday, 22 October 2013, 03:22AM

Upgrade done


Comment by OVH - Tuesday, 22 October 2013, 08:25AM

Following the maintenance, we have noticed an issue with the vlan management of some hosts (a twenties). Thereafter troubleshooting, it doesn't seem like the network is at the origin of the issue. There's actually a board effect due to the maintenance which has an impact on these hosts. We are carrying on the troubleshooting along with the VMware.


Comment by OVH - Wednesday, 23 October 2013, 00:53AM

We will update the N7 in order to see if this fixes the issue.

We are starting the maintenance at 00:00 (CEST).


Comment by OVH - Wednesday, 23 October 2013, 00:53AM

We are starting the update.


Comment by OVH - Wednesday, 23 October 2013, 00:54AM

The update starts on pcc-1a-n7.

1 yes non-disruptive rolling
2 yes non-disruptive rolling
3 yes non-disruptive rolling
4 yes non-disruptive rolling
5 yes non-disruptive reset
6 yes non-disruptive reset
7 yes non-disruptive rolling
8 yes non-disruptive rolling
9 yes non-disruptive rolling
10 yes non-disruptive rolling


Comment by OVH - Wednesday, 23 October 2013, 01:56AM

One of the supervisor cards did a reboot loop.

We are suspending the update.

We are opening a ticket with the supplier.


Comment by OVH - Wednesday, 23 October 2013, 03:02AM

The card is failing. The supplier has sent a new one.


Comment by OVH - Wednesday, 23 October 2013, 04:14AM

We have received the new card. We are replacing it.


Comment by OVH - Wednesday, 23 October 2013, 07:39AM

The new card is inserted. It is being synchronised.


Comment by OVH - Wednesday, 23 October 2013, 07:40AM

The new card couldn't be synchronised.

We are contacting the supplier.

pcc-1a-n7# 2013 Oct 23 04:27:17 pcc-1a-n7 %SYSMGR-2-GSYNC_SNAPSHOT_SRVFAILED: Service "spm" on active supervisor failed to store its snapshot (error-id 0x4048000C).
2013 Oct 23 04:27:17 pcc-1a-n7 %SYSMGR-2-STANDBY_BOOT_FAILED: Standby supervisor failed to boot up.


Comment by OVH - Thursday, 24 October 2013, 00:20AM

We will update switches cold swap.

We will switch traffic on pcc-1b-n7 in order to upgrade pcc-1a-n7.


Comment by OVH - Thursday, 24 October 2013, 00:20AM

Traffic is switched.

We will start the update.


Comment by OVH - Thursday, 24 October 2013, 00:21AM

We rebooted it.


Comment by OVH - Thursday, 24 October 2013, 00:21AM

N7 is up. Linecards booted.


Comment by OVH - Thursday, 24 October 2013, 00:22AM

N7 is up. We are resuming traffic.


Comment by OVH - Thursday, 24 October 2013, 00:23AM

Traffic was resumed. Some ports are down.


Comment by OVH - Thursday, 24 October 2013, 02:14AM

We switched traffic on pcc-1a-n7 in order to upgrade pcc-1b-n7.


Comment by OVH - Thursday, 24 October 2013, 09:45AM

There were some incidents during the switchover. Some of the N5s stopped forwarding the traffic onto pcc-1a-n7.

We have stopped the update for tonight in order to investigate this N5 forwarding issue. p


Comment by OVH - Thursday, 24 October 2013, 15:33PM

oles: I'm taking over the maintenance.


Comment by OVH - Thursday, 24 October 2013, 15:52PM

We're shutting down all ports on chassis A, which is in 6.2,
so that the traffic flows only via 6.1.

We will update A to move it onto 6.1 (hot or cold swap).

We will put get all ports back up and everything
should work as before the update.

We will then launch the instant update of B onto
version 6.2, and we'll see if it still crashes on the
2nd supervisor. If so, we will let Cisco fix the problem for us
however long it takes.


Comment by OVH - Thursday, 24 October 2013, 15:53PM

Go:
We're cutting all the ports on A.
Traffic will flow via B.
No downtime expected.


Comment by OVH - Thursday, 24 October 2013, 17:21PM

All done, we're in 6.1. We 'll talk to Cisco
to find out why it crashed.