Get webhook notifications whenever Network & Infrastructure creates an incident, updates an incident, resolves an incident or changes a component status.
We are going to update the network equipment that manage
the vrack.
There is no expected crash.
The soft update will be done urgently.
Update(s):
Date: 2011-03-01 16:02:42 UTC up.
and the problem is fixed:
Local suspended VLANs - - -
We had to go the way end of the approach yesterday and force the restart of B since it doesn't ping. It might be the B which has hardware problem, that's why we couldn't put it on the spot?
We'll know when we'll update the new version. We'll do it on A.
Date: 2011-03-01 15:56:09 UTC up. has generated 30 seconds of unavailability while reactivating. We intervened to re-cut. it's not automatic automatic ...
We're reactivating.
Date: 2011-03-01 15:53:26 UTC The problem:
Local suspended VLANs - 2009,2174,2215,2241 -
The B in reboot. During reboot:
Local suspended VLANs - - -
Date: 2011-03-01 15:15:14 UTC We are facing some problems to activate the new vrack.
We are going to cut one of the 2 sides then restart it.
Date: 2011-03-01 10:10:26 UTC Everything is up ... ouff ! what an adventure !
Date: 2011-03-01 10:09:22 UTC We will reactivate the double attach
Date: 2011-03-01 10:08:45 UTC Updated.
The A does not ping. It functions, there is no problem, yet, it does not ping.
The B responds to pings...
Date: 2011-03-01 10:07:14 UTC There is still 3 modules. We will put them now on the spot.
Date: 2011-03-01 10:06:02 UTC 105 up
02/28/2011 23:13:19.462460: Deleting route to FEX
02/28/2011 23:13:19.471097: Module disconnected
02/28/2011 23:13:19.473100: Deleting route to FEX
02/28/2011 23:13:19.482795: Module disconnected
02/28/2011 23:13:19.483923: Offlining Module
02/28/2011 23:13:19.484820: Module Offline Sequence
02/28/2011 23:13:57.18916: Module Offline
02/28/2011 23:15:15.289667: Module register received
02/28/2011 23:15:15.290642: Image Version Mismatch
02/28/2011 23:15:15.290760: Registration response sent
02/28/2011 23:15:15.299115: Requesting satellite to download image
02/28/2011 23:20:54.557415: Image preload successful.
02/28/2011 23:21:09.469765: Deleting route to FEX
02/28/2011 23:21:09.477274: Module disconnected
02/28/2011 23:21:09.479010: Module Offline
02/28/2011 23:21:09.509092: Deleting route to FEX
02/28/2011 23:21:09.516864: Module disconnected
02/28/2011 23:21:09.519378: Offlining Module
02/28/2011 23:21:09.571852: Deleting route to FEX
02/28/2011 23:21:09.579704: Module disconnected
02/28/2011 23:21:09.582199: Offlining Module
02/28/2011 23:23:18.760250: Module register received
02/28/2011 23:23:18.761914: Registration response sent
02/28/2011 23:23:18.950577: Module Online Sequence
02/28/2011 23:23:47.432548: Module Online
Date: 2011-03-01 10:05:37 UTC 106: up
02/28/2011 23:13:18.153540: Deleting route to FEX
02/28/2011 23:13:18.161159: Module disconnected
02/28/2011 23:13:18.163105: Deleting route to FEX
02/28/2011 23:13:18.180811: Module disconnected
02/28/2011 23:13:18.181901: Offlining Module
02/28/2011 23:13:18.182778: Module Offline Sequence
02/28/2011 23:13:57.39114: Module Offline
02/28/2011 23:15:53.785707: Module register received
02/28/2011 23:15:53.786738: Image Version Mismatch
02/28/2011 23:15:53.786860: Registration response sent
02/28/2011 23:15:53.796040: Requesting satellite to download image
02/28/2011 23:20:43.597792: Image preload successful.
02/28/2011 23:21:00.306001: Deleting route to FEX
02/28/2011 23:21:00.313451: Module disconnected
02/28/2011 23:21:00.315127: Module Offline
02/28/2011 23:21:00.345386: Deleting route to FEX
02/28/2011 23:21:00.352948: Module disconnected
02/28/2011 23:21:00.356469: Offlining Module
02/28/2011 23:21:00.416404: Deleting route to FEX
02/28/2011 23:21:00.424884: Module disconnected
02/28/2011 23:21:00.427329: Offlining Module
02/28/2011 23:22:40.732535: Module register received
02/28/2011 23:22:40.733821: Registration response sent
02/28/2011 23:22:40.995007: Module Online Sequence
02/28/2011 23:23:08.963722: Module Online
Date: 2011-03-01 10:05:16 UTC 108: up
02/28/2011 23:13:16.659772: Deleting route to FEX
02/28/2011 23:13:16.667174: Module disconnected
02/28/2011 23:13:16.668717: Deleting route to FEX
02/28/2011 23:13:16.676241: Module disconnected
02/28/2011 23:13:16.677497: Offlining Module
02/28/2011 23:13:16.678461: Module Offline Sequence
02/28/2011 23:13:56.454277: Module Offline
02/28/2011 23:14:02.346991: Module register received
02/28/2011 23:14:02.347976: Image Version Mismatch
02/28/2011 23:14:02.348095: Registration response sent
02/28/2011 23:14:02.357064: Requesting satellite to download image
02/28/2011 23:19:04.297376: Image preload successful.
02/28/2011 23:19:21.113000: Deleting route to FEX
02/28/2011 23:19:21.120607: Module disconnected
02/28/2011 23:19:21.122301: Module Offline
02/28/2011 23:19:21.153038: Deleting route to FEX
02/28/2011 23:19:21.161373: Module disconnected
02/28/2011 23:19:21.163916: Offlining Module
02/28/2011 23:19:21.215645: Deleting route to FEX
02/28/2011 23:19:21.223132: Module disconnected
02/28/2011 23:19:21.225972: Offlining Module
02/28/2011 23:21:01.452135: Module register received
02/28/2011 23:21:01.453408: Registration response sent
02/28/2011 23:21:01.675645: Module Online Sequence
02/28/2011 23:21:29.509689: Module Online
Date: 2011-03-01 09:52:17 UTC We will start the inter
Date: 2011-03-01 09:51:45 UTC The updating is not going well as expected. We have an equipment
type which does not support (apparently) the updating on the
spot. Or, is it not good? In all cases, this is not like all the
updatings which has generated 0 downtime.
We will have to update tonight with a reboot, thus, with a cut off
of about 1-3 minutes while rebooting the equipment
in question and updating them.
There are 10 equipments which we will do 1 by 1.
Finally, since this is a problem which we have had before
and which we cannot reproduce in the labo (we no longer have in stock
this equipment series) 1-3 minutes at the best by equipment.
We will have all these starting from midnight.
Date: 2011-02-28 16:40:19 UTC The FEX type N2K-C2148T-1GE has problems while
N2K-C2248TP-1GE and N2K-C2232PP-10GE has no problems.
Date: 2011-02-28 16:36:55 UTC Nothing to do. The update is blocked.
Date: 2011-02-28 16:36:18 UTC Let's go again
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive none
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
Images will be upgraded according to following table:
Module Image Running-Version New-Version
Upg-Required
------ ---------- ---------------------- ----------------------
------------
1 system 5.0(2)N2(1) 5.0(2)N2(1)
no
1 kickstart 5.0(2)N2(1) 5.0(2)N2(1)
no
1 bios v1.3.0(09/08/09) v1.3.0(09/08/09)
no
100 fex 4.2(1)N1(1) 5.0(2)N2(1)
yes
101 fex 4.2(1)N1(1) 5.0(2)N2(1)
yes
102 fex 4.2(1)N1(1) 5.0(2)N2(1)
yes
103 fex 4.2(1)N1(1) 5.0(2)N2(1)
yes
104 fexth 4.2(1)N1(1) 5.0(2)N2(1)
yes
105 fex 4.2(1)N1(1) 5.0(2)N2(1)
yes
106 fex 4.2(1)N1(1) 5.0(2)N2(1)
yes
107 fex 4.2(1)N1(1) 5.0(2)N2(1)
yes
108 fex 4.2(1)N1(1) 5.0(2)N2(1)
yes
109 fexth 4.2(1)N1(1) 5.0(2)N2(1)
yes
110 fexth 4.2(1)N1(1) 5.0(2)N2(1)
yes
1 power-seq v1.2 v1.2
no
Date: 2011-02-28 16:35:24 UTC 2011 Feb 28 16:40:21 cloud-1b-n5 %SATCTRL-FEX100-2-SATCTRL_IMAGE:
FEX100 Image update in progress.
2011 Feb 28 16:40:31 cloud-1b-n5 %SATCTRL-FEX101-2-SATCTRL_IMAGE:
FEX101 Image update in progress.
2011 Feb 28 16:40:41 cloud-1b-n5 %SATCTRL-FEX102-2-SATCTRL_IMAGE:
FEX102 Image update in progress.
2011 Feb 28 16:40:51 cloud-1b-n5 %SATCTRL-FEX103-2-SATCTRL_IMAGE:
FEX103 Image update in progress.
2011 Feb 28 16:41:01 cloud-1b-n5 %SATCTRL-FEX104-2-SATCTRL_IMAGE:
FEX104 Image update in progress.
2011 Feb 28 16:41:11 cloud-1b-n5 %SATCTRL-FEX105-2-SATCTRL_IMAGE:
FEX105 Image update in progress.
2011 Feb 28 16:41:21 cloud-1b-n5 %SATCTRL-FEX106-2-SATCTRL_IMAGE:
FEX106 Image update in progress.
2011 Feb 28 16:41:31 cloud-1b-n5 %SATCTRL-FEX107-2-SATCTRL_IMAGE:
FEX107 Image update in progress.
2011 Feb 28 16:41:41 cloud-1b-n5 %SATCTRL-FEX108-2-SATCTRL_IMAGE:
FEX108 Image update in progress.
2011 Feb 28 16:41:51 cloud-1b-n5 %SATCTRL-FEX109-2-SATCTRL_IMAGE:
FEX109 Image update in progress.
2011 Feb 28 16:42:01 cloud-1b-n5 %SATCTRL-FEX110-2-SATCTRL_IMAGE:
FEX110 Image update in progress.
2011 Feb 28 16:42:05 cloud-1b-n5 %SATCTRL-FEX101-2-SATCTRL_IMAGE:
FEX101 Image update failed [/isan/plugin_img/fex.bin]: File transfer
error
2011 Feb 28 16:42:16 cloud-1b-n5 %SATCTRL-FEX100-2-SATCTRL_IMAGE:
FEX100 Image update failed [/isan/plugin_img/fex.bin]: File transfer
error
2011 Feb 28 16:42:20 cloud-1b-n5 %SATCTRL-FEX102-2-SATCTRL_IMAGE:
FEX102 Image update failed [/isan/plugin_img/fex.bin]: File transfer
error
2011 Feb 28 16:42:21 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4669]
2011 Feb 28 16:42:21 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4672]
2011 Feb 28 16:42:22 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4677]
2011 Feb 28 16:42:22 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4541]
2011 Feb 28 16:42:23 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4680]
2011 Feb 28 16:42:23 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4686]
2011 Feb 28 16:42:24 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4682]
2011 Feb 28 16:42:25 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4664]
2011 Feb 28 16:42:26 cloud-1b-n5 %DAEMON-3-SYSTEM_MSG: tftpd:
read(ack): Connection refused - in.tftpd[4665]
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 100 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 101 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 102 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 103 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 104 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 105 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 106 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 108 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 109 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 110 is online
2011 Feb 28 16:43:26 cloud-1b-n5 %PFMA-2-FEX_STATUS: Fex 107 is online
2011 Feb 28 16:43:56 cloud-1b-n5 %SATCTRL-FEX107-2-SATCTRL_IMAGE:
FEX107 Image update failed [/isan/plugin_img/fex.bin]: File transfer
error
2011 Feb 28 16:44:12 cloud-1b-n5 %SATCTRL-FEX106-2-SATCTRL_IMAGE:
FEX106 Image update failed [/isan/plugin_img/fex.bin]: File transfer
error
2011 Feb 28 16:46:42 cloud-1b-n5 %SATCTRL-FEX104-2-SATCTRL_IMAGE:
FEX104 Image update complete.
2011 Feb 28 16:47:24 cloud-1b-n5 %SATCTRL-FEX103-2-SATCTRL_IMAGE:
FEX103 Image update complete.
2011 Feb 28 16:47:36 cloud-1b-n5 %SATCTRL-FEX108-2-SATCTRL_IMAGE:
FEX108 Image update complete.
2011 Feb 28 16:48:04 cloud-1b-n5 %SATCTRL-FEX109-2-SATCTRL_IMAGE:
FEX109 Image update complete.
2011 Feb 28 16:48:17 cloud-1b-n5 %SATCTRL-FEX105-2-SATCTRL_IMAGE:
FEX105 Image update complete.
2011 Feb 28 16:48:18 cloud-1b-n5 %SATCTRL-FEX110-2-SATCTRL_IMAGE:
FEX110 Image update complete.
2011 Feb 28 16:48:42 cloud-1b-n5 %SYSMGR-3-SERVICE_TERMINATED: Service
\"installer\" (PID 4463) has finished with error code
SYSMGR_EXITCODE_FAILURE_NOCALLHOME (20).
not so ... good ...
Date: 2011-02-28 16:23:30 UTC Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling