OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#5300 — pdc1-1-c1
Incident Report for Network & Infrastructure
Resolved
We have a problem at the level of the hard for this router.

Update(s):

Date: 2011-04-02 23:31:45 UTC
The router is now back to the normal status.

The chassis itself seems out of the cause but we failed to reproduce the boot problem of origin sups cards (and of the 1st spare card) on the test chassis.
It is probable in any case that we'd cumulate many problems at a time.

We do apologize for this downtime of our customers on pdc1-1 in 1x2. Clients in 2x2 or 2x4 have not been impacted.


Date: 2011-04-02 22:36:24 UTC
The new card #2 has booted successfully. We re-descend the setting of the routing module m2.


Date: 2011-04-02 22:30:09 UTC
We are going to intervene within few minutes to reinsert a new card on the slot #2. Probably the chassis crash is the problem's origin. Since we have been able to boot the new card without problem #1, we had good reasons to hope that it would be no problem now. Nevertheless, at this stage and regarding the problems that we faced, we could not be ensure anything. If the chassis is the problem's origin, we'll replace it.


Date: 2011-04-02 19:48:07 UTC
The chassis is up on a new card #1. The configuration is synchronised.
We are going to test the precedent cards in lab and prepare a new card #2
from scratch.

We will intervene most probably during the night in order
to reinsert the card #2.

Date: 2011-04-02 17:39:16 UTC
After investigation, it turned out that we are dealing with different
crashes of different levels rather than with a chassis problem.
The chassis was booted on a new spare. We are downgrading the configuration
of the chassis manually.

Date: 2011-04-02 17:22:48 UTC
Impossible to start any card on the chassis (!).
We are preparing ourselves to replace the whole chassis urgently.

Date: 2011-04-02 16:53:35 UTC
Neither the original card #2 nor the spare card does restart in the slot1 or 2. We are reinserting the original card #1.

Date: 2011-04-02 16:35:55 UTC
We are doing the reboot to have a cold restart of card #2.

Date: 2011-04-02 16:35:21 UTC
We are doing the hard reboot of the chassis in 30 minutes. Another incident is in progress on p19 (#5301).
The chassis works now in degraded mode but it is stable.

Date: 2011-04-02 16:30:47 UTC
Summary of the actions that we did so far:
The router runs currently on the card #1. We tried to replace the card #2 that was crashed and displayed by the router as a default.
As we insert the card of the spare in the slot #2 , the chassis bloks againa. So, we restarted it in hard only on the card #1 in order to have a cold restart of card #1.

After this restarting, new trial of insertion of the spare card in slot #2 , the whole ports of the chassis are in default (!). We take off this time only the card #2 and we have a normal functionning.

We suspect the card #1 to be the origin of the problem even though it works currently in standalone.
A new reboot hard of the chassis will be necessary to restart on the card #2. We will restart after that the card #1 by a spare.
Posted Apr 02, 2011 - 15:52 UTC