The server is not responding.
We are rebooting it.
Update(s):
Date: 2014-03-27 14:51:38 UTC All accounts have been migrated.
Date: 2014-03-05 08:28:22 UTC 99% of the accounts haves been migrated.
We are checking the remaining accounts.
Date: 2014-03-03 16:06:07 UTC 80% of the website have been moved to a new filer in read-write mode with the backup data. The migrations should be completed tonight.
We are also still working to recover the data from the original filer.
Date: 2014-02-27 14:01:46 UTC We had a series of hardware problems on the server which
created a ZFS filesystem corruption. Data is readable but
the server is unstable (the system is crashing every 30 minutes).
We're trying to restabilise the system and begin to recover the
data on a new filer. But we need to find a way to block all
automatic ZFS operations, to making the pool read-only so
that it doesn't make everything crash again.
We're also taking down the last backup stored in Roubaix
The operation would take 24 hours so to speed things up, we
retrieved the backup disks directly in Roubaix and we will go
directly to Paris with them. It will be faster.
So in 3-4 hours times, the filer and the backup data should be up.
This will give life to the 1209 websites affected by the failure.
We hope that refreshing this backup with data from the instable
filer we think we would retrieve in approx. 24 hours.
We need to look into or even patch the ZFS code to make the filer
stable at least in read-only mode.
We apologise for the trouble. Total failure of a filer is very very
rare. In this instance the backup is there, we have it so there's
nothing to worry about, and our engineers are working on the most
recent data on the filer.
Date: 2014-02-27 06:36:46 UTC The filer is having instabilities again, we are intervening
Date: 2014-02-26 11:29:48 UTC The service is still unstable for this filer, we have to disable it.
We are enabling a cluster which will be dedicate to filerz55.
Date: 2014-02-26 09:16:20 UTC We have doubled the log disk redundancy and launched a verification of the whole data pool.
The service is up but remains disrupted by the operation in progress which should take over 6 hours.
Date: 2014-02-26 09:14:27 UTC The system is unstable.
We are changing the given pool configuration.
Date: 2014-02-26 09:13:51 UTC We continue to monitor the server to check that the problem does not recur.
Date: 2014-02-26 09:12:26 UTC The server is up again.
Date: 2014-02-26 06:50:41 UTC We are transferring the data disks in the new system.
Date: 2014-02-26 06:50:01 UTC The whole cluster is affected by the filer
Date: 2014-02-26 06:49:36 UTC We are replacing the server with a spare.
Date: 2014-02-26 06:49:15 UTC The server is back to normal.
Date: 2014-02-26 06:48:57 UTC We have detected a failure on the server.
We are performing a hardware check.