OVHcloud Web Hosting Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#9542 — Telephone exchange
Incident
Report for Web Cloud
Resolved
There is an incident on the telephone exchange.
Update(s):
Date: 2013-12-19 17:51:14 UTC
We have adapted the configuration of our devices following
feedback on white noise.
Date: 2013-12-19 09:14:43 UTC
We are reaching 50,000 lines on the new infrastructure A, which is just below the objective of 30% usage.
We are going to launch the migrations onto the new infrastructure B and we will also prepare to set up a third infrastructure.
Date: 2013-12-17 14:27:39 UTC
The new infrastructure B is being deployed in the cluster.
Date: 2013-12-11 17:12:31 UTC
The synchronisation was completed successfully. The actions requested from the manager are being worked through.
Date: 2013-12-11 15:48:24 UTC
We are proceeding to the cluster synchronisation phase.
During this phase, latency of several minutes may be
detectable in line configuration changes via the manager.
Date: 2013-12-05 21:15:17 UTC
We are starting data synchronisation between the devices.
Date: 2013-11-22 13:41:09 UTC
We starting the configuration and validation of the new device.
Date: 2013-11-20 16:20:38 UTC
We have received the second infrastructure of the new cluster. We will physically install it tonight.
Date: 2013-11-14 10:47:04 UTC
The migrations continue. A second new infrastructure arrives this week and will be added to the cluster. This will enable us to finish the migrations with the usage limitations per infrastructure that we have defined.
Date: 2013-11-07 12:39:43 UTC
The patch is in place on the new infrastructure. All problems listed previously have been fixed: white label, easy/mini pabx in // mode, SVA number redirection. We are preparing new migrations.
Date: 2013-11-06 17:25:21 UTC
We are also going to put in place a new version of SIP gateway tonight on the new infrastructure.
This version fix the problems experienced on some configurations with unconditional transfers.
Date: 2013-11-06 17:24:03 UTC
We're preparing the migration of the MGCP individual lines to the new infrastructure tonight.
Date: 2013-11-06 00:35:02 UTC
Maintenance completed.
Date: 2013-11-06 00:34:46 UTC
We started applying the patch.
Date: 2013-11-05 17:08:52 UTC
We are going to fix some supervision problems on the old infrastructure.
A patch will be put in place tonight.
Date: 2013-11-04 19:34:39 UTC
We have detected and fixed an issue with the Swiss lines on the new infrastructure.
Date: 2013-10-30 22:42:01 UTC
The specific issue of the DDI redirection to a line on the new infrastructure is fixed.
Date: 2013-10-30 15:36:49 UTC
We're working to resolve the last bugs bugs detected on the new infrastructure: no white label, cal pb on DDI specific configurations and
easy/mini pabx mode//, redirection number SVA. All lines affected are being migrated automatically onto the old infrastructure by a robot until resolution is complete.
At the same time we are setting up a second new infrastructure to follow-up the migration.
Date: 2013-10-25 08:19:02 UTC
The new configuration has been set up.
Date: 2013-10-25 08:18:33 UTC
We have updated the infrastructure configuration for the purposes of optimisation.
New calls will not be possible for a few seconds during the reload, no impact on communications in progress.
Date: 2013-10-22 22:59:58 UTC
We have fixed the call issue of a line on the new infrastructure to a cloudIvr or cloudHunting.
Regarding the forwarding of few call issues of a line on the new infrastructure to a gateway number or DDI redirection, we will perform bases resynchronisation.
Date: 2013-10-21 13:29:01 UTC
We have improved our traffic filtering rules, especially for bad packets on the MGCP. This will bring significant improvements for managing MGCP traffic.
Date: 2013-10-21 12:48:23 UTC
The affected lines have been migrated and the SIP rebooted on the modems.
Date: 2013-10-21 11:40:18 UTC
We are having further problems with the functioning of DDI redirections for the lines that have been migrated onto the new infrastructure.
We are reversing the migration for these lines and trying to find a solution, such as using Cloud Hunting for the redirects.
Date: 2013-10-20 03:41:19 UTC
Switching completed.
Date: 2013-10-20 03:41:03 UTC
We switched individual lines.
Date: 2013-10-20 03:40:44 UTC
We will switch compatible individual lines on the new infrastructure tonight, when there will be no more calls. The items will be re-registered at their next request, a reboot of the item can accelerate the re-registeration if necessary.
Date: 2013-10-20 03:37:06 UTC
We are preparing to switch individual compatible lines.
Date: 2013-10-20 03:36:03 UTC
Incoming/outgoing calls to OVH and to the external, as well as all the functionalities (answering machine / forwarding / time slots / reject / busy / ...) are fully operational.
We will monitor the stability of the infrastructure.
Date: 2013-10-20 03:33:02 UTC
Calls are correctly incoming and outgoing. We boosted our tests.
Date: 2013-10-20 03:32:12 UTC
We restarted services.
Date: 2013-10-20 03:31:48 UTC
We have implemented a new configuration. We will monitor its stability.
Date: 2013-10-20 03:31:19 UTC
We tested other settings to improve stability.
Date: 2013-10-20 03:30:42 UTC
We have reports about call errors. Called number busy, an incoming call impossible. We will investigate.
Date: 2013-10-20 03:29:02 UTC
Regarding the potential problem on the call forwarding or answering machine, we fixed it by setting an older version of SIP gateway.
Everything should work smoothly.
Date: 2013-10-19 06:25:07 UTC
SIP modems reloaded. We will keep monitoring the stability of the new infrastructure.
Date: 2013-10-19 06:24:20 UTC
100% of lines were switched, we will force reloading SIP on modems.
Date: 2013-10-19 06:23:08 UTC
50% of lines were switched.
Date: 2013-10-19 00:20:29 UTC
We switched SIP xDSL lines on the new infrastructure.
Date: 2013-10-19 00:19:59 UTC
Tests with the patch are successful for the moment.
We will keep testing.
Date: 2013-10-19 00:19:02 UTC
We tested a patch on the new infrastructure for call redirections and forward to the answering machine.
Date: 2013-10-19 00:17:34 UTC
We restarted the backup machine, we are aiming to reset all the services and the memory of the machine.
Date: 2013-10-18 21:18:29 UTC
Tests of the manager and calls are done successfully.
We are checking the incorporation of data between both infrastructures.
Date: 2013-10-18 19:38:04 UTC
We are following the tests on all internal adsl
lines of the parc as well as for individual lines,
non compatible with call forwarding.
We are actually starting the tests on the manager's
functionalities in order to limit the board effects.
Date: 2013-10-18 19:10:49 UTC
That's too much! As of 1 year, we had a new infrastructure
of the manufacturer which has been ordered, and set but
no longer in production due to a minor bug (the call
forwarding doesn't work properly). The patch which does fix
the issue isn't developed. We can not wait anymore: We will
switch ADSL BOX SIP customers in OVH to a new infrastructure
as expected a long time ago. This will unload the main
infrastructure which visible is overloaded (while we're
turning around 10% of tech specs of the manufacturer).
We are performing our last checks before resetting the SIP
packets coming from the ADSL BOX in OVH.
We will put on tonight and during the weekend to stabilise
the infrastructure and avoid new break-downs.
Meanwhile, we are proceeding to set 2 new infrastructures (we
will have a total of 4) and reduce the number of customers per
infra. We do know that all works properly with less than 100000
lines per infra. With a security margin, we aim to go down
to less than 30000 lines per infra. With this line, the problems
will not reoccur.
There we go ! it's going to be a long night ..
Date: 2013-10-18 16:53:18 UTC
An MGCP gateway crash has caused instabilities on the SIP. The problem is being analysed by the contractor.
All MGCP displays have been restarted.
Date: 2013-10-18 15:26:41 UTC
We have deactivated the probe.
Date: 2013-10-18 15:26:26 UTC
We're restarting the MGCP displays.
Date: 2013-10-18 15:26:01 UTC
We have a problem similar to yesterday, despite having changed the probe settings.
The SIP line are back up, the MGCP lines are being worked on.
Update(s):
Date: 2013-12-19 17:51:14 UTC
We have adapted the configuration of our devices following
feedback on white noise.
Date: 2013-12-19 09:14:43 UTC
We are reaching 50,000 lines on the new infrastructure A, which is just below the objective of 30% usage.
We are going to launch the migrations onto the new infrastructure B and we will also prepare to set up a third infrastructure.
Date: 2013-12-17 14:27:39 UTC
The new infrastructure B is being deployed in the cluster.
Date: 2013-12-11 17:12:31 UTC
The synchronisation was completed successfully. The actions requested from the manager are being worked through.
Date: 2013-12-11 15:48:24 UTC
We are proceeding to the cluster synchronisation phase.
During this phase, latency of several minutes may be
detectable in line configuration changes via the manager.
Date: 2013-12-05 21:15:17 UTC
We are starting data synchronisation between the devices.
Date: 2013-11-22 13:41:09 UTC
We starting the configuration and validation of the new device.
Date: 2013-11-20 16:20:38 UTC
We have received the second infrastructure of the new cluster. We will physically install it tonight.
Date: 2013-11-14 10:47:04 UTC
The migrations continue. A second new infrastructure arrives this week and will be added to the cluster. This will enable us to finish the migrations with the usage limitations per infrastructure that we have defined.
Date: 2013-11-07 12:39:43 UTC
The patch is in place on the new infrastructure. All problems listed previously have been fixed: white label, easy/mini pabx in // mode, SVA number redirection. We are preparing new migrations.
Date: 2013-11-06 17:25:21 UTC
We are also going to put in place a new version of SIP gateway tonight on the new infrastructure.
This version fix the problems experienced on some configurations with unconditional transfers.
Date: 2013-11-06 17:24:03 UTC
We're preparing the migration of the MGCP individual lines to the new infrastructure tonight.
Date: 2013-11-06 00:35:02 UTC
Maintenance completed.
Date: 2013-11-06 00:34:46 UTC
We started applying the patch.
Date: 2013-11-05 17:08:52 UTC
We are going to fix some supervision problems on the old infrastructure.
A patch will be put in place tonight.
Date: 2013-11-04 19:34:39 UTC
We have detected and fixed an issue with the Swiss lines on the new infrastructure.
Date: 2013-10-30 22:42:01 UTC
The specific issue of the DDI redirection to a line on the new infrastructure is fixed.
Date: 2013-10-30 15:36:49 UTC
We're working to resolve the last bugs bugs detected on the new infrastructure: no white label, cal pb on DDI specific configurations and
easy/mini pabx mode//, redirection number SVA. All lines affected are being migrated automatically onto the old infrastructure by a robot until resolution is complete.
At the same time we are setting up a second new infrastructure to follow-up the migration.
Date: 2013-10-25 08:19:02 UTC
The new configuration has been set up.
Date: 2013-10-25 08:18:33 UTC
We have updated the infrastructure configuration for the purposes of optimisation.
New calls will not be possible for a few seconds during the reload, no impact on communications in progress.
Date: 2013-10-22 22:59:58 UTC
We have fixed the call issue of a line on the new infrastructure to a cloudIvr or cloudHunting.
Regarding the forwarding of few call issues of a line on the new infrastructure to a gateway number or DDI redirection, we will perform bases resynchronisation.
Date: 2013-10-21 13:29:01 UTC
We have improved our traffic filtering rules, especially for bad packets on the MGCP. This will bring significant improvements for managing MGCP traffic.
Date: 2013-10-21 12:48:23 UTC
The affected lines have been migrated and the SIP rebooted on the modems.
Date: 2013-10-21 11:40:18 UTC
We are having further problems with the functioning of DDI redirections for the lines that have been migrated onto the new infrastructure.
We are reversing the migration for these lines and trying to find a solution, such as using Cloud Hunting for the redirects.
Date: 2013-10-20 03:41:19 UTC
Switching completed.
Date: 2013-10-20 03:41:03 UTC
We switched individual lines.
Date: 2013-10-20 03:40:44 UTC
We will switch compatible individual lines on the new infrastructure tonight, when there will be no more calls. The items will be re-registered at their next request, a reboot of the item can accelerate the re-registeration if necessary.
Date: 2013-10-20 03:37:06 UTC
We are preparing to switch individual compatible lines.
Date: 2013-10-20 03:36:03 UTC
Incoming/outgoing calls to OVH and to the external, as well as all the functionalities (answering machine / forwarding / time slots / reject / busy / ...) are fully operational.
We will monitor the stability of the infrastructure.
Date: 2013-10-20 03:33:02 UTC
Calls are correctly incoming and outgoing. We boosted our tests.
Date: 2013-10-20 03:32:12 UTC
We restarted services.
Date: 2013-10-20 03:31:48 UTC
We have implemented a new configuration. We will monitor its stability.
Date: 2013-10-20 03:31:19 UTC
We tested other settings to improve stability.
Date: 2013-10-20 03:30:42 UTC
We have reports about call errors. Called number busy, an incoming call impossible. We will investigate.
Date: 2013-10-20 03:29:02 UTC
Regarding the potential problem on the call forwarding or answering machine, we fixed it by setting an older version of SIP gateway.
Everything should work smoothly.
Date: 2013-10-19 06:25:07 UTC
SIP modems reloaded. We will keep monitoring the stability of the new infrastructure.
Date: 2013-10-19 06:24:20 UTC
100% of lines were switched, we will force reloading SIP on modems.
Date: 2013-10-19 06:23:08 UTC
50% of lines were switched.
Date: 2013-10-19 00:20:29 UTC
We switched SIP xDSL lines on the new infrastructure.
Date: 2013-10-19 00:19:59 UTC
Tests with the patch are successful for the moment.
We will keep testing.
Date: 2013-10-19 00:19:02 UTC
We tested a patch on the new infrastructure for call redirections and forward to the answering machine.
Date: 2013-10-19 00:17:34 UTC
We restarted the backup machine, we are aiming to reset all the services and the memory of the machine.
Date: 2013-10-18 21:18:29 UTC
Tests of the manager and calls are done successfully.
We are checking the incorporation of data between both infrastructures.
Date: 2013-10-18 19:38:04 UTC
We are following the tests on all internal adsl
lines of the parc as well as for individual lines,
non compatible with call forwarding.
We are actually starting the tests on the manager's
functionalities in order to limit the board effects.
Date: 2013-10-18 19:10:49 UTC
That's too much! As of 1 year, we had a new infrastructure
of the manufacturer which has been ordered, and set but
no longer in production due to a minor bug (the call
forwarding doesn't work properly). The patch which does fix
the issue isn't developed. We can not wait anymore: We will
switch ADSL BOX SIP customers in OVH to a new infrastructure
as expected a long time ago. This will unload the main
infrastructure which visible is overloaded (while we're
turning around 10% of tech specs of the manufacturer).
We are performing our last checks before resetting the SIP
packets coming from the ADSL BOX in OVH.
We will put on tonight and during the weekend to stabilise
the infrastructure and avoid new break-downs.
Meanwhile, we are proceeding to set 2 new infrastructures (we
will have a total of 4) and reduce the number of customers per
infra. We do know that all works properly with less than 100000
lines per infra. With a security margin, we aim to go down
to less than 30000 lines per infra. With this line, the problems
will not reoccur.
There we go ! it's going to be a long night ..
Date: 2013-10-18 16:53:18 UTC
An MGCP gateway crash has caused instabilities on the SIP. The problem is being analysed by the contractor.
All MGCP displays have been restarted.
Date: 2013-10-18 15:26:41 UTC
We have deactivated the probe.
Date: 2013-10-18 15:26:26 UTC
We're restarting the MGCP displays.
Date: 2013-10-18 15:26:01 UTC
We have a problem similar to yesterday, despite having changed the probe settings.
The SIP line are back up, the MGCP lines are being worked on.