OVHcloud Web Hosting Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#8371 — p19-72/p19-77
Scheduled Maintenance Report for Web Cloud
Completed
We received the new images Ax52_upg_2_7_0-P3_33926_25, which fix 2 problems that we escalated:
1. Memory Leak
2. Slowloris/Torhammer attack

We update devices and we will pout the web hosting traffic and ovh.* on it as usual.

Update(s):

Date: 2013-04-04 22:03:19 UTC
They are stable.

Date: 2013-04-04 22:02:51 UTC
We compared to the sessions we had before on ACE and switching and it is the same. So it works properly and the numbers we had before on the a10 were bad.

So the situation seems stable. We will keep monitoring the AX while restoring them finally in production (after 6 months of instability).

Date: 2013-04-04 09:41:36 UTC
It's stable, but important. We should be at 110K, not 400K

p19-77-a10#sh session brief
Traffic Type Total
--------------------------------------------
TCP Established 406857
TCP Half Open 53375
--------------------------------------

Date: 2013-04-04 09:41:02 UTC
We perform at 400K simultaneously. We check why.

Apr 04 2013 10:59:52 Warning [AX]:conn proxy queue depth exceeds limit
(16001)
Apr 04 2013 10:58:34 Warning [AX]:conn proxy queue depth exceeds limit
(15001)
Apr 04 2013 10:57:14 Warning [AX]:conn proxy queue depth exceeds limit
(14001)
Apr 04 2013 10:55:42 Warning [AX]:conn proxy queue depth exceeds limit
(13001)
Apr 04 2013 10:54:39 Warning [AX]:conn proxy queue depth exceeds limit
(12001)
Apr 04 2013 10:53:08 Warning [AX]:conn proxy queue depth exceeds limit
(11001)
Apr 04 2013 10:51:15 Warning [AX]:conn proxy queue depth exceeds limit
(10001)
Apr 04 2013 10:49:40 Warning [AX]:conn proxy queue depth exceeds limit
(9001)
Apr 04 2013 10:48:26 Warning [AX]:conn proxy queue depth exceeds limit
(8001)
Apr 04 2013 10:46:24 Warning [AX]:conn proxy queue depth exceeds limit
(7001)
Apr 04 2013 10:45:21 Warning [AX]:conn proxy queue depth exceeds limit
(6001)
Apr 04 2013 10:44:23 Info [SYSTEM]:Control CPU Usage is OK. Current
value is 54
Apr 04 2013 10:44:12 Critica [SYSTEM]:Control CPU Usage is over
threshold limit(90). Current value is 100.
Apr 04 2013 10:43:48 Info [SYSTEM]:Control CPU Usage is OK. Current
value is 44
Apr 04 2013 10:43:38 Critica [SYSTEM]:Control CPU Usage is over
threshold limit(90). Current value is 100.
Apr 04 2013 10:43:18 Warning [AX]:conn proxy queue depth exceeds limit
(5001)
Apr 04 2013 10:41:33 Info [SYSTEM]:Control CPU Usage is OK. Current
value is 22
Apr 04 2013 10:41:28 Critica [SYSTEM]:Control CPU Usage is over
threshold limit(90). Current value is 100.
Apr 04 2013 10:41:02 Warning [AX]:conn proxy queue depth exceeds limit
(4001)
Apr 04 2013 10:39:29 Warning [AX]:conn proxy queue depth exceeds limit
(3001)
Apr 04 2013 10:37:29 Warning [AX]:conn proxy queue depth exceeds limit
(2001)
Apr 04 2013 10:35:49 Warning [AX]:conn proxy queue depth exceeds limit
(1001)
Apr 04 2013 10:33:36 Warning [AX]:conn proxy queue depth exceeds limit
(1)


Date: 2013-04-04 09:40:20 UTC
The whole traffic is on a device.

The device is stable. no crash. However the number of connections increase slowly. There is still 1 bug to fix.
Posted Apr 04, 2013 - 09:39 UTC