OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#11995 — internal DDoS
Incident Report for Network & Infrastructure
Resolved
We have a congestion problem on the internal network
due to the hack of hundreds of dedicated servers which are attacking
many internal targets. Overall, the hack generated 100Gbps
of traffic between RBX (source) and BHS (target)

input 35.22 Gbps
output 71.21 Gbps


The detection of the hacked servers which are participating
to the attack is done automatically but it is too
slow regarding the number of hacked servers.


We cut the traffic between RBX and the internal
network that's why the attack goes out by the public
network. So no more congestion.
However, the ips which go through the VAC are having
a loop.



We put back the internal network. It remains 70Gbps .

input 26.49 Gbps
output 43.17 Gbps

Update(s):

Date: 2014-11-10 09:43:14 UTC
All servers who participated in the attack are
in rescue and customers contacted. The servers
were haced in root with shellshock hack (bash).
In all, 800 servers were involved in the attack
this morning generating over 120Gbps peak in the
internal network. We have 60Gbps between EU and BHS
and is what caused the congestion.Obviously it is
time to go to 2x100G on the private network between
Europe and Canada.

Date: 2014-11-10 09:39:10 UTC
We have identified a new series of hacked servers that took part in the attack. We are currently blocking these machines (around 500 machines).

Date: 2014-11-10 09:37:06 UTC
Less than 1,000 dedicated servers
participated in the attack. a dozen of
IP DST were the target.

There are still about 150 servers closed.

Date: 2014-11-09 19:36:49 UTC
We put in DPI the 2 IPs which are controlling the attack
to find all the other ips which are still sending the packets.
And we are comparing with the anti-hack system. We find the sames
hacked servers (ouch). The problem comes from the speed
of handling of the rescue mode of the attacking servers.


We are going to review the system starting from Monday.

Date: 2014-11-09 19:32:41 UTC
We are still interrupting the hacked servers.
There is no congestion anymore.
The VAC is operational.
Posted Nov 09, 2014 - 19:31 UTC