I'm not sure the lawsuit has much merit. OVH's datacenter designs are custom, widely described by themselves, and on their terms. You sign up for their services based on what they provide, they didn't promise much, and when their SLA was broken they paid up.
That's how they remain so low cost, they skimp on everything they can, like Ryanair.
If i can draw an analogy, it'd be like buying food from a food truck selling very cheap seafood, which follows all relevant regulations, but has a booklet describing how that seafood takes a week to get to them, and you get sick. Yes, they should do their job better, but nobody can act surprised.
"OVHcloud fire report: SBG2 data center had wooden ceilings, no extinguisher, and no power cut-out"[1]
I suspect that opens up negligence, which may route you around legal barriers like SLAs and TOS, depending on the laws where this lawsuit is happening. And of course, you don't have to win in court to win. You can just be enough of a cost, nuisance, or PR problem to get a settlement.
The class action is only for lost data.. but data loss is an inherent risk of all computer systems… even if ovh never had a fire, these clients could have lost data… but they took no precautions, not even by purchasing a $50 hard drive.
It’s kind of incredible that they wouldn’t spend $50 or $100 to backup their data, but suddenly it’s worth an average of $70,000.
> even if ovh never had a fire, these clients could have lost data
On the other hand, would OVH had power-kill switches and extinguishers, some clients may have ended up not losing their data, even as the fire occurred.
This DC was designed for no power loss and with nah-there-wont-be-any-fire hopetimism. Lawsuit is imo fully deserved.
I suspect that'd only be the case if these are in contravention of building/fire codes. Is that the case? Given that the building is a datacenter, it might be subject to less regulations compared to something like an apartment.
In DE you do not need to have a fire extinguisher in your apartment. Where i work, we have extinguishers, hidrants, regular training and automatically calling the emergency. It's just a regular office building.
But there are a lot of people in an office. The data center could be evacuated immediately and was located in a harbor with no inhabited buildings anywhere close to it. The only damage they did was to their own property.
At least where I live negligence is a thing but this may vary wildly. If no reasonable person would expect you to do it that way then you can claim negligence, YMMV how much you get out of it. The problem usually is that you have to either go after the business which goes belly up as soon as you try and you still get nothing or after the owners money and claim irresponsible management which is far harder to do (and has a criminal code part).
> That's how they remain so low cost, they skimp on everything they can, like Ryanair.
Ryanair has arguably the best safety record of any airline in the world (with no fatal accidents or incidents in their history, and an order of magnitude more flights/PAX PA than the other contenders such as Hawaiian), so they definitely don't skimp on everything.
"Everything they can" isn't the same as "everything." "Everything they can" means "everything they can get away with as a long-term-viable business strategy." For an airline, safety is not one of those things.
For OVH, though, the "safety" of their servers (i.e. availability/durability) is one of those things they can skimp on — because unlike for people, for servers, "safety" is just another fungible thing to optimize as part of OpEx.
One of the fundamental tenets of IT operations is that hardware can just fail, datacenters can just catch fire(!), backhoes can just cut cables, etc. — and so businesses that care, architect their stacks to be resilient to these failure modes, using multi-AZ replication and so forth. And if everybody who actually cares is already doing that... then it really doesn't matter if your DC goes up in smoke.
(And also, to these same companies, low cost can have a "safety" all its own. Rather than lowering OpEx, you can instead see the calculus as: for the price of one vertically-scaled AWS instance, you can get four equivalent OVH boxes in different regions, and replicate between them. Which one of those options is "safer"?)
Mind you, if anything got burnt up that had real SLAs on it — for example, if they were offering an DBaaS service out of that datacenter — they'd be in trouble. But some leased bare-metal boxes? Nah.
Ryanair had incidents from dangerously little fuel on board for the conditons at the destination to risk of controlled flight into ground. Directly caused by aggressive cost saving. Blaming pilots for carrying extra fuel or being late. In such an incident it is often luck what the outcome is. So far they have been lucky, but their history as a mass carrier is much shorter than that of other airlines.
I seriously doubt that they have systematically better safety than many other European companies.
Fuel was too low for regulation, but from all I read there was never a realistic risk of an accident due to low fuel. The consequence rather was that these flights then had to be prioritized at the destination airport, affecting schedules and other airlines.
> Ryanair has arguably the best safety record of any airline in the world (with no fatal accidents or incidents in their history, and an order of magnitude more flights/PAX PA than the other contenders such as Hawaiian), so they definitely don't skimp on everything.
Ryanair is also a relatively young company compared to other air companies, and flights are mostly within Europe, not inter continents.
I took a Ryanair plane once, that plane was a complete wreck and I thought I'd die. No more. And all things considered they aren't even that cheap since they often fly from remote airports in the middle of nowhere, so you need to get there first, wasting money and time.
That's so crazy to me that it could even be allowed after an inspection for a certificate of occupancy. I worked for a company that leased space in a very corporate filled with laywers/finance type of tennants, but this company, a video post house, was very much not either of those. The building kept requiring expense after expense (partly in hopes of making it so expensive they'd choose to go somewhere else). Part of these expenses were that all cabling had to be run through conduit including CAT5, phone cabling, 75ohm coax video cables. In the room that doubled as the machine room and small data center, they required smoke detectors every 6 feet.
And these knuckleheads got to build one in what sounds like a wooden hut with no electrical cut off? Sheesh!
What does nobody living in the datacenter have to do with the price of tea in China?
In order to receive a CO in the US, an inspection has to be done to show that the build was done according to the plans on file. Part of that is a fire safety plan. You have to show max occupancy for any room in the space. That max number is determined by the size of the room, the number of egress points, the size of the doors, etc. Then the fire marshal sends a rep to insepct everything.
So just from all of that exeprience, it has nothing to do with people living there and surprised me that it is not done in a similar fashion in the EU
A datacenter, esp. when you store the UPS batteries near, is a ticking dirty bomb. Many of the systems have rechargeable batteries (UPS batteries & RAID controllers' battery backup), a lot of rare earth metals which release toxic fumes when burnt, etc.
It's extremely hard to extinguish a full-blown electrical fire, backed by UPS batteries and other nasty chemicals and supported by a multi-megawatt power connection.
You don't need to harm people directly. The fumes' effects are enough of an hazard for people already.
That's why the fire at the OVH dc took so long to extinguish. They had to stay away for hours and get boats to provide more water. Sure there were toxic gases, but I'm not aware of people being harmed by it, especially with no one living anywhere close.
What’s funny is I’ve seen several HN posts disown hyperscale cloud providers and encourage others to use OVH based simply on cost, without any analysis of what they lose in the process
Another poster pointed out that the data center had wooden ceilings! That’s insane!
If your server goes down it doesn't matter if it's because of a faulty host or because of a dc fire. You need redundancy either way. People affected by this fire had likely all assets in one data center, which I also wouldn't recommend with AWS & Co.
It's slightly different, because AWS have AZs. Each region is composed of at least 3 AZs, each one being at least a datacenter, at a distance from one another, but the region is usable as a single entity. An OVH DC is separate, but can be next to other DCs in an area ( e.g. the Strasbourg DCs were all next to each other) which is OK because they don't sell any AZ or regional features.
What most of these posts also say is that you need a good strategy for handling catastrophic failure of single data centers. Even if you deploy your workload 3x redundant with offsite backups, you're likely still paying less than on the 3 big clouds in traffic alone.
Except that most countries have public (state and/or local) surveillance of food safety and those not complying will be fined or can be closed. With reliability of IT services such thing probably doesn't exist in any country. Only when it comes cybersecurity they could be fined for violating GDPR. But I don't think they leaked anything, they just lost it.
That's why i said, they followed all relevant regulations ( which for datacenters basically stop at fire safety). Their implementation was really bad, but compliant.
It will be interesting to see. If for example there were somewhat relevant fire or electrical regulations that weren't properly enforced by local government, I guess the owner or builder is still the only one liable?
You can never guarantee that your food doesn’t make people sick. You can and must follow all regulations, guidelines and best-practices but that’s on a best-effort basis.
I'd never trust multiple datacenters on one site. If they're not at least a mile from each other, I treat them as one datacenter. OVH has 4 locations in France alone, there's no reason not to use different cities instead of just SBG1+2.
Or have a good DR strategy/a second cheap cloud provider ready, potentially giving you more reliability than a single expensive one (see e.g. https://news.ycombinator.com/item?id=17431609 and the horror stories posted in the comments there).
Harder if you're heavily relying on the more advanced services with often provider-specific APIs, but I think if you host with OVH you're likely to be primarily renting machines.
Yeah, I was affected by another datacenter fire (WebNX Utah). I had hourly back up, everything was back and running on Digital Ocean within an hour. There are reasons to use AWS/Google Cloud/Azure but reliability is not it.
Unless you want to be cost-effective and have the expertise, then bare-metal is the way to go. As things seem right now, companies are gonna get a lot better at being cost-effective and cloud services are not.
I don't whether you're also including "build your own data center" inside bare-metal, but a data center has many moving parts. Keeping servers up and running is the easiest of all.
None of these clients owned these servers… so the class action is only for lost data. But none of these people had backups? They weren’t worried about a failed hard drive or a power outage causing data corruption? Or a bad stick of ram? Or cosmic rays? Or software bugs? Or hardware bugs? Or hackers? Or ransomware? Or user error? Or …
Seriously.. weren’t these people just waiting for something bad to happen? And now that something bad did happen, they want someone else to pay for it.. because they wouldn’t pay for the cheapest of measures to protect their property they claim was so valuable. Have you looked at how cheap hard drives are today?
I know for a fact that I do. Yes, I was affected by the fire, and had to pull the backups and rebuild the whole stack - we got a few hours of downtime. And yet I stayed with OVH, the main reason being that, even taking into account this unfortunate event, you get a great value for your money.
Of course, some businesses can't afford any downtime: in this case, whichever provider one chooses, they have to go multi-cloud, or at least distribute their setup on multiple regions. I have been burned (figuratively!) in the past all the same with big names cloud provider... Shit just happens, be prepared!
Because OVH's negligence transpires more widely than just in DC fires. I've had my primary business domain name almost expire twice due to OVH's incompetence. I should have run away the first time. I finally did the second time. Never again, thankyouverymuch.
Some low cost providers do offer acceptable quality. OVH was (is?) not one of those.
What is acceptable varies. Some things are very low-stakes. Even if your domain-name is important you could manage it with another registrar and still use OVH for their cheap metal.
Of course. They're much cheaper, mostly good quality and have a lot of locations. With any cloud provider you should deploy in multiple data centers (as in separate cities, not just different buildings on the same site). Doesn't matter if the server goes down because of a faulty host or because of a fire. And OVH saves lots of money.
As long as you don't need any of the services that only the big 3 offer, I don't see a reason why I would prefer AWS over OVH or Hetzner.
Yes, many people deployed a lot in OVH. Their reputation wasn’t perfect but much better before the fire (and the loss of the managed backups drama). They were also big before AWS or Azure arrived in Europe.
I wonder how cheap a VPS provider can go. In Spain none of the companies can match OVH, but datacenters in Spain are pretty old regular ones, even the small ones like NixVal.
https://lowendbox.com/ is the go-to place for finding the most shoestring VPS available, and they can come very cheap, either in absolute dollars or in compute-per-dollar.
Or lately https://lowendspirit.com/ - split off from the above. I find the providers there slightly less shady. Though again fly by night is entirely possible so backup accordingly
Hetzner can match OVH. To match their prices you need scale to build your own datacenters and servers. Small providers can't do that, they need to colocate and that'll usually be more expensive than owning your dc.
In addition to the video posted: Their DCs tend to be outside of cities (cheaper space), low density and single story (much easier to cool). If you buy cheap land and keep server density low, both building the dc as well as cooling is much cheaper (you can mostly use ambient air with lower grade fans).
Hetzner is also known to do everything in-house, slows them down building new dcs and expanding to new locations but keeps cost down and the company independent.
That's how they remain so low cost, they skimp on everything they can, like Ryanair.
If i can draw an analogy, it'd be like buying food from a food truck selling very cheap seafood, which follows all relevant regulations, but has a booklet describing how that seafood takes a week to get to them, and you get sick. Yes, they should do their job better, but nobody can act surprised.