Thread

AWS says it can't restore some data from mideast facilities struck by Iran

530 points · 452 comments · berkeleyjunk

curuinor · 1789512623 · focus · HN ↗
They guaranteed 11 9's durability, didn't they?
e: Yep
<a href="https://aws.amazon.com/s3/storage-classes/" rel="nofollow">https://aws.amazon.com/s3/storage-classes/
1. [deleted] · 1789520134 · focus · HN ↗
  [deleted]
2. jonahx · 1789594332 · focus · HN ↗
  I don't think this has any teeth. They don't compensate in the event of loss afaict.
3. rbanffy · 1789594616 · focus · HN ↗
  Considering all the data they have globally, they might still be compliant.
4. stackskipton · 1789594884 · focus · HN ↗
  Even if they have payable SLA on this, most SLAs have Acts of God and Acts of War exemption.
  1. lbreakjai · 1789599437 · focus · HN ↗
    But do they have Act of Special Operation exemptions?
    eastbound · 1789631370 · focus · HN ↗
    I'm going to reword my Terms of Service this second to add "any military operation" next to "acts of war". But I'm sure we'll then have to demonstrate whether paramilitary are assimilated to the military.
    MisterMunchkin · 1789655692 · focus · HN ↗
    I was just reading an insurance policy and it said "any war, including undeclared wars"
    The insurers always know how to weasel out of it.
  2. Y-bar · 1789620956 · focus · HN ↗
    > US refuses payout for soldiers who died in Iran 'because it isn't a war'
    > In response to another question, Mr Vance rejected using the word “war” to characterise US operations in Iran, saying there was “no active shooting”.
    <a href="https://news.ycombinator.com/item?id=49596955">https://news.ycombinator.com/item?id=49596955
5. advisedwang · 1789595354 · focus · HN ↗
  They say it's "designed for" 11 9s, not guaranteed.
  1. the8472 · 1789597696 · focus · HN ↗
    But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system. At some point you need to start worrying about asteroid defense too.
    JCharante · 1789628663 · focus · HN ↗
    Ah so that's why we need a lunar base. To uphold S3's 11 9s of availability
6. beejiu · 1789595396 · focus · HN ↗
  The SLA excludes force majeure.
  1. LastTrain · 1789597242 · focus · HN ↗
    This
  2. the8472 · 1789597464 · focus · HN ↗
    Making a probabilistic claim while excluding a factor that dominates those statistics is... is quite creative accounting.
    mpyne · 1789598879 · focus · HN ↗
    Are you saying that most data loss happens because your data center gets blown up in a shooting war? Like, AWS is the first digital service provider to lose data in decades?
    the8472 · 1789600366 · focus · HN ↗
    I'm saying that if you have eliminated more mundane failures like dying harddrives, cosmic rays and so on from your systems and your calculation ends up with 11 nines then actually those "force majeure" events are probable enough that they dominate whatever other residuals are supposedly hiding in those last 0.0000000001%.
    The region has seen a bunch of wars in the last 100 years, so the annual war-rate is > 1%. Even if we generously add the assumption that only 1 in 100 wars affects a datacenter you can see that wars become a major source of correlated hardware failures that they need to solve to actually deliver that kind of reliability.
    mpyne · 1789603427 · focus · HN ↗
    Cosmic rays and dying hard drives are not force majeure though.
    ployable7 · 1789607549 · focus · HN ↗
    You don’t want to blend probabilities like this, because the tactics you use as a consumer vary between the two. If you consider 11 9s like “object AFR”, you might build systems that are resilient to very occasional single object loss. And it’s useful to know at what rate that might occur.
    Whereas with these force majeure events you’d want a complete DR setup, and it’s typically an async recovery. Here it is useful to understand the fault domain (single server or single building or multi-building) so you can plan.
    Blending the two numbers doesn’t help you build better against the systems. And the force majeure events are rare enough that they won’t happen … until they do. I’m not sure that knowing the precise probability that Iran would attack a gulf nation would change the fact that if they do, you need to have a DR story.
    the8472 · 1789645247 · focus · HN ↗
    Seems like begging the question to me. You can't blend the numbers because amazon didn't blend the numbers. If they did and miraculously still arrived at 11 9s then that would also cover things such as wars and natural catastrophes, e.g. because they do offsite backups internally.
    ployable7 · 1789651491 · focus · HN ↗
    I’m not saying you can’t blend the numbers, I’m saying you shouldn’t blend the numbers. Because one number doesn’t communicate what you actually need to know to build.
    You want to know how reliable the service is in steady state. For example it’s useful to know that S3 is effectively lossless in steady state whereas EBS volumes have an AFR of about 0.1%. You build your apps very differently between S3 and EBS knowing this. You can build highly resilient applications on each, but you code them differently, informed by these design goals.
    You separately want to understand the failure modes that will require you to fully recover from backup. For example knowing that cloud storage is resilient to everything but region failure would inform you that your backups should be out of the region, not just a bucket in the same region. You don’t get that perspective from just a 9s number.
    mjr00 · 1789600592 · focus · HN ↗
    Force majeure carveouts are really common in every type of contract.
    You should check your home insurance contract, for instance... It likely would not cover an ICBM strike.
    sire-vc · 1789601610 · focus · HN ↗
    Somehow I feel like the biggest post-apocalyptic problem will be the loss of home equity due to uninsured damage causing a collapse of financial markets.
    eli · 1789601030 · focus · HN ↗
    I think it's what most people comparing provider SLAs would expect
    unethical_ban · 1789601220 · focus · HN ↗
    Are you suggesting that their technical documents have separate availability numbers to predict geopolitical events and war?
    throwawaythekey · 1789605072 · focus · HN ↗
    The sales pitch should change from "probabilistically we will NEVER lose your data" to "you are most likely to lose your data due to wars, terrorists, software bugs, someone losing the master encryption key, the government forcing us to...".
    Offsite backups are sadly rare these days, and aws sales is the main reason why.
    mitxela · 1789602125 · focus · HN ↗
    Creative accounting works and is good because it works. If your customers give you more money because you lied to them, but it's legal, then it's good.
  3. oatmeal1 · 1789617145 · focus · HN ↗
    Excluding war as force majeure in the Middle East is the same as excluding high tide as force majeure building a sandcastle at low tide.
    Y-bar · 1789638777 · focus · HN ↗
    This is very on point. While I am not legally trained in US law, and especially not in any Middle Eastern law, I have a enough knowledge on the law in my country here where I live…
    Invoking force majeure requires the entity to prove all three following to be true:
    A. That the event was unexpected and therefore unavoidable.
    B. That the event was outside the control of the entity.
    C. That the event made it impossible for the company to resolve the issue.
    War in the region is as you say rather common unfortunately. The fact that AWS is used by the IDF (<a href="https://www.972mag.com/cloud-israeli-army-gaza-amazon-google-microsoft/" rel="nofollow">https://www.972mag.com/cloud-israeli-army-gaza-amazon-google...) should be considered a factor whether or not their data centre became a more likely target or not. What remains is the ability or not for AWS to do multi-location reduncancy.
    roryirvine · 1789657938 · focus · HN ↗
    As someone who ran a tech company in Northern Ireland at a time when violence was much more common than it is today, I can tell you that our force majeure clauses always disclaimed liability for "riot, violent disorder, civil commotion, and acts of unlawful civil unrest and terrorism".
    How could it be otherwise - do you expect AWS to have its own army and missile defence system?
    oatmeal1 · 1789663324 · focus · HN ↗
    The difference between you and Amazon is hundreds of millions of dollars in political donations and lobbying. Amazon certainly has the political connections to make defense of its datacenters a national security priority of the strongest military on earth if it made it a priority.
7. jameshart · 1789595753 · focus · HN ↗
  The footnote which says that is the design durability against equipment failure literally begins:
  > In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss
8. [deleted] · 1789595831 · focus · HN ↗
  [deleted]
9. shepherdjerred · 1789610523 · focus · HN ↗
  It seems unreasonable to blame Amazon here. The AZ was destroyed. Are they supposed to have missile/drone defense?
  I'm not going to complain to DoorDash if my order is delayed due to a car crash
  1. Edman274 · 1789622359 · focus · HN ↗
    If a senior leader at Doordash swore to God that your sandwich would 100% guaranteed make it to you, regardless of whether or not there was a car crash, then yeah, maybe you should complain
    shepherdjerred · 1789623406 · focus · HN ↗
    Yeah I agree with you. If they made a specific promise to some unlikely case out of their control, then I would expect compensation.
    Did Amazon make such a promise? They didn't as far as I know. My understanding is they provide specific guarantees like given an AZ outage, your data is still safe (provided you architect correctly).
    earth-tattoo · 1789655149 · focus · HN ↗
    Correct analogy would be: they promised to make 3 copies of my sandwich, from 3 different restaurants, and deliver them by 3 different cars, taking different routes, in case one of them gets in a crash. But in this case all 3 restaurants were bombed by Iran!
  2. jLaForest · 1789661300 · focus · HN ↗
    Wild berries has missle defence now, why not Amazon?
10. kmeisthax · 1789616491 · focus · HN ↗
  Hot take: if you have a service that is 11 nines reliable, but there is an underlying component whose reliability is lower, cap the nines to that component.
11. ciberado · 1789626229 · focus · HN ↗
  If they haven't changed it recently, 11 nines is the durability target by design, but it is not set in any SLA. the S3 SLA is focused on availability.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.