Thread

AWS says it can't restore some data from mideast facilities struck by Iran

530 points · 452 comments · berkeleyjunk

cmiles8 · 1789596948 · focus · HN ↗
This interview with an AWS leader isn’t aging well, from CBS Sunday morning:
Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"
<a href="https://www.cbsnews.com/news/cloud-computing-loudoun-county-virginia/" rel="nofollow">https://www.cbsnews.com/news/cloud-computing-loudoun-county-...
1. krick · 1789598998 · focus · HN ↗
  That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.
  1. houssc · 1789599866 · focus · HN ↗
    The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.
    marcosdumay · 1789607481 · focus · HN ↗
    No, that quotation on the GP clearly states that AWS has enough redundancy within the same region that they will continue all services running on it if a datacenter is destroyed.
    It's very clearly not about you being able to set-up redundancy for yourself.
    cyberax · 1789608831 · focus · HN ↗
    That's actually true. AWS is designed to survive one datacenter being offline (which happened more than once, btw). When the first DC in ME was hit, AWS continued working normally, with only a few services experiencing issues.
    But it's not designed to survive TWO datacenters going offline, and in a permanent fashion.
    Dylan16807 · 1789625332 · focus · HN ↗
    Aren't they supposed to have 3 copies of everything? If so, two datacenters going offline at the same time should mean almost zero data loss.
    dotancohen · 1789628696 · focus · HN ↗
    No.
    _You_ are supposed to have three copies of everything - and two of those should be off AWS.
    Dylan16807 · 1789629486 · focus · HN ↗
    Yeah yeah I know about 3-2-1.
    But when I put data on normal S3, isn't my money supposedly paying for triple redundancy?
    Look, here's the documentation: <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html" rel="nofollow">https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
    "S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region."
    As far as I've heard before that's not with parity and three zones means three copies. But when I search now I see things about 5+4 parity, any insight here?
    cyberax · 1789666563 · focus · HN ↗
    Yes, normal S3 is at least triply-redundant. But it's possible for S3 to lose some data if two DCs are down and a disk dies in the single remaining DC.
    AWS doesn't use parity for S3, they actually mirror the data.
    cyberax · 1789664399 · focus · HN ↗
    Some systems in AWS do, but they typically need a quorum of at least two copies to work. Other systems only have 2 copies.
    And in this case, it looks like all three DCs were damaged.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.