The unfortunate part of that graph is that I think it's not updated for today's prices given the memory shortage/crunch we're experiencing that's driven up the prices for all kinds of memory.
But also while it should be faster than it is, how would an S3 designed around SSDs look differently to an API user? I would think the API is basically the same.
That's S3 Express One Zone. Directory buckets are SSD-backed and regular buckets are HDD-backed. Some of the API differences off the top of my head:
- Directory entries are no longer returned in sorted order in ListObjectsV2
It's hard to imagine how these API differences can be explained by the different underlying block device. I don't see any good reason you couldn't support these operations on a HDD.
I suspect it's more to do with the fact that with One Zone is a clean rewrite of large parts of the application stack that makes up S3.
S3 is made up of hundreds of microservices[0], there probably isn't anyone at Amazon that actually understands the whole system. Refactoring it to support these features probably requires coordination between a lot of different teams.
They might have petabytes of metadata, making a change to how metadata is persisted probably requires a massive risky data migration.
[0]: "All in, S3 today is composed of hundreds of microservices" - <a href="https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html" rel="nofollow">https://www.allthingsdistributed.com/2023/07/building-and-op...
0xCMP · · focus · HN ↗
But also while it should be faster than it is, how would an S3 designed around SSDs look differently to an API user? I would think the API is basically the same.
pugz · · focus · HN ↗
- Directory entries are no longer returned in sorted order in ListObjectsV2
- There's an AppendObject API
- There's a RenameObject API
WatchDog · · focus · HN ↗
I suspect it's more to do with the fact that with One Zone is a clean rewrite of large parts of the application stack that makes up S3.
S3 is made up of hundreds of microservices[0], there probably isn't anyone at Amazon that actually understands the whole system. Refactoring it to support these features probably requires coordination between a lot of different teams. They might have petabytes of metadata, making a change to how metadata is persisted probably requires a massive risky data migration.
[0]: "All in, S3 today is composed of hundreds of microservices" - <a href="https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html" rel="nofollow">https://www.allthingsdistributed.com/2023/07/building-and-op...
pugz · · focus · HN ↗