> It's not "cheap". 6 months of S3 is at around price of outright buying 4TB SSD at retail price. That before you do any IOPS to it
This is badly misunderstanding the problem: S3 is a highly available service with geographic redundancy and a huge range of integrated features. Your comparison would need to be updated to include multiple running servers in addition to redundant storage, and the software stack implementing things like immutability, not to mention all of the security features.
That’s not to say that you can’t build equivalents for the parts you use but you either need massive scale or giving up features to do that. For example, if you can tolerate bitrot or long access times if hardware fails, you can definitely get a lower cost per terabyte.
The reason why most people don’t do that, even when they have scale, is that it adds cost and risk everywhere else if you need engineering/ops people working on storage. If you’re, say, the internet archive that might make sense—it’s quite literally why your organization exists—but most other places are going to see all of the integrated features that they don’t have to build and operate paying for the difference between S3 and physical media pricing. For example, if I want to process files as soon as they’re uploaded or have immutability, I can just turn that on rather than having to build more services.
Try doing the math and ask why your time costs. You need to buy a lot of storage to pay for the engineering work and most places would prefer to spend that time and attention on the product rather than shaving a few percent off of the storage bill (especially since the savings will be negative for quite some time).
I have done the math as part of my job and often found that yes, it was literally worth the money to do our own storage in cases. Running a highly available, durable storage system is not particularly difficult. There was an era when it was a common skill for a sysadmin. In the current era there are off the shelf solutions for it, both open and proprietary.
The biggest reason people pay for S3 is because of data transfer cost to other cloud services they are using.
Yes, I’ve also built those services. As I said before, it is totally possible to beat S3 on pricing but the margin narrows considerably when you factor in the full costs — e.g. think about how many places had replicated storage but not bit-rot protection or versioning and built that at the application layer instead — and the savings is often not big enough to make it your top engineering pick for the money & attention required.
Fundamentally you’re playing a commodity game and that means you need a consistent edge to be worth the fixed costs. If your fully-loaded advantage is less than, say, 30% you’re better off negotiating with your AWS rep and investing your engineering resources in something your customers want.
PunchyHamster · · focus · HN ↗
It's not "cheap". 6 months of S3 is at around price of outright buying 4TB SSD at retail price. That before you do any IOPS to it
S3 is terrible deal on any front. it's just easy
acdha · · focus · HN ↗
This is badly misunderstanding the problem: S3 is a highly available service with geographic redundancy and a huge range of integrated features. Your comparison would need to be updated to include multiple running servers in addition to redundant storage, and the software stack implementing things like immutability, not to mention all of the security features.
That’s not to say that you can’t build equivalents for the parts you use but you either need massive scale or giving up features to do that. For example, if you can tolerate bitrot or long access times if hardware fails, you can definitely get a lower cost per terabyte.
The reason why most people don’t do that, even when they have scale, is that it adds cost and risk everywhere else if you need engineering/ops people working on storage. If you’re, say, the internet archive that might make sense—it’s quite literally why your organization exists—but most other places are going to see all of the integrated features that they don’t have to build and operate paying for the difference between S3 and physical media pricing. For example, if I want to process files as soon as they’re uploaded or have immutability, I can just turn that on rather than having to build more services.
ehe78qhe · · focus · HN ↗
acdha · · focus · HN ↗
ehe78qhe · · focus · HN ↗
The biggest reason people pay for S3 is because of data transfer cost to other cloud services they are using.
acdha · · focus · HN ↗
Fundamentally you’re playing a commodity game and that means you need a consistent edge to be worth the fixed costs. If your fully-loaded advantage is less than, say, 30% you’re better off negotiating with your AWS rep and investing your engineering resources in something your customers want.
everfrustrated · · focus · HN ↗
Only someone who had never done this would say this.
ehe78qhe · · focus · HN ↗