Something that's interested me is how hard drive capacities have continuously gone up, but read and write speeds have not really changed. To the point where it would take multiple days to read these new high capacity drives.
I wonder what the use case for them is, they aren't ideal for RAID as they would take far too long to rebuild the array in a drive failure, but they also aren't great for long term cold storage. Perhaps they are ideal for periodic backups where you want to take a daily 1TB backup of a server or CCTV where you don't really care about backing up the data as it's all deleted on schedule anyway.
When you're regularly working with largely sequential datasets with sizes ranging from 100GB to >1TB that you're not processing at speeds in excess of 100MB/sec, hard drives are very much good enough.
Especially now that the same 4TB NVMe SSDs I was buying for $300 are now selling for over $1,000.
In my case, the source of the data is S3, so there's no need for redundancy, as the hard drive is effectively a local cache for cloud storage and the work isn't nearly so time-critical that an occasional failure would be more than a mild annoyance to a customer, if that.
Same for archival backups. Faster recovery time than Glacier Deep Archive, less cost over a period of years, simpler and less expensive than tape for modest volume.
I'm not doubting the usefulness of hard drives in general, just the ones above 20TB where it would take you most of a week to write to the drive. Hard drive fill times have been steadily going up to the point it becomes almost impractical to use these 30tb drives.
I guess if you have it all backed up in S3 it's fine, but if you put this thing in a NAS and one drive dies, your NAS is going to spend the whole week rebuilding where it's running at 100%, and another drive dying during this means data loss.
It depends? In something like a 800TB Ceph cluster a failure of a single 20TB drive is not felt at all if configured redundantly enough. Then it does not matter how long replacing it takes, only how many bucks each TB costs and/or how much electricity each drive needs.
If the cluster as whole can stomach the drive failures without losing data and is overall fast enough, there is no need to run many small drives where running fewer big drives is cheaper.
Can't you just not run a Raid 6 or whatever ? If it takes a week to rebuild but is just making a new redundant copy of the old failed drive then it doesn't seem like a huge deal ?
Gigachad · · focus · HN ↗
I wonder what the use case for them is, they aren't ideal for RAID as they would take far too long to rebuild the array in a drive failure, but they also aren't great for long term cold storage. Perhaps they are ideal for periodic backups where you want to take a daily 1TB backup of a server or CCTV where you don't really care about backing up the data as it's all deleted on schedule anyway.
jasomill · · focus · HN ↗
Especially now that the same 4TB NVMe SSDs I was buying for $300 are now selling for over $1,000.
In my case, the source of the data is S3, so there's no need for redundancy, as the hard drive is effectively a local cache for cloud storage and the work isn't nearly so time-critical that an occasional failure would be more than a mild annoyance to a customer, if that.
Same for archival backups. Faster recovery time than Glacier Deep Archive, less cost over a period of years, simpler and less expensive than tape for modest volume.
Gigachad · · focus · HN ↗
I guess if you have it all backed up in S3 it's fine, but if you put this thing in a NAS and one drive dies, your NAS is going to spend the whole week rebuilding where it's running at 100%, and another drive dying during this means data loss.
atoav · · focus · HN ↗
If the cluster as whole can stomach the drive failures without losing data and is overall fast enough, there is no need to run many small drives where running fewer big drives is cheaper.
Every application will have different trade offs
lostlogin · · focus · HN ↗
I’ve got a Synology that’s mostly full of 20TB drives. As you say, adding a drive takes about a week. Expanding the array is even longer.
Melatonic · · focus · HN ↗