I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest.
All my hosts run the same CoreOS setup (<a href="https://github.com/ebrahim37/infra-template" rel="nofollow">https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.
I plan to implement it like this:
vps1:
- restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours
homelab:
- backrest container, to back up volumes/, do prune/check, replicate repo to offsite
- rest-server container, will store backups from vps1, homelab, offsite
offsite:
- restic container, backs up volumes/ to homelab every 24 hours
- rest-server container, store copy of backups from homelab
Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc.
The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost
I've been trying to find a solution for this too! I was considering using Rclone but too many things are using SQLite for me to trust rsync. I was also going to go with CoreOS but I'm leaning towards Fedora Cloud now in case I need to manage things a bit more (and "auto updating" is not something I want as that suggests auto rebooting).
Your secrets.yaml makes me nervous though - too easy to miss a key and leave something exposed. Why not just add the whole file to the vault?
I like CoreOS because of the fact that all config, etc files, sysctls are in one place.
Previously, I was using artix and had this “etc” directory[1] checked in to keep track of system configuration, but there was no good way to keep track of config drift (other than remembering to update this dir).
Haven’t gone through a CoreOS update yet (been using for ~2 months), but doubtful it would break anything. I’ve tested to make sure all my containers shutdown gracefully etc.
The SOPS (secrets.yaml) pattern is more common in NixOS configs, and I found it works nicely here too. In the artix setup, I had a bunch of .example files strewn around [2], which I had to remember to sync with the real versions.
Encrypting a key is just prefixing it with “enc_priv_”, SOPS will encrypt and decrypt it automatically. I keep “public” values plaintext to maybe help someone setting this up for themselves. Just have to double-check git diff before committing.
ebrahimh · · focus · HN ↗
All my hosts run the same CoreOS setup (<a href="https://github.com/ebrahim37/infra-template" rel="nofollow">https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.
I plan to implement it like this:
Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc.The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost
zenoprax · · focus · HN ↗
Your secrets.yaml makes me nervous though - too easy to miss a key and leave something exposed. Why not just add the whole file to the vault?
ebrahimh · · focus · HN ↗
Previously, I was using artix and had this “etc” directory[1] checked in to keep track of system configuration, but there was no good way to keep track of config drift (other than remembering to update this dir).
Haven’t gone through a CoreOS update yet (been using for ~2 months), but doubtful it would break anything. I’ve tested to make sure all my containers shutdown gracefully etc.
The SOPS (secrets.yaml) pattern is more common in NixOS configs, and I found it works nicely here too. In the artix setup, I had a bunch of .example files strewn around [2], which I had to remember to sync with the real versions.
Encrypting a key is just prefixing it with “enc_priv_”, SOPS will encrypt and decrypt it automatically. I keep “public” values plaintext to maybe help someone setting this up for themselves. Just have to double-check git diff before committing.
[1] <a href="https://github.com/ebrahim37/infra-template/tree/00eccff06aed3b65287ab348fb4a725dccd51986/etc" rel="nofollow">https://github.com/ebrahim37/infra-template/tree/00eccff06ae... [2] <a href="https://github.com/ebrahim37/infra-template/blob/00eccff06aed3b65287ab348fb4a725dccd51986/rybbit/env/backend.env.example" rel="nofollow">https://github.com/ebrahim37/infra-template/blob/00eccff06ae...