How I Backup

There is this old 3-2-1 rule for backups: 3 copies, 2 different media, 1 copy off site. I try to make sure my strategy at least fulfils this. Currently I have up to 6 copies, 5 devices, and 3 locations. It may sound like a lot to manage, but it is actually quite simple to setup and use if you know how to setup SSH access, use rsync¹ and make simple bash scripts. Performing the backups themselves takes very little time, even though it's done manually.

To give some context, I start by explaining my previous strategy and then go on to explain my current one.

The previous strategy

• Photos and videos: iPhone automatically synchronized to iCloud. Intermittently I added photos from my Sony RX100 to the Apple Photos library (and thus icloud), and at the same time backing up the whole library to an external harddrive. Shared photos from friends are usually shared through Google Photos, so once in a while I downloaded these onto the external harddrive.
• Code projects: Backed up to GitHub. Some of it also to the external harddrive.
• Documents: Backed up to external harddrive.
• Everything on my laptop and external HDD were then automatically backed up to Backblaze.

There were a few things I was unhappy about:

• Backblaze running in the background, slowing down my laptop and draining the battery faster.
• Sometimes there were issues with Backblaze not recognizing an external HDD when it had not been plugged in for a while.
• I'm not the biggest fan of automatic synchronization. I like the control manual backups give me.
• It was a bit unstructured and too much reliance on other companies handling the data in proprietary formats. My Github account were messy with many private projects that didn't really need to be there, they were only on github as a backup.

What I did like:

• I liked the Apple Photos app, and that all photos were automagically available an all (Mac) devices through iCloud. However, I have switched to a thinkpad laptop running Ubuntu. If I go back to having a MacOS laptop/desktop and iPhone I will probably include iCloud and Apple photos in my backup strategy.

The current setup

The new strategy is based around a few points:

• To backup files to a local or remote destination I use the program rsync (or rclone² for Hetzner StorageBox).
• For storage devices, I use a mix of different devices at home together with a virtual private server with attached SSD on Vultr and a StorageBox on Hetzner.
• For each location I have made a bash script that contains:
  1. a check to make sure the destination is available.
  2. the rsync or rclone command (filtering in/out certain folders).
  3. and lastly logging the date of backup to a logfile (if the backup was successful).
• Nothing is automatically synchronized and I no longer use iCloud.
• I have two locations for data I want to backup: Everyday stuff I want to backup every day, and things like photos/videos I rarely access. Separating it like this makes backing up every day items very fast, and I can use/re-use smaller devices for storing everyday data (like old laptops, small HDDs, or even USB sticks).
• When I want to backup, I call the corresponding script for location and if the data is everyday or rarely-accessed. E.g.: $ bel and $ beo for backing up every day items to the laptop partition and the Vultr VPS, or $ bkt and $ bkh for backing up rarely-ccessed stuff to external SSD and Hetzner.

Right now my devices are:

• Main laptop SSD, 200GB (source for every day items).
• One partition on my laptop, 800GB. (source for rarely accessed items)
• One 2TB SSD.
• One 2015 macbook pro with MacOS's built-in FileVault encryption, 500GB. It is stashed away, but always powered on and accessible via ssh/rsync on the wifi network
• One 40GB encrypted SSD connected to my virtual private server on Vultr (which also serves erict.org).
• One 2TB StorageBox on Hetzner.
• My Android smartphone.

Every day items:

• My home folder on my laptop (excluding some folders) serves as the "source" for everyday items.
• These items I backup every day or so. I backup to the encrypted partition, to the macbook pro, to the virtual private server, and sometimes to the storagebox on hetzner.
• For each device I have made a bash script that makes a backup to that device using rsync. So when I want to back up to a device i just enter $ bel, $ bem, $ beo, or $ beh. Within 15 seconds I have backed up to 3-4 locations. During the day I usually just backup to one location, which takes one second, then I backup to all of them at the end of the day.

Rarely accessed, but want to keep long-term

• I keep the encrypted partition on my laptop as the "source", so whenever I want to add long term stuff like photos, I add them to this partition. Every day items is also backed up to this partition.
• Once in a while, maybe once a month, I backup photos and videos from my phone to this folder using localsend ³. And when necessary I import photos and videos from my Sony RX100 camera. Afterwards, I just run the scripts $ bkt and $ bkh to backup the partition to my external SSD and Hetzner's StorageBox.
• When transferring over photos and videos from the phone, I now actively filter out photos to throw away, so my photos library is no longer littered with unwanted photos.

Some thoughts on this setup

• I don't trust that Hetzner or Vultr or any other cloud provider will never be hacked or otherwise make my data available to adversaries. So I use encryption for all remote storage. I also like to do it for most of my local devices.
• I like the manual transfer of photos and videos, because it makes me take an active part in the photos I want to keep and the photos to throw away.
• I like the organization of having two different sources of truth for backup: the everyday, and the rarely accessed.
• It makes things quite modular. It's easy to add another device to the mix. If it is remote, just add SSH access. Then add a script using rsync/rclone.
• There is noting stopping me from just combining all the small scripts for each location into one bigger script for backing up everywhere. But I like the control of backing up to specific locations when I do my backups.

Reference script

The following script is what I use to backup every day items to my MacBook Pro, which is accessible over my local WiFi. I use very similar scripts for the partition, Vultr VPS, and Hetzner StorageBox (although rclone for the StorageBox).

#!/bin/bash

# Backup everyday folders to MacBookPro

# Folders:
# - .config
# - .ssh
# - bin
# - Dev
# - htdocs
# - log
# - Notes
# - Writing
# - Templates

# Files:
# - all dotfiles

# Configuration
REMOTE_PATH="eric@pro.local:/Users/eric/Backup-Everyday/"
LOG_FILE="/home/eric/log/backup-everyday.log"

# Function to log messages
log_msg() {
  printf "~/bin/bem: %s - %s\n" "$(date '+%Y-%m-%d %H:%M:%S')" "$1" | tee -a $LOG_FILE
}

# Add timestamp to log entries
log_msg "Starting sync of everyday items to MacBookPro"

# Folders
## Notes: If subfolders like .config/nvim/ and its content should be included, all the directories until
## that path must be included for the file path to work for rsync.
## /.config/ means that the .config at the relative root is included, but not its content
rsync -avz \
  --include='/*' \
  --include='/.config/' \
  --include='/.config/rclone/' \
  --include='/.config/rclone/**' \
  --include='/.config/nvim/' \
  --include='/.config/nvim/**' \
  --include='/.config/Typora/' \
  --include='/.config/Typora/**' \
  --include='/.config/Code/' \
  --include='/.config/Code/**' \
  --include='/.ssh/**' \
  --include='/bin/**' \
  --include='/Dev/**' \
  --include='/htdocs/**' \
  --include='/log/**' \
  --include='/Notes/**' \
  --include='/Writing/**' \
  --include='/Templates/**' \
  --exclude='/.steam*' \
  --exclude='/.sudo_*' \
  --exclude='/snap' \
  --exclude='/tmp' \
  --exclude='**/node_modules' \
  --exclude='**/venv' \
  --exclude='**/.venv' \
  --exclude='**/taulag/tmp/**' \
  --exclude='*' \
  --prune-empty-dirs \
  /home/eric/ \
  "$REMOTE_PATH"

# Check for rsync errors
rsync_exit_code=$?
if [ $rsync_exit_code -ne 0 ]; then
  log_msg "Error: rsync failed with exit code $rsync_exit_code"
  printf "exiting due to error"
  exit 1
fi

log_msg "Finished successful sync to MacBookPro"
log_msg "----------"