Time Machine Backups with a Raspberry Pi and External Drives

August 18, 2020

Tags: raspberry-pi • Categories: Learning

Table of Contents

As I was reviewing my backup strategy, I realized I hadn’t completed a Time Machine backup on my machines in a long time. Plugging in the drive was just enough friction to forget doing it completely.

The Airport Express has a USB port to plug hard drives, printers, etc into. These devices would be magically broadcasted to the network. It was awesome, and then Apple killed the device. The Eero I upgraded to is great, but the USB port is useless.

But, there’s silver lining! I’ve been looking for a good excuse to buy a Raspberry Pi and mounting external hard drives on the network fit the bill! $35 for a tiny computer more powerful that anything I had growing up and more powerful than a $5 DigitalOcean or AWS VPS. What’s not to like?

Purchasing the Hardware

Raspberry PI 4 2GB. $45. I didn’t end up using the USB-C => micro USB connector and the eBook was useless. HDMI connector was helpful.
Case, fan, and power supply. $12. The 5V 3A power supply required isn’t common, so you’ll most likely need to buy one. Having a case is really nice.
You’ll also need a micro SD card, but I had an extra 16GB card.

So not exactly the $35 sticker price that is advertised, but still cheap.

Setting up Raspberry Pi for Remote VNC & SSH Access

My goal was to run the Pi headless. Here’s how I got the Pi setup for VNC access over the network that works across reboots:

Download http://downloads.raspberrypi.org/NOOBS_latest. Unzip and put it on the SD card. Make sure the SD card is FAT formatted. Make sure you don’t put the unzipped folder on the root directory, but rather the contents of the unzipped folder.
- Startup the Pi. You’ll want a monitor connected via HDMI and a (wired) keyboard to complete the setup process. You don’t need a mouse.
- Setup VNC & ssh. Open up a terminal and run sudo raspi-config. Navigate to "Interfacing Options", enable VNC & SSH. Set boot options to desktop for easy VNC usage. Here’s more info
- You also want to set the default resolution via raspi-config or VNC won’t work when you reboot without a monitor.
- On your mac brew cask install vnc-viewer. Username: pi, password is what you used during the on-screen setup. You should be able to manage the device right from your mac.

At this point, you’ll have access to the PI without a keyboard and mouse. Let’s setup the Pi to serve up the hard drives over the network!

Setting Up External Hard Drives as Network Attached Storage (NAS)

Here’s a couple articles I found that were helpful:

None of the articles seemed to completely match by setup. Here’s what I wanted to setup:

I have two external drives. I wanted to use one as a networked time machine drive and the other as general storage.
One of the drives had a power supply and the other did not.

Here’s how I ended up serving my two hard drives on the network:

Install the packages we’ll need: sudo apt-get --assume-yes install netatalk
A quick note on HFS+ formatted drives: I ended up corrupting the drives on HFS+ mode, most likely because I aggressively turned the power on/off without unmounting the drives. I’d recommend against using HFS+ formatted drives and instead format to the linux-native Ext4. I’ve documented this below.
Run netatalk -v to make sure you have a recent version and get the location to the config file. Latest version is indicated here: http://netatalk.sourceforge.net
sudo nano /etc/netatalk/afp.conf to edit the config file. This is the location for version 3.1. Pull the location of the config from the output of the previous netatalk command we ran if you run into issues finding this file. The instructions inlined in the config file are pretty straightforward.
- In Global add mimic model = TimeCapsule6,106. This broadcasts the time machine drive to look like a ‘real’ time machine device.
- Here’s a list of other options you can use. Neat! https://wiki.archlinux.org/index.php/Netatalk
You’ll also want to edit sudo nano /etc/nsswitch.conf and append mdns4 mdns to the line with dns. This broadcasts the drive availability on the network.
Get a list of all services running on your Pi with sudo service --status-all
sudo shutdown -r now or sudo reboot to restart the system from the command line
Change your AFP config? sudo service netatalk restart
I did find the MacOS finder was pretty glitchy when I restarted services on the Pi. I ended up force quitting the finder a couple times to pick up new drive configurations.
sudo chown -R pi:pi /media/pi/MikeExternalStorage to fix strange permission issues when accessing the drive. This may have had to do with attempting to use HFS+ formatting at first, so you most likely don’t need to do this.

Here’s the final /etc/nsswitch.conf:

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files
group:          files
shadow:         files
gshadow:        files

hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4 mdns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

And /etc/netatalk/afp.conf:

;
; Netatalk 3.x configuration file
;

[Global]
  mimic model = TimeCapsule6,106

; [Homes]
; basedir regex = /xxxx

[ExternalStorage]
  path = /media/pi/ExternalStorage

[TimeMachine]
  path = /media/pi/TimeMachine
  time machine = yes

A Warning on Filesystem Types

I had everything working, and then I accidentally restarted the Pi and everything was mounted as a read-only drive. I ran mount and saw type hfsplus (ro... in the info line. RO = read only.

After some googling I found that this seemed to be caused by the HFS+ filesystem (i.e. macos’s default) that I attempted to support by installing hfsprogs hfsplus hfsutils. Don’t try to host a HFS drive via the Pi. The drivers are not great and I believe this is why I ran intro trouble here.

Here’s what I tried to fix the problem:

After some googling I found: sudo fsck.hfsplus -f /dev/sda1. This didn’t do anything for me.
I tried https://askubuntu.com/questions/333287/how-to-fix-external-hard-disk-read-only. This didn’t seem to work for me because I had partitions
lsblk, blkid, sudo fdisk -l, ls -l /dev/disk/by-uuid/ are useful tools for inspecting what disks/devices are mounted.
sudo cp fstab fstab.bak, edit fstab with nano, UUID=1-1-1-1 /media/pi/ExternalStorage hfsplus force,rw. No quotes around uuid. Tried sudo umount /dev/sda2 && sudo mount /dev/sda2. No luck.
Determine how large a folder is: sudo du -sh
Tried disabling sudo service netatalk stop && sudo service avahi-daemon stop and restarting the computer. No dice.

Uh oh. Not good.

I plugged the drive into my MacOS computer and told me the drive was corrupted. I tried to repair the disk and it gave me esoteric error. I think accidentally turning off the power may have corrupted the disk. HFS+ isn’t supported natively and seems to be prone to corruption issues if drives are not unmounted properly. Ext4 is the recommended file system format.

Some reference articles:

It’s surprising to me, that in 2020, file system formats still matter across operating systems. I feel like I’m transported back to the 90s. Why isn’t this a solved problem yet?

Here’s what I did to fix the issue:

Luckily, although I can’t write to my drives, I can read from them and was able duplicate the data other devices.
I pulled the data off to various drives and computers using rsync. It was a huge pain: I needed to spread the data across a couple of different devices. Here’s the commands I used: sudo rsync -avh --progress pi@192.168.2.200:/media/pi/MikeExternalStorage/MikeiTunes ~/Desktop/MikeiTunes
After the data was moved off I reformatted the drive:
- Unmount the drive: sudo umount /dev/sba2
- Wipe the drive and format sudo mkfs.ext4 /dev/sba
- Create a mount point sudo mkdir /media/TimeMachine && sudo chmod 777 /media/TimeMachine
- Add the table entry: sudo nano /etc/fstab, then /dev/sda /media/TimeMachine auto defaults 0 2. fstab => "File System Table"
- Add a label to the drive sudo e2label /dev/sda TimeMachine https://www.raspberrypi.org/forums/viewtopic.php?t=67896
- Mount the drive sudo mount /dev/sda
After this was done, I rsync’d the data back to the drive. In some cases I needed to use sudo on the Pi to avoid any permission issues. sudo rsync -avh --progress --rsync-path="sudo rsync" ~/Desktop/EmilyBackup pi@192.168.7.200:/media/ExternalStorage/
I found it helpful to use screen to manage long running sessions on the PI:
- sudo apt install screen
- screen to start a new session
- screen ls to list all sessions.
- Attach to a session like screen -r THE_ID
- tmux is also great for this (and probably better)
I also needed a way to unzip some folders. There’s not a built-in util that provides unzipping with a progress indicator. I found 7z which fits the bill: 7z x Projects_old.zip -o./. https://askubuntu.com/questions/909918/q-how-to-show-unzip-progress

Summary: don’t use HFS/mac formatted hard drives on linux!

A Note on Using Old Drives for Storage

I couldn’t get my 2TB drive to pass the SMART monitoring (details in a future blog post!) ‘long’ test. I did a bit of research and external drives tend to only last ~5 years. The 2TB drive was ~10 years old and had exhibiting some glitchy behavior. I ended up replacing it with a much smaller 2TB drive for $60.

This is all to say: it’s worth replacing drives every 5 years or so and ensure they are monitoring by SMART to catch any failures early on.

Fixing Raspberry Pi’s Emergency Mode

I connected the new drive to replace my old one, formatted it, and setup the fstab config just like the other drives. The UI started glitching out and I had two mounts setup for my old drive. I figured restarting the Pi would fix the issue, but that was a bad idea.

The Pi wouldn’t connect to the network and appeared dead. I tried unplugging the drives, but that didn’t help.

I ended up having to plug it back into a monitor and found the following message:

You are in emergency mode. After logging in, "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into default mode.

Here’s what I found:

If any of the devices in fstab cannot be found, it will hang the boot process and you’ll be kicked into emergency mode. This was surprising to me: looks like linux is not too forgiving with bad configuration.
Here’s how to fix the issue:
- Plug your SD card into another computer, edit cmdline.txt on the root of the card, and add init=/bin/sh to the end of it. Looks like the Pi reads that txt file to determine how to boot. I believe this is a Pi-specific config file.
- Plug the SD card back into the pi, run mount -o remount,rw / when the prompt appears and comment out custom lines in /etc/fstab.
- Reboot the Pi and you’ll be back in action.
I ran journalctl -xb but couldn’t find any errors specifically identifying the drive. /var/log/syslog is also a good place to look.
sudo findmnt --verify --verbose is a way to verify your fstab config
If you specify default,nofail in fstab it looks like you can avoid this problem. I’m not sure what the side effects of this approach is.
I don’t understand why fstab definitions are necessary if the default drive config is working fine. All drives automount when connected. I ended up removing all fstab entries and using the autogenerated mount points at /media/pi.

Resources:

Under Voltage & USB-Powered Devices

When I replaced my old hard drive, I grabbed a USB powered one off of Amazon. However, the Pi can only support powering a single external drive drive.

You can determine if this is happening by searching the syslogs:

cat /var/log/kern.log | grep -i 'voltage'

Some references:

The solution is buying a USB hub that is externally powered, like this one.

Spotlight Indexing on NAS

It’s not possible:

As an aside, I’ve also learned it’s not possible to exclude folders with a specific pattern (such as node_modules) or with a specific dot file within the folder (.metadata-no-index). You can only control what’s indexed via the control panel.

Wow, this seems like a lot of work? Was this even a good idea?

Yes, it was. Took way more time than I expected. Probably not a great idea! If you just want to get a networked time machine up and running quickly, I wouldn’t do this.

But…I learned a bunch, which was the fun part for me.

Why is Linux still hard to use?

Way back before Heroku & AWS were a thing, I used to manage server config for various apps I developed. It was a massive pain. I remember clearly staring blankly into my terminal editing files in /etc/* as instructed by obscure blog posts across the internet and hoping things worked. Once I had things working, I left them alone.

Now, to be sure, things have gotten better. Ansible, Terraform, CDK, etc all allow you to configure servers and cloud services with code rather than manually editing files. However, these abstractions are simply that—abstractions. Many times you’ll run into issues with the underlying system config that you need to correct.

The Pi experience, which I’m assuming mirrors the general state of Linux config in general, is really bad. I forgot how incredibly valuable it is to have sane, smart defaults configured on MacOS that is tailored to the hardware which it’s running on. A Given the slow decay of Mac devices (high hopes for Apple Silicon, but overall Apple machines have gotten worse over the years), I’ve thought about moving to Linux, but this experience has eliminated that thought from my mind.

Maybe some of this pain can be chalked up to the Pi OS, but I can’t imagine things are many orders-of-magnitude better on other Linux variants. I hope I’m wrong, and I hope Linux desktops can eventually get to the ‘just works’ state that MacOS maintains.