The empty mountpoint trap

A shared drive lived on a large USB disk, mounted at boot from fstab with nofail, and backed up hourly with rsync --delete to a second machine. After a power cut, files stopped landing where everyone thought they were.

What happened

After the power came back, the USB enclosure took fifteen minutes to appear. systemd waited its usual 90 seconds for the disk, gave up, and carried on booting, which is exactly what nofail asks for. It does not try the mount again when the disk turns up later. Meanwhile the mountpoint was just an empty folder on the system disk. The file share and scheduled jobs kept using it: some failed on permissions, and the ones running as root wrote straight into it. Earlier writes made the same way sat hidden underneath the drive once it was mounted.

The near miss

The hourly backup was rsync --delete from the share to the backup machine. Run against an empty mountpoint, that tells rsync the share is now empty, so it deletes everything on the other side. This one kept deleted files in a history folder, so it would have moved the whole backup there, not erased it. It didn't happen, only because the backup machine's own disk had been left unmounted days earlier and every run was failing quietly: a second problem, found the same day.

The guards

First, the backup refuses to run unless both ends are real mounts, and says why in its log. Second, the empty folder under the mount is made immutable. Mounting over it works as normal, but when the disk is missing every writer gets an error instead of quietly filling the system disk.

# at the top of the backup script
mountpoint -q /srv/share || { log "skip: /srv/share not mounted"; exit 1; }
ssh backup 'mountpoint -q /mnt/share' || { log "skip: backup disk not mounted"; exit 1; }

# once, with the disk unmounted: lock the empty folder underneath
sudo chattr +i /srv/share

What I took from it

• nofail keeps the machine booting. It also hides the failure, so something else has to notice.
• Anything that deletes to match a source has to check that the source is really there.
• Make the failure loud: an error on write beats a full system disk discovered days later.