Skip to content
Linux Administration
Lab 27 of 27·45mIntermediate

Take a backup and prove the restore

Archive a directory with tar, mirror it with rsync, then restore into an empty tree and verify byte-for-byte that nothing was lost.

You need

  • A Linux system with sudo access
  • rsync (apt-get install -y rsync)

Do first

An untested backup is a belief, not a backup. This lab spends more time on the restore than the backup, because that is the half that fails.

1. Something worth backing up

mkdir -p ~/labs/backup/{source/{config,data,logs},dest,restore} && cd ~/labs/backup
printf 'listen=8080\nworkers=4\n' > source/config/app.conf
printf 'secret=do-not-lose-this\n' > source/config/secrets.env
chmod 600 source/config/secrets.env
for i in $(seq 1 50); do
  echo "record $i" > "source/data/record-$i.txt"
done
dd if=/dev/urandom of=source/data/blob.bin bs=1K count=800 status=none
ln -s ../config/app.conf source/data/app.conf.link
echo "noisy" > source/logs/app.log
find source -type f | wc -l

Note the 600 file and the symlink. Both are things a careless backup loses.

Verify

find source -type f | wc -l; stat -c '%a' source/config/secrets.env # 53 # 600

2. Archive with tar, preserving what matters

tar -czf dest/backup.tar.gz \
  --exclude='logs/*' \
  -C source .
ls -lh dest/backup.tar.gz
tar -tzf dest/backup.tar.gz | head -n 8
tar -tzf dest/backup.tar.gz | wc -l

-C source . archives the _contents_ of source with relative paths, which is what you want — archiving /home/you/labs/backup/source bakes absolute paths in and restores to the wrong place. --exclude drops the logs. -t lists without extracting, and reading that list before you trust the archive is a thirty-second habit worth having.

Permissions and symlinks survive by default, which is why tar beats zip here.

Verify

tar -tzf dest/backup.tar.gz | grep -c "logs/" # 0 — the exclusion held

3. Restore into an empty tree and verify

tar -xzf dest/backup.tar.gz -C restore
find restore -type f | wc -l
stat -c '%a %n' restore/config/secrets.env
ls -l restore/data/app.conf.link

Now the part people skip. Compare the trees rather than eyeballing them:

diff -r --no-dereference source restore
echo "diff exit=$?"

diff -r recurses; --no-dereference compares the symlink itself rather than following it. The only differences should be the excluded logs.

Verify

diff -r --no-dereference source restore 2>&1 | grep -v "logs" | head -n 3 # no output — everything except the excluded logs matches

Verify content by checksum, which catches corruption a size comparison misses:

( cd source && find . -type f -not -path "./logs/*" -exec sha256sum {} + \
  | sort -k2 ) > /tmp/src.sums
( cd restore && find . -type f -exec sha256sum {} + | sort -k2 ) > /tmp/dst.sums
diff /tmp/src.sums /tmp/dst.sums && echo "all checksums match"

Verify

diff /tmp/src.sums /tmp/dst.sums && echo "IDENTICAL" # IDENTICAL

4. Mirror with rsync instead, and understand the trailing slash

rsync -aAX --delete --exclude 'logs/' source/ dest/mirror/
find dest/mirror -type f | wc -l
stat -c '%a %n' dest/mirror/config/secrets.env

The flags: -a is archive mode (recursive, preserves permissions, times, symlinks, owner and group), -A adds ACLs, -X adds extended attributes. -aAX is the combination that preserves everything an ext4 file can carry.

The trailing slash is the whole trap. source/ copies the _contents_ into the destination. source without the slash copies the _directory itself_, creating dest/mirror/source/. Getting this wrong is how a mirror ends up one level deeper on every run.

--delete makes the destination match the source exactly, including removals. It is also the flag that turns a typo in the source path into data loss — always --dry-run first:

rsync -aAX --delete --exclude 'logs/' --dry-run -v source/ dest/mirror/ | tail -n 5

Verify

test -d dest/mirror/config && echo "contents copied, not the directory" # contents copied, not the directory

5. Prove incremental behaviour

echo "changed" >> source/config/app.conf
rsync -aAX --delete --exclude 'logs/' --stats source/ dest/mirror/ \
  | grep -E "Number of regular files transferred|Total transferred"

One file transferred, not 53. rsync compares size and mtime and only sends what differs — which is why it, not tar, is what you schedule every fifteen minutes.

Then prove --delete really deletes:

rm source/data/record-1.txt
rsync -aAX --delete --exclude 'logs/' source/ dest/mirror/
test -e dest/mirror/data/record-1.txt \
  && echo "STILL THERE" || echo "removed from mirror too"

That is the property you want in a mirror and the reason a mirror is not a backup: a deletion propagates, so ransomware or rm -rf propagates too. A mirror protects against hardware failure. Only versioned or offsite copies protect against a mistake.

Verify

test -e dest/mirror/data/record-1.txt && echo "STILL THERE" || echo "removed" # removed

6. What a real backup needs beyond this

This lab covered the mechanics. A backup you can rely on also needs:

PropertyWhy it matters
Offsite copyA fire, or a compromised account, takes the local copy with it
VersioningYesterday's good state, when today's backup captured the damage
ImmutabilityObject-lock or append-only, so an attacker cannot delete history
Tested restoreOn a schedule, into a clean host, timed — that is your RTO
EncryptionAt rest and in transit, with the key stored somewhere else

The one to schedule is the restore test. Every organisation that lost data had backups; what they did not have was a restore anyone had performed.

Clean up

cd ~ && rm -rf ~/labs/backup /tmp/src.sums /tmp/dst.sums

Where this goes next

The remaining labs in this group are the incident-response set: hardened service units, SSH, growing a disk under pressure, log limits, network path debugging, and load triage.