Ask any hosting provider whether they take backups and the answer is yes. Ask when they last performed a full restore of a client site from those backups, under time pressure, and watch the answer get vaguer.
This is not cynicism. It is the structural weakness of backup systems: taking a backup produces visible output every day, and restoring from one produces nothing until the day you desperately need it. Only one of those two operations gets exercised in normal use, and it is not the one that matters.
For an agency holding hosting for fifty or two hundred client sites, this matters more than it does for a single site owner, because you are the party the client will call. This article is about the operational side — what a backup regime for client sites needs to cover, and how to test it so that the answer to the question above is a date rather than a shrug.
Start with two numbers
Before choosing any tool, agree two figures per client tier.
Recovery Point Objective is how much data you can afford to lose, measured in time. A nightly backup means an RPO of up to 24 hours — if the failure happens at 23:00 and the last backup ran at 02:00, you lose 21 hours of orders, comments and form submissions.
Recovery Time Objective is how long a restore may take. This is where honest measurement matters: restoring 80 GB from an object store over a domestic-grade connection is not a fifteen-minute job, whatever the marketing says.
These two numbers determine everything else. A brochure site can live with RPO 24 hours and RTO a day. A shop taking orders cannot, and needs either more frequent database backups or a replica. Pricing the hosting without pricing these is how agencies end up absorbing the cost of an expectation they never agreed to.
3-2-1, and the two extra digits
The classic rule: three copies of the data, on two different media or platforms, with one held off-site.
Two additions have become necessary since it was coined.
One copy immutable or offline. Ransomware now specifically targets backup repositories, and credentials found on a compromised server are used to delete them before the encryption starts. A backup that the compromised machine can delete is not a backup against this threat.
Zero errors on verification. A copy that exists but does not restore is not a copy. This has to be checked mechanically, because it will not announce itself.
The immutability requirement is the one that changes architecture. In practice it means one of:
- Object storage with an object-lock or retention policy, so that objects cannot be deleted before a set date even with valid credentials.
- A repository the client machine can write to but not delete from —
borg serve --append-only, or arest-serverinstance running in append-only mode for restic. - Application keys scoped to write-only, with deletion and pruning performed by a separate, more privileged process elsewhere.
The general shape is the same in all three: the machine being backed up should not hold credentials capable of destroying its own backups.
What actually needs backing up
Files and databases are the obvious two, and the ones every tool covers. The list that gets forgotten:
- Database routines, triggers and events.
mysqldumpdoes not include these by default. A dump without them restores a schema that looks right and behaves wrong. - Mail. If you host mailboxes, they are usually a separate system from the web hosting and often a separate backup regime. Check.
- DNS zones. If your control panel is authoritative for client zones, a lost server means lost zones. Export them.
- TLS private keys, or at minimum the ACME account key, so that certificates can be reissued rather than re-validated from scratch.
- Control panel configuration — PHP versions, cron jobs, mail routing, redirects, aliases. A restore of files and database onto a bare server reproduces the site’s content and none of its environment.
- Cron jobs. Frequently the only record of a scheduled task exists in the crontab itself.
For a database, the dump invocation that actually captures everything:
mysqldump --single-transaction --routines --triggers --events \
--default-character-set=utf8mb4 clientdb | zstd -T0 > clientdb.sql.zst
--single-transaction gives a consistent snapshot on InnoDB without locking writes for the duration. It does not do so for MyISAM tables, which is one of several reasons not to have any.
For databases beyond a few gigabytes, a logical dump becomes slow to restore — the import replays every insert. A physical backup tool such as Mariabackup or Percona XtraBackup produces something you can restore by copying files back, which is dramatically faster at scale.
Snapshot backups, briefly
Both restic and borg do the same core job well: deduplicated, encrypted, incremental snapshots to local or remote storage. Either is a reasonable choice. A restic repository against object storage:
export RESTIC_REPOSITORY="s3:https://s3.example-region.backblazeb2.com/agency-backups/client01"
export RESTIC_PASSWORD_FILE=/etc/restic/client01.pass
restic backup /var/www/client01 /var/backups/client01 \
--exclude-file=/etc/restic/excludes.txt --tag nightly
Exclusions matter more than they look — caches, node_modules, generated thumbnails and log files can be the majority of a web root by size and are worthless in a restore.
Retention is a separate operation, and should run from a host with delete permissions rather than from the client server:
restic forget --tag nightly --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune
The repository password is the encryption key. Lose it and the backups are unrecoverable, permanently. It belongs in your password manager and in a second, independent place.
Verification, in three tiers
Tier one: does the repository still make sense? Structural integrity, run nightly, cheap.
restic check
Tier two: does the data actually read back? restic check by default verifies metadata and structure, not the contents of every data blob. Reading everything is expensive; reading a rotating sample is not:
restic check --read-data-subset=5%
Run weekly. Over a few months this reads the entire repository, and it catches bit rot and truncated uploads that structural checks miss.
Tier three: does a restore produce a working site? This is the tier that matters, and the only one that answers the question the client will ask.
Automate it. Once a month, per client tier, restore the most recent snapshot into a scratch environment, load the database, point a test hostname at it via /etc/hosts, and assert that the homepage returns 200 and contains a known string. That last check is the important one — a restore can complete "successfully" and produce a database connection error, and an unattended job that only checks the exit code of the restore command will report success.
restic restore latest --target /srv/restore-test --tag nightly
zstd -dc /srv/restore-test/var/backups/client01/clientdb.sql.zst | mysql restoretest
curl -sf --resolve client01.test:443:127.0.0.1 https://client01.test/ | grep -q 'Known Footer Text'
Record the date and the elapsed time. The elapsed time is your real RTO, and it is almost always longer than the one in the proposal.
Restoring under pressure
Two lessons that only appear when you do this for real.
Restore to a new location, never over the top of the damaged one. If the restore is incomplete or you have picked the wrong snapshot, overwriting the current state removes your ability to try again. Restore alongside, verify, then swap.
Know which snapshot you want before you start. In a compromise, the most recent backup contains the compromise. Restoring it faithfully reinstates the problem. This is why retention depth is a security control and not just a convenience: if the intrusion is discovered three weeks after it happened, seven daily snapshots are not enough. The --keep-monthly 12 in the retention policy above exists for this.
Before restoring after a suspected compromise, take a forensic copy of the current state first. It is the only evidence of what happened, and once you have restored over it, it is gone.
The GDPR footnote
Client sites hold personal data, so backups hold personal data. Two practical consequences.
Retention should be defined and finite. "We keep everything forever" is not a defensible position when your lawful basis for holding the data has a defined lifetime.
Erasure requests are the awkward case, because you cannot surgically delete one person’s record from an encrypted, deduplicated snapshot. The ICO’s guidance is pragmatic here: where deletion from backup is not immediately feasible, the data can be put beyond use — not accessed, not used for any purpose — and deleted when the backup cycles out in the ordinary course. What matters is that you have a documented retention period after which it genuinely does cycle out.
The question worth being able to answer
Not "do you take backups?" — everyone says yes.
The question is: when did you last restore a client site from backup, how long did it take, and who watched it happen?
If you can answer that with a date, an elapsed time and a name, you have a backup system. If you can only point at a dashboard of green ticks, you have a backup process and an untested assumption.
FXRM’s reseller platform includes automated backups with off-site retention, and we test restores rather than assuming them. See reseller hosting →
