design walkthrough · a trip expense app

Getting the backups off the server

designopsbackupspostgres

A trip expense-splitting app runs as a handful of containers on one small server. Its backup scripts were written and tested, but they had nowhere off the server to write, so they never ran. This design picks where the copies live, installs the nightly schedule and adds the checks that say when a night goes wrong, without changing a line of the backup scripts.

04:00
nightly backup, UTC
14
days of restore points
$0
a month under 10 GB
0
lines changed in the scripts
the problem

A backup with nowhere to go

Two backup scripts already existed and were tested. They had never run once.

One script dumps the database; the other archives the receipt photos. Both are needed: receipts live on disk, outside Postgres, so a database-only restore brings back expenses whose photos point at files that no longer exist.

Both scripts insist on writing to a mounted, off-box directory and refuse to run when it isn't there. That is deliberate. Creating the folder on the fly would quietly turn an unplugged mount into a plain folder on the server's own disk, and every night would look like a success. With no off-box storage chosen, the schedule was never installed.

The rule this design is built around

"An unrestored backup does not count as a backup." From the project's own backup runbook.

the system

The system, and what this adds

Everything production runs: the containers, the jobs on the server's schedule and the services the app calls. Green outlines are new; amber is changed. Click a box for what it does.

actorTrip membersphone browser
externalDNS providerdomain + DNS
VPS118 changed
Docker Compose stack6
serviceCaddyHTTPS, the only open port
serviceExpense appNext.js
storePostgresledger + pg_cron
storeUploads volumereceipt photos, PDFs
Ledger checkone-shot container
Receipts pruneone-shot container
~ modifiedserviceHost croncrontab
Database dumpbackup script
Receipts archivebackup script
Restore drillrestore script
+ addedOff-box mountrclone + encryption
+ addedstoreBackblaze B2 bucketObject Lock, 14 days
+ addedexternalHeartbeat monitoralerts on a missed night
externalContainer registrybuilt by CI
externalOAuth providersign-in
externalFX rates APIcurrency rates
externalDocument-reading APIreads documents on request
connections (24)
  1. Trip members → DNS provider look up the domain
  2. Trip members → Caddy HTTPS
  3. Caddy → Expense app reverse proxy
  4. Expense app → Postgres queries
  5. Expense app → Uploads volume store receipt files
  6. Expense app → OAuth provider sign-in
  7. Expense app → FX rates API FX rates
  8. Expense app → Document-reading API read a document
  9. Container registry → Expense app image pull
  10. Host cron → Ledger check 03:15 nightly
  11. Ledger check → Postgres read-only check
  12. Host cron → Receipts prune 03:45 nightly
  13. Receipts prune → Uploads volume remove orphaned files
  14. Database dump → Postgres pg_dump in the container
  15. Receipts archive → Uploads volume tar the volume
  16. Restore drill → Postgres restore to a scratch DB
  17. Host cron → Database dump 04:00 nightly+ added
  18. Host cron → Receipts archive 04:05 nightly+ added
  19. Host cron → Restore drill monthly drill+ added
  20. Database dump → Off-box mount write .partial, rename+ added
  21. Receipts archive → Off-box mount write archive+ added
  22. Restore drill → Off-box mount read newest dump+ added
  23. Off-box mount → Backblaze B2 bucket encrypted upload (async)+ added
  24. Host cron → Heartbeat monitor ping after each job (async)+ added
actorservicestoreexternalasyncaddedmodified

the decision

Where the copies live

Every candidate can be mounted with rclone, so the scripts stay exactly as they are. The choice came down to four questions.

What does it cost? Could a hacked server delete the backups? Could one company's bad day take out both the app and its backups? And what does a restore cost? Scores are out of 100, and higher is better on every axis.

ChosenA
Backblaze B2
Object Lock, 10 GB free, independent
B
Cloudflare R2
Free downloads, 10 GB free
C
AWS S3
Nearest region, highest price

Backblaze B2

The cheapest per GB of the three, free at today's size, and the only one whose delete protection could be confirmed on the vendor's own docs. It also sits with a company that runs neither the server nor the domain.

Pros
  • Object Lock, opt-in per bucket, no extra fee
  • First 10 GB free, then $6.95 per TB
  • Free downloads up to 3x the data stored
Cons
  • Needs rclone; no native mount
  • Region list not confirmed (one upload a night, so distance doesn't matter)
Cost
95
Can't delete
90
Independence
95
Restore cost
85
Easy setup
70

Cloudflare R2

Free to download from, and free at today's size. But its delete lock could not be confirmed in this review, and past the free tier it costs about twice as much as B2.

Pros
  • No download fees at all
  • First 10 GB a month free
Cons
  • Lock not confirmed in this review
  • $0.015 per GB after the free tier, about twice B2
Cost
85
Can't delete
50
Independence
90
Restore cost
100
Easy setup
70

AWS S3

Next door to the server, with mature Object Lock and versioning, at the highest storage price of the three. Downloads past the free 100 GB a month cost $0.12 per GB.

Pros
  • Object Lock and versioning
  • A region beside the server
  • First 100 GB of downloads a month free
Cons
  • $0.025 per GB a month, the highest here
  • More access-policy setup than the other two
Cost
55
Can't delete
90
Independence
95
Restore cost
75
Easy setup
60

What storage costs once the free tier runs out

storagelist price, 100 GB for a monthper month
Backblaze B2
first 10 GB free
$0.70
Cloudflare R2
first 10 GB free
$1.50
AWS S3
$2.50

At today's size, under 10 GB, B2 and R2 cost nothing and S3 is 2.5 cents a GB. List prices are from each vendor's own pricing page, checked in October 2026.

Also considered

The server host's own object storage has a nearby region and object lock, but it is the same company as the server: one account suspension or billing failure would take the app and its backups together. Hetzner's Storage Box offers SFTP and built-in snapshots, but its current price couldn't be confirmed, so it is left off the chart.

the schedule

One night, in order

The order is on purpose: check the ledger, prune, then copy, so the archive never carries files that were about to be deleted.

Ledger check
03:15 UTC
Read-only queries confirm the ledger balances before anything is copied.
Receipts prune
03:45 UTC
Orphaned files past the retention window are removed, so the archive never carries them.
Database dump
04:00 UTC
pg_dump runs inside the Postgres container, the dump is checked with pg_restore --list, written as .partial and renamed into the encrypted mount. Then a heartbeat ping.
Receipts archive
04:05 UTC
The uploads volume goes into one .tar.gz in the same mount, five minutes after the dump. Then a heartbeat ping.
Upload to B2
in the background
rclone uploads the encrypted files to the bucket, where each copy is locked for 14 days.
Restore drill
1st of the month
The newest dump is restored into a scratch database and the ledger check runs against it. A failed drill alerts like a missed night.
the rules

The rules it keeps

Four rules were already in the code and the design keeps them. Three are new.

✓ Back up the receipts, not just the database›
Receipts live on disk, outside Postgres. A test restores a dump on its own and shows the restored expense pointing at a photo that no longer exists, then shows the same photo coming back byte for byte once the archive is extracted too.
✓ Refuse to run when the mount is missing›
Neither script creates its destination. A dropped mount means a missing directory and a refused run, never a quiet backup onto the server's own disk.
✓ Never leave a half-written backup›
Each file is written as a hidden .partial and renamed only when complete. A test aims the script at an unreachable destination and checks that nothing of any size is left behind.
✓ Run pg_dump inside the container›
The host's own client could be older than the server and refuse to dump, and production publishes no database port anyway.
+ Encrypt before anything leaves›
The mount is an rclone remote with encryption layered on top, so the storage company only ever holds ciphertext. The backup scripts did not change.
+ Alert on silence, not only on errors›
Cron mails only when a job prints an error. A heartbeat that expects a ping every night also catches the job that never started.
+ Rehearse a restore every month›
The repo's own rule is that an unrestored backup does not count. The existing rehearsal script now runs on a schedule, against the newest dump.
still open

Still to prove

1
The first drill against production's own Postgres image. Until it passes, the copies exist but the restore half is unproven.
2
Where the encryption key lives. It has to be somewhere other than the server, or losing the server loses the key to its backups too.
3
Who receives the heartbeat alert, and how quickly they are expected to act on it.
4
Receipt growth. A booking can carry five files and a PDF can be 25 MB, so re-check the bill once the archive passes the free 10 GB.
component

Database dump script

Dumps the database and hands the file to the off-box mount without ever leaving a half-written copy behind.

  • Refuses to run unless the destination is an existing, writable directory, so an unplugged mount can never become a folder on the server's own disk.
  • Runs pg_dump -Fc inside the Postgres container, then checks the dump with pg_restore --list before copying it anywhere.
  • Writes a hidden .partial file and renames it into place, so a failed night leaves nothing that looks like a backup.
  • Deletes copies older than the retention window, 14 days by default.
component

Restore drill script

Proves a backup can come back. It restores a dump into a fresh scratch database, then runs the ledger check against that copy and passes on its exit code.

  • pg_cron's schema is filtered out first. pg_cron only installs in the app's main database, so a scratch database can't take it, and leaving it in would fail every restore.
  • No --single-transaction, so each failure surfaces on its own instead of one opaque rollback.
store · added

Backblaze B2 bucket

Where the copies live. Object Lock is on, and each copy is locked for 14 days to match the retention window, so a stolen server key can't delete a backup early.

  • $6.95 per TB a month, and the first 10 GB are free.
  • Downloads are free up to three times the data stored, so the monthly drill costs nothing.
  • The server's key can write and list, but has no permission to bypass the lock.
before this design

The system, before

The same production system before the off-box copy existed. The three backup scripts are already here, but nothing schedules them and they have nowhere to write.

actorTrip membersphone browser
externalDNS providerdomain + DNS
VPS10
Docker Compose stack6
serviceCaddyHTTPS, the only open port
serviceExpense appNext.js
storePostgresledger + pg_cron
storeUploads volumereceipt photos, PDFs
Ledger checkone-shot container
Receipts pruneone-shot container
serviceHost croncrontab
Database dumpbackup script
Receipts archivebackup script
Restore drillrestore script
externalContainer registrybuilt by CI
externalOAuth providersign-in
externalFX rates APIcurrency rates
externalDocument-reading APIreads documents on request
connections (16)
  1. Trip members → DNS provider look up the domain
  2. Trip members → Caddy HTTPS
  3. Caddy → Expense app reverse proxy
  4. Expense app → Postgres queries
  5. Expense app → Uploads volume store receipt files
  6. Expense app → OAuth provider sign-in
  7. Expense app → FX rates API FX rates
  8. Expense app → Document-reading API read a document
  9. Container registry → Expense app image pull
  10. Host cron → Ledger check 03:15 nightly
  11. Ledger check → Postgres read-only check
  12. Host cron → Receipts prune 03:45 nightly
  13. Receipts prune → Uploads volume remove orphaned files
  14. Database dump → Postgres pg_dump in the container
  15. Receipts archive → Uploads volume tar the volume
  16. Restore drill → Postgres restore to a scratch DB
actorservicestoreexternal