Green is not fresh

Between May and October the lab's backups reported healthy at least four times while the copy was empty or weeks old. Every signal that said fresh came from something the backup job itself wrote. How that happens, a script that reproduces it in a second, and the check that catches it.

As of 15 October 2026

To take away: A signal the backup job writes (a log line, the time on a link, the date in a folder name) proves that the job ran, not what it copied. Check something inside the copy.
: When data moves, its backup has to move in the same change. Two of the four cases were a rename and a move; they left the lab without a current automated offsite copy for 61 days and for 19 days.
: Know what is normal for your tools. An exit code that means files vanished during the copy is routine on a live database, and copies inherit file times, so a rotation that sorts by time can delete the newest snapshot.
The full record: The backup chain, an exploration with every date and number

The false signal

Exit code 0Folder named for tonightLink younger than 36 hoursMarker inside the copy

Five nights of the script below. Plain dots: the signal says fresh. Blue: the copy is new. Red: it holds an old night. Hollow: it is empty.

A backup job usually leaves three traces that look like health. It exits with code 0 or logs OK, it creates a folder named after tonight, and it moves a link called current to that folder. A dashboard that watches any of these turns green on every night the job runs. None of them looks inside the copy.

In the lab, each of the three said fresh at least once while the copy was old.

Four times

31 May to 31 JulyRenamed database, old names copied into nothing.
9 to 23 AugustRotation deleted the newest snapshot.
27 to 31 AugustA normal exit code treated as failure.
18 September to 7 OctoberData moved, backup stayed.

Four cases from the record, May to October 2026. The exploration lists the others.

61 days, until 31 July. A database had been renamed on 31 May. The backup script went on copying the old names into nothing, with its error output silenced, and logged OK every night. The copy was 16 kilobytes; the database was 1.2 gigabytes.
Fourteen nights, until 23 August. The rotation that keeps seven generations sorted the snapshots by modification time. Copies inherit the time of their source, so the newest snapshot sorted into the slot that gets deleted, every night. The health check measured the age of the current link, which the job set anew every night: fourteen nights of healthy on a link to nothing.
Four days, 27 to 31 August. The copy tool returned exit code 24, which means files vanished during the run. On a live database that compacts while it works, that is normal. The script treated it as failure and never advanced its anchor again, and the offsite copy stored the same snapshot of 27 August three nights running under three new dates.
19 days, until 7 October. On 18 September the lab’s memory moved to a new database. The offsite job went on copying the old one, which stopped changing on 25 September, every night under a new date. An internal review on 3 October called the chain done. The new database reached the backup machine for the first time on 7 October; until then its only copy elsewhere was one taken by hand on 22 September.

Reproduce it

The script builds a source and a nightly backup in a temporary folder, breaks the copy on night three the way two of the lab’s copies broke, and runs two checks every night: the age of the current link, and a marker read from inside the copy. A second run with five healthy nights is the negative control. A third run skips one night, so you can see the link check go red for the only thing it can see: a job that did not run. It needs bash, cp, ln and find, and deletes its folder when it ends.

Download check.sh, then run bash check.sh. It prints:

Five nights; the copy breaks on night 3 and stays broken:
  night  run     link   content
  1      ok      green  green
  2      ok      green  green
  3      frozen  green  red: copy holds night 2, source is at night 3
  4      frozen  green  red: copy holds night 2, source is at night 4
  5      empty   green  red: no marker in the copy

Negative control, five healthy nights:
  night  run     link   content
  1      ok      green  green
  2      ok      green  green
  3      ok      green  green
  4      ok      green  green
  5      ok      green  green

The job skips night 3:
  night  run     link   content
  1      ok      green  green
  2      ok      green  green
  3      skipped red    red: copy holds night 2, source is at night 3

Broken run: the link check was green on 5 of 5 nights, the content check red on 3.
Healthy run: the link check was green on 5 of 5 nights, the content check red on 0.
Skipped night: the link check was red on 1 night, the content check red on 1.
As described: the link cannot see a broken copy, the marker can, and it stays quiet on a healthy one.

The link check turns red only when the job did not run; the marker turns red whenever the copy is old. The script exits with 1 if the checks do not behave that way.

The check that would have caught all four

Let the live service write a marker into the data it owns, for example a row holding the current time, rewritten every hour. The backup carries it along like everything else. The check reads it from the newest copy and raises an alarm when it is older than the backup interval plus a margin:

# The live service rewrites this as part of its data: date +%s > "$DATA/backup-marker"
written=$(cat /backup/current/backup-marker 2>/dev/null || echo 0)
age_h=$(( ($(date +%s) - written) / 3600 ))
if [ "$age_h" -gt 36 ]; then
  echo "backup content is $age_h hours old" >&2
  exit 1
fi

Where the marker lives decides what it can see. In September a marker inside the old database would have stopped on 25 September and turned the check red within two days. A marker in a file next to the database, rewritten by a timer, would have kept everything green. Write it from the service whose data you care about, into the store you care about.

Two cheap floors catch what a marker misses: size, because a copy that shrank by 99 % is not a copy, and counts at both ends. Since 7 October the lab’s new offsite job reports success only when the copy holds as many export files as the source and more than 100 megabytes; pointed at a wrong source path on purpose, it failed. And the only proof is a restore: pull a copy back, open it, count, compare with the live system.

Checklist

The live service writes a marker into its own data, at least as often as you back up.
The health check reads that marker from inside the newest copy, not from the job’s log, link or folder name.
A size floor and a count at both ends.
Every exit code of the copy tool is classified on purpose. For rsync, 24 (files vanished) is normal on a live database.
Rotation sorts by name or by an explicit date, never by file time.
Every change that renames, moves or replaces a data store names its backup in the same change, and the next morning someone opens the copy.
The check itself is tested: break a copy on purpose and watch it turn red, run it on a healthy copy and watch it stay quiet.
Restores happen on a schedule: pull back, open, count, compare.

Where the lab stands

The chain has been proven by restores, not by a dashboard. On 1 September the old memory and its vectors were pulled back from the offsite copy, opened, counted and compared with the live system. The new database’s nightly export was restored in tests on 20 September and again on 8 October, and since 7 October it has its own offsite job with a count floor and a negative control. Next on the list is the lab’s own health check: it still reads the age of a link, and on 15 October it reported the retired copy as healthy and 17 hours old while the file inside was from 25 September. The marker from this note is how that gets fixed.