15 - Backup System

Date: 2026-02-11 Updated: 2026-10-21 Hostname: pve IP address: 192.168.0.109

Overview¶

Backup strategy - two targets, two types of data:

What	Tool	Primary location	Secondary (Nobara)
LXC/VM backups	vzdump	`/mnt/storage/backup/proxmox/`	`/mnt/hdd/Backup/proxmox-vms/`
Proxmox host OS	Restic	`/mnt/disk1/backup/proxmox-host/`	`/mnt/hdd/Backup/proxmox-host/`

Nobara is not always on - rsync skips gracefully if offline.

1. LXC/VM Backups (vzdump)¶

Configured in the Proxmox GUI under Datacenter - Backup.

Storage: backup-hdd - path /mnt/storage/backup/proxmox/

Backups are stored as .tar.zst files (vzdump format).

2. Proxmox Host OS Backup (Restic)¶

Backs up the Proxmox root filesystem (/) to a local restic repository.

Script: `/root/backup-proxmox-restic.sh`¶

#!/bin/bash
REPO="/mnt/disk1/backup/proxmox-host"
export RESTIC_PASSWORD_FILE="/root/.secrets/restic-password"

restic -r $REPO backup / \
  --exclude /mnt/disk1 \
  --exclude /mnt/disk2 \
  --exclude /mnt/disk3 \
  --exclude /mnt/disk4 \
  --exclude /mnt/storage \
  --exclude /mnt/pve \
  --exclude /var/lib/vz \
  --exclude /tmp \
  --exclude /dev \
  --exclude /proc \
  --exclude /sys \
  --exclude /run \
  --exclude '*.img' \
  --exclude '*.qcow2' \
  --exclude /var/tmp

restic -r $REPO forget \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 3 \
  --prune

restic -r $REPO check

The repository password¶

Stored in /root/.secrets/restic-password (chmod 600) and, since 2026-08-27, also in Vaultwarden as the secure note Proxmox host restic backup - repo jelszo.

Until that date it existed in exactly one place: inside the repository it unlocks. The host root filesystem was being backed up with a key that only lived on that same host root filesystem, so a dead system disk would have left ~27 GiB of verified, unopenable snapshots. This is the same failure the K3s gpg passphrase had until 2026-08-24 and it was fixed the same way - see scripts/README.md.

Do not delete the file thinking the Vaultwarden copy replaces it. The weekly cron reads the file; the Vaultwarden copy is the human one for the day pve is gone. The note carries the repo path, the Nobara copy path, and the file-less restore form, because in a real disaster you will be running restic from some other machine with the USB disk attached:

# no RESTIC_PASSWORD_FILE - restic prompts for it
restic -r /mnt/disk1/backup/proxmox-host snapshots
restic -r /mnt/disk1/backup/proxmox-host restore latest --target /mnt/restore

The password is 6 characters. The file is 7 bytes and restic strips the trailing newline - confirmed on 2026-08-27 by opening the repo with RESTIC_PASSWORD="$(cat ...)", which succeeds. That is thin for a repository living on the MergerFS pool, which is both a Samba share and an NFS export to the whole 192.168.0.0/24 with rw,no_root_squash, and which is rsynced to Nobara weekly. Rotation is cheap, since restic key re-wraps the master key rather than re-encrypting the 27 GiB:

restic -r $REPO key add --new-password-file /root/.secrets/restic-password.new
RESTIC_PASSWORD_FILE=/root/.secrets/restic-password.new restic -r $REPO snapshots --latest 1
# store the new password, verify you can read it back, only then:
restic -r $REPO key list && restic -r $REPO key remove <old-id>

It was deliberately not done on 2026-08-27 - getting the existing password out of its single point of failure was the whole point of that pass, and a rotation in the same pass would have meant storing one password and then immediately invalidating it. restic key remove of the last known key is unrecoverable; restic has no master-key escrow, so the Vaultwarden entry has to exist and be readable before the old key goes.

Cron (on Proxmox host)¶

0 4 * * 0 /root/backup-proxmox-restic.sh >> /var/log/restic-backup.log 2>&1

Runs Sundays at 04:00.

Check snapshots¶

RESTIC_PASSWORD_FILE=/root/.secrets/restic-password restic -r /mnt/disk1/backup/proxmox-host snapshots

Note restic will not read REPO from the backup script's environment, so -r is required even when running as root on pve. Without it the command fails with Fatal: Please specify repository location, which looks like a broken repo but is not.

Verified 2026-08-10: 7 weekly snapshots, unbroken from 2026-06-28 to 2026-08-09, latest 24.834 GiB. The Sunday run does backup then forget/prune then check in one pass, and the log ends with the line worth grepping for:

check snapshots, trees and blobs
[0:00] 100.00%  7 / 7 snapshots
no errors were found

The restic repo lives on /mnt/disk1, not on the NVMe, so it is unaffected by the pve/data thin pool problems in section 7.

3. Rsync to Nobara PC¶

After local backups run, a cron job rsyncs both the vzdump files and the restic repo to Nobara.

Script: `/root/sync-to-nobara.sh`¶

#!/bin/bash

if mountpoint -q /mnt/pve/nobara-backup; then
  echo "$(date) - Syncing to Nobara..." >> /var/log/nobara-sync.log

  rsync -av --delete /mnt/storage/backup/proxmox/ \
    /mnt/pve/nobara-backup/proxmox-vms/ >> /var/log/nobara-sync.log 2>&1

  rsync -av --delete /mnt/disk1/backup/proxmox-host/ \
    /mnt/pve/nobara-backup/proxmox-host/ >> /var/log/nobara-sync.log 2>&1

  echo "$(date) - Sync done" >> /var/log/nobara-sync.log
else
  echo "$(date) - NFS not mounted, skipping" >> /var/log/nobara-sync.log
fi

Cron (on Proxmox host)¶

0 11,19 * * 0 /root/sync-to-nobara.sh

Runs Sundays at 11:00 and 19:00. If Nobara is offline, it logs and skips.

The 11:00 run skipping is normal, not a fault - Nobara is a desktop and is usually still off in the morning. That is exactly why there is a second attempt at 19:00. Both 2026-08-02 and 2026-08-09 followed this pattern: NFS not mounted, skipping at 11:00, full sync at 19:00.

While Nobara is off, the soft mount retries in the background and fills the journal with nfs: server 192.168.0.100 not responding, timed out (~600/hour), and mountpoint /mnt/pve/nobara-backup returns Input/output error rather than a clean "is not a mountpoint". Both are expected noise from soft,x-systemd.automount and recover on their own when Nobara boots. Do not "fix" this by unmounting or by switching to a hard mount.

NFS mount¶

See 15_NFS-Setup_Documentation.md for mount configuration.

4. Weekly Schedule (Sundays)¶

Time	Job
03:00	SnapRAID sync
04:00	Restic host OS backup
11:00	Rsync to Nobara
19:00	Rsync to Nobara (second attempt, in case Nobara was offline at 11:00)

Because the offsite copy is weekly while vzdump is nightly, the Nobara copy can be up to 7 days behind the local one. That is by design, but it matters when reading a failure: after the 2026-08-09 outage the Sunday sync ran while 9 of the 10 daily dumps were missing, so Nobara carried 2026-08-08 dumps for those guests until the following Sunday, even though the local copy was already current again on 2026-08-10. When judging whether an incident cost real redundancy, check both copies, not just /mnt/storage/backup/proxmox/.

5. Monitoring¶

# Restic backup log
tail -f /var/log/restic-backup.log

# Nobara sync log
tail -f /var/log/nobara-sync.log

# List LXC backups
ls -lh /mnt/storage/backup/proxmox/dump/

# Check NFS mount
mountpoint /mnt/pve/nobara-backup

Backups being written is not the same as backups being restorable. The restic repository is verified weekly by a restore test, and the daily digest checks vzdump coverage per guest rather than just counting files: see 30 - Backup Verification + Restore Test.

5. Immich Database Backup¶

Immich stores its data at /mnt/storage/immich, not in /srv/docker-data, and that path reaches LXC 100 as the bind mount mp0: /mnt/storage,mp=/mnt/storage. vzdump skips bind mounts, so nothing under it has ever been in a container archive - only the 52 GB rootfs is. Until 2026-08-12 the Postgres data directory was therefore protected by SnapRAID parity alone, which for a running database means a torn copy of files that are being written: enough to restore the blocks, not enough to start Postgres from them.

Two things fixed that, in this order.

The dump¶

/root/immich-pgdump.sh on LXC 100, daily at 00:30 UTC (02:30 CEST) via /etc/cron.d/immich-pgdump:

docker exec immich_postgres pg_dumpall --clean --if-exists -U postgres \
  | gzip > "$D/immich-$(date +%Y%m%d).sql.gz.tmp"
mv "$D/immich-$(date +%Y%m%d).sql.gz.tmp" "$D/immich-$(date +%Y%m%d).sql.gz"
ls -t "$D"/immich-*.sql.gz | tail -n +8 | xargs -r rm -f

$D is /mnt/storage/immich/pgdump, inside the SnapRAID-protected pool, so the nightly sync picks the dump up as an ordinary file. Roughly 61 MB gzipped, seven kept.

Two details that are not decoration. The dump writes to .tmp and is renamed only on success, so an interrupted run cannot leave a truncated file under a name that looks complete. And the cron hour is UTC while the SnapRAID schedule is CEST: LXC 100 runs on UTC, pve does not, so 30 2 in that crontab would have fired at 04:30 CEST, an hour after the 03:00 sync it is supposed to precede. The sync moved from Sunday to nightly on 2026-08-29, which does not change that arithmetic - only how often it matters.

The exclusion¶

/etc/snapraid.conf now carries exclude /immich/pgdata/ in place of the narrower exclude /immich/pgdata/pg_stat_tmp/. This is not only about wasted parity. A live Postgres changes pg_wal, pg_control and pg_xact while SnapRAID reads them, which produces Unexpected attribute change soft errors; any soft error makes sync exit with warning, and the daemon's maintenance chain stops there without running the scrub. One such sync on 2026-08-12 carried 68 of them. Excluding the data directory removes the only recurring source of them.

Changing this exclusion costs one manual run: the 1697 already-indexed pgdata files show up as deletions on the next sync and would trip sync_threshold_deletes, so the first sync after the change needs ignore_thresholds.

What is NOT backed up by this¶

The photo library at /mnt/storage/immich/library is on the MergerFS pool, protected by SnapRAID parity. Thumbnails and encoded videos are excluded from any offsite backup - they are regenerable via Administration > Jobs in the Immich UI.

Every Docker service, audited once¶

Immich was found by accident. On 2026-08-27 the same question was answered for all of LXC 100 by walking every running container's bind mounts instead of reading compose files, which is the only way to see what is actually mounted:

for c in $(docker ps --format '{{.Names}}'); do
  docker inspect "$c" --format '{{range .Mounts}}{{if eq .Type "bind"}}'"$c"'|{{.Source}}|{{.Destination}}
{{end}}{{end}}'
done | grep '|/mnt/storage'

Everything else - sonarr, radarr, qbittorrent, jellyfin, calibre-web-automated, shelfmark, homepage - only mounts media from there and keeps its own config on the rootfs, so the daily vzdump has it. Application state outside the vzdump is exactly three things:

Path	Container	Covered by
`/mnt/storage/immich/pgdata`	`immich_postgres`	the nightly `pg_dumpall` above
`/mnt/storage/immich/library`	`immich_server`	SnapRAID parity only
`/mnt/storage/syncthing`	`syncthing`	SnapRAID parity only, plus whatever peers hold a copy

(garage also writes to /mnt/storage/backup/garage, but that is the Longhorn S3 target for the K3s cluster, not homelab state - see scripts/README.md.)

The databases that are on the rootfs - rails-lab-db (postgres:17-alpine), dawarich_db (postgis:17-3.5) - need no separate dump. vzdump runs in snapshot mode, the LVM snapshot is atomic across the whole filesystem, and Postgres treats that as a power cut and replays WAL on start. Immich was different only because its data directory was on the bind mount and therefore in no archive at all.

This is also the answer to "should we add a restic layer for the Docker volumes": no, it would duplicate the vzdump. scripts/backup.sh used to describe exactly that layer without ever being deployed, and was deleted on 2026-08-12 for misleading a reader into thinking Immich was covered.

6. Disk Layout¶

Proxmox¶

nvme1n1 Samsung PM991a 256GB, p3 = VG pve (237GB LVM)
├─ pve-root (60GB) -> /
├─ pve-swap (8GB)
└─ pve-data (164.94GB thin pool) -> storage local-lvm, every guest except CT 100

nvme0n1 Kingston KC3000 1TB = VG nvmethin (since 2026-10-21)
└─ nvmethin (934.67GB thin pool) -> storage nvmethin, CT 100

sda1 (5.5TB) -> /mnt/disk1
sdb1 (5.5TB) -> /mnt/disk2
sdc1 (931GB) -> /mnt/disk3
sdd1 (1.8TB) -> /mnt/disk4
/mnt/storage -> MergerFS pool (backup-hdd lives here)

Nobara PC¶

/mnt/hdd/Backup (3.7TB) -> NFS export, rsync target

7. LVM Thin Pool Capacity Incidents¶

pve/data is a 164.94GB LVM thin pool hosting every guest disk. Being thin-provisioned, its Data% reflects actual allocated blocks, not declared guest disk sizes - and vzdump snapshot-mode backups need free pool space to create a temporary LVM snapshot per guest. When the pool gets too full, backups fail outright.

2026-07-07: first hit (96.14%)¶

Grew to 96.14% (NetData alert). Extended manually:

lvextend -L +15G pve/data

brought it to 87.39%, but VG (pve, 237GB total) had only ~1GB free afterward - not a real fix, just breathing room. Enabled thin pool autoextend as a safety net in /etc/lvm/lvm.conf:

thin_pool_autoextend_threshold = 80
thin_pool_autoextend_percent = 10

Note: autoextend can only pull from VG free space - with the VG nearly full, it has nothing to extend into. The only durable fix is physical disk expansion (see private/todo.md - second NVMe in the HP EliteDesk 800 G4's free M.2 SSD2 slot).

2026-07-16: pool full again, nightly backup failed for 9/10 guests¶

Pool had grown back to 89.51% (above the 80% autoextend threshold), and with VG free space still ~1GB, LVM's safety check blocked new thin volume creation entirely. The 02:00 vzdump job failed for LXCs 100, 102, 103, 105, 106, 107, 109, 110, 111 (only the HAOS VM 101 succeeded - QEMU snapshot mode doesn't need an LVM snapshot the way LXC rootfs backups do):

ERROR: Backup of VM XXX failed - lvcreate snapshot 'pve/snap_vm-XXX-disk-X_vzdump' error:
  Cannot create new thin volume, free space in thin pool pve/data reached threshold.

Recovery - fstrim to reclaim already-freed-but-unreturned blocks:

# Per container (safe - only discards blocks the guest filesystem already marked free)
pct fstrim <vmid>

Run across all running LXCs, this reclaimed enough space to drop the pool from 89.51% to 74.35% (no guest data touched - fstrim only returns blocks the guest filesystem itself already considers free, e.g. deleted files that were never discarded back to the thin pool). Re-ran the failed backups manually afterward:

vzdump <vmid1> <vmid2> ... --storage backup-hdd --mode snapshot --compress zstd \
  --prune-backups keep-daily=7,keep-last=7,keep-monthly=3,keep-weekly=4

Also found and fixed: the 2026-07-07 lvm.conf edit had left a duplicate thin_pool_autoextend_percent line (both uncommented), causing WARNING: Ignoring duplicate config value on every LVM command. Removed the duplicate.

2026-07-24: automated the fstrim stopgap, `discard=on` does not apply to LXC¶

Pool was at 69.93%. Tried enabling discard=on on all 10 LXC rootfs entries so trim would happen live instead of needing periodic pct fstrim:

pct set <vmid> -rootfs local-lvm:vm-<vmid>-disk-0,size=<X>G,discard=on

Rejected by the Proxmox API on every container:

400 Parameter verification failed.
rootfs.discard: property is not defined in schema and the schema does not allow additional properties

discard=on is a QEMU/VM disk option only. It works for VMs because the guest talks to a virtual scsi/virtio disk and the discard flag lets that passthrough reach the host's thin volume. LXC containers share the host kernel and mount the thin LV's filesystem directly - there's no virtual disk layer to carry a discard flag, so the option doesn't exist for rootfs at all. There is no live-discard equivalent for LXC; periodic fstrim is the only mechanism.

Automated the existing manual workaround instead - added /etc/cron.weekly/lxc-fstrim on the Proxmox host:

#!/bin/bash
for id in $(pct list | awk 'NR>1 && $2=="running"{print $1}'); do
  pct fstrim $id
done

First run reclaimed ~25.3GB (69.93% -> 65.65%); CT 100 (docker-host, 13.2GB) and CT 109 (6.4GB) were the largest single reclaims.

2026-08-09: backup failed again, and the weekly fstrim cron had never run¶

Pool hit 81.24% and the nightly vzdump failed for the same 9/10 guests with the identical Cannot create new thin volume error. The 10th guest (VM 101, haos) succeeded because it is a QEMU VM and uses vma streaming, not an LVM snapshot.

The important part is not the pool percentage but why it climbed back at all: the /etc/cron.weekly/lxc-fstrim job added on 2026-07-24 had never executed once. Two lines had been appended to /etc/crontab without the username field:

0 4 * * 0 /root/backup-proxmox-restic.sh
0 11,19 * * 0 /root/sync-to-nobara.sh

/etc/crontab and /etc/cron.d/* take six fields, where the sixth is the user to run as. A user crontab (crontab -e) takes five. Cron parsed /root/backup-proxmox-restic.sh as a username, failed, and discarded the entire file rather than just the bad lines. Everything driven from /etc/crontab stopped: cron.hourly, cron.daily, cron.weekly, cron.monthly, and with them logrotate, apt-compat, man-db, netdata-updater, sysstat and lxc-fstrim.

Nothing about this is visible from the outside. systemctl is-active cron reports active, and jobs in /etc/cron.d/ and in root's user crontab keep running normally, so the scheduler looks healthy. The one reliable check:

# 0 means /etc/crontab is being rejected wholesale
journalctl --since '30 days ago' | grep -c run-parts

Both offending lines were duplicates of entries already present and correct in root's user crontab, so removing them lost no functionality:

cp /etc/crontab /etc/crontab.bak
sed -i '/backup-proxmox-restic.sh/d; /sync-to-nobara.sh/d' /etc/crontab

Cron logged (*system*) RELOAD (/etc/crontab) within a minute. Confirmed working when cron.hourly fired at the next :17 and logged (root) CMD (cd / && run-parts --report /etc/cron.hourly).

Manual pct fstrim across all 10 containers in the meantime took the pool from 79.89% to 68.02%, about 19.6GB reclaimed (CT 100 13.1GB, CT 111 10.6GB, CT 113 7.4GB).

The 9 missed backups were deliberately not re-run by hand. The 2026-08-10 nightly job completed all 10/10 on its own, so every guest has a current restore point and retention (keep-daily=7) still covers the window. A manual catch-up would only have added a duplicate of a day already covered, at the cost of pushing the pool back up while it was the thing being fixed.

Why 80% is a hard failure line, not a warning¶

/etc/lvm/lvm.conf has thin_pool_autoextend_threshold = 80, but VG pve has only 1.00GB free, so the autoextend can never actually fire. Above 80% LVM refuses to create any new thin volume, which means every vzdump LXC snapshot fails immediately. The whole backup job aborts in about 70 seconds.

With a 164.94GB pool, the gap between "backups fine" and "no backups at all" is roughly 0.1%, about 180MB. Observed growth is around 1%/day, so from a post-trim 68% there are roughly 12 days of headroom.

Status¶

fstrim is a recurring stopgap, not a fix - the pool will fill again within days/weeks under normal guest disk growth. The second NVMe was the actual fix. It went in on 2026-10-21 and CT 100 moved onto it, which took the pool from 71.33% to 41.66% (see the last subsection). The trim job stays: the remaining guests still need it.

The trim job was moved off run-parts entirely on 2026-08-10, because both cron.weekly (Sundays 06:47) and cron.daily (06:25) fire after the 02:00 vzdump. If the pool crossed 80% overnight the backup would still fail once before the trim cleaned up. It now runs daily at 01:30, half an hour before the backup:

mv /etc/cron.weekly/lxc-fstrim /usr/local/bin/lxc-fstrim
# in root's crontab:
30 1 * * * /usr/local/bin/lxc-fstrim >> /var/log/homelab/lxc-fstrim.log 2>&1

Growth is roughly 1%/day against a ~12% cushion from a post-trim 68%, so a daily trim keeps the pool clear of the threshold with room to spare. Check /var/log/homelab/lxc-fstrim.log if the pool climbs anyway.

2026-08-14: the daily trim ran every night and trimmed nothing¶

The daily digest reported the pool at 78.69%, up 9.2 points in the three days since the 2026-08-11 low of 69.49%. The trim job was supposed to be holding it flat.

It was not broken in any of the ways the previous incident taught us to check. The crontab entry was correct, journalctl -u cron showed the job firing at 01:30 on the 12th, 13th and 14th, and the log file was being written. The log content was the problem:

/usr/local/bin/lxc-fstrim: line 2: pct: command not found

pct lives in /usr/sbin, and cron's default PATH is /usr/bin:/bin. The script had always been called with a bare pct, which works in an interactive root shell because the login profile puts /usr/sbin on the path, and fails silently under cron. The job had therefore never once succeeded since being moved to root's crontab on 2026-08-10 - the ten successful trim lines at the top of the log were the manual run from that day, into the same file.

Fixed with the absolute path, plus a timestamp line so the log says which run produced which numbers:

#!/bin/bash
# FIGYELEM: abszolut utvonal kell. A cron PATH-jaban nincs /usr/sbin, ahol a pct lakik.
echo "=== $(date +%F\ %T) ==="
for id in $(/usr/sbin/pct list | awk 'NR>1 && $2=="running"{print $1}'); do
  /usr/sbin/pct fstrim "$id"
done

Verified by running it under a stripped environment rather than from the current shell, which is the only way to reproduce what cron actually does:

env -i PATH=/usr/bin:/bin HOME=/root /usr/local/bin/lxc-fstrim

13.4GB came back, 78.71% to 70.59%. CT 100 alone gave up 12.5GiB and CT 106 (karakeep) 7GiB.

Proving it, without waiting for 01:30¶

env -i reproduces cron's environment, but it is still not cron. Given that this job had already been declared fixed twice before, the fix was verified end to end with a temporary every-minute entry, then the crontab restored from a backup taken first:

crontab -l > /root/crontab.bak-20260814
(crontab -l; echo '* * * * * /usr/local/bin/lxc-fstrim >> /var/log/homelab/lxc-fstrim.log 2>&1  # TEMP') | crontab -
# wait for one firing, then:
crontab /root/crontab.bak-20260814
diff <(crontab -l) /root/crontab.bak-20260814   # must be empty

The evidence is the pairing of the two logs - cron says it launched the wrapper, the job's own log says what the command inside it did:

Aug 14 10:56:01 pve CRON[2388286]: (root) CMD (/usr/local/bin/lxc-fstrim ...)

=== 2026-08-14 10:56:01 ===
/var/lib/lxc/100/rootfs/: 7 GiB (7519141888 bytes) trimmed
/var/lib/lxc/109/rootfs/: 1.2 GiB (1241018368 bytes) trimmed

The same run confirmed arping-keepalive.sh: six real cron firings, and its new error log still 0 bytes, which it could not be if arping were unresolvable.

fstrim does not always finish in one pass on a live filesystem. The second run twenty minutes later reclaimed another 7GiB from CT 100, which looks like runaway growth and is not - the container's own df went 38G to 37G across both runs, so nothing was consuming space. The first pass simply had five days of backlog to work through. Daily runs do not hit this.

Two things worth carrying forward:

This is the third distinct way the same job has failed to run, after a malformed /etc/crontab and a run-parts schedule that fired after the backup. The verification that would have caught all three is the same one: read the log for output, not for existence. "The file is there and cron logged the CMD" proves the wrapper ran, not the command inside it.
Test cron jobs with env -i. Running the script by hand proves nothing about the environment it will actually execute in. The identical bug hit the ai-digest job two days earlier with claude in ~/.local/bin, so every script in a crontab on this host was audited in the same pass rather than waiting for the next incident.

The audit found a second one. /usr/local/bin/arping-keepalive.sh, which sends a gratuitous ARP every five minutes to keep the router's MAC table from being corrupted by the RE605X extender, calls a bare arping - also /usr/sbin. It had never worked either, and it was better hidden than the trim job, because of these two lines together:

arping -c 1 -U -I vmbr0 192.168.0.109 > /dev/null 2>&1
exit 0

2>&1 to /dev/null discards the command not found, and the hard-coded exit 0 means cron sees a success and never mails. A script that discards stderr and forces exit 0 cannot report that it did nothing - it is indistinguishable from one that works, in the logs, in cron's mail, and in its own exit status. Fixed with the absolute path and stderr redirected to /var/log/homelab/arping-keepalive.err instead of /dev/null, keeping exit 0 so a transient failure does not mail every five minutes. An empty error log now means it ran.

Everything else in root's crontab (restic, sync-to-nobara.sh, restore-test.sh) resolves under PATH=/usr/bin:/bin and was left alone. Quick check for the whole set:

env -i PATH=/usr/bin:/bin bash -c 'command -v restic arping pct qm vzdump'

2026-10-17: the second NVMe, chosen and ordered¶

The fix the Status section keeps pointing at is now ordered: a Kingston KC3000 1TB (SKC3000S/1024G). It was installed on 2026-10-21, see the last subsection.

The free slot was confirmed from the OS before anyone opened the case. dmidecode -t 9 lists both M.2 storage slots and their state:

Designation: Slot6 / M2 SSD     Type: PCI Express 3 x4   Current Usage: In Use
Designation: Slot7 / M2 SSD     Type: PCI Express 3 x4   Current Usage: Available
Designation: Slot5 / M2 WLAN/BT Type: PCI Express 3 x1   Current Usage: Available

HP's QuickSpecs (c06040430) say the same for the SFF: "(2) M.2 PCIe x4 2280/2230 Combo (for storage)". The service guide (c06472102) shows them as SSD1 and SSD2. Slot5 is the x1 WLAN slot and is not meant for a drive.

The screw was the only open question, and only the open case could answer it. HP lists no M.2 screw as a spare part, and forum reports disagree: some owners had no standoff, one quotes an unconfirmed special part number (L54548-001). Photos of the open board settled it:

SSD2 is connector J40, right next to SSD1 (J39), where the existing Samsung sits.
The 2280 mounting point for SSD2 (NUT7) has a standoff already pressed in, in line with NUT10, which holds the Samsung. A flat pad halfway along is the shorter, roughly 2242 position.
So only the screw is missing, a standard M2 (about M2x3 or M2x4, Phillips). K24.sk does not sell M.2 screws, so it comes from another shop.
The SATA data cable for the two HGST drives runs diagonally across the SSD2 area. It has to be moved aside on install, not unplugged.

Why this drive, at this size. Requirements were TLC NAND with an onboard DRAM cache, M.2 2280 and NVMe. QLC and DRAM-less drives are the usual warning for thin-pool VM storage, which sees small writes all day. Speed did not matter: the slot is PCIe 3.0, so any Gen4 drive is capped at about 3.5 GB/s. Endurance was measured rather than guessed. The current Samsung shows 14.0 TB written in 2,373 power-on hours, about 50 TB a year. At that rate even a 400 TBW drive would outlast a five-year warranty, so TBW was not the deciding factor.

The 1TB, not the 512GB: everything currently provisioned adds up to 188 GB across 11 guests, with 117 GB actually used, so 512GB would be enough today. The 1TB cost about 60 EUR more for twice the space and twice the TBW (800 vs 400), so I bought the bigger one. Prices compared on heureka.sk on 2026-10-17:

1TB drive	Lowest price	DRAM	TBW
Kingston KC3000	191.76-199.90 EUR	yes (Phison E18)	800
Samsung 990 PRO	187.13 EUR	yes	600
Samsung 990 EVO Plus	183.84 EUR	no (HMB)	600
WD Black SN850X	~250 EUR	yes	600
Crucial T500	320 EUR	yes	600

The 990 PRO was a few euro cheaper, but the KC3000 has a third more TBW for practically the same money. Used drives were ruled out: a used PM9A1 at 135 EUR saved about 60 EUR and came with no warranty, for the disk every guest would live on. I also skipped the Allegro marketplace offer: its buyer protection covers non-delivery and wrong items, but a warranty claim goes to whichever seller listed it. It was 8 EUR cheaper than K24.sk, a Slovak shop.

Plan for install day:

Shut down outside the 01:30 trim and 02:00 vzdump window.
Fit the drive into J40 and screw it to NUT7.
Check nvme list and smartctl -a. Data Units Written should be close to zero, and the link should be Gen3 x4.
Create a separate LVM-thin storage on the new disk, rather than adding it to VG pve. A VG that spans two disks dies with either one, and with it every guest.
Move CT 100 across with pct move-volume, and VM 101 with it. The next subsection explains why these two, and not simply the largest. CT 109 (25 GB) and CT 113 (20 GB) follow only if space gets tight.

How long the current NVMe lasts, and what actually needs to move¶

The question before install day was whether the Samsung PM991a has to be retired and everything migrated off it. It does not. Wear is not the problem. Capacity is the problem, and so is where the writes come from.

Health, from smartctl -a /dev/nvme0 on 2026-10-17:

Field	Value
Percentage Used	11%
Available Spare	100% (threshold 10%)
Media and Data Integrity Errors	0
Data Units Written	14.0 TB
Unsafe Shutdowns	49
Thermal Temp. 1 / 2 Total Time	37,257 s / 23 s

Thermal throttling adds up to about ten hours of light throttling and 23 seconds of heavy throttling over the drive's whole life, which is negligible.

The write rate comes from Prometheus, not from SMART. Power On Hours reads 2,373, which is far less than the ten months the host has been running. Samsung drives report low power-on counts with APST enabled. That is my guess at the cause, and it is unconfirmed. Either way, the per-hour rate the counter implies is wrong. The host metrics in the K3s Prometheus give the real figure:

increase(node_disk_written_bytes_total{instance="pve",device="nvme0n1"}[7d]) / 1e9 / 7

That gives 45.8 GB/day, about 16.7 TB a year. It matches the drive's lifetime figure: 14.0 TB in the 302 days since Proxmox was installed on 2025-12-19 is about 46 GB/day. So the drive was new when it went in.

One trap here: a [30d] window returned 24.7 GB/day. Prometheus only has data since 2026-10-01, so increase() spread about 16 days of writes over a 30-day divisor. Check prometheus_tsdb_lowest_timestamp_seconds before trusting any long window.

Remaining life. Samsung publishes no TBW for this OEM part. Retailers quote 150 TB, one quotes 300. The drive's own meter is more useful: 11% for 14.0 TB implies about 127 TB to 100%. At 16.7 TB a year, the remaining 89% lasts about 6.8 years.

Who writes. The same Prometheus metric per device-mapper device, averaged over 7 days and mapped to LV names through /sys/block/dm-*/dm/name:

LV	GB/day
thin pool total (`pve-data-tpool`)	38.3
CT 100 docker-host	25.4 (66% of the pool)
pve-root (host, outside the pool)	5.7
VM 101 haos	3.4
CT 110 caddy	2.0
CT 103 vaultwarden	1.6
CT 109 claude-mgmt	1.4
everything else	under 1.2 each

The dm-N names were read after a reboot that falls inside the 7-day window. If that reboot renumbered them, part of the window could be attributed to the wrong LV. The picture is consistent with CT 100 running 26 Compose stacks, so I do not think it did.

What this changes in the plan. Moving CT 100 alone takes about two thirds of the pool's writes off the old drive, from about 46 to roughly 17-20 GB/day. The remaining 89% would then last 15-18 years instead of 7. VM 101 is the next largest writer and is small, so it moves too. CT 109 and CT 113 are large but barely write, so they only move if space requires it. The Proxmox root stays where it is: reinstalling onto the new drive would add risk for no benefit.

The real exposure is the 49 unsafe shutdowns, not the wear. The PM991a is DRAM-less with no power-loss protection, and neither is the KC3000. A power cut mid-write can corrupt whatever is in flight. The new drive does not change that. A UPS would.

2026-10-21: installed, CT 100 moved, old pool at 41.66%¶

The install-day plan above ran as written, except step 5: only CT 100 moved.

The device names swapped. After the install the Kingston enumerates as nvme0n1 and the Samsung as nvme1n1. Everything above that says nvme0 from before 2026-10-21 means the Samsung. The PromQL write-rate query in the previous subsection now needs device="nvme1n1" to keep measuring the old drive.

Step 3, the drive check:

Field	Value
Firmware	EIFK51.2
Data Units Written	0
Media and Data Integrity Errors	0
Percentage Used	0%
Link (`lspci -vv`)	`LnkSta: Speed 8GT/s (downgraded), Width x4`

"Downgraded" is expected: the drive is Gen4 (16GT/s), the J40 slot is Gen3 x4.

Step 4, the storage. One call through the Proxmox API creates the PV, the VG, the thin pool and the storage entry, the same thing the GUI's Disks > LVM-Thin does:

pvesh create /nodes/pve/disks/lvmthin --device /dev/nvme0n1 --name nvmethin --add_storage 1

That gave VG nvmethin with a 934.67GB thin pool, and an lvmthin: nvmethin entry in /etc/pve/storage.cfg with content images,rootdir.

Step 5, CT 100 only. pct move-volume needs the container stopped:

pct shutdown 100 --timeout 180 && pct move-volume 100 rootfs nvmethin; pct start 100

Downtime was 4.5 minutes, 4m15s of it the copy. Between two thin pools on different VGs it is a file copy (mkfs + rsync, 47.2 GB, about 190 MB/s), not a block copy. Without --delete the old volume stays behind as unused0, which keeps a one-line rollback: put rootfs: local-lvm:vm-100-disk-0,size=57G back in the config.

The old volume was deleted only after checking:

the same 37 containers running before and after (diff of docker ps names), none unhealthy
all 23 Caddy backends on 192.168.0.110 answering 200/302/307
no ext4 or I/O errors in the pve kernel log since the move
a manual snapshot vzdump of CT 100 from the new storage: 5m53s, 23.86GB, finished successfully. This is what the 02:00 job does, so there was no reason to wait a night for it.

Then pct set 100 --delete unused0, and pve/data dropped from 71.33% to 41.66%.

VM 101 stayed on the old pool. It holds 8.7 GB, about 5 points of the pool, so moving it buys little space. It can move online with qm disk move 101 scsi0 nvmethin and qm disk move 101 efidisk0 nvmethin if local-lvm passes 60% again.

CT 100 rootfs grown to 77G. It was 86% full inside (46G of 56G), unrelated to the move. pct resize 100 rootfs +20G runs online (resize2fs on the mounted filesystem) and took it to 63%. LXC rootfs cannot be shrunk, so this is one-way.

Two failed units in CT 100 disabled. nvmf-autoconnect (from nvme-cli) and openipmi fail on every boot. They came in on 2026-08-30 as Recommends of prometheus-node-exporter-collectors (ipmitool, nvme-cli, and ipmitool in turn recommends openipmi). An LXC has no NVMe-oF targets and cannot load the IPMI kernel modules. They failed the same way on the boot before the move, so they were not caused by it. systemctl disable both, packages left installed.

8. Bare-Metal Restore of pve¶

Not written down until 2026-08-27, and it is the procedure you need on the worst day.

Restic restores files, not a bootable disk. There is no one-click bare-metal restore in Proxmox VE 9 - the Proxmox wiki covers guests, not the host. The path is reinstall, then restore over it:

Install PVE 9.1 on the new system disk with the same hostname (pve) and the same IP (192.168.0.109). The node name is baked into the /etc/pve/nodes/<name>/ tree and into the SSH host keys, so a different one turns a restore into a migration.
Attach the USB disks, mount /mnt/disk1, restore the snapshot to a staging directory rather than straight over /.
Copy back /etc/network/interfaces, /etc/fstab, /etc/hosts, /etc/default/grub, /etc/snapraid.conf, /root/.secrets, /root/*.sh and /var/spool/cron/crontabs/root.
Restore the cluster config. Stop pve-cluster first - config.db is a live SQLite database and overwriting it under a running pmxcfs corrupts it:
```
systemctl stop pve-cluster
cp /mnt/restore/var/lib/pve-cluster/config.db /var/lib/pve-cluster/config.db
rm -f /var/lib/pve-cluster/config.db-wal /var/lib/pve-cluster/config.db-shm
systemctl start pve-cluster
```
The snapshot also carries /etc/pve as plain files, because restic was never given --one-file-system and follows the pmxcfs FUSE mount. If config.db turns out to be torn - it is copied live - that file tree is the better source: recreate the guest configs from /etc/pve/lxc/*.conf and /etc/pve/qemu-server/*.conf by hand.
Restore the guests from /mnt/storage/backup/proxmox/dump/ with pct restore / qmrestore. This is the only step that has been exercised for real, several times.

Verified on 2026-08-27 that the weekly snapshot actually contains every file this procedure needs (restic ls latest): /etc/pve including priv/, lxc/, qemu-server/ and jobs.cfg; /var/lib/pve-cluster/config.db; /etc/network/interfaces; /etc/fstab; /etc/default/grub; /etc/snapraid.conf; /root/.secrets; /var/spool/cron/crontabs/root. The published host-backup checklists (saturnme, aremesch/pve-host-backup) name nothing that is not already inside it, so no extra config-dump script is needed here.

What this still does not solve¶

Every copy is in the same flat on the same power feed: the vzdump target on the MergerFS pool, the restic repo on /mnt/disk1, and the Sunday rsync to Nobara at 192.168.0.100. Section 3 calls Nobara "offsite" and that is wrong - it is a second machine, not a second site. Fire, theft or one surge takes all three. Vaultwarden is on LXC 103, in the same flat, so the password custody above has the same shape of hole: the Bitwarden mobile client keeps a read-only offline cache of the vault, but only while logged in and only for 90 days of offline use.

Fixing this needs a real offsite target and is tracked separately, not in this document.

15 - Backup System

Overview¶

1. LXC/VM Backups (vzdump)¶

2. Proxmox Host OS Backup (Restic)¶

Script: /root/backup-proxmox-restic.sh¶

The repository password¶

Cron (on Proxmox host)¶

Check snapshots¶

3. Rsync to Nobara PC¶

Script: /root/sync-to-nobara.sh¶

Cron (on Proxmox host)¶

NFS mount¶

4. Weekly Schedule (Sundays)¶

5. Monitoring¶

5. Immich Database Backup¶

The dump¶

The exclusion¶

What is NOT backed up by this¶

Every Docker service, audited once¶

6. Disk Layout¶

Proxmox¶

Nobara PC¶

7. LVM Thin Pool Capacity Incidents¶

2026-07-07: first hit (96.14%)¶

2026-07-16: pool full again, nightly backup failed for 9/10 guests¶

2026-07-24: automated the fstrim stopgap, discard=on does not apply to LXC¶

2026-08-09: backup failed again, and the weekly fstrim cron had never run¶

Why 80% is a hard failure line, not a warning¶

Status¶

2026-08-14: the daily trim ran every night and trimmed nothing¶

Proving it, without waiting for 01:30¶

2026-10-17: the second NVMe, chosen and ordered¶

How long the current NVMe lasts, and what actually needs to move¶

2026-10-21: installed, CT 100 moved, old pool at 41.66%¶

8. Bare-Metal Restore of pve¶

What this still does not solve¶

Script: `/root/backup-proxmox-restic.sh`¶

Script: `/root/sync-to-nobara.sh`¶

2026-07-24: automated the fstrim stopgap, `discard=on` does not apply to LXC¶