Server observability & hardening (post-HTTPS rollout)

[ulises-c]

Post-HTTPS-rollout improvements identified once every linux-server service was
wired up with a Tailscale sidecar. Plan tracked in TODO.md →
"Server observability & hardening (post-HTTPS rollout)"; per-service HTTPS pattern
in linux-server/HTTPS.md. To be tackled in follow-up PR(s) (can be split).

1. Watchtower observability — "what updated, and when"

Watchtower auto-updates containers (3am daily) and is monitored in Uptime Kuma via
its /v1/metrics API — but that's cumulative counters only
(watchtower_containers_updated / _failed / _scanned, watchtower_scans_total):
no record of which containers updated, to what version, or when. The "what &
when" must come from notifications/logs.

• Tier 1 — ntfy notifications (quick win, reuses the existing ntfy). Set
  WATCHTOWER_NOTIFICATION_URL (shoutrrr ntfy URL → our ntfy, topic e.g.
  watchtower) + WATCHTOWER_NOTIFICATION_REPORT=true. Persistent, timestamped
  per-run report (containers updated/failed/skipped, old→new image) + phone push.
  Keep it metrics-only on the HTTP API (no update endpoint) so the schedule keeps
  running.
• Tier 2 — Prometheus + Grafana on the existing metrics endpoint. Trend the
  counters, alert on watchtower_containers_failed > 0. Counts only — pairs with
  Tier 1. Also a home for other metrics (glances, node-exporter, cAdvisor).
• Tier 3 (optional) — dedicated update-tracker UI (What's Up Docker / Diun)
  for a per-container available/applied-update view.

2. Resilience

• Pin the Tailscale sidecar image. All ~13 sidecars run
  tailscale/tailscale:latest and watchtower auto-updates them — a bad release
  could drop every HTTPS front door at once. Pin a stable tag (bump deliberately)
  or exclude the sidecars from watchtower.
• Backups. Done in linux-server/backup/ (6965c1b): nightly restic backup
  of all persistent server state to the dedicated 1TB drive (+ optional 14TB second
  copy), encrypted/deduplicated/pruned, with ntfy alerts and a homepage status card.
  Covers Forgejo (repos+LFS+DB), all SQLite app DBs (online .backup, no downtime),
  Portainer BoltDB, certs/configs, and every .env. ts-state/ deliberately
  excluded (re-auth via TS_AUTHKEY regenerates node keys). systemd timer (03:30,
  Persistent) + OnFailure alert; restore runbook in the README.

3. Maintainability

• DRY the sidecar boilerplate. ~13 near-identical <svc>-ts blocks +
  ts-serve.json (differ only by hostname/port). Use Compose extends from a
  shared base so a global change is one edit, not 13. (Touches all stacks — needs
  live re-verify.)
• One shared TS_AUTHKEY instead of the same OAuth secret copied into ~13
  .env files (rotation/rebuild pain).
• Validation script for the server stacks (CI, like dryrun-smoke.sh):
  assert every linux-server/*/ has matching compose + ts-serve.json +
  .env.example, valid YAML/JSON, serve port == container port, ts-state/
  gitignored.

4. Security

• Tighten the Tailscale ACL — least-privilege for the tag:container nodes
  (currently default allow-all).
• Forward-auth for the NPM public edge (Authelia/Authentik) — bundle with the
  *.home.ulises-c.me NPM trusted-HTTPS setup, since filebrowser/glances etc.
  have weak/no auth once exposed off-tailnet.