kvitt/infra/README.md

infra: Kvitt infrastructure

This file covers the development stack on the owner's Mac (infra/dev/, milestone M2c-dev, D-001), the Firebase setup (SPEC §10.7) and the Pi stack (infra/pi/, milestone M2c-pi, D-100): a self-contained bundle the owner copies to the Kvitt Pi and runs there. No agent and no Mac script connects to the Pis; the owner runs every Pi command from the checklist in "Pi: what the owner does".

Layout

Path	What
dev/compose.yaml	Dev stack: server (image kvitt-server:dev) and the on-demand backup job
dev/.env.example	Dev settings, no secret values; copy to dev/.env (git-ignored) to override
dev/build.sh	Builds kvitt-server:dev from server/Dockerfile and kvitt-backup:dev from backup/Dockerfile
dev/up.sh	Starts the server, waits for /health, prints the Mac and phone URLs; --down stops the stack
dev/logs.sh	Follows the logs (dev/logs.sh --tail 100 server)
dev/backup.sh	Runs the backup job once and checks the snapshot with PRAGMA integrity_check
dev/copy-google-services.sh	Copies secrets/google-services.json to android/app/google-services.json
dev/lib.sh	Helpers sourced by the scripts
backup/Dockerfile, backup/backup.sh, backup/healthcheck.sh	Backup image and job of SPEC §10.9 (shared with the Pi stack); Drive upload and D-002 deletion
cert-sync/Dockerfile, cert-sync/sync.sh, cert-sync/entrypoint.sh	Certificate sync image of SPEC §10.5.1
pi/	The Pi bundle (copied to the Kvitt Pi): compose.yaml, Caddyfile, .env.example, install.sh, deploy.sh, bootstrap.sh, backup-now.sh, install-apk.sh, lib.sh; git-ignored pi/.env, pi/secrets/, pi/out/
build-images.sh	Mac fallback for the image build: linux/arm64, push to the Gitea registry
publish-apk.sh	Mac: release APK, mapping and latest.json into pi/out/apk/ for the owner to copy to the Pi
RESTORE.md	Restore from a Drive backup, step by step (SPEC §10.9)
../.gitea/workflows/build-images.yaml	Gitea Actions image build (the normal path)
secrets/	Git-ignored: fcm-service-account.json, google-services.json, ha_ssh_key(.pub), rclone.conf, optional registry.env. Never committed

All Mac commands below run from the repository root of the main checkout (/Users/jhag/projects/kvitt).

Development stack

Choices. No Caddy and no TLS: debug builds accept http:// to private LAN addresses (AC-2.8), so the server publishes its own port (default 8080) on all of the Mac's interfaces. The server keeps the hardening of SPEC §8.9 (uid 10001, read-only root, no capabilities, no-new-privileges), the 160 MiB limit, json-file logging (10 MB x 3) and TZ=Europe/Stockholm. Data lives in the bind-mounted folder infra/dev/data/ (git-ignored) so it is easy to inspect and reset. The secrets folder is mounted read-only at /secrets; if fcm-service-account.json is missing the server runs with push disabled (§10.4).

On this Mac docker is Podman (docker compose runs podman-compose); the files work with Docker Desktop as well.

Build

infra/dev/build.sh            # both images
infra/dev/build.sh server     # only kvitt-server:dev; needs server/Dockerfile (milestone M2a)
infra/dev/build.sh backup     # only kvitt-backup:dev

The server build is docker build --platform linux/arm64 -f server/Dockerfile -t kvitt-server:dev server (override the context with KVITT_SERVER_BUILD_CONTEXT). The script refuses to run while a Gradle build holds /tmp/kvitt-gradle.lock (CLAUDE.md).

Start, stop, logs

infra/dev/up.sh               # start; prints health, the Mac URL and the phone URL
infra/dev/logs.sh             # follow logs (Ctrl-C to stop following)
infra/dev/up.sh --down        # stop and remove the containers (data stays in infra/dev/data/)

KVITT_PUBLIC_URL (used in invite links) is left empty in .env.example; up.sh sets it to http://<Mac LAN address>:<port>. Pin it in infra/dev/.env if the Mac's address changes often.

Finding the Mac's LAN address

up.sh prints it. By hand: ipconfig getifaddr en0 (Wi-Fi on most Macs), or the interface of the default route: route -n get default | grep interface, then ipconfig getifaddr <interface>. It must be a private address (192.168.*, 10.*, 172.16.* to 172.31.*) for a debug build to accept http:// (AC-2.8).

The first time a phone connects, macOS may ask whether the container runtime (Podman's gvproxy, or Docker) may accept incoming network connections: answer Allow.

Bootstrap and pairing a debug build

The bootstrap and pair-device subcommands (SPEC §10.4, AC-1.1, AC-1.6) arrive with the server milestones.

C="docker compose -f infra/dev/compose.yaml --env-file infra/dev/.env.example"   # or --env-file infra/dev/.env
$C exec server kvitt-server bootstrap                      # first admin invite: code, link, expiry
$C exec server kvitt-server pair-device --user <user_id>   # extra device invite for an existing user
$C run --rm server migrate                                 # when the server is stopped (entrypoint is kvitt-server)

On the phone (home Wi-Fi, debug build se.hagstedt.kvitt.debug): open the pairing screen, enter the server URL http://<Mac LAN address>:8080 and the printed code (AC-2.7). Check reachability first by opening http://<Mac LAN address>:8080/health in Chrome on the phone: it shows "status":"ok".

Backup (local, no Drive upload)

infra/dev/backup.sh

Runs infra/backup/backup.sh once in the backup image (compose profile backup, so up never starts it) with RCLONE_FOLDER_ID empty: .backup snapshot, PRAGMA integrity_check must print ok, then infra/dev/backups/kvitt-<UTC stamp>.sqlite.gz, pictures-<stamp>.tar.gz and last-success; the Drive upload is skipped with a log line and the files are kept (SPEC §10.9, D-001). The wrapper then checks the new snapshot again. To check by hand: gunzip -c infra/dev/backups/kvitt-<stamp>.sqlite.gz > /tmp/k.sqlite && sqlite3 /tmp/k.sqlite "PRAGMA integrity_check".

Resetting the dev data

infra/dev/up.sh --down
rm -rf infra/dev/data         # the dev database, pictures and APKs; never the phones' data
infra/dev/up.sh               # fresh database; run bootstrap again

Paired debug builds then hold a token the new database does not know: re-pair them (clear the debug app's data on the phone, or reinstall the debug build).

Worktrees

Agents work in .claude/worktrees/agent-<id>, which has no infra/secrets/ unless copied (CLAUDE.md). Either copy it from the main checkout, or set KVITT_SECRETS_DIR=/Users/jhag/projects/kvitt/infra/secrets in infra/dev/.env. infra/dev/copy-google-services.sh /Users/jhag/projects/kvitt copies the Firebase config from the main checkout into the worktree's android/app/.

Firebase (SPEC §10.7)

Project kvitt-1337 (Firebase added to the existing Google Cloud project). Free tier only; nothing billable is created by this setup.

Resource	Name
APIs enabled	firebase.googleapis.com, fcm.googleapis.com, cloudresourcemanager.googleapis.com, iam.googleapis.com
Android app	"Kvitt", package se.hagstedt.kvitt
Android app	"Kvitt debug", package se.hagstedt.kvitt.debug
Service account	kvitt-fcm@kvitt-1337.iam.gserviceaccount.com ("Kvitt FCM sender"), role roles/firebasecloudmessaging.admin (the narrowest predefined role holding cloudmessaging.messages.create)
Key	infra/secrets/fcm-service-account.json (git-ignored, mode 600)
Config	infra/secrets/google-services.json, one file with both Android clients (git-ignored)

Put the config where Gradle reads it: infra/dev/copy-google-services.sh.

The setup used only gcloud and the Firebase Management REST API. To repeat or inspect it:

T=$(gcloud auth print-access-token)
H=(-H "Authorization: Bearer $T" -H "x-goog-user-project: kvitt-1337")
API=https://firebase.googleapis.com/v1beta1/projects/kvitt-1337
curl -sS "${H[@]}" "$API/androidApps"                                 # list the apps (ids, packages)
curl -sS "${H[@]}" "$API/androidApps/<appId>/config"                   # configFileContents is base64 google-services.json
gcloud iam service-accounts keys list --iam-account kvitt-fcm@kvitt-1337.iam.gserviceaccount.com --project kvitt-1337

Rotating the key: create a new one with gcloud iam service-accounts keys create infra/secrets/fcm-service-account.json --iam-account kvitt-fcm@kvitt-1337.iam.gserviceaccount.com --project kvitt-1337, restart the server, then delete the old key id with gcloud iam service-accounts keys delete <key id> .... Copying the key to the Pi (§10.7 step 7, mode 0440, group 10001) is done by pi/install.sh ("Pi: what the owner does", step 6).

Pi stack (SPEC §10.1 to §10.10, D-100)

How it fits together

Where	What runs	Who runs it
Gitea Actions (runner label host, a Raspberry Pi)	.gitea/workflows/build-images.yaml: builds kvitt-server, kvitt-cert-sync, kvitt-backup for linux/arm64, pushes them to gitea.lan/jacobh2/... tagged vMAJOR.MINOR.PATCH and latest, pushes the git tag	Gitea, on a push to server/**, infra/cert-sync/**, infra/backup/**, .gitea/workflows/**, or by hand (Actions, "Run workflow")
Owner's Mac	infra/build-images.sh [tag] (fallback image build), infra/publish-apk.sh (release APK into infra/pi/out/apk/)	The owner (or an agent, for the APK build only)
Kvitt Pi, /opt/kvitt/	compose.yaml, Caddyfile, .env, secrets/, import/, apk/, and the scripts deploy.sh, bootstrap.sh, backup-now.sh, install-apk.sh (copied there by install.sh)	The owner, in their own SSH session
Home Assistant Pi	Its own certbot renews icehack.asuscomm.com; cert-sync reads the two PEM files over SSH with a dedicated key	Untouched apart from one authorized_keys entry

Stack (pi/compose.yaml, project name kvitt): caddy (publishes 8443, the only published port), server, cert-sync, backup; volumes data, certs, caddy_data, backups; network kvitt; every service restart: unless-stopped, json-file logs 10 MB x 3, TZ=Europe/Stockholm. Images are ${KVITT_REGISTRY}/kvitt-server|kvitt-cert-sync|kvitt-backup:${KVITT_TAG} with KVITT_REGISTRY=gitea.lan/jacobh2 (D-104; the names the workflow pushes).

Versions (resolved 2026-10-18)

Component	Version	Source
Caddy	caddy:2.11.7@sha256:685ae3d7c34d9b5f8caeddc5d14051db7bbd88ff1f5a2fa0c16bd8ca328fb7af (multi-arch index; has linux/arm64/v8)	Docker Hub official image; tag 2.11.7 = 2.11 = latest, pushed 2026-10-14
Alpine (cert-sync, backup)	alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6	Docker Hub official image; 3.24 = latest
cert-sync packages (Alpine v3.24)	openssh-client-default 10.3_p1-r1, curl 8.22.0-r0, openssl 3.5.9-r0, supercronic 0.2.49-r1, tzdata 2026e-r0	apk policy in the pinned image
backup packages (Alpine v3.24)	sqlite 3.53.4, rclone 1.74.1, tar 1.35, tzdata 2026e (M2c-dev), supercronic 0.2.49-r1	apk policy in the pinned image
Docker Engine on the Pi	Docker's Debian apt repository, arm64 packages (docker-ce 5:29.8.2 current): docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin	docs.docker.com/engine/install/raspberry-pi-os/: that page is for 32-bit (armhf) only; 64-bit Raspberry Pi OS uses the Debian instructions (docs.docker.com/engine/install/debian/)

Why supercronic, not busybox crond (D-104). Both sidecars run as non-root users (cert-sync uid 1000, backup uid 10001). Busybox crond started as a non-root user in alpine 3.24.2 never ran a job (tested 2026-10-18: an every-minute job produced no output in two minutes). supercronic is a cron for containers that runs jobs as the user it runs as and honours TZ; the schedules are unchanged (0 4 * * * cert-sync, 30 3 * * * backup).

Backup uid (D-105, SPEC §10.2 "M2c-pi reconciles the two")

backup runs as uid 10001, the server's uid, on the Pi as on the Mac: sqlite3 .backup on a WAL database must open (and write) kvitt.sqlite-shm, which the server creates as 10001 with mode 0644, so uid 1000 could not back up the live database without loosening the server's files. Therefore /opt/kvitt/secrets/rclone.conf is mode 0600 owned by 10001:10001 (not 1000). cert-sync stays uid 1000 with ha_ssh_key and ha_known_hosts mode 0600 owned by 1000:1000 (ssh refuses a key not owned by its user or readable by others). rclone writes refreshed access tokens back to its config; the mounted file is read-only, so backup.sh works on a private copy in /tmp each run. install.sh gives the named volumes their owners once (/data, /data/apk, /backups to 10001, /certs to 1000) with a one-off root container (docker compose run --user 0:0 --cap-add CHOWN). Caddy runs as root (official image) and reads the 0640 certificate files.

What was checked on the Mac (2026-10-18)

• shellcheck on every infra/**/*.sh: clean.
• podman-compose -f infra/pi/compose.yaml --env-file <filled sample> config: exit 0.
• cert-sync and backup images built for linux/arm64 with Podman (not pushed).
• End-to-end rehearsal with Podman: a stand-in Home Assistant sshd with the variant A key below, the cert-sync image as uid 1000 with StrictHostKeyChecking=yes, the pinned Caddy with pi/Caddyfile in front of a stand-in server. The first fetch installed both files (0640); GET /health through Caddy answered over TLS; a renewed certificate was installed and Caddy reloaded through its admin API without a restart; an unchanged certificate was left alone; an older certificate and a key that does not match the certificate were refused with the old pair kept; TLS 1.1 was refused. Caddy logged the request headers with "Authorization":["REDACTED"] and the token never appeared in its log (SPEC §10.5.2).
• Backup rehearsal with a local rclone remote: upload, lsf confirmation by name and size, local deletion only after both files were confirmed (D-002); files kept and last-success unchanged when the upload failed; the leftovers uploaded by the next good run; healthcheck green on a fresh start and after a run, red at 26 h.
• Not checked (needs the real hosts): the Home Assistant add-on's handling of authorized_keys options, the Gitea workflow (not yet run), pulls from gitea.lan, the router forward, Drive.

Gitea Actions workflow (.gitea/workflows/build-images.yaml), not yet run

Same setup as the owner's working example: runs-on: host, sh steps, apk add git docker-cli docker-cli-buildx, docker login gitea.lan with the repository secret kvitt_cicd (created by the owner) and ${{ github.actor }}, a buildx builder whose buildkitd.toml sets http = true for gitea.lan (created once and reused, so its cache survives), actions/checkout@v4 with set-safe-directory and fetch-depth: 0, the version from the newest v1.0.* git tag plus one, and the git tag pushed after the build. That version is what deploy.sh takes.

Risk: the runner is a Raspberry Pi 3 with 1 GB RAM. The server image compiles about 232 Rust crates, aws-lc-sys's C code among them. The workflow passes --build-arg CARGO_BUILD_JOBS=1; it takes effect only once server/Dockerfile declares ARG CARGO_BUILD_JOBS in its build stage (requested from the server's owner; until then buildx warns about an unused build argument and cargo uses all four cores, which may run out of memory). The Dockerfile already keeps the cargo registry and target directory in BuildKit cache mounts, so only the first build is a full one. The first build's duration on the Pi is unknown (a cold arm64 build takes 48 s on the M4 Mac). If it fails for memory, add swap on the runner Pi, or build on the Mac with infra/build-images.sh.

Mac fallback: infra/build-images.sh [tag]

Builds the same three images for linux/arm64 and pushes them when infra/secrets/registry.env exists (KVITT_REGISTRY=gitea.lan/jacobh2, optional KVITT_REGISTRY_USER and KVITT_REGISTRY_TOKEN with write:package); otherwise it builds locally and prints the exact push commands. Tag: the short git SHA by default, plus latest. Refuses a dirty tree and does not build while a Gradle build holds the gradle-serial lock. With Podman (this Mac) it pushes with --tls-verify=false; with Docker it creates a buildx builder that allows plain HTTP to the registry.

Publishing an APK: infra/publish-apk.sh

Version rule (SPEC §10.6): version_code = git rev-list --count HEAD, version_name = YYYY.MM.DD-<short sha> of HEAD's commit date (Europe/Stockholm), passed as -Pkvitt.versionCode and -Pkvitt.versionName. The image tags of the Gitea workflow (v1.0.N) are separate and say nothing about the APK. The script refuses a dirty tree, builds :app:assembleRelease through tools/gradle-serial.sh (JDK 25 on PATH), refuses an unsigned APK, and writes into infra/pi/out/apk/: kvitt-<version_code>.apk, kvitt-<version_code>.mapping.txt (the R8 mapping, for reading stack traces; never served) and latest.json (temp file, then rename); it keeps the newest 5 APKs with their mappings. Then the owner, on the Mac:

cd /Users/jhag/projects/kvitt
scp infra/pi/out/apk/* <KVITT_PI_SSH_USER>@<KVITT_PI_HOST>:/opt/kvitt/apk/
ssh <KVITT_PI_SSH_USER>@<KVITT_PI_HOST> /opt/kvitt/install-apk.sh

install-apk.sh checks the SHA-256 against latest.json, copies the APKs into the data volume (/data/apk/) and latest.json last with a rename, keeps the newest 5 there and prints GET /apk/latest.

Pi: what the owner does

Fill these in first (in infra/pi/.env on the Mac; the bundle carries the file to the Pi):

Placeholder in infra/pi/.env	What	Where to find it
KVITT_PI_HOST	LAN address of the Kvitt Pi (give it a static DHCP lease)	ASUS router web UI, LAN, DHCP Server, client list
KVITT_PI_SSH_USER	Your user on the Kvitt Pi	The user set in Raspberry Pi Imager
HA_HOST	LAN address of the Home Assistant Pi	Router client list, or Home Assistant, Settings, System, Network
HA_PORT	SSH port of the SSH add-on (default 22)	Home Assistant, Settings, Add-ons, the SSH add-on, Configuration, Network
HA_USER	SSH user of the add-on (root for "Terminal & SSH"; the configured username for "Advanced SSH & Web Terminal")	Same add-on, Configuration
HA_CERT_DIR	Directory holding fullchain.pem and privkey.pem on the Home Assistant Pi (/ssl on Home Assistant OS)	In the add-on's terminal: ls -l /ssl
KVITT_TAG	Image tag to run: the newest v1.0.N the Gitea workflow pushed (or the SHA from build-images.sh)	Gitea, the repository, Packages, kvitt-server
KVITT_REGISTRY_USER, KVITT_REGISTRY_TOKEN	Only if the Gitea packages are private: a Gitea user and a token with read:package	Gitea, Settings, Applications, Generate New Token
KVITT_REGISTRY	Already gitea.lan/jacobh2
RCLONE_REMOTE, RCLONE_FOLDER_ID	Already gdrive: and 1jDp1dGE5R3H2DBcXCqIPN1iWzQnPkdlV

1. Provision the Kvitt Pi

Raspberry Pi Imager on the Mac: Raspberry Pi 3, Raspberry Pi OS Lite (64-bit); in its settings set the host name, your user and password, the network, time zone Europe/Stockholm, and enable SSH with your public key. Boot it, give it a static DHCP lease in the router, then from the Mac: ssh <KVITT_PI_SSH_USER>@<KVITT_PI_HOST>.

2. Install Docker Engine on the Pi (docs.docker.com, Debian arm64 repository)

On the Pi:

sudo apt update && sudo apt install -y ca-certificates curl python3
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker "$USER"
exit

Log in again; docker compose version prints a version and docker run --rm hello-world prints "Hello from Docker!".

3. Put the cert-sync key on the Home Assistant SSH add-on

Home Assistant, Settings, Add-ons, your SSH add-on, Configuration, authorized_keys: add one new entry. Try variant A first (replace /ssl if your HA_CERT_DIR differs):

Variant A (restricted, preferred):

restrict,command="tar -C /ssl -cf - fullchain.pem privkey.pem" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMiXj2dU8GU0Xmdj91WxLhVpPOI1zRWvBF5h2MbE4S8V kvitt-cert-sync

Save, restart the add-on. If the add-on refuses to save or start with it (its Log tab shows an error about the key), replace it with variant B:

Variant B (plain key):

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMiXj2dU8GU0Xmdj91WxLhVpPOI1zRWvBF5h2MbE4S8V kvitt-cert-sync

cert-sync works with both unchanged: it always asks for exactly that tar command. Tell the orchestrator which variant the add-on accepted, so it is recorded here. The private key is /Users/jhag/projects/kvitt/infra/secrets/ha_ssh_key (git-ignored; never paste it anywhere).

4. Fill the .env and stage the secrets, on the Mac

cd /Users/jhag/projects/kvitt
cp infra/pi/.env.example infra/pi/.env
open -e infra/pi/.env            # replace every CHANGE_ME (table above), save
mkdir -p infra/pi/secrets
cp infra/secrets/fcm-service-account.json infra/secrets/ha_ssh_key infra/secrets/rclone.conf infra/pi/secrets/

rclone.conf is already generated from your rclone authorize "drive" (remote [gdrive], type = drive, scope = drive, the token); the Drive folder id is RCLONE_FOLDER_ID in .env, not in the conf. infra/pi/.env and infra/pi/secrets/ are git-ignored.

5. Copy the bundle to the Pi and record the Home Assistant host key

On the Mac:

scp -r infra/pi <KVITT_PI_SSH_USER>@<KVITT_PI_HOST>:kvitt-pi
rm -r infra/pi/secrets           # the originals stay in infra/secrets/

On the Pi:

cd ~/kvitt-pi
ssh-keyscan -p <HA_PORT> <HA_HOST> > secrets/ha_known_hosts
wc -l secrets/ha_known_hosts     # at least 1 line

6. Install and start

On the Pi:

cd ~/kvitt-pi && ./install.sh

It creates /opt/kvitt/{secrets,import,apk}, installs the four secret files (fcm-service-account.json 0440 root:10001; ha_ssh_key, ha_known_hosts 0600 1000:1000; rclone.conf 0600 10001:10001), adds gitea.lan to insecure-registries in /etc/docker/daemon.json (and restarts Docker if that changed), copies compose.yaml, Caddyfile, .env (0600) and the scripts to /opt/kvitt, pulls, starts and waits. Expected end: server: healthy, cert-sync: healthy, caddy: healthy, a docker compose ps table, the line GET https://icehack.asuscomm.com:8443/health (via this Pi): {"status":"ok",...}, then "Kvitt is running.". Then rm -r ~/kvitt-pi/secrets (the script reminds you). Running install.sh again is safe.

7. Router forward

ASUS router web UI, WAN, Virtual Server / Port Forwarding: add a rule, protocol TCP, external port 8443, internal IP <KVITT_PI_HOST>, internal port 8443; Apply. Then on the phone with Wi-Fi off, open https://icehack.asuscomm.com:8443/health in Chrome: it shows "status":"ok" without a certificate warning (yes/no).

8. First backup and bootstrap

On the Pi:

/opt/kvitt/backup-now.sh      # ends with "last-success: <stamp>" and the newest kvitt-*.sqlite.gz and pictures-*.tar.gz on Drive
/opt/kvitt/bootstrap.sh       # first install only: prints the admin invite code, link and expiry

9. Deploying a new version

After the Gitea workflow pushed v1.0.N (or infra/build-images.sh printed a tag), on the Pi:

/opt/kvitt/deploy.sh v1.0.N

It backs up first (Drive confirmed) on every deploy, because any tag may carry a migration (SPEC §10.10), sets KVITT_TAG, pulls, starts, waits for server: healthy and prints /health. If the server does not come up: /opt/kvitt/deploy.sh "$(cat /opt/kvitt/previous-tag)" --skip-backup; if a migration had partially applied, restore the pre-deploy backup (infra/RESTORE.md). A newer bundle with a changed compose.yaml or Caddyfile is installed by copying the bundle again (step 5, without the secrets) and running ./install.sh.

Fallback: Caddy internal CA (SPEC §10.5.3, not deployed)

Only if cert-sync cannot be made to work. In /opt/kvitt/Caddyfile replace the line tls /certs/fullchain.pem /certs/privkey.pem with tls internal, then docker compose restart caddy in /opt/kvitt. Copy Caddy's root to the Mac once: on the Pi docker compose cp caddy:/data/caddy/pki/authorities/local/root.crt /tmp/kvitt_root_ca.crt, then on the Mac scp <KVITT_PI_SSH_USER>@<KVITT_PI_HOST>:/tmp/kvitt_root_ca.crt android/app/src/release/res/raw/kvitt_root_ca.crt (a public certificate, committed by an Android agent) and set kvitt.tls.pinnedRoot=true in android/gradle.properties (D-082). The root is valid 10 years. Caddy still waits for cert-sync to be healthy, which needs the two files in the certs volume: keep the last fetched pair there, or remove the cert-sync condition from depends_on in /opt/kvitt/compose.yaml.

Operations (SPEC §10.10), on the Pi in /opt/kvitt

Task	Command
Status	docker compose ps
Logs	docker compose logs --tail 100 <caddy, server, cert-sync or backup>
Backup now	/opt/kvitt/backup-now.sh
Upgrade	/opt/kvitt/deploy.sh <tag>
Roll back	/opt/kvitt/deploy.sh "$(cat /opt/kvitt/previous-tag)" --skip-backup
Certificate now	docker compose exec cert-sync /usr/local/bin/sync.sh
Certificate expiry and issuer	docker compose exec cert-sync openssl x509 -in /certs/fullchain.pem -noout -enddate -issuer
Extra device invite	docker compose exec server kvitt-server pair-device --user <user_id>
Memory	docker stats --no-stream (server RSS under 100 MB with both phones syncing)
Disk	df -h / and docker system df; SQLite a few MB, pictures under 1 MB, APKs about 20 MB each, 5 kept
No token in Caddy's log	docker compose logs caddy | grep -c 'kvt_' prints 0
Restore	infra/RESTORE.md

Troubleshooting

• cert-sync unhealthy, so caddy does not start: docker compose logs cert-sync. "Host key verification failed": redo ssh-keyscan (step 5) and run ./install.sh again. "Permission denied (publickey)": the add-on does not have the key (step 3). "did not hold fullchain.pem and privkey.pem": wrong HA_CERT_DIR, or HA_USER cannot read privkey.pem.
• Pull fails with "http: server gave HTTP response to HTTPS client": gitea.lan is missing from /etc/docker/daemon.json (install.sh adds it; check with docker info | grep -A3 'Insecure Registries').
• Pull fails with "unauthorized": the packages are private; set KVITT_REGISTRY_USER and KVITT_REGISTRY_TOKEN in ~/kvitt-pi/.env and run ./install.sh again.
• install.sh says /opt/kvitt/secrets/<file> is a directory: Docker created it because the file was missing at a start; sudo rmdir it, put the file in ~/kvitt-pi/secrets/, run ./install.sh.