kvitt/infra/README.md

# infra: Kvitt infrastructure

This file covers the **development stack on the owner's Mac** (`infra/dev/`, milestone M2c-dev, D-001), the
**Firebase setup** (SPEC §10.7) and the **Pi stack** (`infra/pi/`, milestone M2c-pi, D-100): a self-contained
compose project the owner copies to `/home/jacobhagstedt/kvitt` on the Kvitt Pi and runs there (D-125), behind the owner's Nginx Proxy Manager on the Home Assistant
Pi (D-120). No agent and no Mac script connects to the Pis; the owner runs every Pi command and every NPM setting
from the checklist in "Pi: what the owner does".

## Layout

| Path | What |
|---|---|
| `dev/compose.yaml` | Dev stack: `server` (image `kvitt-server:dev`) and the on-demand `backup` job |
| `dev/.env.example` | Dev settings, no secret values; copy to `dev/.env` (git-ignored) to override |
| `dev/build.sh` | Builds `kvitt-server:dev` from `server/Dockerfile` and `kvitt-backup:dev` from `backup/Dockerfile` |
| `dev/up.sh` | Starts the server, waits for `/health`, prints the Mac and phone URLs; `--down` stops the stack |
| `dev/logs.sh` | Follows the logs (`dev/logs.sh --tail 100 server`) |
| `dev/backup.sh` | Runs the backup job once and checks the snapshot with `PRAGMA integrity_check` |
| `dev/copy-google-services.sh` | Copies `secrets/google-services.json` to `android/app/google-services.json` |
| `dev/lib.sh` | Helpers sourced by the scripts |
| `dev/secrets/` | Git-ignored: only `fcm-service-account.json`, copied there by `dev/up.sh` and mounted read-only at `/secrets` |
| `backup/Dockerfile`, `backup/backup.sh`, `backup/healthcheck.sh` | Backup image and job of SPEC §10.9 (shared with the Pi stack); Drive upload and D-002 deletion |
| `pi/` | **The Pi project** (copied to `~/kvitt` on the Kvitt Pi): `compose.yaml`, `.env.example`; git-ignored `pi/.env`, `pi/secrets/`, `pi/out/` |
| `build-images.sh` | Mac fallback for the image build: linux/arm64, push to the Gitea registry |
| `publish-apk.sh` | Mac: release APK, mapping and `latest.json` into `pi/out/apk/` for the owner to copy to the Pi |
| `RESTORE.md` | Restore from a Drive backup, step by step (SPEC §10.9) |
| `../.gitea/workflows/build-images.yaml` | Gitea Actions image build (the normal path) |
| `secrets/` | Git-ignored: `fcm-service-account.json`, `google-services.json`, `rclone.conf`, optional `registry.env` (and `ha_ssh_key(.pub)`, unused since D-120). Never committed |

All Mac commands below run from the repository root of the main checkout (`/Users/jhag/projects/kvitt`).

## Development stack

**Choices.** No Caddy and no TLS: debug builds accept `http://` to private LAN addresses (AC-2.8), so the server
publishes its own port (default 8080) on all of the Mac's interfaces. The server keeps the hardening of SPEC §8.9
(uid 10001, read-only root, no capabilities, `no-new-privileges`), the 160 MiB limit, `json-file` logging
(10 MB x 3) and `TZ=Europe/Stockholm`. Data lives in the bind-mounted folder `infra/dev/data/` (git-ignored) so it
is easy to inspect and reset. Only the FCM key reaches the container (D-117): `up.sh` copies
`${KVITT_SECRETS_DIR}/fcm-service-account.json` (default `infra/secrets/`) into `infra/dev/secrets/` (git-ignored,
nothing else in it) and that folder is mounted read-only at `/secrets`, so the server never sees `rclone.conf`,
the registry credentials or `google-services.json`. If the key is missing the folder is
empty and the server runs with push disabled (§10.4). Run `up.sh` once before the bare `docker compose` commands
below (they mount the same folder); after rotating the key, run `up.sh` again.

On this Mac `docker` is Podman (`docker compose` runs `podman-compose`); the files work with Docker Desktop as well.

### Build

```bash
infra/dev/build.sh            # both images
infra/dev/build.sh server     # only kvitt-server:dev; needs server/Dockerfile
infra/dev/build.sh backup     # only kvitt-backup:dev
```

The server build is `docker build --platform linux/arm64 -f server/Dockerfile -t kvitt-server:dev server`
(override the context with `KVITT_SERVER_BUILD_CONTEXT`). The script refuses to run while a Gradle build holds
`/tmp/kvitt-gradle.lock` (CLAUDE.md).

### Start, stop, logs

```bash
infra/dev/up.sh               # start; prints health, the Mac URL and the phone URL
infra/dev/logs.sh             # follow logs (Ctrl-C to stop following)
infra/dev/up.sh --down        # stop and remove the containers (data stays in infra/dev/data/)
```

`KVITT_PUBLIC_URL` (used in invite links) is left empty in `.env.example`; `up.sh` sets it to
`http://<Mac LAN address>:<port>`. Pin it in `infra/dev/.env` if the Mac's address changes often.

### Finding the Mac's LAN address

`up.sh` prints it. By hand: `ipconfig getifaddr en0` (Wi-Fi on most Macs), or the interface of the default route:
`route -n get default | grep interface`, then `ipconfig getifaddr <interface>`. It must be a private address
(`192.168.*`, `10.*`, `172.16.*` to `172.31.*`) for a debug build to accept `http://` (AC-2.8).

The first time a phone connects, macOS may ask whether the container runtime (Podman's `gvproxy`, or Docker)
may accept incoming network connections: answer Allow.

### Bootstrap and pairing a debug build

The `bootstrap` and `pair-device` subcommands (SPEC §10.4, AC-1.1, AC-1.6) arrive with the server milestones.

```bash
C="docker compose -f infra/dev/compose.yaml --env-file infra/dev/.env.example"   # or --env-file infra/dev/.env
$C exec server kvitt-server bootstrap                      # first admin invite: code, link, expiry
$C exec server kvitt-server pair-device --user <user_id>   # extra device invite for an existing user
$C exec server kvitt-server invite-partner                 # partner invite by the admin, as E-12 (D-113)
$C run --rm server migrate                                 # when the server is stopped (entrypoint is kvitt-server)
```

On the phone (home Wi-Fi, debug build `se.hagstedt.kvitt.debug`): open the pairing screen, enter the server URL
`http://<Mac LAN address>:8080` and the printed code (AC-2.7). Check reachability first by opening
`http://<Mac LAN address>:8080/health` in Chrome on the phone: it shows `"status":"ok"`.

### Backup (local, no Drive upload)

```bash
infra/dev/backup.sh
```

Runs `infra/backup/backup.sh` once in the `backup` image (compose profile `backup`, so `up` never starts it) with
`RCLONE_FOLDER_ID` empty: `.backup` snapshot, `PRAGMA integrity_check` must print `ok`, then
`infra/dev/backups/kvitt-<UTC stamp>.sqlite.gz`, `pictures-<stamp>.tar.gz` and `last-success`; the Drive upload
is skipped with a log line and the files are kept (SPEC §10.9, D-001). The wrapper then checks the new snapshot
again. The dev env file has no rclone settings (`RCLONE_REMOTE` is no longer in `dev/.env.example`): only the Pi's
`.env` names the remote. To check by hand: `gunzip -c infra/dev/backups/kvitt-<stamp>.sqlite.gz > /tmp/k.sqlite && sqlite3 /tmp/k.sqlite "PRAGMA integrity_check"`.

### Resetting the dev data

```bash
infra/dev/up.sh --down
rm -rf infra/dev/data         # the dev database, pictures and APKs; never the phones' data
infra/dev/up.sh               # fresh database; run bootstrap again
```

Paired debug builds then hold a token the new database does not know: re-pair them (clear the debug app's data on
the phone, or reinstall the debug build).

### Worktrees

Agents work in `.claude/worktrees/agent-<id>`, which has no `infra/secrets/` unless copied (CLAUDE.md). Either copy
it from the main checkout, or set `KVITT_SECRETS_DIR=/Users/jhag/projects/kvitt/infra/secrets` in `infra/dev/.env`.
`infra/dev/copy-google-services.sh /Users/jhag/projects/kvitt` copies the Firebase config from the main checkout
into the worktree's `android/app/`.

## Firebase (SPEC §10.7)

Project `kvitt-1337` (Firebase added to the existing Google Cloud project). Free tier only; nothing billable is
created by this setup.

| Resource | Name |
|---|---|
| APIs enabled | `firebase.googleapis.com`, `fcm.googleapis.com`, `cloudresourcemanager.googleapis.com`, `iam.googleapis.com` |
| Android app | "Kvitt", package `se.hagstedt.kvitt` |
| Android app | "Kvitt debug", package `se.hagstedt.kvitt.debug` |
| Service account | `kvitt-fcm@kvitt-1337.iam.gserviceaccount.com` ("Kvitt FCM sender"), role `roles/firebasecloudmessaging.admin` (the narrowest predefined role holding `cloudmessaging.messages.create`) |
| Key | `infra/secrets/fcm-service-account.json` (git-ignored, mode 600) |
| Config | `infra/secrets/google-services.json`, one file with both Android clients (git-ignored) |

Put the config where Gradle reads it: `infra/dev/copy-google-services.sh`.

The setup used only `gcloud` and the Firebase Management REST API. To repeat or inspect it:

```bash
T=$(gcloud auth print-access-token)
H=(-H "Authorization: Bearer $T" -H "x-goog-user-project: kvitt-1337")
API=https://firebase.googleapis.com/v1beta1/projects/kvitt-1337
curl -sS "${H[@]}" "$API/androidApps"                                 # list the apps (ids, packages)
curl -sS "${H[@]}" "$API/androidApps/<appId>/config"                   # configFileContents is base64 google-services.json
gcloud iam service-accounts keys list --iam-account kvitt-fcm@kvitt-1337.iam.gserviceaccount.com --project kvitt-1337
```

Rotating the key: create a new one with
`gcloud iam service-accounts keys create infra/secrets/fcm-service-account.json --iam-account kvitt-fcm@kvitt-1337.iam.gserviceaccount.com --project kvitt-1337`,
restart the server, then delete the old key id with `gcloud iam service-accounts keys delete <key id> ...`.
The key goes to the Pi as `~/kvitt/secrets/fcm-service-account.json` ("Pi: what the owner does", step 2).

## Pi stack (SPEC §10.1 to §10.10, D-100)

### How it fits together

| Where | What runs | Who runs it |
|---|---|---|
| Gitea Actions (runner label `host`, a Raspberry Pi) | `.gitea/workflows/build-images.yaml`: builds `kvitt-server` and `kvitt-backup` for linux/arm64, pushes them to `gitea.lan/jacobh2/...` tagged `vMAJOR.MINOR.PATCH` and `latest`, pushes the git tag | Gitea, on a push to `server/**`, `infra/backup/**`, `.gitea/workflows/**`, or by hand (Actions, "Run workflow") |
| Owner's Mac | `infra/build-images.sh [tag]` (fallback image build), `infra/publish-apk.sh` (release APK into `infra/pi/out/apk/`) | The owner (or an agent, for the APK build only) |
| Kvitt Pi, `/home/jacobhagstedt/kvitt/` | `compose.yaml`, `.env`, `secrets/`, and the bind-mounted `data/` (SQLite, pictures, `apk/`), `backups/`, `import/`; nothing outside this folder (D-125) | The owner, in their own SSH session, with plain `docker compose` commands |
| Home Assistant Pi | Nginx Proxy Manager (NPM): a proxy host for `<KVITT_PUBLIC_HOST>` with its own Let's Encrypt certificate, TLS on 443, plain HTTP to `<kvitt-pi-address>:8080` over the LAN (D-120) | The owner, in NPM's web UI (step 4) |
| Router (ASUS) | No new rule: the existing external TCP 443 forward to the Home Assistant Pi (NPM, for Home Assistant) serves Kvitt too, NPM picks the proxy host by name; external 80 already reaches NPM for Let's Encrypt; nothing is forwarded to the Kvitt Pi (D-124) | The owner checks it (step 5) |

Stack (`pi/compose.yaml`, project name `kvitt`): `server` (publishes `8080:8080` on the Pi's LAN interfaces, the
only published port; `KVITT_PUBLIC_URL=https://${KVITT_PUBLIC_HOST}` (default port 443, D-124), `KVITT_TRUSTED_PROXY=${HA_HOST}`) and
`backup`; relative bind mounts `./data:/data`, `./backups:/backups` (backup), `./import:/import:ro` (server), and one
secret file each, read-only: `secrets/fcm-service-account.json` for the server, `secrets/rclone.conf` for the backup
(D-117: the server never sees the Drive token); both containers run as `${KVITT_UID}:${KVITT_GID}`, the owner's
uid and gid, with a read-only root, a 16 MB tmpfs `/tmp`, no capabilities and `no-new-privileges`; every service
`restart: unless-stopped`, json-file logs 10 MB x 3, `TZ=Europe/Stockholm`; the compose default network. Images are
`${KVITT_REGISTRY}/kvitt-server|kvitt-backup:${KVITT_TAG}` with `KVITT_REGISTRY=gitea.lan/jacobh2` (D-104; the names
the workflow pushes). Caddy and cert-sync were removed by D-120: TLS terminates in NPM, which already runs on the Home
Assistant Pi. There is no install script and nothing on the Pi outside `~/kvitt` is changed (D-125).

**Client IPs (SPEC §8.5, §8.6).** NPM's request to the server carries `X-Forwarded-For: <what the client sent>,
<client address>`; the server takes the last entry only when the TCP peer is `KVITT_TRUSTED_PROXY`, so `HA_HOST`
must be NPM's LAN IPv4 address (the server accepts an IP literal or a resolvable name, not a CIDR; use the
address, because the container cannot resolve an mDNS name such as `homeassistant.local`). The
router's forward keeps the internet client's source address, and Docker's published port keeps NPM's (DNAT, no
SNAT, for traffic arriving on the LAN interface), so the per-IP limits of D-032, D-115 and D-119 see real
addresses. The server does not use `X-Forwarded-Proto` or `Host`: links are built from `KVITT_PUBLIC_URL` (E-12).
Anyone on the home LAN can also reach `http://<kvitt-pi-address>:8080` directly; such requests count against their own
address, and only `HA_HOST` may set `X-Forwarded-For`.

### Versions (resolved 2026-10-18)

| Component | Version | Source |
|---|---|---|
| Nginx Proxy Manager | The owner's existing install on the Home Assistant Pi; the block of step 4 was written against NPM v2.16.0 (2026-09-24) templates | github.com/NginxProxyManager/nginx-proxy-manager, `backend/templates/proxy_host.conf` |
| Alpine (backup) | `alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6` | Docker Hub official image; `3.24` = `latest` |
| backup packages (Alpine v3.24) | sqlite 3.53.4, rclone 1.74.1, tar 1.35, tzdata 2026e (M2c-dev), supercronic 0.2.49-r1 | `apk policy` in the pinned image |
| Docker Engine on the Pi | Docker's Debian apt repository, arm64 packages (`docker-ce` 5:29.8.2 current): `docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin` | docs.docker.com/engine/install/raspberry-pi-os/: that page is for 32-bit (armhf) only; 64-bit Raspberry Pi OS uses the Debian instructions (docs.docker.com/engine/install/debian/) |

**Why supercronic, not busybox crond (D-104).** The backup sidecar runs as a non-root user (the owner's uid on the Pi, D-125). Busybox
crond started as a non-root user in alpine 3.24.2 never ran a job (tested 2026-10-18: an every-minute job produced
no output in two minutes). supercronic is a cron for containers that runs jobs as the
user it runs as and honours `TZ`; the schedule is unchanged (`30 3 * * *`).

### Container uid (D-105, D-125)

`server` and `backup` run as the **same uid**, because `sqlite3 .backup` on a WAL database must open (and write)
`kvitt.sqlite-shm`, which the server creates with mode 0644 (D-105). On the Pi that uid is the owner's own
(`KVITT_UID`, `KVITT_GID` in `.env`, `id -u` / `id -g`), so everything in `data/` and `backups/` belongs to the
owner, the secret files can simply be the owner's with mode 0600, and a restore works with plain `mv` and `cp`
(D-125). The images need no particular uid: the server image is distroless with `USER 10001` only as its default
(compose's `user:` overrides it; the binary needs no account, no home and nothing writable outside `/data`), and the
backup image's scheduler supercronic runs as any uid. The owner's uid has no account inside either image; the
backup container gets `HOME=/tmp` for rclone. Folders the containers write to must exist before the first start
(`mkdir -p data/apk backups import`): a bind-mount source Docker creates itself belongs to root. rclone writes
refreshed access tokens back to its config; the mounted file is read-only, so `backup.sh` works on a private copy
in the tmpfs `/tmp` each run. The development stack on the Mac keeps uid 10001 (`infra/dev/`).

### What was checked on the Mac

2026-10-20 (D-125, self-contained compose project):

* `podman-compose -f infra/pi/compose.yaml --env-file <filled sample> config`: exit 0.
* Rehearsal with Podman in a scratch copy of the project folder, `KVITT_UID`/`KVITT_GID` = the Mac user's 501:20
  (not 10001), images `kvitt-server` from `server/Dockerfile` and `kvitt-backup` from `infra/backup/Dockerfile`:
  both containers healthy with read-only roots; `GET /health` 200 with `db_ok: true`; `kvitt.sqlite`, its `-wal`
  and `-shm` and `data/apk` owned by 501 on the host; `kvitt-server bootstrap` through `exec` printed a code, a
  `https://<KVITT_PUBLIC_HOST>/i/` link and an expiry; an APK and `latest.json` copied into `data/apk/` were served
  at once (`/apk/latest`, `/apk/files/kvitt-<n>.apk` 200 `application/vnd.android.package-archive`); `/backup.sh`
  through `exec`: with `RCLONE_FOLDER_ID` empty it kept both files and wrote `last-success`; with a local rclone
  remote under `/tmp` it uploaded, confirmed and deleted the local copies; with the remote on the read-only root it
  failed, kept the files and left `last-success` unchanged; the archives belong to 501. The restore steps of
  `RESTORE.md` (stop, `gunzip`, `integrity_check` `ok`, `mv` aside, `cp`, `tar -x`, `run --rm server
  rotate-epoch`, start) ended with `/health` 200. Podman-compose ignores `create_host_path: false` (it created a
  directory for a missing secret file); Docker Compose honours it and refuses to start instead, not checked here.
* Not checked (needs the Pi): Docker Engine and Docker Compose themselves, pulls from `gitea.lan`, Drive.

2026-10-19 (D-120, Nginx Proxy Manager instead of Caddy and cert-sync):

* `shellcheck -S style` on every script under `infra/` and `tools/`: clean.
* `podman-compose -f infra/pi/compose.yaml --env-file <filled sample> config` and the same for `infra/dev/`: exit 0.
* `nginx -t` (nginx:alpine 1.31.7) of the step 4 block inside a minimal server block, and inside a copy of NPM
  v2.16.0's `nginx.conf` with the proxy host rendered as NPM's template renders it for the step 4 settings (Block
  Common Exploits, Force SSL, HTTP/2, HSTS, the block; NPM's regex `^(?:.*;)?\s*?location\s*?\/\s*?{` finds the
  block's `location /`, so NPM leaves its own out): both successful.
* Rehearsal with Podman: the server image built from `server/Dockerfile` at this commit (arm64, hardened as on the
  Pi) behind that nginx with a self-signed certificate, `KVITT_TRUSTED_PROXY` = the nginx container's address,
  clients on a non-private test subnet (so NPM's `set_real_ip_from` does not apply, as for internet clients).
  21 checks passed: `GET /health` 200 over HTTP/2 with `Content-Security-Policy`, `X-Content-Type-Options: nosniff`,
  `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer` and `Strict-Transport-Security` once each and no
  `X-Served-By`; the landing page `/i/<code>?from=<name>` 200 with one `Referrer-Policy`; `http://` answered 301 to
  `https://`; a 7 MB body refused by nginx (413), a 2 MB body answered by the server (`payload_too_large`); one
  client limited after 60 requests a minute while a second client was still served, a forged `X-Forwarded-For`
  did not escape the limit, and with `KVITT_TRUSTED_PROXY` set to another address both clients shared one bucket
  (so the limits key on the forwarded address); after `/i/`, `//i/` and `/%69/` requests, and a 502 on `/i/` with
  the server stopped, the invite code, the name and `/i/` occurred 0 times in nginx's access and error logs, while
  other paths were logged with their query strings and a 502 on `/health` was in the error log.
* Not checked (needs the real hosts): NPM's own UI and Let's Encrypt for `<KVITT_PUBLIC_HOST>`, the router
  forward, the Gitea workflow, pulls from `gitea.lan`, Drive.

2026-10-18 (M2c-pi, D-104 to D-106, D-117): backup rehearsal with a local rclone remote: upload, `lsf`
confirmation by name and size, local deletion only after both files were confirmed (D-002); files kept and
`last-success` unchanged when the upload failed; the leftovers uploaded by the next good run; healthcheck green on a
fresh start and after a run, red at 26 h. (The Caddy and cert-sync rehearsals of that day concern the removed stack.)

### Gitea Actions workflow (`.gitea/workflows/build-images.yaml`)

Same setup as the owner's working example: `runs-on: host`, `sh` steps, `apk add git docker-cli docker-cli-buildx`,
`docker login gitea.lan` with `kvitt_cicd` (a user-level secret of the repository owner, created by the owner;
Gitea gives a repository its owner's secrets) and `${{ github.actor }}`, a buildx builder whose `buildkitd.toml`
sets `http = true` for `gitea.lan` (created once and reused, so its cache survives), `actions/checkout@v4` with
`set-safe-directory` and `fetch-depth: 0`, the version from the newest `v1.0.*` git tag plus one, and the git tag
pushed after the build. That version is the `KVITT_TAG` the Pi's `.env` takes.
The checkout is a cone-mode sparse checkout of `server` and `infra/backup` only, the two build contexts (neither
Dockerfile copies anything outside its context), so the action fetches with `--filter=blob:none` and writes only
those files instead of all 2118 (52 MB of Android screenshot goldens among them; about 95 s on the first runs).
`fetch-depth: 0` stays because the version needs every `v1.0.*` tag (commits and trees only, no blobs): with
`fetch-depth: 1`, `fetch-tags: true` brings only tags that point at the fetched commit.
The token is interpolated into the login step's `run` line (`printf '%s' "${{ secrets.kvitt_cicd }}" | docker login
...`), the owner's proven form: the first run on the runner (2026-10-19) got an empty token from the env-based form
(`env: REGISTRY_TOKEN: ${{ secrets.kvitt_cicd }}`) on act_runner 0.6.1 and failed with "password is empty";
act_runner masks the secret in the log. That first run got as far as the login (container, `apk add`, checkout
all worked). `kvitt_cicd` needs only the `write:package` scope: the tag push uses the credentials `actions/checkout` stored (the job's own token).
If the "Tag and Push Version" step fails with a permission error, the job token cannot push tags on this Gitea;
then add `write:repository` to the token (owner decision below). The step "Print image digests" prints each
image's `sha256:` digest, which must match the digest the Pi pulled (Operations, "Image digests"; D-117).

The builder's `buildkitd.toml` is read only when the builder is created. After changing the registry setting in the
workflow, remove the builder once on the runner Pi: `docker buildx rm local-http-builder` (this also drops its
build cache, so the next server build is a full one).

**Risk: the runner is a Raspberry Pi 3 with 1 GB RAM.** The server image compiles about 232 Rust crates,
aws-lc-sys's C code among them. The workflow passes `--build-arg CARGO_BUILD_JOBS=1`; it takes effect only once
`server/Dockerfile` declares `ARG CARGO_BUILD_JOBS` in its build stage (requested from the server's owner; until
then buildx warns about an unused build argument and cargo uses all four cores, which may run out of memory). The
Dockerfile already keeps the cargo registry and target directory in BuildKit cache mounts, so only the first build
is a full one. The first build's duration on the Pi is unknown (a cold arm64 build takes 48 s on the M4 Mac). If
it fails for memory, add swap on the runner Pi, or build on the Mac with `infra/build-images.sh`.

### Mac fallback: `infra/build-images.sh [tag]`

Builds the same two images for linux/arm64 and pushes them when `infra/secrets/registry.env` exists
(`KVITT_REGISTRY=gitea.lan/jacobh2`, optional `KVITT_REGISTRY_USER` and `KVITT_REGISTRY_TOKEN` with
write:package); otherwise it builds locally and prints the exact push commands. Tag: the short git SHA by default,
plus `latest`. Refuses a dirty tree and does not build while a Gradle build holds the gradle-serial lock. With
Podman (this Mac) it pushes with `--tls-verify=false`; with Docker it creates a buildx builder that allows plain
HTTP to the registry.

### Publishing an APK: `infra/publish-apk.sh`

Version rule (SPEC §10.6): `version_code` = `git rev-list --count HEAD`, `version_name` = `YYYY.MM.DD-<short sha>`
of HEAD's commit date (Europe/Stockholm), passed as `-Pkvitt.versionCode` and `-Pkvitt.versionName`; the script
reads `KVITT_PUBLIC_HOST` from `infra/pi/.env` (refusing a missing file or key or a `CHANGE_ME` value) and passes it as
`-Pkvitt.publicHost`, the app's release server and invite-link host (D-121). The image
tags of the Gitea workflow (`v1.0.N`) are separate and say nothing about the APK. The script refuses a dirty tree,
builds `:app:assembleRelease` through `tools/gradle-serial.sh` (JDK 25 on `PATH`), refuses an unsigned APK, and
writes into `infra/pi/out/apk/`: `kvitt-<version_code>.apk`, `kvitt-<version_code>.mapping.txt` (the R8 mapping,
for reading stack traces; never served) and `latest.json` (temp file, then rename); it keeps the newest 5 APKs
with their mappings. Then the owner, on the Mac, copies the APK first and `latest.json` last into the server's
`data/apk/` (the server reads `latest.json` on every `GET /apk/latest`, so it must never name a file that is not
there yet):

```bash
cd /Users/jhag/projects/kvitt
scp infra/pi/out/apk/kvitt-<version_code>.apk jacobhagstedt@<kvitt-pi-address>:kvitt/data/apk/
scp infra/pi/out/apk/latest.json jacobhagstedt@<kvitt-pi-address>:kvitt/data/apk/
curl -s http://<kvitt-pi-address>:8080/apk/latest     # shows the new version_code
```

The mapping files stay on the Mac. Older APKs on the Pi are removed by hand when wanted
(`ssh jacobhagstedt@<kvitt-pi-address> ls kvitt/data/apk`, then `rm` the old `kvitt-<n>.apk` files; about 10 MB each).

## Pi: what the owner does

Kvitt runs from `/home/jacobhagstedt/kvitt` on the Kvitt Pi (D-125): `compose.yaml`, `.env`, `secrets/` and the
folders `data/`, `backups/`, `import/`. Nothing outside that folder is installed or changed. `<kvitt-pi-address>` is
the Kvitt Pi's LAN IPv4 address (static DHCP lease; ASUS router web UI, LAN, DHCP Server, client list), also NPM's
Forward Hostname / IP (step 4).

### 1. Prerequisites (already on the Pi)

* Docker Engine with the compose plugin, usable by `jacobhagstedt` without sudo: `docker compose version` prints a
  version.
* Registry trust: the images come over plain HTTP from `gitea.lan`. If `docker compose pull` says "http: server
  gave HTTP response to HTTPS client", add `gitea.lan` to `insecure-registries` in `/etc/docker/daemon.json` (the
  Pi already has it for another project; Kvitt never edits it).
* If `docker compose pull` says "unauthorized" (private packages): once, `docker login gitea.lan` with your Gitea
  user and a token with `read:package` (Gitea, Settings, Applications, Generate New Token).

### 2. On the Mac: `.env`, secrets, copy

```bash
cd /Users/jhag/projects/kvitt
cp infra/pi/.env.example infra/pi/.env
open -e infra/pi/.env            # set HA_HOST; KVITT_UID/KVITT_GID = `id -u` / `id -g` on the Pi (1000 for the first user)
mkdir -p infra/pi/secrets
cp infra/secrets/fcm-service-account.json infra/secrets/rclone.conf infra/pi/secrets/
scp -r infra/pi/compose.yaml infra/pi/.env infra/pi/secrets jacobhagstedt@<kvitt-pi-address>:kvitt/
rm -r infra/pi/secrets           # the originals stay in infra/secrets/
```

| Key in `infra/pi/.env` | Value |
|---|---|
| `HA_HOST` | LAN IPv4 address of the Home Assistant Pi (NPM), for example `192.168.1.20`, not `homeassistant.local` (router client list, or Home Assistant, Settings, System, Network) |
| `KVITT_UID`, `KVITT_GID` | `id -u` and `id -g` on the Kvitt Pi as `jacobhagstedt` (default 1000, 1000) |
| `KVITT_TAG` | `v1.0.0`, or the newest `v1.0.N` the Gitea workflow pushed (Gitea, the repository, Packages, `kvitt-server`) |
| `KVITT_PUBLIC_HOST`, `KVITT_REGISTRY`, `RCLONE_FOLDER_ID` | Already `kvitt.suorra.duckdns.org`, `gitea.lan/jacobh2`, `1jDp1dGE5R3H2DBcXCqIPN1iWzQnPkdlV` |

`rclone.conf` is generated from your `rclone authorize "drive"` (remote `[gdrive]`, `scope = drive`); the Drive
folder id is `RCLONE_FOLDER_ID` in `.env`. `infra/pi/.env` and `infra/pi/secrets/` are git-ignored. To change
`compose.yaml` or `.env` later, copy just that file again the same way.

### 3. On the Pi: start

```bash
cd ~/kvitt
chmod 700 secrets && chmod 600 secrets/*
mkdir -p data/apk backups import
docker compose pull
docker compose up -d
docker compose ps
curl -s localhost:8080/health
```

Expected: `docker compose ps` lists `server` (ports `0.0.0.0:8080->8080/tcp`) and `backup`, both `(healthy)` within
a minute; `curl` prints `{"status":"ok",...,"db_ok":true}`.

### 4. Nginx Proxy Manager on the Home Assistant Pi (already set up)

NPM terminates TLS for Kvitt with its own Let's Encrypt certificate for `<KVITT_PUBLIC_HOST>` and forwards plain
HTTP over the home LAN to the Kvitt Pi's port 8080 (plain HTTP between the two Pis is accepted, D-120). First make
sure `<KVITT_PUBLIC_HOST>` points at the home's public address (your DNS or DuckDNS page shows the same address as
`icehack.asuscomm.com`). Let's Encrypt's HTTP-01 challenge reaches NPM through external port 80, which already
works: it is how NPM got the certificate for Home Assistant. Serving Kvitt under a path of `icehack.asuscomm.com`
(for example `/kvitt/`) is not supported. NPM listens on the default HTTPS port 443 for every proxy host and
picks the host by the name the client asks for, so Kvitt and Home Assistant share that port (D-124).

Open NPM's web UI (Home Assistant, Settings, Add-ons, Nginx Proxy Manager, Open Web UI), then Hosts, Proxy Hosts,
Add Proxy Host.

Details tab:

| Field | Value |
|---|---|
| Domain Names | `<KVITT_PUBLIC_HOST>` |
| Scheme | `http` |
| Forward Hostname / IP | The Kvitt Pi's LAN IPv4 address (`<kvitt-pi-address>`) |
| Forward Port | `8080` |
| Cache Assets | off |
| Block Common Exploits | on |
| Websockets Support | off |
| Access List | Publicly Accessible |

Custom Locations tab: leave empty.

SSL tab:

| Field | Value |
|---|---|
| SSL Certificate | Request a new Certificate (Let's Encrypt, HTTP challenge: Use DNS Challenge off); Email address: yours; if a Let's Encrypt Terms of Service checkbox is shown, tick it |
| Force SSL | on |
| HTTP/2 Support | on |
| HSTS Enabled | on |
| HSTS Sub-domains | off |
| Trust Upstream Forwarded Proto Headers | off |

Advanced tab, Custom Nginx Configuration: paste this block exactly (all of it, comments included):

```nginx
# Kvitt (SPEC §10.5.2, D-120). NPM, Proxy Host <KVITT_PUBLIC_HOST>, Advanced tab, Custom Nginx Configuration.
# Server body: 1 MiB, PUT /me/picture 5 MiB (SPEC §8.6); the server answers anything larger with its own 413.
client_max_body_size 6m;
# No long-poll in the API (SPEC §14.3); 60 s between two reads from the server.
proxy_read_timeout 60s;
# Response headers (D-117). They live at server level, so no location below may use add_header (nginx would then
# drop all of these in it); NPM's HSTS line (SSL tab) sits at the same level. The server's own copy is hidden, so
# each header appears once (the landing page sends Referrer-Policy itself).
proxy_hide_header Content-Security-Policy;
proxy_hide_header X-Content-Type-Options;
proxy_hide_header X-Frame-Options;
proxy_hide_header Referrer-Policy;
add_header Content-Security-Policy "default-src 'none'; style-src 'unsafe-inline'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always;
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options DENY always;
add_header Referrer-Policy no-referrer always;
# Replaces NPM's own "location /" (NPM leaves it out when this field has one), whose add_header X-Served-By would
# drop the headers above. $forward_scheme, $server and $port are NPM's Details tab values.
location / {
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_pass $forward_scheme://$server:$port$request_uri;
}
# Invite links /i/<code>?from=<name>: neither the code nor the inviter's name may reach a log (SPEC §8.7). NPM's
# access log format writes the whole $request_uri, query string included, so this path is not logged at all, and
# its error log (which would carry the request line on a 502) goes nowhere.
location ^~ /i/ {
  access_log off;
  error_log /dev/null crit;
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_pass $forward_scheme://$server:$port$request_uri;
}
```

Save. The new row under Proxy Hosts shows the status Online and an SSL entry for Let's Encrypt (yes/no). Then on
the Kvitt Pi:

```bash
curl -fsS --resolve <KVITT_PUBLIC_HOST>:443:<HA_HOST> https://<KVITT_PUBLIC_HOST>/health
```

It prints `{"status":"ok",...}` (no `-k`: this also checks the certificate).

**Why the block has its own `location /`.** nginx drops every `add_header` inherited from the server level in a
location that has an `add_header` of its own, and NPM's default `location /` has one (`X-Served-By`), which would
silently drop the four security headers (D-117). NPM leaves its default location out when the Custom Nginx
Configuration contains `location /`, so the block brings its own without `add_header`; NPM's HSTS line from the SSL
tab stays at server level and still applies. The block's `proxy_pass` uses NPM's Forward Hostname / IP and Forward
Port, so change those on the Details tab as usual. If a later NPM release changes its proxy host template, check
the headers again (Operations, "Headers through NPM").

**What NPM logs (SPEC §8.7).** NPM writes one access log line per request to `/data/logs/proxy-host-<id>_access.log`
inside NPM, in its `proxy` format: time, cache and upstream status, status, method, scheme, host,
`"$request_uri"` (the path **with** its query string), `[Client <address>]`, length, gzip ratio,
`[Sent-to <kvitt-pi-address>]`, user agent and referer. It logs no request headers, so the `Authorization` token is never
logged. Query strings are logged for every path except `/i/`; the other query parameters of the API are `since` and
`limit` (`GET /sync`) and `size` and `v` (pictures), which hold no personal content. `/i/<code>?from=<name>` is not
logged at all: the block turns the access log off for `/i/` and sends its error log to `/dev/null`, so neither the
invite code nor the inviter's name reaches a log. Errors on other paths (for example a 502 while the server is down)
go to `proxy-host-<id>_error.log` with the request line.

**HSTS.** HSTS tells a browser to use only HTTPS for `<KVITT_PUBLIC_HOST>`, on every port, for two years
(`max-age=63072000; preload`, NPM's value), and to refuse certificate warnings for it. Because the name is Kvitt's
own, nothing else is affected; but if you ever serve anything else under `<KVITT_PUBLIC_HOST>` (any port), it must be
HTTPS with a valid certificate too, and turning HSTS off later does not clear browsers that already stored it. The
app itself does not rely on HSTS (release builds allow no cleartext, SPEC §8.5); it protects the landing page in a
browser.

**Accepted: NPM trusts `X-Real-IP` from the LAN.** NPM's `nginx.conf` takes the client address from `X-Real-IP` when
the connection comes from 10/8, 172.16/12, 192.168/16 (and the CDN ranges it downloads), so a device on the home
LAN can choose its own client address for the rate limits. Internet clients arrive with their public address (the
router's forward keeps the source address), so their `X-Real-IP` is ignored.

### 5. Router: no new rule

No new rule (D-124): external TCP 443 already reaches the Home Assistant Pi (NPM) for Home Assistant, and NPM picks
Kvitt's proxy host by name on the same port. Verify it exists: ASUS router web UI, WAN, Virtual Server / Port
Forwarding: the list has a TCP rule with External Port `443` and Internal IP Address `<HA_HOST>` (yes/no). Nothing is
forwarded to the Kvitt Pi: remove a rule to `<kvitt-pi-address>` if you made one for the Caddy version, and a rule with
External Port `8443` if you made one for the earlier NPM version (D-120). Then on the phone with Wi-Fi off, open
`https://<KVITT_PUBLIC_HOST>/health` in Chrome: it shows `"status":"ok"` without a certificate warning (yes/no).

### 6. From outside, first backup, bootstrap

On the phone with Wi-Fi off (or from the Mac), `https://kvitt.suorra.duckdns.org/health` shows `"status":"ok"`
(yes/no). Then on the Pi:

```bash
cd ~/kvitt
docker compose exec backup /backup.sh                 # ends with "backup: ok <stamp>" after "uploaded and confirmed on Drive" twice
docker compose exec server kvitt-server bootstrap     # first install only: prints the admin invite code, link and expiry
```

The invite link starts with `https://<KVITT_PUBLIC_HOST>/i/` (no port). The release app opens such links and connects to
that host only once its Android build carries the host name (`kvitt.publicHost`, D-121) and the default port 443
(D-124; builds made for port 8443 do not match a link without a port): tell the orchestrator before pairing a
release build. The partner's invite, once the admin is paired: `docker compose exec server kvitt-server invite-partner`.

### 7. Deploying a new version

After the Gitea workflow pushed `v1.0.N` (or `infra/build-images.sh` printed a tag), on the Pi:

```bash
cd ~/kvitt
docker compose exec backup /backup.sh                 # backup first, every deploy: any tag may carry a migration (SPEC §10.10)
sed -i 's/^KVITT_TAG=.*/KVITT_TAG=v1.0.N/' .env       # or edit .env; note the previous tag for a rollback
docker compose pull && docker compose up -d
docker compose ps && curl -s localhost:8080/health
```

Continue only when the backup ended with `backup: ok`. The images come over plain HTTP from `gitea.lan` (D-001), so
compare the pulled digests (Operations, "Image digests") with the workflow's "Print image digests" step for that
run (D-117). Rollback: set the previous tag in `.env` the same way and run `docker compose pull && docker compose
up -d`; if a migration had partially applied, restore the pre-deploy backup (`infra/RESTORE.md`).

### Fallback (SPEC §10.5.3): when NPM is down

* NPM or the Home Assistant Pi down means Kvitt is unreachable from outside, and also from home over
  `https://<KVITT_PUBLIC_HOST>` (that path goes through NPM too); Home Assistant is then unreachable from outside as
  well (same NPM, same port 443). The server on the Kvitt Pi keeps running and nothing there needs to change. The phones keep working on their full local copy and sync when NPM is back.
* On the home Wi-Fi only **debug builds** can reach the server directly, at `http://<kvitt-pi-address>:8080` (AC-2.8
  allows `http://` to private LAN addresses in debug builds only). **Release builds require HTTPS with a publicly
  trusted certificate (SPEC §8.5) and cannot use that address: there is no LAN fallback for the release app.** A
  debug build is a separate app (`se.hagstedt.kvitt.debug`) with its own pairing and data, not a stand-in for the
  release app.
* If NPM cannot renew the Let's Encrypt certificate: fix it in NPM (SSL Certificates, the certificate, Renew Now;
  port 80 must reach NPM). The pinned-root switch of the app (`kvitt.tls.pinnedRoot`, D-082) still exists for a
  self-made root whose certificate would be uploaded to NPM as a Custom certificate; it is not prepared: tell the
  orchestrator.

### Operations (SPEC §10.10), on the Pi in `~/kvitt`

| Task | Command |
|---|---|
| Status | `docker compose ps` |
| Logs | `docker compose logs --tail 100 <server or backup>`; NPM's own log files are inside NPM on the Home Assistant Pi |
| Backup now | `docker compose exec backup /backup.sh` (ends with `backup: ok <stamp>`) |
| Upgrade | Step 7: backup, set `KVITT_TAG` in `.env`, `docker compose pull && docker compose up -d` |
| Roll back | Set the previous `KVITT_TAG` in `.env`, `docker compose pull && docker compose up -d` |
| Health through NPM | `curl -fsS --resolve <KVITT_PUBLIC_HOST>:443:<HA_HOST> https://<KVITT_PUBLIC_HOST>/health` |
| Headers through NPM | `curl -sS -D - -o /dev/null --resolve <KVITT_PUBLIC_HOST>:443:<HA_HOST> https://<KVITT_PUBLIC_HOST>/health` lists `content-security-policy`, `x-content-type-options`, `x-frame-options`, `referrer-policy`, `strict-transport-security` once each and no `x-served-by` |
| Certificate expiry and issuer | `echo \| openssl s_client -connect <HA_HOST>:443 -servername <KVITT_PUBLIC_HOST> 2>/dev/null \| openssl x509 -noout -enddate -issuer` (NPM renews it; NPM, SSL Certificates shows the same) |
| Partner invite | `docker compose exec server kvitt-server invite-partner` |
| Extra device invite | `docker compose exec server kvitt-server pair-device --user <user_id>` |
| Memory | `docker stats --no-stream` (server RSS under 100 MB with both phones syncing) |
| Disk | `df -h ~` and `du -sh ~/kvitt/*`; SQLite a few MB, pictures under 1 MB, APKs about 10 MB each since R8, 5 kept |
| Image digests | `docker compose images` names the images; `docker image inspect --format '{{.RepoDigests}}' gitea.lan/jacobh2/kvitt-server:<tag>` (and `kvitt-backup`) prints the `sha256:` digest to compare with the workflow's |
| Client IPs are real (once, after step 5) | Open `https://<KVITT_PUBLIC_HOST>/health` on the phone with Wi-Fi off, then within a minute on the Kvitt Pi `sudo apt install -y conntrack` (once) and `sudo conntrack -L -p tcp --orig-port-dst 8080`: in every line the first `src=` is `<HA_HOST>` and the second half (from the second `src=`) has `dst=<HA_HOST>`, no `172.*` there, so the server sees NPM's address and trusts its `X-Forwarded-For`. If the second half shows a `172.*` address instead, every internet client shares one per-IP limit (SPEC §8.6): tell the orchestrator |
| Restore | `infra/RESTORE.md` |

**APK downloads (security review L-3, D-119).** `GET /apk/files/<name>` is public (the landing page links it) and
about 11 MB per response; the server allows 10 per minute per client IP. If one address still fetches
`/apk/files/` over and over (NPM's access log), disable Kvitt's proxy host in NPM (Hosts, Proxy Hosts, the row's
menu, Disable) until it stops; the router's 443 forward also serves Home Assistant, so leave it in place.

### Owner decisions (open)

Backup encryption (security review M-1) is decided: no rclone `crypt`, the `drive` scope stays (SPEC §8.7, 2026-10-19).

1. **Gitea token scope.** Does the workflow's tag push work with `kvitt_cicd` limited to `write:package`? If not,
   is adding `write:repository` acceptable?
2. **Kvitt's host name (D-120).** Which name is `<KVITT_PUBLIC_HOST>`? It goes into the release app and
   `docs/HANDOFF.md`.

### Troubleshooting

* The step 4 check says /health through NPM failed: "Could not resolve host" or "SSL certificate problem":
  the proxy host has no certificate for `<KVITT_PUBLIC_HOST>` yet (SSL tab) or a different name; a `502 Bad Gateway`:
  NPM cannot reach `<kvitt-pi-address>:8080` (wrong Forward Hostname / IP or Forward Port, or the server is down:
  `docker compose ps`); the Home Assistant page instead of `{"status":"ok"}`: Domain Names does not match the name
  you requested (NPM fell back to another host).
* "Request a new Certificate" fails in NPM: `<KVITT_PUBLIC_HOST>` does not point at the home's public address yet,
  or external port 80 does not reach NPM.
* Pull fails with "http: server gave HTTP response to HTTPS client": `gitea.lan` is missing from `insecure-registries`
  (step 1; check with `docker info | grep -A3 'Insecure Registries'`).
* Pull fails with "unauthorized": the packages are private; `docker login gitea.lan` once (step 1).
* `docker compose up` says "bind source path does not exist" for a file in `secrets/`: the file was not copied;
  copy it (step 2) and run `docker compose up -d` again.
* The server or backup log says "Permission denied" or "readonly database" on `/data` or `/backups`: the folder was
  created by Docker as root, or `KVITT_UID`/`KVITT_GID` differ from the folder's owner: `ls -ln ~/kvitt` shows the
  owner, `id -u` / `id -g` yours; `sudo chown -R "$(id -u):$(id -g)" ~/kvitt/data ~/kvitt/backups ~/kvitt/import` once.