diff --git a/infra/README.md b/infra/README.md index 2cc6f26..700867c 100644 --- a/infra/README.md +++ b/infra/README.md @@ -2,8 +2,9 @@ This file covers the **development stack on the owner's Mac** (`infra/dev/`, milestone M2c-dev, D-001), the **Firebase setup** (SPEC §10.7) and the **Pi stack** (`infra/pi/`, milestone M2c-pi, D-100): a self-contained -bundle the owner copies to the Kvitt Pi and runs there. No agent and no Mac script connects to the Pis; the owner -runs every Pi command from the checklist in "Pi: what the owner does". +bundle the owner copies to the Kvitt Pi and runs there, behind the owner's Nginx Proxy Manager on the Home Assistant +Pi (D-120). No agent and no Mac script connects to the Pis; the owner runs every Pi command and every NPM setting +from the checklist in "Pi: what the owner does". ## Layout @@ -19,13 +20,12 @@ runs every Pi command from the checklist in "Pi: what the owner does". | `dev/lib.sh` | Helpers sourced by the scripts | | `dev/secrets/` | Git-ignored: only `fcm-service-account.json`, copied there by `dev/up.sh` and mounted read-only at `/secrets` | | `backup/Dockerfile`, `backup/backup.sh`, `backup/healthcheck.sh` | Backup image and job of SPEC §10.9 (shared with the Pi stack); Drive upload and D-002 deletion | -| `cert-sync/Dockerfile`, `cert-sync/sync.sh`, `cert-sync/entrypoint.sh` | Certificate sync image of SPEC §10.5.1 | -| `pi/` | **The Pi bundle** (copied to the Kvitt Pi): `compose.yaml`, `Caddyfile`, `.env.example`, `install.sh`, `deploy.sh`, `bootstrap.sh`, `backup-now.sh`, `install-apk.sh`, `lib.sh`; git-ignored `pi/.env`, `pi/secrets/`, `pi/out/` | +| `pi/` | **The Pi bundle** (copied to the Kvitt Pi): `compose.yaml`, `.env.example`, `install.sh`, `deploy.sh`, `bootstrap.sh`, `backup-now.sh`, `install-apk.sh`, `lib.sh`; git-ignored `pi/.env`, `pi/secrets/`, `pi/out/` | | `build-images.sh` | Mac fallback for the image build: linux/arm64, push to the Gitea registry | | `publish-apk.sh` | Mac: release APK, mapping and `latest.json` into `pi/out/apk/` for the owner to copy to the Pi | | `RESTORE.md` | Restore from a Drive backup, step by step (SPEC §10.9) | | `../.gitea/workflows/build-images.yaml` | Gitea Actions image build (the normal path) | -| `secrets/` | Git-ignored: `fcm-service-account.json`, `google-services.json`, `ha_ssh_key(.pub)`, `rclone.conf`, optional `registry.env`. Never committed | +| `secrets/` | Git-ignored: `fcm-service-account.json`, `google-services.json`, `rclone.conf`, optional `registry.env` (and `ha_ssh_key(.pub)`, unused since D-120). Never committed | All Mac commands below run from the repository root of the main checkout (`/Users/jhag/projects/kvitt`). @@ -37,8 +37,8 @@ publishes its own port (default 8080) on all of the Mac's interfaces. The server (10 MB x 3) and `TZ=Europe/Stockholm`. Data lives in the bind-mounted folder `infra/dev/data/` (git-ignored) so it is easy to inspect and reset. Only the FCM key reaches the container (D-117): `up.sh` copies `${KVITT_SECRETS_DIR}/fcm-service-account.json` (default `infra/secrets/`) into `infra/dev/secrets/` (git-ignored, -nothing else in it) and that folder is mounted read-only at `/secrets`, so the server never sees the Home Assistant -SSH key, `rclone.conf`, the registry credentials or `google-services.json`. If the key is missing the folder is +nothing else in it) and that folder is mounted read-only at `/secrets`, so the server never sees `rclone.conf`, +the registry credentials or `google-services.json`. If the key is missing the folder is empty and the server runs with push disabled (§10.4). Run `up.sh` once before the bare `docker compose` commands below (they mount the same folder); after rotating the key, run `up.sh` again. @@ -153,7 +153,7 @@ gcloud iam service-accounts keys list --iam-account kvitt-fcm@kvitt-1337.iam.gse Rotating the key: create a new one with `gcloud iam service-accounts keys create infra/secrets/fcm-service-account.json --iam-account kvitt-fcm@kvitt-1337.iam.gserviceaccount.com --project kvitt-1337`, restart the server, then delete the old key id with `gcloud iam service-accounts keys delete ...`. -Copying the key to the Pi (§10.7 step 7, mode 0440, group 10001) is done by `pi/install.sh` ("Pi: what the owner does", step 6). +Copying the key to the Pi (§10.7 step 7, mode 0440, group 10001) is done by `pi/install.sh` ("Pi: what the owner does", step 5). ## Pi stack (SPEC §10.1 to §10.10, D-100) @@ -161,89 +161,95 @@ Copying the key to the Pi (§10.7 step 7, mode 0440, group 10001) is done by `pi | Where | What runs | Who runs it | |---|---|---| -| Gitea Actions (runner label `host`, a Raspberry Pi) | `.gitea/workflows/build-images.yaml`: builds `kvitt-server`, `kvitt-cert-sync`, `kvitt-backup` for linux/arm64, pushes them to `gitea.lan/jacobh2/...` tagged `vMAJOR.MINOR.PATCH` and `latest`, pushes the git tag | Gitea, on a push to `server/**`, `infra/cert-sync/**`, `infra/backup/**`, `.gitea/workflows/**`, or by hand (Actions, "Run workflow") | +| Gitea Actions (runner label `host`, a Raspberry Pi) | `.gitea/workflows/build-images.yaml`: builds `kvitt-server` and `kvitt-backup` for linux/arm64, pushes them to `gitea.lan/jacobh2/...` tagged `vMAJOR.MINOR.PATCH` and `latest`, pushes the git tag | Gitea, on a push to `server/**`, `infra/backup/**`, `.gitea/workflows/**`, or by hand (Actions, "Run workflow") | | Owner's Mac | `infra/build-images.sh [tag]` (fallback image build), `infra/publish-apk.sh` (release APK into `infra/pi/out/apk/`) | The owner (or an agent, for the APK build only) | -| Kvitt Pi, `/opt/kvitt/` | `compose.yaml`, `Caddyfile`, `.env`, `secrets/`, `import/`, `apk/`, and the scripts `deploy.sh`, `bootstrap.sh`, `backup-now.sh`, `install-apk.sh` (copied there by `install.sh`) | The owner, in their own SSH session | -| Home Assistant Pi | Its own certbot renews `icehack.asuscomm.com`; cert-sync reads the two PEM files over SSH with a dedicated key | Untouched apart from one `authorized_keys` entry | +| Kvitt Pi, `/opt/kvitt/` | `compose.yaml`, `.env`, `secrets/`, `import/`, `apk/`, and the scripts `deploy.sh`, `bootstrap.sh`, `backup-now.sh`, `install-apk.sh` (copied there by `install.sh`) | The owner, in their own SSH session | +| Home Assistant Pi | Nginx Proxy Manager (NPM): a proxy host for `` with its own Let's Encrypt certificate, TLS on 443, plain HTTP to `:8080` over the LAN (D-120) | The owner, in NPM's web UI (step 6) | +| Router (ASUS) | External TCP 8443 to the Home Assistant Pi's 443 (NPM); external 80 already reaches NPM for Let's Encrypt | The owner (step 7) | -Stack (`pi/compose.yaml`, project name `kvitt`): `caddy` (publishes 8443, the only published port), `server`, -`cert-sync`, `backup`; volumes `data`, `certs`, `caddy_data`, `backups`; network `kvitt`; every service -`restart: unless-stopped`, json-file logs 10 MB x 3, `TZ=Europe/Stockholm`. Images are -`${KVITT_REGISTRY}/kvitt-server|kvitt-cert-sync|kvitt-backup:${KVITT_TAG}` with `KVITT_REGISTRY=gitea.lan/jacobh2` -(D-104; the names the workflow pushes). +Stack (`pi/compose.yaml`, project name `kvitt`): `server` (publishes `8080:8080` on the Pi's LAN interfaces, the +only published port; `KVITT_PUBLIC_URL=https://${KVITT_PUBLIC_HOST}:8443`, `KVITT_TRUSTED_PROXY=${HA_HOST}`) and +`backup`; volumes `data`, `backups`; network `kvitt`; every service `restart: unless-stopped`, json-file logs +10 MB x 3, `TZ=Europe/Stockholm`. Images are `${KVITT_REGISTRY}/kvitt-server|kvitt-backup:${KVITT_TAG}` with +`KVITT_REGISTRY=gitea.lan/jacobh2` (D-104; the names the workflow pushes). Caddy and cert-sync were removed by +D-120: TLS terminates in NPM, which already runs on the Home Assistant Pi. + +**Client IPs (SPEC §8.5, §8.6).** NPM's request to the server carries `X-Forwarded-For: , +`; the server takes the last entry only when the TCP peer is `KVITT_TRUSTED_PROXY`, so `HA_HOST` +must be NPM's LAN IPv4 address (the server accepts an IP literal or a resolvable name, not a CIDR; `install.sh` +requires the address because the container cannot resolve an mDNS name such as `homeassistant.local`). The +router's forward keeps the internet client's source address, and Docker's published port keeps NPM's (DNAT, no +SNAT, for traffic arriving on the LAN interface), so the per-IP limits of D-032, D-115 and D-119 see real +addresses. The server does not use `X-Forwarded-Proto` or `Host`: links are built from `KVITT_PUBLIC_URL` (E-12). +Anyone on the home LAN can also reach `http://:8080` directly; such requests count against their own +address, and only `HA_HOST` may set `X-Forwarded-For`. ### Versions (resolved 2026-10-18) | Component | Version | Source | |---|---|---| -| Caddy | `caddy:2.11.7@sha256:685ae3d7c34d9b5f8caeddc5d14051db7bbd88ff1f5a2fa0c16bd8ca328fb7af` (multi-arch index; has linux/arm64/v8) | Docker Hub official image; tag `2.11.7` = `2.11` = `latest`, pushed 2026-10-14 | -| Alpine (cert-sync, backup) | `alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6` | Docker Hub official image; `3.24` = `latest` | -| cert-sync packages (Alpine v3.24) | openssh-client-default 10.3_p1-r1, curl 8.22.0-r0, openssl 3.5.9-r0, supercronic 0.2.49-r1, tzdata 2026e-r0 | `apk policy` in the pinned image | +| Nginx Proxy Manager | The owner's existing install on the Home Assistant Pi; the block of step 6 was written against NPM v2.16.0 (2026-09-24) templates | github.com/NginxProxyManager/nginx-proxy-manager, `backend/templates/proxy_host.conf` | +| Alpine (backup) | `alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6` | Docker Hub official image; `3.24` = `latest` | | backup packages (Alpine v3.24) | sqlite 3.53.4, rclone 1.74.1, tar 1.35, tzdata 2026e (M2c-dev), supercronic 0.2.49-r1 | `apk policy` in the pinned image | | Docker Engine on the Pi | Docker's Debian apt repository, arm64 packages (`docker-ce` 5:29.8.2 current): `docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin` | docs.docker.com/engine/install/raspberry-pi-os/: that page is for 32-bit (armhf) only; 64-bit Raspberry Pi OS uses the Debian instructions (docs.docker.com/engine/install/debian/) | -**Why supercronic, not busybox crond (D-104).** Both sidecars run as non-root users (cert-sync uid 1000, backup -uid 10001). Busybox crond started as a non-root user in alpine 3.24.2 never ran a job (tested 2026-10-18: an -every-minute job produced no output in two minutes). supercronic is a cron for containers that runs jobs as the -user it runs as and honours `TZ`; the schedules are unchanged (`0 4 * * *` cert-sync, `30 3 * * *` backup). +**Why supercronic, not busybox crond (D-104).** The backup sidecar runs as a non-root user (uid 10001). Busybox +crond started as a non-root user in alpine 3.24.2 never ran a job (tested 2026-10-18: an every-minute job produced +no output in two minutes). supercronic is a cron for containers that runs jobs as the +user it runs as and honours `TZ`; the schedule is unchanged (`30 3 * * *`). ### Backup uid (D-105, SPEC §10.2 "M2c-pi reconciles the two") `backup` runs as **uid 10001**, the server's uid, on the Pi as on the Mac: `sqlite3 .backup` on a WAL database must open (and write) `kvitt.sqlite-shm`, which the server creates as 10001 with mode 0644, so uid 1000 could not back up the live database without loosening the server's files. Therefore `/opt/kvitt/secrets/rclone.conf` is -mode 0600 **owned by 10001:10001** (not 1000). cert-sync stays uid 1000 with `ha_ssh_key` and `ha_known_hosts` -mode 0600 owned by 1000:1000 (ssh refuses a key not owned by its user or readable by others). rclone writes +mode 0600 **owned by 10001:10001** (not 1000). rclone writes refreshed access tokens back to its config; the mounted file is read-only, so `backup.sh` works on a private copy in `/tmp` each run. `install.sh` gives the named volumes their owners once (`/data`, `/data/apk`, `/backups` to -10001, `/certs` to 1000 with mode 0750, Caddy's `/data` to 10002) with one-off root containers -(`docker compose run --user 0:0 --cap-add CHOWN`, plus `FOWNER` for the `chmod`). +10001) with a one-off root container (`docker compose run --user 0:0 --cap-add CHOWN`). -**Caddy uid (D-117).** Caddy runs as **uid 10002** with the supplementary group **1000** (cert-sync's group): the -certificate files are 0640 `1000:1000` and `/certs` is 0750 `1000:1000`, so cert-sync writes and Caddy only reads. -Read-only root filesystem, `no-new-privileges`, `cap_drop: [ALL]` with `cap_add: [NET_BIND_SERVICE]`, `/config` -and `/tmp` as tmpfs, `caddy_data` writable by 10002. Port 8443 needs no capability, but the official image's -`/usr/bin/caddy` has the file capability `cap_net_bind_service=ep`, and the kernel refuses to exec it when that -capability is outside the bounding set ("exec container process /usr/bin/caddy: Operation not permitted" with -`cap_drop: [ALL]` alone, checked 2026-10-18), so it is the one capability kept. +### What was checked on the Mac -### What was checked on the Mac (2026-10-18) +2026-10-19 (D-120, Nginx Proxy Manager instead of Caddy and cert-sync): -* `shellcheck` on every `infra/**/*.sh`: clean. -* `podman-compose -f infra/pi/compose.yaml --env-file config`: exit 0. -* `cert-sync` and `backup` images built for linux/arm64 with Podman (not pushed). -* End-to-end rehearsal with Podman: a stand-in Home Assistant sshd with the variant A key below, the cert-sync - image as uid 1000 with `StrictHostKeyChecking=yes`, the pinned Caddy with `pi/Caddyfile` in front of a stand-in - server. The first fetch installed both files (0640); `GET /health` through Caddy answered over TLS; a renewed - certificate was installed and Caddy reloaded through its admin API without a restart; an unchanged certificate - was left alone; an older certificate and a key that does not match the certificate were refused with the old pair - kept; TLS 1.1 was refused. **Caddy logged the request headers with `"Authorization":["REDACTED"]` and the token - never appeared in its log** (SPEC §10.5.2). -* Backup rehearsal with a local rclone remote: upload, `lsf` confirmation by name and size, local deletion only - after both files were confirmed (D-002); files kept and `last-success` unchanged when the upload failed; the - leftovers uploaded by the next good run; healthcheck green on a fresh start and after a run, red at 26 h. -* Caddy hardening rehearsal (D-117, 2026-10-18, Podman, pinned Caddy image, `pi/Caddyfile`, a self-signed pair in - a `certs` volume set up as `install.sh` and cert-sync do, a stand-in upstream named `server`): Caddy runs as - `uid=10002 gid=10002 groups=1000,10002`, `NoNewPrivs: 1`, capability sets `0x400` (NET_BIND_SERVICE only), the root - filesystem refuses writes, the healthcheck command succeeds, memory about 12 MB. `GET /i/?from=` with a - bearer header answered 200 with `Content-Security-Policy`, `X-Content-Type-Options: nosniff`, - `X-Frame-Options: DENY` and one `Referrer-Policy: no-referrer` (the upstream sent it too). The access log showed - `"uri":"/i/REDACTED?REDACTED"` and `"/sync/pull?REDACTED"`, `"Authorization":["REDACTED"]`; with the upstream - stopped (502) Caddy's error log carried `/i/REDACTED?REDACTED` too; the code, the name, `kvt_` and the query value - occurred 0 times in either log. A reload through the admin API from another container (`POST /load`, as - cert-sync does) succeeded. TLS 1.1 refused. -* Not checked (needs the real hosts): the Home Assistant add-on's handling of `authorized_keys` options, the - Gitea workflow (not yet run), pulls from `gitea.lan`, the router forward, Drive. +* `shellcheck -S style` on every script under `infra/` and `tools/`: clean. +* `podman-compose -f infra/pi/compose.yaml --env-file config` and the same for `infra/dev/`: exit 0. +* `nginx -t` (nginx:alpine 1.31.7) of the step 6 block inside a minimal server block, and inside a copy of NPM + v2.16.0's `nginx.conf` with the proxy host rendered as NPM's template renders it for the step 6 settings (Block + Common Exploits, Force SSL, HTTP/2, HSTS, the block; NPM's regex `^(?:.*;)?\s*?location\s*?\/\s*?{` finds the + block's `location /`, so NPM leaves its own out): both successful. +* Rehearsal with Podman: the server image built from `server/Dockerfile` at this commit (arm64, hardened as on the + Pi) behind that nginx with a self-signed certificate, `KVITT_TRUSTED_PROXY` = the nginx container's address, + clients on a non-private test subnet (so NPM's `set_real_ip_from` does not apply, as for internet clients). + 21 checks passed: `GET /health` 200 over HTTP/2 with `Content-Security-Policy`, `X-Content-Type-Options: nosniff`, + `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer` and `Strict-Transport-Security` once each and no + `X-Served-By`; the landing page `/i/?from=` 200 with one `Referrer-Policy`; `http://` answered 301 to + `https://`; a 7 MB body refused by nginx (413), a 2 MB body answered by the server (`payload_too_large`); one + client limited after 60 requests a minute while a second client was still served, a forged `X-Forwarded-For` + did not escape the limit, and with `KVITT_TRUSTED_PROXY` set to another address both clients shared one bucket + (so the limits key on the forwarded address); after `/i/`, `//i/` and `/%69/` requests, and a 502 on `/i/` with + the server stopped, the invite code, the name and `/i/` occurred 0 times in nginx's access and error logs, while + other paths were logged with their query strings and a 502 on `/health` was in the error log. +* Not checked (needs the real hosts): NPM's own UI and Let's Encrypt for ``, the router + forward, the Gitea workflow, pulls from `gitea.lan`, Drive. -### Gitea Actions workflow (`.gitea/workflows/build-images.yaml`), not yet run +2026-10-18 (M2c-pi, D-104 to D-106, D-117): backup rehearsal with a local rclone remote: upload, `lsf` +confirmation by name and size, local deletion only after both files were confirmed (D-002); files kept and +`last-success` unchanged when the upload failed; the leftovers uploaded by the next good run; healthcheck green on a +fresh start and after a run, red at 26 h. (The Caddy and cert-sync rehearsals of that day concern the removed stack.) + +### Gitea Actions workflow (`.gitea/workflows/build-images.yaml`) Same setup as the owner's working example: `runs-on: host`, `sh` steps, `apk add git docker-cli docker-cli-buildx`, `docker login gitea.lan` with the repository secret `kvitt_cicd` (created by the owner) and `${{ github.actor }}`, a buildx builder whose `buildkitd.toml` sets `http = true` for `gitea.lan` (created once and reused, so its cache survives), `actions/checkout@v4` with `set-safe-directory` and `fetch-depth: 0`, the version from the newest `v1.0.*` git tag plus one, and the git tag pushed after the build. That version is what `deploy.sh` takes. -The token reaches `docker login` through an environment variable, never as text in the script. `kvitt_cicd` needs -only the `write:package` scope: the tag push uses the credentials `actions/checkout` stored (the job's own token). +The token is interpolated into the login step's `run` line (`printf '%s' "${{ secrets.kvitt_cicd }}" | docker login +...`), the owner's proven form: the first run on the runner (2026-10-19) got an empty token from the env-based form +(`env: REGISTRY_TOKEN: ${{ secrets.kvitt_cicd }}`) on act_runner 0.6.1 and failed with "password is empty"; +act_runner masks the secret in the log. That first run got as far as the login (container, `apk add`, checkout +all worked). `kvitt_cicd` needs only the `write:package` scope: the tag push uses the credentials `actions/checkout` stored (the job's own token). If the "Tag and Push Version" step fails with a permission error, the job token cannot push tags on this Gitea; then add `write:repository` to the token (owner decision below). The step "Print image digests" prints each image's `sha256:` digest, which must match what `deploy.sh` prints on the Pi (D-117). @@ -262,7 +268,7 @@ it fails for memory, add swap on the runner Pi, or build on the Mac with `infra/ ### Mac fallback: `infra/build-images.sh [tag]` -Builds the same three images for linux/arm64 and pushes them when `infra/secrets/registry.env` exists +Builds the same two images for linux/arm64 and pushes them when `infra/secrets/registry.env` exists (`KVITT_REGISTRY=gitea.lan/jacobh2`, optional `KVITT_REGISTRY_USER` and `KVITT_REGISTRY_TOKEN` with write:package); otherwise it builds locally and prints the exact push commands. Tag: the short git SHA by default, plus `latest`. Refuses a dirty tree and does not build while a Gradle build holds the gradle-serial lock. With @@ -294,12 +300,10 @@ Fill these in first (in `infra/pi/.env` on the Mac; the bundle carries the file | Placeholder in `infra/pi/.env` | What | Where to find it | |---|---|---| -| `KVITT_PI_HOST` | LAN address of the Kvitt Pi (give it a static DHCP lease) | ASUS router web UI, LAN, DHCP Server, client list | +| `KVITT_PUBLIC_HOST` | Kvitt's own public host name (D-120), for example a DuckDNS name or a name in your own domain, pointing at the home's public address like `icehack.asuscomm.com` does. Not `icehack.asuscomm.com` itself: that name is NPM's proxy host for Home Assistant, and NPM chooses the proxy host by name, not by port | Your choice; tell the orchestrator the name (it goes into the app and `docs/HANDOFF.md`) | +| `KVITT_PI_HOST` | LAN IPv4 address of the Kvitt Pi (give it a static DHCP lease); also NPM's Forward Hostname / IP | ASUS router web UI, LAN, DHCP Server, client list | | `KVITT_PI_SSH_USER` | Your user on the Kvitt Pi | The user set in Raspberry Pi Imager | -| `HA_HOST` | LAN address of the Home Assistant Pi | Router client list, or Home Assistant, Settings, System, Network | -| `HA_PORT` | SSH port of the SSH add-on (default 22) | Home Assistant, Settings, Add-ons, the SSH add-on, Configuration, Network | -| `HA_USER` | SSH user of the add-on (`root` for "Terminal & SSH"; the configured `username` for "Advanced SSH & Web Terminal"); `root` only with the restricted key of step 3, otherwise a dedicated non-root user | Same add-on, Configuration | -| `HA_CERT_DIR` | Directory holding `fullchain.pem` and `privkey.pem` on the Home Assistant Pi (`/ssl` on Home Assistant OS) | In the add-on's terminal: `ls -l /ssl` | +| `HA_HOST` | LAN IPv4 address of the Home Assistant Pi, where NPM runs (static DHCP lease); the server trusts `X-Forwarded-For` only from it. An address such as `192.168.1.20`, not `homeassistant.local` | Router client list, or Home Assistant, Settings, System, Network | | `KVITT_TAG` | Image tag to run: the newest `v1.0.N` the Gitea workflow pushed (or the SHA from `build-images.sh`) | Gitea, the repository, Packages, `kvitt-server` | | `KVITT_REGISTRY_USER`, `KVITT_REGISTRY_TOKEN` | Only if the Gitea packages are private: a Gitea user and a token with `read:package` | Gitea, Settings, Applications, Generate New Token | | `KVITT_REGISTRY` | Already `gitea.lan/jacobh2` | | @@ -337,43 +341,23 @@ exit Log in again; `docker compose version` prints a version and `docker run --rm hello-world` prints "Hello from Docker!". -### 3. Put the cert-sync key on the Home Assistant SSH add-on - -Home Assistant, Settings, Add-ons, your SSH add-on, Configuration, `authorized_keys`: add **one** new entry, the -restricted key (variant A; replace `/ssl` if your `HA_CERT_DIR` differs): - -``` -restrict,command="tar -C /ssl -cf - fullchain.pem privkey.pem" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMiXj2dU8GU0Xmdj91WxLhVpPOI1zRWvBF5h2MbE4S8V kvitt-cert-sync -``` - -Optionally also limit it to the Kvitt Pi: start the line with `restrict,from="",command=...`. -Save, restart the add-on; its Log tab shows no error about the key. - -**A plain key for `root` (the old variant B) is not acceptable (D-117).** The Home Assistant SSH add-ons log in as -root, and without `restrict,command=` whoever holds this key (a compromised cert-sync container, or a copy of the -key on the Mac or the Pi) gets an unrestricted root shell on the Home Assistant Pi, which also owns ports 80/443 -and the house's automation. If the add-on refuses the restricted line, use a dedicated non-root user instead: a -user with no `sudo` that can only read `HA_CERT_DIR` (set `HA_USER` to it), with the plain key line in that user's -`authorized_keys`. If neither is possible, stop and tell the orchestrator; the fallback is Caddy's internal CA -(SPEC §10.5.3). cert-sync works unchanged with both: it always asks for exactly that `tar` command. Tell the -orchestrator which one the add-on accepted (restricted root key, or the dedicated user), so it is recorded here. -The private key is `/Users/jhag/projects/kvitt/infra/secrets/ha_ssh_key` (git-ignored; never paste it anywhere). - -### 4. Fill the `.env` and stage the secrets, on the Mac +### 3. Fill the `.env` and stage the secrets, on the Mac ```bash cd /Users/jhag/projects/kvitt cp infra/pi/.env.example infra/pi/.env open -e infra/pi/.env # replace every CHANGE_ME (table above), save mkdir -p infra/pi/secrets -cp infra/secrets/fcm-service-account.json infra/secrets/ha_ssh_key infra/secrets/rclone.conf infra/pi/secrets/ +cp infra/secrets/fcm-service-account.json infra/secrets/rclone.conf infra/pi/secrets/ ``` +Only these two secret files go to the Pi. `infra/secrets/ha_ssh_key` and `ha_ssh_key.pub` are not used since +D-120: do not copy them (and no SSH add-on or `authorized_keys` entry on the Home Assistant Pi is needed). `rclone.conf` is already generated from your `rclone authorize "drive"` (remote `[gdrive]`, `type = drive`, `scope = drive`, the token); the Drive folder id is `RCLONE_FOLDER_ID` in `.env`, not in the conf. `infra/pi/.env` and `infra/pi/secrets/` are git-ignored. -### 5. Copy the bundle to the Pi and record the Home Assistant host key +### 4. Copy the bundle to the Pi On the Mac: @@ -382,15 +366,7 @@ scp -r infra/pi @:kvitt-pi rm -r infra/pi/secrets # the originals stay in infra/secrets/ ``` -On the Pi: - -```bash -cd ~/kvitt-pi -ssh-keyscan -p > secrets/ha_known_hosts -wc -l secrets/ha_known_hosts # at least 1 line -``` - -### 6. Install and start +### 5. Install and start On the Pi: @@ -398,20 +374,146 @@ On the Pi: cd ~/kvitt-pi && ./install.sh ``` -It creates `/opt/kvitt/{secrets,import,apk}`, installs the four secret files (`fcm-service-account.json` 0440 -root:10001; `ha_ssh_key`, `ha_known_hosts` 0600 1000:1000; `rclone.conf` 0600 10001:10001), adds `gitea.lan` to -`insecure-registries` in `/etc/docker/daemon.json` (and restarts Docker if that changed), copies `compose.yaml`, -`Caddyfile`, `.env` (0600) and the scripts to `/opt/kvitt`, pulls, starts and waits. Expected end: -`server: healthy`, `cert-sync: healthy`, `caddy: healthy`, a `docker compose ps` table, the line -`GET https://icehack.asuscomm.com:8443/health (via this Pi): {"status":"ok",...}`, then "Kvitt is running.". +It checks the `.env` (`KVITT_PUBLIC_HOST` a host name, `HA_HOST` an IPv4 address), creates +`/opt/kvitt/{secrets,import,apk}`, installs the two secret files (`fcm-service-account.json` 0440 root:10001; +`rclone.conf` 0600 10001:10001), adds `gitea.lan` to `insecure-registries` in `/etc/docker/daemon.json` (and +restarts Docker if that changed), copies `compose.yaml`, `.env` (0600) and the scripts to `/opt/kvitt`, pulls, +starts and waits. If an earlier bundle with Caddy and cert-sync was installed, it also removes its leftovers +(`/opt/kvitt/Caddyfile`, `secrets/ha_ssh_key`, `secrets/ha_known_hosts`, the containers and the volumes +`kvitt_certs` and `kvitt_caddy_data`) and says so. Expected end: `server: healthy`, a `docker compose ps` table with +`server` (ports `0.0.0.0:8080->8080/tcp`) and `backup`, the line `GET http://localhost:8080/health: {"status":"ok",...}`, +then "Not reachable through Nginx Proxy Manager yet" (expected on the first install, before step 6) or +`GET https://:443/health (via NPM at ): {"status":"ok",...}`, then "Kvitt is running.". Then `rm -r ~/kvitt-pi/secrets` (the script reminds you). Running `install.sh` again is safe. Re-running `install.sh` keeps the `KVITT_TAG` that `deploy.sh` set; change the tag only with `deploy.sh `. +### 6. Nginx Proxy Manager on the Home Assistant Pi + +NPM terminates TLS for Kvitt with its own Let's Encrypt certificate for `` and forwards plain +HTTP over the home LAN to the Kvitt Pi's port 8080 (plain HTTP between the two Pis is accepted, D-120). First make +sure `` points at the home's public address (your DNS or DuckDNS page shows the same address as +`icehack.asuscomm.com`). Let's Encrypt's HTTP-01 challenge reaches NPM through external port 80, which already +works: it is how NPM got the certificate for Home Assistant. Serving Kvitt under a path of `icehack.asuscomm.com` +(for example `/kvitt/`) is not supported. + +Open NPM's web UI (Home Assistant, Settings, Add-ons, Nginx Proxy Manager, Open Web UI), then Hosts, Proxy Hosts, +Add Proxy Host. + +Details tab: + +| Field | Value | +|---|---| +| Domain Names | `` | +| Scheme | `http` | +| Forward Hostname / IP | `` (the IPv4 address) | +| Forward Port | `8080` | +| Cache Assets | off | +| Block Common Exploits | on | +| Websockets Support | off | +| Access List | Publicly Accessible | + +Custom Locations tab: leave empty. + +SSL tab: + +| Field | Value | +|---|---| +| SSL Certificate | Request a new Certificate (Let's Encrypt, HTTP challenge: Use DNS Challenge off); Email address: yours; if a Let's Encrypt Terms of Service checkbox is shown, tick it | +| Force SSL | on | +| HTTP/2 Support | on | +| HSTS Enabled | on | +| HSTS Sub-domains | off | +| Trust Upstream Forwarded Proto Headers | off | + +Advanced tab, Custom Nginx Configuration: paste this block exactly (all of it, comments included): + +```nginx +# Kvitt (SPEC §10.5.2, D-120). NPM, Proxy Host , Advanced tab, Custom Nginx Configuration. +# Server body: 1 MiB, PUT /me/picture 5 MiB (SPEC §8.6); the server answers anything larger with its own 413. +client_max_body_size 6m; +# No long-poll in the API (SPEC §14.3); 60 s between two reads from the server. +proxy_read_timeout 60s; +# Response headers (D-117). They live at server level, so no location below may use add_header (nginx would then +# drop all of these in it); NPM's HSTS line (SSL tab) sits at the same level. The server's own copy is hidden, so +# each header appears once (the landing page sends Referrer-Policy itself). +proxy_hide_header Content-Security-Policy; +proxy_hide_header X-Content-Type-Options; +proxy_hide_header X-Frame-Options; +proxy_hide_header Referrer-Policy; +add_header Content-Security-Policy "default-src 'none'; style-src 'unsafe-inline'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'" always; +add_header X-Content-Type-Options nosniff always; +add_header X-Frame-Options DENY always; +add_header Referrer-Policy no-referrer always; +# Replaces NPM's own "location /" (NPM leaves it out when this field has one), whose add_header X-Served-By would +# drop the headers above. $forward_scheme, $server and $port are NPM's Details tab values. +location / { + proxy_set_header Host $host; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; + proxy_pass $forward_scheme://$server:$port$request_uri; +} +# Invite links /i/?from=: neither the code nor the inviter's name may reach a log (SPEC §8.7). NPM's +# access log format writes the whole $request_uri, query string included, so this path is not logged at all, and +# its error log (which would carry the request line on a 502) goes nowhere. +location ^~ /i/ { + access_log off; + error_log /dev/null crit; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; + proxy_pass $forward_scheme://$server:$port$request_uri; +} +``` + +Save. The new row under Proxy Hosts shows the status Online and an SSL entry for Let's Encrypt (yes/no). Then on +the Kvitt Pi: + +```bash +curl -fsS --resolve :443: https:///health +``` + +It prints `{"status":"ok",...}` (no `-k`: this also checks the certificate). `deploy.sh` and `install.sh` run the +same check. + +**Why the block has its own `location /`.** nginx drops every `add_header` inherited from the server level in a +location that has an `add_header` of its own, and NPM's default `location /` has one (`X-Served-By`), which would +silently drop the four security headers (D-117). NPM leaves its default location out when the Custom Nginx +Configuration contains `location /`, so the block brings its own without `add_header`; NPM's HSTS line from the SSL +tab stays at server level and still applies. The block's `proxy_pass` uses NPM's Forward Hostname / IP and Forward +Port, so change those on the Details tab as usual. If a later NPM release changes its proxy host template, check +the headers again (Operations, "Headers through NPM"). + +**What NPM logs (SPEC §8.7).** NPM writes one access log line per request to `/data/logs/proxy-host-_access.log` +inside NPM, in its `proxy` format: time, cache and upstream status, status, method, scheme, host, +`"$request_uri"` (the path **with** its query string), `[Client
]`, length, gzip ratio, +`[Sent-to ]`, user agent and referer. It logs no request headers, so the `Authorization` token is never +logged. Query strings are logged for every path except `/i/`; the other query parameters of the API are `since` and +`limit` (`GET /sync`) and `size` and `v` (pictures), which hold no personal content. `/i/?from=` is not +logged at all: the block turns the access log off for `/i/` and sends its error log to `/dev/null`, so neither the +invite code nor the inviter's name reaches a log. Errors on other paths (for example a 502 while the server is down) +go to `proxy-host-_error.log` with the request line. + +**HSTS.** HSTS tells a browser to use only HTTPS for ``, on every port, for two years +(`max-age=63072000; preload`, NPM's value), and to refuse certificate warnings for it. Because the name is Kvitt's +own, nothing else is affected; but if you ever serve anything else under `` (any port), it must be +HTTPS with a valid certificate too, and turning HSTS off later does not clear browsers that already stored it. The +app itself does not rely on HSTS (release builds allow no cleartext, SPEC §8.5); it protects the landing page in a +browser. + +**Accepted: NPM trusts `X-Real-IP` from the LAN.** NPM's `nginx.conf` takes the client address from `X-Real-IP` when +the connection comes from 10/8, 172.16/12, 192.168/16 (and the CDN ranges it downloads), so a device on the home +LAN can choose its own client address for the rate limits. Internet clients arrive with their public address (the +router's forward keeps the source address), so their `X-Real-IP` is ignored. + ### 7. Router forward -ASUS router web UI, WAN, Virtual Server / Port Forwarding: add a rule, protocol TCP, external port 8443, internal -IP ``, internal port 8443; Apply. Then on the phone with Wi-Fi off, open -`https://icehack.asuscomm.com:8443/health` in Chrome: it shows `"status":"ok"` without a certificate warning (yes/no). +ASUS router web UI, WAN, Virtual Server / Port Forwarding: add a rule: Service Name `Kvitt`, Protocol TCP, External +Port `8443`, Internal Port `443` (NPM's HTTPS port; if the add-on's Network settings map `443/tcp` to another host +port, use that one), Internal IP Address ``; Apply. No rule forwards to the Kvitt Pi (remove one to +`` if you made it for the Caddy version). Then on the phone with Wi-Fi off, open +`https://:8443/health` in Chrome: it shows `"status":"ok"` without a certificate warning (yes/no). ### 8. First backup and bootstrap @@ -422,6 +524,10 @@ On the Pi: /opt/kvitt/bootstrap.sh # first install only: prints the admin invite code, link and expiry ``` +The invite link starts with `https://:8443/i/`. The release app opens such links and connects to +that host only once an Android build carries the new host name (SPEC §8.5; until then the app knows +`icehack.asuscomm.com`): tell the orchestrator before pairing a release build. + ### 9. Deploying a new version After the Gitea workflow pushed `v1.0.N` (or `infra/build-images.sh` printed a tag), on the Pi: @@ -432,51 +538,52 @@ After the Gitea workflow pushed `v1.0.N` (or `infra/build-images.sh` printed a t It backs up first (Drive confirmed) on every deploy, because any tag may carry a migration (SPEC §10.10), sets `KVITT_TAG`, pulls, prints the `sha256:` digest of every image and appends it to `/opt/kvitt/deployed-digests`, -starts, waits for `server: healthy` and prints `/health`. The images come over plain HTTP from `gitea.lan` -(D-001), so compare the printed digests with the workflow's "Print image digests" step for that run; a line -starting `WARNING:` means a tag that was deployed before now points at a different image: do not continue, roll -back and find out why (D-117). If the server does not come up: +starts, waits for `server: healthy` and prints `/health` on port 8080 and through NPM (a failure through NPM is a +WARNING: the deploy succeeded, but Kvitt is not reachable from outside). The images come over plain HTTP from +`gitea.lan` (D-001), so compare the printed digests with the workflow's "Print image digests" step for that run; a +line starting `WARNING:` about a digest means a tag that was deployed before now points at a different image: do not +continue, roll back and find out why (D-117). If the server does not come up: `/opt/kvitt/deploy.sh "$(cat /opt/kvitt/previous-tag)" --skip-backup`; if a migration had partially applied, -restore the pre-deploy backup (`infra/RESTORE.md`). A newer bundle with a changed `compose.yaml` or `Caddyfile` is -installed by copying the bundle again (step 5, without the secrets) and running `./install.sh`. +restore the pre-deploy backup (`infra/RESTORE.md`). A newer bundle with a changed `compose.yaml` is installed by +copying the bundle again (step 4, without the secrets) and running `./install.sh`. -### Fallback: Caddy internal CA (SPEC §10.5.3, not deployed) +### Fallback (SPEC §10.5.3): when NPM is down -Only if cert-sync cannot be made to work. In `/opt/kvitt/Caddyfile` replace the line -`tls /certs/fullchain.pem /certs/privkey.pem` with `tls internal`, then `docker compose restart caddy` in -`/opt/kvitt`. Copy Caddy's root to the Mac once: on the Pi -`docker compose cp caddy:/data/caddy/pki/authorities/local/root.crt /tmp/kvitt_root_ca.crt`, then on the Mac -`scp @:/tmp/kvitt_root_ca.crt android/app/src/release/res/raw/kvitt_root_ca.crt` -(a public certificate, committed by an Android agent) and set `kvitt.tls.pinnedRoot=true` in -`android/gradle.properties` (D-082). The root is valid 10 years. Caddy still waits for cert-sync to be healthy, -which needs the two files in the `certs` volume: keep the last fetched pair there, or remove the `cert-sync` -condition from `depends_on` in `/opt/kvitt/compose.yaml`. +* NPM or the Home Assistant Pi down means Kvitt is unreachable from outside, and also from home over + `https://:8443` (that path goes through NPM too). The server on the Kvitt Pi keeps running and + nothing there needs to change. The phones keep working on their full local copy and sync when NPM is back. +* On the home Wi-Fi only **debug builds** can reach the server directly, at `http://:8080` (AC-2.8 + allows `http://` to private LAN addresses in debug builds only). **Release builds require HTTPS with a publicly + trusted certificate (SPEC §8.5) and cannot use that address: there is no LAN fallback for the release app.** A + debug build is a separate app (`se.hagstedt.kvitt.debug`) with its own pairing and data, not a stand-in for the + release app. +* If NPM cannot renew the Let's Encrypt certificate: fix it in NPM (SSL Certificates, the certificate, Renew Now; + port 80 must reach NPM). The pinned-root switch of the app (`kvitt.tls.pinnedRoot`, D-082) still exists for a + self-made root whose certificate would be uploaded to NPM as a Custom certificate; it is not prepared: tell the + orchestrator. ### Operations (SPEC §10.10), on the Pi in `/opt/kvitt` | Task | Command | |---|---| | Status | `docker compose ps` | -| Logs | `docker compose logs --tail 100 ` | +| Logs | `docker compose logs --tail 100 `; NPM's own log files are inside NPM on the Home Assistant Pi | | Backup now | `/opt/kvitt/backup-now.sh` | | Upgrade | `/opt/kvitt/deploy.sh ` | | Roll back | `/opt/kvitt/deploy.sh "$(cat /opt/kvitt/previous-tag)" --skip-backup` | -| Certificate now | `docker compose exec cert-sync /usr/local/bin/sync.sh` | -| Certificate expiry and issuer | `docker compose exec cert-sync openssl x509 -in /certs/fullchain.pem -noout -enddate -issuer` | +| Health through NPM | `curl -fsS --resolve :443: https:///health` | +| Headers through NPM | `curl -sS -D - -o /dev/null --resolve :443: https:///health` lists `content-security-policy`, `x-content-type-options`, `x-frame-options`, `referrer-policy`, `strict-transport-security` once each and no `x-served-by` | +| Certificate expiry and issuer | `echo \| openssl s_client -connect :443 -servername 2>/dev/null \| openssl x509 -noout -enddate -issuer` (NPM renews it; NPM, SSL Certificates shows the same) | | Extra device invite | `docker compose exec server kvitt-server pair-device --user ` | | Memory | `docker stats --no-stream` (server RSS under 100 MB with both phones syncing) | | Disk | `df -h /` and `docker system df`; SQLite a few MB, pictures under 1 MB, APKs about 10 MB each since R8, 5 kept | -| No token in Caddy's log | `docker compose logs caddy \| grep -c 'kvt_'` prints `0` | -| No invite code or name in Caddy's log | `docker compose logs caddy \| grep '"uri":"/i/' \| grep -vc '/i/REDACTED'` prints `0` | | Image digests deployed | `tail -n 8 /opt/kvitt/deployed-digests` | -| Client IPs are real (once, after the router forward) | open `https://icehack.asuscomm.com:8443/health` on the phone with Wi-Fi off, then `docker compose logs --tail 20 caddy \| grep -o '"remote_ip":"[^"]*"'`: the address is your mobile network's public address, not `172.*` or `10.*`. If every request shows the Docker bridge gateway, all internet clients share one per-IP rate limit (SPEC §8.6): tell the orchestrator | +| Client IPs are real (once, after the router forward) | Open `https://:8443/health` on the phone with Wi-Fi off, then within a minute on the Kvitt Pi `sudo apt install -y conntrack` (once) and `sudo conntrack -L -p tcp --orig-port-dst 8080`: in every line the first `src=` is `` and the second half (from the second `src=`) has `dst=`, no `172.*` there, so the server sees NPM's address and trusts its `X-Forwarded-For`. If the second half shows a `172.*` address instead, every internet client shares one per-IP limit (SPEC §8.6): tell the orchestrator | | Restore | `infra/RESTORE.md` | -**APK downloads (security review L-3).** `GET /apk/files/` is public (the landing page links it) and about -11 MB per response; the only limit is the general 60 requests per minute per IP of SPEC §8.6, so one stranger can -pull several hundred MB a minute and fill the home uplink. A lower per-IP limit for this path is a server change -and a SPEC number that is not decided yet (owner decision below). Until then, if `docker compose logs caddy` -shows one address fetching `/apk/files/` over and over, remove the router forward (step 7) until it stops. +**APK downloads (security review L-3, D-119).** `GET /apk/files/` is public (the landing page links it) and +about 11 MB per response; the server allows 10 per minute per client IP. If one address still fetches +`/apk/files/` over and over (NPM's access log), remove the router forward (step 7) until it stops. ### Owner decisions (open) @@ -485,19 +592,20 @@ shows one address fetching `/apk/files/` over and over, remove the router forwar whole Drive). Should the backups go through an rclone `crypt` remote (two passwords kept in `rclone.conf`, restore through the same remote) and should rclone be re-authorised with the narrower `drive.file` scope? Both change SPEC §8.7 and §10.9. Until you decide, the backup scripts stay as they are. -2. **APK download limit (security review L-3).** May `/apk/files/*` get its own lower per-IP limit (for example a - few downloads per minute per IP)? The number goes into SPEC §8.6 and the server; it is not set here. -3. **cert-sync key (security review M-2, step 3).** Which form did the Home Assistant add-on accept: the restricted - root key, or a dedicated non-root user? -4. **Gitea token scope.** Does the workflow's tag push work with `kvitt_cicd` limited to `write:package`? If not, +2. **Gitea token scope.** Does the workflow's tag push work with `kvitt_cicd` limited to `write:package`? If not, is adding `write:repository` acceptable? +3. **Kvitt's host name (D-120).** Which name is ``? It goes into the release app and + `docs/HANDOFF.md`. ### Troubleshooting -* `cert-sync` unhealthy, so `caddy` does not start: `docker compose logs cert-sync`. "Host key verification - failed": redo `ssh-keyscan` (step 5) and run `./install.sh` again. "Permission denied (publickey)": the add-on - does not have the key (step 3). "did not hold fullchain.pem and privkey.pem": wrong `HA_CERT_DIR`, or `HA_USER` - cannot read `privkey.pem`. +* Step 6 check or `deploy.sh` says /health through NPM failed: "Could not resolve host" or "SSL certificate problem": + the proxy host has no certificate for `` yet (SSL tab) or a different name; a `502 Bad Gateway`: + NPM cannot reach `:8080` (wrong Forward Hostname / IP or Forward Port, or the server is down: + `docker compose ps`); the Home Assistant page instead of `{"status":"ok"}`: Domain Names does not match the name + you requested (NPM fell back to another host). +* "Request a new Certificate" fails in NPM: `` does not point at the home's public address yet, + or external port 80 does not reach NPM. * Pull fails with "http: server gave HTTP response to HTTPS client": `gitea.lan` is missing from `/etc/docker/daemon.json` (`install.sh` adds it; check with `docker info | grep -A3 'Insecure Registries'`). * Pull fails with "unauthorized": the packages are private; set `KVITT_REGISTRY_USER` and `KVITT_REGISTRY_TOKEN` diff --git a/infra/RESTORE.md b/infra/RESTORE.md index 6943877..f7e570b 100644 --- a/infra/RESTORE.md +++ b/infra/RESTORE.md @@ -71,10 +71,11 @@ It prints the new epoch. The phones see it and re-push their data (§14.4.8). ```bash docker compose start server docker compose ps server # healthy within a minute -curl -fsS --resolve icehack.asuscomm.com:8443:127.0.0.1 https://icehack.asuscomm.com:8443/health +curl -fsS http://localhost:8080/health ``` -The last line prints `{"status":"ok",...}`. +The last line prints `{"status":"ok",...}` (the server's own port; the path through Nginx Proxy Manager is +unchanged by a restore). ## 7. The phones