Pi: minimal self-contained compose project (D-125)
The Kvitt Pi runs Kvitt from /home/jacobhagstedt/kvitt: compose.yaml, .env, secrets/ and relative bind mounts ./data, ./backups, ./import. Both containers run as the owner's uid/gid (KVITT_UID, KVITT_GID). No install script, no /opt, no Docker daemon or system changes; operations are plain docker compose commands (README "Pi: what the owner does", RESTORE.md). Removed install.sh, deploy.sh, bootstrap.sh, backup-now.sh, install-apk.sh, lib.sh. .env holds only the keys compose.yaml reads; RCLONE_REMOTE is fixed to gdrive: in compose.yaml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TQd3pEP9UeN2uifbHknQ7B
This commit is contained in:
+130
-141
@@ -2,7 +2,7 @@
|
||||
|
||||
This file covers the **development stack on the owner's Mac** (`infra/dev/`, milestone M2c-dev, D-001), the
|
||||
**Firebase setup** (SPEC §10.7) and the **Pi stack** (`infra/pi/`, milestone M2c-pi, D-100): a self-contained
|
||||
bundle the owner copies to the Kvitt Pi and runs there, behind the owner's Nginx Proxy Manager on the Home Assistant
|
||||
compose project the owner copies to `/home/jacobhagstedt/kvitt` on the Kvitt Pi and runs there (D-125), behind the owner's Nginx Proxy Manager on the Home Assistant
|
||||
Pi (D-120). No agent and no Mac script connects to the Pis; the owner runs every Pi command and every NPM setting
|
||||
from the checklist in "Pi: what the owner does".
|
||||
|
||||
@@ -20,7 +20,7 @@ from the checklist in "Pi: what the owner does".
|
||||
| `dev/lib.sh` | Helpers sourced by the scripts |
|
||||
| `dev/secrets/` | Git-ignored: only `fcm-service-account.json`, copied there by `dev/up.sh` and mounted read-only at `/secrets` |
|
||||
| `backup/Dockerfile`, `backup/backup.sh`, `backup/healthcheck.sh` | Backup image and job of SPEC §10.9 (shared with the Pi stack); Drive upload and D-002 deletion |
|
||||
| `pi/` | **The Pi bundle** (copied to the Kvitt Pi): `compose.yaml`, `.env.example`, `install.sh`, `deploy.sh`, `bootstrap.sh`, `backup-now.sh`, `install-apk.sh`, `lib.sh`; git-ignored `pi/.env`, `pi/secrets/`, `pi/out/` |
|
||||
| `pi/` | **The Pi project** (copied to `~/kvitt` on the Kvitt Pi): `compose.yaml`, `.env.example`; git-ignored `pi/.env`, `pi/secrets/`, `pi/out/` |
|
||||
| `build-images.sh` | Mac fallback for the image build: linux/arm64, push to the Gitea registry |
|
||||
| `publish-apk.sh` | Mac: release APK, mapping and `latest.json` into `pi/out/apk/` for the owner to copy to the Pi |
|
||||
| `RESTORE.md` | Restore from a Drive backup, step by step (SPEC §10.9) |
|
||||
@@ -153,7 +153,7 @@ gcloud iam service-accounts keys list --iam-account kvitt-fcm@kvitt-1337.iam.gse
|
||||
Rotating the key: create a new one with
|
||||
`gcloud iam service-accounts keys create infra/secrets/fcm-service-account.json --iam-account kvitt-fcm@kvitt-1337.iam.gserviceaccount.com --project kvitt-1337`,
|
||||
restart the server, then delete the old key id with `gcloud iam service-accounts keys delete <key id> ...`.
|
||||
Copying the key to the Pi (§10.7 step 7, mode 0440, group 10001) is done by `pi/install.sh` ("Pi: what the owner does", step 5).
|
||||
The key goes to the Pi as `~/kvitt/secrets/fcm-service-account.json` ("Pi: what the owner does", step 2).
|
||||
|
||||
## Pi stack (SPEC §10.1 to §10.10, D-100)
|
||||
|
||||
@@ -163,21 +163,25 @@ Copying the key to the Pi (§10.7 step 7, mode 0440, group 10001) is done by `pi
|
||||
|---|---|---|
|
||||
| Gitea Actions (runner label `host`, a Raspberry Pi) | `.gitea/workflows/build-images.yaml`: builds `kvitt-server` and `kvitt-backup` for linux/arm64, pushes them to `gitea.lan/jacobh2/...` tagged `vMAJOR.MINOR.PATCH` and `latest`, pushes the git tag | Gitea, on a push to `server/**`, `infra/backup/**`, `.gitea/workflows/**`, or by hand (Actions, "Run workflow") |
|
||||
| Owner's Mac | `infra/build-images.sh [tag]` (fallback image build), `infra/publish-apk.sh` (release APK into `infra/pi/out/apk/`) | The owner (or an agent, for the APK build only) |
|
||||
| Kvitt Pi, `/opt/kvitt/` | `compose.yaml`, `.env`, `secrets/`, `import/`, `apk/`, and the scripts `deploy.sh`, `bootstrap.sh`, `backup-now.sh`, `install-apk.sh` (copied there by `install.sh`) | The owner, in their own SSH session |
|
||||
| Home Assistant Pi | Nginx Proxy Manager (NPM): a proxy host for `<KVITT_PUBLIC_HOST>` with its own Let's Encrypt certificate, TLS on 443, plain HTTP to `<kvitt-pi-address>:8080` over the LAN (D-120) | The owner, in NPM's web UI (step 6) |
|
||||
| Router (ASUS) | No new rule: the existing external TCP 443 forward to the Home Assistant Pi (NPM, for Home Assistant) serves Kvitt too, NPM picks the proxy host by name; external 80 already reaches NPM for Let's Encrypt; nothing is forwarded to the Kvitt Pi (D-124) | The owner checks it (step 7) |
|
||||
| Kvitt Pi, `/home/jacobhagstedt/kvitt/` | `compose.yaml`, `.env`, `secrets/`, and the bind-mounted `data/` (SQLite, pictures, `apk/`), `backups/`, `import/`; nothing outside this folder (D-125) | The owner, in their own SSH session, with plain `docker compose` commands |
|
||||
| Home Assistant Pi | Nginx Proxy Manager (NPM): a proxy host for `<KVITT_PUBLIC_HOST>` with its own Let's Encrypt certificate, TLS on 443, plain HTTP to `<kvitt-pi-address>:8080` over the LAN (D-120) | The owner, in NPM's web UI (step 4) |
|
||||
| Router (ASUS) | No new rule: the existing external TCP 443 forward to the Home Assistant Pi (NPM, for Home Assistant) serves Kvitt too, NPM picks the proxy host by name; external 80 already reaches NPM for Let's Encrypt; nothing is forwarded to the Kvitt Pi (D-124) | The owner checks it (step 5) |
|
||||
|
||||
Stack (`pi/compose.yaml`, project name `kvitt`): `server` (publishes `8080:8080` on the Pi's LAN interfaces, the
|
||||
only published port; `KVITT_PUBLIC_URL=https://${KVITT_PUBLIC_HOST}` (default port 443, D-124), `KVITT_TRUSTED_PROXY=${HA_HOST}`) and
|
||||
`backup`; volumes `data`, `backups`; network `kvitt`; every service `restart: unless-stopped`, json-file logs
|
||||
10 MB x 3, `TZ=Europe/Stockholm`. Images are `${KVITT_REGISTRY}/kvitt-server|kvitt-backup:${KVITT_TAG}` with
|
||||
`KVITT_REGISTRY=gitea.lan/jacobh2` (D-104; the names the workflow pushes). Caddy and cert-sync were removed by
|
||||
D-120: TLS terminates in NPM, which already runs on the Home Assistant Pi.
|
||||
`backup`; relative bind mounts `./data:/data`, `./backups:/backups` (backup), `./import:/import:ro` (server), and one
|
||||
secret file each, read-only: `secrets/fcm-service-account.json` for the server, `secrets/rclone.conf` for the backup
|
||||
(D-117: the server never sees the Drive token); both containers run as `${KVITT_UID}:${KVITT_GID}`, the owner's
|
||||
uid and gid, with a read-only root, a 16 MB tmpfs `/tmp`, no capabilities and `no-new-privileges`; every service
|
||||
`restart: unless-stopped`, json-file logs 10 MB x 3, `TZ=Europe/Stockholm`; the compose default network. Images are
|
||||
`${KVITT_REGISTRY}/kvitt-server|kvitt-backup:${KVITT_TAG}` with `KVITT_REGISTRY=gitea.lan/jacobh2` (D-104; the names
|
||||
the workflow pushes). Caddy and cert-sync were removed by D-120: TLS terminates in NPM, which already runs on the Home
|
||||
Assistant Pi. There is no install script and nothing on the Pi outside `~/kvitt` is changed (D-125).
|
||||
|
||||
**Client IPs (SPEC §8.5, §8.6).** NPM's request to the server carries `X-Forwarded-For: <what the client sent>,
|
||||
<client address>`; the server takes the last entry only when the TCP peer is `KVITT_TRUSTED_PROXY`, so `HA_HOST`
|
||||
must be NPM's LAN IPv4 address (the server accepts an IP literal or a resolvable name, not a CIDR; `install.sh`
|
||||
requires the address because the container cannot resolve an mDNS name such as `homeassistant.local`). The
|
||||
must be NPM's LAN IPv4 address (the server accepts an IP literal or a resolvable name, not a CIDR; use the
|
||||
address, because the container cannot resolve an mDNS name such as `homeassistant.local`). The
|
||||
router's forward keeps the internet client's source address, and Docker's published port keeps NPM's (DNAT, no
|
||||
SNAT, for traffic arriving on the LAN interface), so the per-IP limits of D-032, D-115 and D-119 see real
|
||||
addresses. The server does not use `X-Forwarded-Proto` or `Host`: links are built from `KVITT_PUBLIC_URL` (E-12).
|
||||
@@ -188,34 +192,55 @@ address, and only `HA_HOST` may set `X-Forwarded-For`.
|
||||
|
||||
| Component | Version | Source |
|
||||
|---|---|---|
|
||||
| Nginx Proxy Manager | The owner's existing install on the Home Assistant Pi; the block of step 6 was written against NPM v2.16.0 (2026-09-24) templates | github.com/NginxProxyManager/nginx-proxy-manager, `backend/templates/proxy_host.conf` |
|
||||
| Nginx Proxy Manager | The owner's existing install on the Home Assistant Pi; the block of step 4 was written against NPM v2.16.0 (2026-09-24) templates | github.com/NginxProxyManager/nginx-proxy-manager, `backend/templates/proxy_host.conf` |
|
||||
| Alpine (backup) | `alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6` | Docker Hub official image; `3.24` = `latest` |
|
||||
| backup packages (Alpine v3.24) | sqlite 3.53.4, rclone 1.74.1, tar 1.35, tzdata 2026e (M2c-dev), supercronic 0.2.49-r1 | `apk policy` in the pinned image |
|
||||
| Docker Engine on the Pi | Docker's Debian apt repository, arm64 packages (`docker-ce` 5:29.8.2 current): `docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin` | docs.docker.com/engine/install/raspberry-pi-os/: that page is for 32-bit (armhf) only; 64-bit Raspberry Pi OS uses the Debian instructions (docs.docker.com/engine/install/debian/) |
|
||||
|
||||
**Why supercronic, not busybox crond (D-104).** The backup sidecar runs as a non-root user (uid 10001). Busybox
|
||||
**Why supercronic, not busybox crond (D-104).** The backup sidecar runs as a non-root user (the owner's uid on the Pi, D-125). Busybox
|
||||
crond started as a non-root user in alpine 3.24.2 never ran a job (tested 2026-10-18: an every-minute job produced
|
||||
no output in two minutes). supercronic is a cron for containers that runs jobs as the
|
||||
user it runs as and honours `TZ`; the schedule is unchanged (`30 3 * * *`).
|
||||
|
||||
### Backup uid (D-105, SPEC §10.2 "M2c-pi reconciles the two")
|
||||
### Container uid (D-105, D-125)
|
||||
|
||||
`backup` runs as **uid 10001**, the server's uid, on the Pi as on the Mac: `sqlite3 .backup` on a WAL database
|
||||
must open (and write) `kvitt.sqlite-shm`, which the server creates as 10001 with mode 0644, so uid 1000 could not
|
||||
back up the live database without loosening the server's files. Therefore `/opt/kvitt/secrets/rclone.conf` is
|
||||
mode 0600 **owned by 10001:10001** (not 1000). rclone writes
|
||||
`server` and `backup` run as the **same uid**, because `sqlite3 .backup` on a WAL database must open (and write)
|
||||
`kvitt.sqlite-shm`, which the server creates with mode 0644 (D-105). On the Pi that uid is the owner's own
|
||||
(`KVITT_UID`, `KVITT_GID` in `.env`, `id -u` / `id -g`), so everything in `data/` and `backups/` belongs to the
|
||||
owner, the secret files can simply be the owner's with mode 0600, and a restore works with plain `mv` and `cp`
|
||||
(D-125). The images need no particular uid: the server image is distroless with `USER 10001` only as its default
|
||||
(compose's `user:` overrides it; the binary needs no account, no home and nothing writable outside `/data`), and the
|
||||
backup image's scheduler supercronic runs as any uid. The owner's uid has no account inside either image; the
|
||||
backup container gets `HOME=/tmp` for rclone. Folders the containers write to must exist before the first start
|
||||
(`mkdir -p data/apk backups import`): a bind-mount source Docker creates itself belongs to root. rclone writes
|
||||
refreshed access tokens back to its config; the mounted file is read-only, so `backup.sh` works on a private copy
|
||||
in `/tmp` each run. `install.sh` gives the named volumes their owners once (`/data`, `/data/apk`, `/backups` to
|
||||
10001) with a one-off root container (`docker compose run --user 0:0 --cap-add CHOWN`).
|
||||
in the tmpfs `/tmp` each run. The development stack on the Mac keeps uid 10001 (`infra/dev/`).
|
||||
|
||||
### What was checked on the Mac
|
||||
|
||||
2026-10-20 (D-125, self-contained compose project):
|
||||
|
||||
* `podman-compose -f infra/pi/compose.yaml --env-file <filled sample> config`: exit 0.
|
||||
* Rehearsal with Podman in a scratch copy of the project folder, `KVITT_UID`/`KVITT_GID` = the Mac user's 501:20
|
||||
(not 10001), images `kvitt-server` from `server/Dockerfile` and `kvitt-backup` from `infra/backup/Dockerfile`:
|
||||
both containers healthy with read-only roots; `GET /health` 200 with `db_ok: true`; `kvitt.sqlite`, its `-wal`
|
||||
and `-shm` and `data/apk` owned by 501 on the host; `kvitt-server bootstrap` through `exec` printed a code, a
|
||||
`https://<KVITT_PUBLIC_HOST>/i/` link and an expiry; an APK and `latest.json` copied into `data/apk/` were served
|
||||
at once (`/apk/latest`, `/apk/files/kvitt-<n>.apk` 200 `application/vnd.android.package-archive`); `/backup.sh`
|
||||
through `exec`: with `RCLONE_FOLDER_ID` empty it kept both files and wrote `last-success`; with a local rclone
|
||||
remote under `/tmp` it uploaded, confirmed and deleted the local copies; with the remote on the read-only root it
|
||||
failed, kept the files and left `last-success` unchanged; the archives belong to 501. The restore steps of
|
||||
`RESTORE.md` (stop, `gunzip`, `integrity_check` `ok`, `mv` aside, `cp`, `tar -x`, `run --rm server
|
||||
rotate-epoch`, start) ended with `/health` 200. Podman-compose ignores `create_host_path: false` (it created a
|
||||
directory for a missing secret file); Docker Compose honours it and refuses to start instead, not checked here.
|
||||
* Not checked (needs the Pi): Docker Engine and Docker Compose themselves, pulls from `gitea.lan`, Drive.
|
||||
|
||||
2026-10-19 (D-120, Nginx Proxy Manager instead of Caddy and cert-sync):
|
||||
|
||||
* `shellcheck -S style` on every script under `infra/` and `tools/`: clean.
|
||||
* `podman-compose -f infra/pi/compose.yaml --env-file <filled sample> config` and the same for `infra/dev/`: exit 0.
|
||||
* `nginx -t` (nginx:alpine 1.31.7) of the step 6 block inside a minimal server block, and inside a copy of NPM
|
||||
v2.16.0's `nginx.conf` with the proxy host rendered as NPM's template renders it for the step 6 settings (Block
|
||||
* `nginx -t` (nginx:alpine 1.31.7) of the step 4 block inside a minimal server block, and inside a copy of NPM
|
||||
v2.16.0's `nginx.conf` with the proxy host rendered as NPM's template renders it for the step 4 settings (Block
|
||||
Common Exploits, Force SSL, HTTP/2, HSTS, the block; NPM's regex `^(?:.*;)?\s*?location\s*?\/\s*?{` finds the
|
||||
block's `location /`, so NPM leaves its own out): both successful.
|
||||
* Rehearsal with Podman: the server image built from `server/Dockerfile` at this commit (arm64, hardened as on the
|
||||
@@ -245,7 +270,7 @@ Same setup as the owner's working example: `runs-on: host`, `sh` steps, `apk add
|
||||
Gitea gives a repository its owner's secrets) and `${{ github.actor }}`, a buildx builder whose `buildkitd.toml`
|
||||
sets `http = true` for `gitea.lan` (created once and reused, so its cache survives), `actions/checkout@v4` with
|
||||
`set-safe-directory` and `fetch-depth: 0`, the version from the newest `v1.0.*` git tag plus one, and the git tag
|
||||
pushed after the build. That version is what `deploy.sh` takes.
|
||||
pushed after the build. That version is the `KVITT_TAG` the Pi's `.env` takes.
|
||||
The checkout is a cone-mode sparse checkout of `server` and `infra/backup` only, the two build contexts (neither
|
||||
Dockerfile copies anything outside its context), so the action fetches with `--filter=blob:none` and writes only
|
||||
those files instead of all 2118 (52 MB of Android screenshot goldens among them; about 95 s on the first runs).
|
||||
@@ -258,7 +283,7 @@ act_runner masks the secret in the log. That first run got as far as the login (
|
||||
all worked). `kvitt_cicd` needs only the `write:package` scope: the tag push uses the credentials `actions/checkout` stored (the job's own token).
|
||||
If the "Tag and Push Version" step fails with a permission error, the job token cannot push tags on this Gitea;
|
||||
then add `write:repository` to the token (owner decision below). The step "Print image digests" prints each
|
||||
image's `sha256:` digest, which must match what `deploy.sh` prints on the Pi (D-117).
|
||||
image's `sha256:` digest, which must match the digest the Pi pulled (Operations, "Image digests"; D-117).
|
||||
|
||||
The builder's `buildkitd.toml` is read only when the builder is created. After changing the registry setting in the
|
||||
workflow, remove the builder once on the runner Pi: `docker buildx rm local-http-builder` (this also drops its
|
||||
@@ -291,114 +316,76 @@ tags of the Gitea workflow (`v1.0.N`) are separate and say nothing about the APK
|
||||
builds `:app:assembleRelease` through `tools/gradle-serial.sh` (JDK 25 on `PATH`), refuses an unsigned APK, and
|
||||
writes into `infra/pi/out/apk/`: `kvitt-<version_code>.apk`, `kvitt-<version_code>.mapping.txt` (the R8 mapping,
|
||||
for reading stack traces; never served) and `latest.json` (temp file, then rename); it keeps the newest 5 APKs
|
||||
with their mappings. Then the owner, on the Mac:
|
||||
with their mappings. Then the owner, on the Mac, copies the APK first and `latest.json` last into the server's
|
||||
`data/apk/` (the server reads `latest.json` on every `GET /apk/latest`, so it must never name a file that is not
|
||||
there yet):
|
||||
|
||||
```bash
|
||||
cd /Users/jhag/projects/kvitt
|
||||
scp infra/pi/out/apk/* <user>@<kvitt-pi-address>:/opt/kvitt/apk/
|
||||
ssh <user>@<kvitt-pi-address> /opt/kvitt/install-apk.sh
|
||||
scp infra/pi/out/apk/kvitt-<version_code>.apk jacobhagstedt@<kvitt-pi-address>:kvitt/data/apk/
|
||||
scp infra/pi/out/apk/latest.json jacobhagstedt@<kvitt-pi-address>:kvitt/data/apk/
|
||||
curl -s http://<kvitt-pi-address>:8080/apk/latest # shows the new version_code
|
||||
```
|
||||
|
||||
`install-apk.sh` checks the SHA-256 against `latest.json`, copies the APKs into the `data` volume (`/data/apk/`)
|
||||
and `latest.json` last with a rename, keeps the newest 5 there and prints `GET /apk/latest`.
|
||||
The mapping files stay on the Mac. Older APKs on the Pi are removed by hand when wanted
|
||||
(`ssh jacobhagstedt@<kvitt-pi-address> ls kvitt/data/apk`, then `rm` the old `kvitt-<n>.apk` files; about 10 MB each).
|
||||
|
||||
## Pi: what the owner does
|
||||
|
||||
Fill these in first (in `infra/pi/.env` on the Mac; the bundle carries the file to the Pi):
|
||||
Kvitt runs from `/home/jacobhagstedt/kvitt` on the Kvitt Pi (D-125): `compose.yaml`, `.env`, `secrets/` and the
|
||||
folders `data/`, `backups/`, `import/`. Nothing outside that folder is installed or changed. `<kvitt-pi-address>` is
|
||||
the Kvitt Pi's LAN IPv4 address (static DHCP lease; ASUS router web UI, LAN, DHCP Server, client list), also NPM's
|
||||
Forward Hostname / IP (step 4).
|
||||
|
||||
| Placeholder in `infra/pi/.env` | What | Where to find it |
|
||||
|---|---|---|
|
||||
| `KVITT_PUBLIC_HOST` | Kvitt's own public host name (D-120), for example a DuckDNS name or a name in your own domain, pointing at the home's public address like `icehack.asuscomm.com` does. Not `icehack.asuscomm.com` itself: that name is NPM's proxy host for Home Assistant, and NPM chooses the proxy host by name, not by port | Your choice; tell the orchestrator the name (it goes into the app and `docs/HANDOFF.md`) |
|
||||
| `HA_HOST` | LAN IPv4 address of the Home Assistant Pi, where NPM runs (static DHCP lease); the server trusts `X-Forwarded-For` only from it. An address such as `192.168.1.20`, not `homeassistant.local` | Router client list, or Home Assistant, Settings, System, Network |
|
||||
| `KVITT_TAG` | Image tag to run: the newest `v1.0.N` the Gitea workflow pushed (or the SHA from `build-images.sh`) | Gitea, the repository, Packages, `kvitt-server` |
|
||||
| `KVITT_REGISTRY_USER`, `KVITT_REGISTRY_TOKEN` | Only if the Gitea packages are private: a Gitea user and a token with `read:package` | Gitea, Settings, Applications, Generate New Token |
|
||||
| `KVITT_REGISTRY` | Already `gitea.lan/jacobh2` | |
|
||||
| `RCLONE_REMOTE`, `RCLONE_FOLDER_ID` | Already `gdrive:` and `1jDp1dGE5R3H2DBcXCqIPN1iWzQnPkdlV` | |
|
||||
### 1. Prerequisites (already on the Pi)
|
||||
|
||||
Not in `.env` (nothing on the Pi reads them; dropped from `.env` on 2026-10-20): in this README,
|
||||
`<kvitt-pi-address>` is the Kvitt Pi's LAN IPv4 address (give it a static DHCP lease; ASUS router web UI, LAN, DHCP
|
||||
Server, client list), also NPM's Forward Hostname / IP (step 6), and `<user>` is your user on the Kvitt Pi (the user
|
||||
set in Raspberry Pi Imager).
|
||||
* Docker Engine with the compose plugin, usable by `jacobhagstedt` without sudo: `docker compose version` prints a
|
||||
version.
|
||||
* Registry trust: the images come over plain HTTP from `gitea.lan`. If `docker compose pull` says "http: server
|
||||
gave HTTP response to HTTPS client", add `gitea.lan` to `insecure-registries` in `/etc/docker/daemon.json` (the
|
||||
Pi already has it for another project; Kvitt never edits it).
|
||||
* If `docker compose pull` says "unauthorized" (private packages): once, `docker login gitea.lan` with your Gitea
|
||||
user and a token with `read:package` (Gitea, Settings, Applications, Generate New Token).
|
||||
|
||||
### 1. Provision the Kvitt Pi
|
||||
|
||||
Raspberry Pi Imager on the Mac: Raspberry Pi 3, **Raspberry Pi OS Lite (64-bit)**; in its settings set the host
|
||||
name, your user and password, the network, time zone Europe/Stockholm, and enable SSH with your public key. Boot
|
||||
it, give it a static DHCP lease in the router, then from the Mac: `ssh <user>@<kvitt-pi-address>`.
|
||||
|
||||
### 2. Install Docker Engine on the Pi (docs.docker.com, Debian arm64 repository)
|
||||
|
||||
On the Pi:
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt install -y ca-certificates curl python3
|
||||
sudo install -m 0755 -d /etc/apt/keyrings
|
||||
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
|
||||
sudo chmod a+r /etc/apt/keyrings/docker.asc
|
||||
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
|
||||
Types: deb
|
||||
URIs: https://download.docker.com/linux/debian
|
||||
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
|
||||
Components: stable
|
||||
Architectures: $(dpkg --print-architecture)
|
||||
Signed-By: /etc/apt/keyrings/docker.asc
|
||||
EOF
|
||||
sudo apt update
|
||||
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
sudo usermod -aG docker "$USER"
|
||||
exit
|
||||
```
|
||||
|
||||
Log in again; `docker compose version` prints a version and `docker run --rm hello-world` prints
|
||||
"Hello from Docker!".
|
||||
|
||||
### 3. Fill the `.env` and stage the secrets, on the Mac
|
||||
### 2. On the Mac: `.env`, secrets, copy
|
||||
|
||||
```bash
|
||||
cd /Users/jhag/projects/kvitt
|
||||
cp infra/pi/.env.example infra/pi/.env
|
||||
open -e infra/pi/.env # replace every CHANGE_ME (table above), save
|
||||
open -e infra/pi/.env # set HA_HOST; KVITT_UID/KVITT_GID = `id -u` / `id -g` on the Pi (1000 for the first user)
|
||||
mkdir -p infra/pi/secrets
|
||||
cp infra/secrets/fcm-service-account.json infra/secrets/rclone.conf infra/pi/secrets/
|
||||
```
|
||||
|
||||
Only these two secret files go to the Pi. `infra/secrets/ha_ssh_key` and `ha_ssh_key.pub` are not used since
|
||||
D-120: do not copy them (and no SSH add-on or `authorized_keys` entry on the Home Assistant Pi is needed).
|
||||
`rclone.conf` is already generated from your `rclone authorize "drive"` (remote `[gdrive]`, `type = drive`,
|
||||
`scope = drive`, the token); the Drive folder id is `RCLONE_FOLDER_ID` in `.env`, not in the conf.
|
||||
`infra/pi/.env` and `infra/pi/secrets/` are git-ignored.
|
||||
|
||||
### 4. Copy the bundle to the Pi
|
||||
|
||||
On the Mac:
|
||||
|
||||
```bash
|
||||
scp -r infra/pi <user>@<kvitt-pi-address>:kvitt-pi
|
||||
scp -r infra/pi/compose.yaml infra/pi/.env infra/pi/secrets jacobhagstedt@<kvitt-pi-address>:kvitt/
|
||||
rm -r infra/pi/secrets # the originals stay in infra/secrets/
|
||||
```
|
||||
|
||||
### 5. Install and start
|
||||
| Key in `infra/pi/.env` | Value |
|
||||
|---|---|
|
||||
| `HA_HOST` | LAN IPv4 address of the Home Assistant Pi (NPM), for example `192.168.1.20`, not `homeassistant.local` (router client list, or Home Assistant, Settings, System, Network) |
|
||||
| `KVITT_UID`, `KVITT_GID` | `id -u` and `id -g` on the Kvitt Pi as `jacobhagstedt` (default 1000, 1000) |
|
||||
| `KVITT_TAG` | `v1.0.0`, or the newest `v1.0.N` the Gitea workflow pushed (Gitea, the repository, Packages, `kvitt-server`) |
|
||||
| `KVITT_PUBLIC_HOST`, `KVITT_REGISTRY`, `RCLONE_FOLDER_ID` | Already `kvitt.suorra.duckdns.org`, `gitea.lan/jacobh2`, `1jDp1dGE5R3H2DBcXCqIPN1iWzQnPkdlV` |
|
||||
|
||||
On the Pi:
|
||||
`rclone.conf` is generated from your `rclone authorize "drive"` (remote `[gdrive]`, `scope = drive`); the Drive
|
||||
folder id is `RCLONE_FOLDER_ID` in `.env`. `infra/pi/.env` and `infra/pi/secrets/` are git-ignored. To change
|
||||
`compose.yaml` or `.env` later, copy just that file again the same way.
|
||||
|
||||
### 3. On the Pi: start
|
||||
|
||||
```bash
|
||||
cd ~/kvitt-pi && ./install.sh
|
||||
cd ~/kvitt
|
||||
chmod 700 secrets && chmod 600 secrets/*
|
||||
mkdir -p data/apk backups import
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
docker compose ps
|
||||
curl -s localhost:8080/health
|
||||
```
|
||||
|
||||
It checks the `.env` (`KVITT_PUBLIC_HOST` a host name, `HA_HOST` an IPv4 address), creates
|
||||
`/opt/kvitt/{secrets,import,apk}`, installs the two secret files (`fcm-service-account.json` 0440 root:10001;
|
||||
`rclone.conf` 0600 10001:10001), adds `gitea.lan` to `insecure-registries` in `/etc/docker/daemon.json` (and
|
||||
restarts Docker if that changed), copies `compose.yaml`, `.env` (0600) and the scripts to `/opt/kvitt`, pulls,
|
||||
starts and waits. If an earlier bundle with Caddy and cert-sync was installed, it also removes its leftovers
|
||||
(`/opt/kvitt/Caddyfile`, `secrets/ha_ssh_key`, `secrets/ha_known_hosts`, the containers and the volumes
|
||||
`kvitt_certs` and `kvitt_caddy_data`) and says so. Expected end: `server: healthy`, a `docker compose ps` table with
|
||||
`server` (ports `0.0.0.0:8080->8080/tcp`) and `backup`, the line `GET http://localhost:8080/health: {"status":"ok",...}`,
|
||||
then "Not reachable through Nginx Proxy Manager yet" (expected on the first install, before step 6) or
|
||||
`GET https://<KVITT_PUBLIC_HOST>/health (via NPM at <HA_HOST>:443): {"status":"ok",...}`, then "Kvitt is running.".
|
||||
Then `rm -r ~/kvitt-pi/secrets` (the script reminds you). Running `install.sh` again is safe. Re-running
|
||||
`install.sh` keeps the `KVITT_TAG` that `deploy.sh` set; change the tag only with `deploy.sh <tag>`.
|
||||
Expected: `docker compose ps` lists `server` (ports `0.0.0.0:8080->8080/tcp`) and `backup`, both `(healthy)` within
|
||||
a minute; `curl` prints `{"status":"ok",...,"db_ok":true}`.
|
||||
|
||||
### 6. Nginx Proxy Manager on the Home Assistant Pi
|
||||
### 4. Nginx Proxy Manager on the Home Assistant Pi (already set up)
|
||||
|
||||
NPM terminates TLS for Kvitt with its own Let's Encrypt certificate for `<KVITT_PUBLIC_HOST>` and forwards plain
|
||||
HTTP over the home LAN to the Kvitt Pi's port 8080 (plain HTTP between the two Pis is accepted, D-120). First make
|
||||
@@ -486,8 +473,7 @@ the Kvitt Pi:
|
||||
curl -fsS --resolve <KVITT_PUBLIC_HOST>:443:<HA_HOST> https://<KVITT_PUBLIC_HOST>/health
|
||||
```
|
||||
|
||||
It prints `{"status":"ok",...}` (no `-k`: this also checks the certificate). `deploy.sh` and `install.sh` run the
|
||||
same check.
|
||||
It prints `{"status":"ok",...}` (no `-k`: this also checks the certificate).
|
||||
|
||||
**Why the block has its own `location /`.** nginx drops every `add_header` inherited from the server level in a
|
||||
location that has an `add_header` of its own, and NPM's default `location /` has one (`X-Served-By`), which would
|
||||
@@ -519,7 +505,7 @@ the connection comes from 10/8, 172.16/12, 192.168/16 (and the CDN ranges it dow
|
||||
LAN can choose its own client address for the rate limits. Internet clients arrive with their public address (the
|
||||
router's forward keeps the source address), so their `X-Real-IP` is ignored.
|
||||
|
||||
### 7. Router: no new rule
|
||||
### 5. Router: no new rule
|
||||
|
||||
No new rule (D-124): external TCP 443 already reaches the Home Assistant Pi (NPM) for Home Assistant, and NPM picks
|
||||
Kvitt's proxy host by name on the same port. Verify it exists: ASUS router web UI, WAN, Virtual Server / Port
|
||||
@@ -528,38 +514,38 @@ forwarded to the Kvitt Pi: remove a rule to `<kvitt-pi-address>` if you made one
|
||||
External Port `8443` if you made one for the earlier NPM version (D-120). Then on the phone with Wi-Fi off, open
|
||||
`https://<KVITT_PUBLIC_HOST>/health` in Chrome: it shows `"status":"ok"` without a certificate warning (yes/no).
|
||||
|
||||
### 8. First backup and bootstrap
|
||||
### 6. From outside, first backup, bootstrap
|
||||
|
||||
On the Pi:
|
||||
On the phone with Wi-Fi off (or from the Mac), `https://kvitt.suorra.duckdns.org/health` shows `"status":"ok"`
|
||||
(yes/no). Then on the Pi:
|
||||
|
||||
```bash
|
||||
/opt/kvitt/backup-now.sh # ends with "last-success: <stamp>" and the newest kvitt-*.sqlite.gz and pictures-*.tar.gz on Drive
|
||||
/opt/kvitt/bootstrap.sh # first install only: prints the admin invite code, link and expiry
|
||||
cd ~/kvitt
|
||||
docker compose exec backup /backup.sh # ends with "backup: ok <stamp>" after "uploaded and confirmed on Drive" twice
|
||||
docker compose exec server kvitt-server bootstrap # first install only: prints the admin invite code, link and expiry
|
||||
```
|
||||
|
||||
The invite link starts with `https://<KVITT_PUBLIC_HOST>/i/` (no port). The release app opens such links and connects to
|
||||
that host only once its Android build carries the host name (`kvitt.publicHost`, D-121) and the default port 443
|
||||
(D-124; builds made for port 8443 do not match a link without a port): tell the orchestrator before pairing a
|
||||
release build.
|
||||
release build. The partner's invite, once the admin is paired: `docker compose exec server kvitt-server invite-partner`.
|
||||
|
||||
### 9. Deploying a new version
|
||||
### 7. Deploying a new version
|
||||
|
||||
After the Gitea workflow pushed `v1.0.N` (or `infra/build-images.sh` printed a tag), on the Pi:
|
||||
|
||||
```bash
|
||||
/opt/kvitt/deploy.sh v1.0.N
|
||||
cd ~/kvitt
|
||||
docker compose exec backup /backup.sh # backup first, every deploy: any tag may carry a migration (SPEC §10.10)
|
||||
sed -i 's/^KVITT_TAG=.*/KVITT_TAG=v1.0.N/' .env # or edit .env; note the previous tag for a rollback
|
||||
docker compose pull && docker compose up -d
|
||||
docker compose ps && curl -s localhost:8080/health
|
||||
```
|
||||
|
||||
It backs up first (Drive confirmed) on every deploy, because any tag may carry a migration (SPEC §10.10), sets
|
||||
`KVITT_TAG`, pulls, prints the `sha256:` digest of every image and appends it to `/opt/kvitt/deployed-digests`,
|
||||
starts, waits for `server: healthy` and prints `/health` on port 8080 and through NPM (a failure through NPM is a
|
||||
WARNING: the deploy succeeded, but Kvitt is not reachable from outside). The images come over plain HTTP from
|
||||
`gitea.lan` (D-001), so compare the printed digests with the workflow's "Print image digests" step for that run; a
|
||||
line starting `WARNING:` about a digest means a tag that was deployed before now points at a different image: do not
|
||||
continue, roll back and find out why (D-117). If the server does not come up:
|
||||
`/opt/kvitt/deploy.sh "$(cat /opt/kvitt/previous-tag)" --skip-backup`; if a migration had partially applied,
|
||||
restore the pre-deploy backup (`infra/RESTORE.md`). A newer bundle with a changed `compose.yaml` is installed by
|
||||
copying the bundle again (step 4, without the secrets) and running `./install.sh`.
|
||||
Continue only when the backup ended with `backup: ok`. The images come over plain HTTP from `gitea.lan` (D-001), so
|
||||
compare the pulled digests (Operations, "Image digests") with the workflow's "Print image digests" step for that
|
||||
run (D-117). Rollback: set the previous tag in `.env` the same way and run `docker compose pull && docker compose
|
||||
up -d`; if a migration had partially applied, restore the pre-deploy backup (`infra/RESTORE.md`).
|
||||
|
||||
### Fallback (SPEC §10.5.3): when NPM is down
|
||||
|
||||
@@ -576,23 +562,24 @@ copying the bundle again (step 4, without the secrets) and running `./install.sh
|
||||
self-made root whose certificate would be uploaded to NPM as a Custom certificate; it is not prepared: tell the
|
||||
orchestrator.
|
||||
|
||||
### Operations (SPEC §10.10), on the Pi in `/opt/kvitt`
|
||||
### Operations (SPEC §10.10), on the Pi in `~/kvitt`
|
||||
|
||||
| Task | Command |
|
||||
|---|---|
|
||||
| Status | `docker compose ps` |
|
||||
| Logs | `docker compose logs --tail 100 <server or backup>`; NPM's own log files are inside NPM on the Home Assistant Pi |
|
||||
| Backup now | `/opt/kvitt/backup-now.sh` |
|
||||
| Upgrade | `/opt/kvitt/deploy.sh <tag>` |
|
||||
| Roll back | `/opt/kvitt/deploy.sh "$(cat /opt/kvitt/previous-tag)" --skip-backup` |
|
||||
| Backup now | `docker compose exec backup /backup.sh` (ends with `backup: ok <stamp>`) |
|
||||
| Upgrade | Step 7: backup, set `KVITT_TAG` in `.env`, `docker compose pull && docker compose up -d` |
|
||||
| Roll back | Set the previous `KVITT_TAG` in `.env`, `docker compose pull && docker compose up -d` |
|
||||
| Health through NPM | `curl -fsS --resolve <KVITT_PUBLIC_HOST>:443:<HA_HOST> https://<KVITT_PUBLIC_HOST>/health` |
|
||||
| Headers through NPM | `curl -sS -D - -o /dev/null --resolve <KVITT_PUBLIC_HOST>:443:<HA_HOST> https://<KVITT_PUBLIC_HOST>/health` lists `content-security-policy`, `x-content-type-options`, `x-frame-options`, `referrer-policy`, `strict-transport-security` once each and no `x-served-by` |
|
||||
| Certificate expiry and issuer | `echo \| openssl s_client -connect <HA_HOST>:443 -servername <KVITT_PUBLIC_HOST> 2>/dev/null \| openssl x509 -noout -enddate -issuer` (NPM renews it; NPM, SSL Certificates shows the same) |
|
||||
| Partner invite | `docker compose exec server kvitt-server invite-partner` |
|
||||
| Extra device invite | `docker compose exec server kvitt-server pair-device --user <user_id>` |
|
||||
| Memory | `docker stats --no-stream` (server RSS under 100 MB with both phones syncing) |
|
||||
| Disk | `df -h /` and `docker system df`; SQLite a few MB, pictures under 1 MB, APKs about 10 MB each since R8, 5 kept |
|
||||
| Image digests deployed | `tail -n 8 /opt/kvitt/deployed-digests` |
|
||||
| Client IPs are real (once, after step 7) | Open `https://<KVITT_PUBLIC_HOST>/health` on the phone with Wi-Fi off, then within a minute on the Kvitt Pi `sudo apt install -y conntrack` (once) and `sudo conntrack -L -p tcp --orig-port-dst 8080`: in every line the first `src=` is `<HA_HOST>` and the second half (from the second `src=`) has `dst=<HA_HOST>`, no `172.*` there, so the server sees NPM's address and trusts its `X-Forwarded-For`. If the second half shows a `172.*` address instead, every internet client shares one per-IP limit (SPEC §8.6): tell the orchestrator |
|
||||
| Disk | `df -h ~` and `du -sh ~/kvitt/*`; SQLite a few MB, pictures under 1 MB, APKs about 10 MB each since R8, 5 kept |
|
||||
| Image digests | `docker compose images` names the images; `docker image inspect --format '{{.RepoDigests}}' gitea.lan/jacobh2/kvitt-server:<tag>` (and `kvitt-backup`) prints the `sha256:` digest to compare with the workflow's |
|
||||
| Client IPs are real (once, after step 5) | Open `https://<KVITT_PUBLIC_HOST>/health` on the phone with Wi-Fi off, then within a minute on the Kvitt Pi `sudo apt install -y conntrack` (once) and `sudo conntrack -L -p tcp --orig-port-dst 8080`: in every line the first `src=` is `<HA_HOST>` and the second half (from the second `src=`) has `dst=<HA_HOST>`, no `172.*` there, so the server sees NPM's address and trusts its `X-Forwarded-For`. If the second half shows a `172.*` address instead, every internet client shares one per-IP limit (SPEC §8.6): tell the orchestrator |
|
||||
| Restore | `infra/RESTORE.md` |
|
||||
|
||||
**APK downloads (security review L-3, D-119).** `GET /apk/files/<name>` is public (the landing page links it) and
|
||||
@@ -611,16 +598,18 @@ Backup encryption (security review M-1) is decided: no rclone `crypt`, the `driv
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
* Step 6 check or `deploy.sh` says /health through NPM failed: "Could not resolve host" or "SSL certificate problem":
|
||||
* The step 4 check says /health through NPM failed: "Could not resolve host" or "SSL certificate problem":
|
||||
the proxy host has no certificate for `<KVITT_PUBLIC_HOST>` yet (SSL tab) or a different name; a `502 Bad Gateway`:
|
||||
NPM cannot reach `<kvitt-pi-address>:8080` (wrong Forward Hostname / IP or Forward Port, or the server is down:
|
||||
`docker compose ps`); the Home Assistant page instead of `{"status":"ok"}`: Domain Names does not match the name
|
||||
you requested (NPM fell back to another host).
|
||||
* "Request a new Certificate" fails in NPM: `<KVITT_PUBLIC_HOST>` does not point at the home's public address yet,
|
||||
or external port 80 does not reach NPM.
|
||||
* Pull fails with "http: server gave HTTP response to HTTPS client": `gitea.lan` is missing from
|
||||
`/etc/docker/daemon.json` (`install.sh` adds it; check with `docker info | grep -A3 'Insecure Registries'`).
|
||||
* Pull fails with "unauthorized": the packages are private; set `KVITT_REGISTRY_USER` and `KVITT_REGISTRY_TOKEN`
|
||||
in `~/kvitt-pi/.env` and run `./install.sh` again.
|
||||
* `install.sh` says `/opt/kvitt/secrets/<file>` is a directory: Docker created it because the file was missing at a
|
||||
start; `sudo rmdir` it, put the file in `~/kvitt-pi/secrets/`, run `./install.sh`.
|
||||
* Pull fails with "http: server gave HTTP response to HTTPS client": `gitea.lan` is missing from `insecure-registries`
|
||||
(step 1; check with `docker info | grep -A3 'Insecure Registries'`).
|
||||
* Pull fails with "unauthorized": the packages are private; `docker login gitea.lan` once (step 1).
|
||||
* `docker compose up` says "bind source path does not exist" for a file in `secrets/`: the file was not copied;
|
||||
copy it (step 2) and run `docker compose up -d` again.
|
||||
* The server or backup log says "Permission denied" or "readonly database" on `/data` or `/backups`: the folder was
|
||||
created by Docker as root, or `KVITT_UID`/`KVITT_GID` differ from the folder's owner: `ls -ln ~/kvitt` shows the
|
||||
owner, `id -u` / `id -g` yours; `sudo chown -R "$(id -u):$(id -g)" ~/kvitt/data ~/kvitt/backups ~/kvitt/import` once.
|
||||
|
||||
+28
-26
@@ -1,14 +1,13 @@
|
||||
# Restore Kvitt from a Drive backup (SPEC §10.9)
|
||||
|
||||
Run on the Kvitt Pi, in `/opt/kvitt`, as the owner's user. Every command uses the running `backup` container,
|
||||
which has `sqlite3`, `rclone`, the Drive credentials and the `data` and `backups` volumes, and runs as uid 10001,
|
||||
the owner of the server's files. Nothing is deleted: the current database and pictures are moved aside.
|
||||
Do not run it around 03:30, when the nightly backup runs.
|
||||
Run on the Kvitt Pi, in `~/kvitt` (`/home/jacobhagstedt/kvitt`), as `jacobhagstedt`. The files in `data/` and
|
||||
`backups/` are yours (both containers run as your uid, D-125), so they are moved with plain `mv` and `cp`; only
|
||||
`rclone` and `sqlite3` come from the running `backup` container. Nothing is deleted: the current database and
|
||||
pictures are moved aside. Do not run it around 03:30, when the nightly backup runs.
|
||||
|
||||
```bash
|
||||
cd /opt/kvitt
|
||||
B="docker compose exec -T backup sh -c"
|
||||
FOLDER=$(sed -n 's/^RCLONE_FOLDER_ID=//p' .env); REMOTE=$(sed -n 's/^RCLONE_REMOTE=//p' .env)
|
||||
cd ~/kvitt
|
||||
FOLDER=$(sed -n 's/^RCLONE_FOLDER_ID=//p' .env)
|
||||
```
|
||||
|
||||
## 1. Stop the server
|
||||
@@ -22,26 +21,29 @@ docker compose stop server
|
||||
List what is on Drive (newest last) and pick a stamp:
|
||||
|
||||
```bash
|
||||
$B 'c=$(mktemp); cp /config/rclone/rclone.conf "$c"; RCLONE_CONFIG=$c rclone lsf "$1" --drive-root-folder-id "$2" --files-only; rm -f "$c"' sh "$REMOTE" "$FOLDER" | sort
|
||||
docker compose exec -T backup sh -c 'c=$(mktemp); cp /config/rclone/rclone.conf "$c"; RCLONE_CONFIG=$c rclone lsf gdrive: --drive-root-folder-id "$1" --files-only; rm -f "$c"' sh "$FOLDER" | sort
|
||||
STAMP=20261102T013000Z # the stamp of the pair you chose
|
||||
```
|
||||
|
||||
Download both files into the `backups` volume:
|
||||
Download both files into `backups/restore/`:
|
||||
|
||||
```bash
|
||||
$B 'set -e; mkdir -p /backups/restore; c=$(mktemp); cp /config/rclone/rclone.conf "$c"
|
||||
for f in "kvitt-$3.sqlite.gz" "pictures-$3.tar.gz"; do
|
||||
RCLONE_CONFIG=$c rclone copy "$1$f" /backups/restore/ --drive-root-folder-id "$2"
|
||||
done; rm -f "$c"; ls -l /backups/restore' sh "$REMOTE" "$FOLDER" "$STAMP"
|
||||
mkdir -p backups/restore
|
||||
docker compose exec -T backup sh -c 'set -e; c=$(mktemp); cp /config/rclone/rclone.conf "$c"
|
||||
for f in "kvitt-$2.sqlite.gz" "pictures-$2.tar.gz"; do
|
||||
RCLONE_CONFIG=$c rclone copy "gdrive:$f" /backups/restore/ --drive-root-folder-id "$1"
|
||||
done; rm -f "$c"' sh "$FOLDER" "$STAMP"
|
||||
ls -l backups/restore
|
||||
```
|
||||
|
||||
(A file downloaded in the browser instead: copy it to the Pi with `scp`, then
|
||||
`docker compose cp <file> backup:/backups/restore/`.)
|
||||
(A file downloaded in the browser instead: copy it from the Mac with
|
||||
`scp <file> jacobhagstedt@<kvitt-pi-address>:kvitt/backups/restore/`.)
|
||||
|
||||
## 3. Unpack and check
|
||||
|
||||
```bash
|
||||
$B 'set -e; cd /backups/restore; gunzip -f "kvitt-$1.sqlite.gz"; sqlite3 "kvitt-$1.sqlite" "PRAGMA integrity_check"' sh "$STAMP"
|
||||
gunzip -f "backups/restore/kvitt-$STAMP.sqlite.gz"
|
||||
docker compose exec -T backup sqlite3 "/backups/restore/kvitt-$STAMP.sqlite" "PRAGMA integrity_check"
|
||||
```
|
||||
|
||||
It must print `ok`. Anything else: stop here and pick another backup.
|
||||
@@ -49,13 +51,13 @@ It must print `ok`. Anything else: stop here and pick another backup.
|
||||
## 4. Move the current data aside and put the backup in place
|
||||
|
||||
```bash
|
||||
$B 'set -e; aside="/data/pre-restore-$(date -u +%Y%m%dT%H%M%SZ)"; mkdir "$aside"
|
||||
for f in /data/kvitt.sqlite /data/kvitt.sqlite-wal /data/kvitt.sqlite-shm /data/pictures; do
|
||||
[ -e "$f" ] && mv "$f" "$aside/"
|
||||
done
|
||||
cp "/backups/restore/kvitt-$1.sqlite" /data/kvitt.sqlite
|
||||
tar -xzf "/backups/restore/pictures-$1.tar.gz" -C /data
|
||||
echo "previous data kept in $aside"; ls -l /data' sh "$STAMP"
|
||||
aside="data/pre-restore-$(date -u +%Y%m%dT%H%M%SZ)"; mkdir "$aside"
|
||||
for f in data/kvitt.sqlite data/kvitt.sqlite-wal data/kvitt.sqlite-shm data/pictures; do
|
||||
[ -e "$f" ] && mv "$f" "$aside/"
|
||||
done
|
||||
cp "backups/restore/kvitt-$STAMP.sqlite" data/kvitt.sqlite
|
||||
tar -xzf "backups/restore/pictures-$STAMP.tar.gz" -C data
|
||||
echo "previous data kept in $aside"; ls -l data
|
||||
```
|
||||
|
||||
## 5. New sync epoch
|
||||
@@ -84,6 +86,6 @@ epoch and re-pushes the phone's data (if nothing seems to happen, close the app
|
||||
|
||||
## Afterwards
|
||||
|
||||
When the phones agree, the downloaded files can go:
|
||||
`$B 'rm -r /backups/restore'`. The `/data/pre-restore-<stamp>/` folder stays until you decide otherwise
|
||||
(`$B 'ls /data'` shows it). Run `/opt/kvitt/backup-now.sh` once to put the restored state on Drive.
|
||||
When the phones agree, the downloaded files can go: `rm -r backups/restore`. The `data/pre-restore-<stamp>/`
|
||||
folder stays until you decide otherwise. Run `docker compose exec backup /backup.sh` once to put the restored state
|
||||
on Drive.
|
||||
|
||||
+14
-29
@@ -1,32 +1,17 @@
|
||||
# Kvitt Pi settings (SPEC §10.2). Copy to .env in this folder, replace every CHANGE_ME, then run ./install.sh,
|
||||
# which copies it to /opt/kvitt/.env (mode 0600). install.sh refuses a CHANGE_ME left in a key it checks.
|
||||
# No secret files here: those go to /opt/kvitt/secrets/ (infra/README.md "Pi: what the owner does").
|
||||
# Kvitt on the Kvitt Pi (SPEC §10.2, D-125). Copy to .env next to compose.yaml and set HA_HOST; check KVITT_UID and
|
||||
# KVITT_GID against `id -u` and `id -g` on the Pi. Every key is read by compose.yaml and by nothing else.
|
||||
|
||||
# --- Read by compose.yaml -----------------------------------------------------------------------------------------
|
||||
|
||||
# Registry and owner path of the Kvitt images; images are ${KVITT_REGISTRY}/kvitt-server|kvitt-backup.
|
||||
# compose.yaml: image names ${KVITT_REGISTRY}/kvitt-server and /kvitt-backup (D-104).
|
||||
KVITT_REGISTRY=gitea.lan/jacobh2
|
||||
# Image tag to run. deploy.sh sets it; first install: the newest version tag the Gitea workflow pushed (vMAJOR.MINOR.PATCH),
|
||||
# or the short git SHA that infra/build-images.sh printed.
|
||||
KVITT_TAG=CHANGE_ME_TAG
|
||||
# Kvitt's own public host name (D-120), owner-chosen: a domain or DuckDNS name pointing at the home's public address.
|
||||
# NPM holds its Let's Encrypt certificate and serves it on the default HTTPS port 443, which the router already
|
||||
# forwards to NPM for Home Assistant (D-124). Invite links and the landing page use https://<KVITT_PUBLIC_HOST>
|
||||
# (KVITT_PUBLIC_URL, no port). Not icehack.asuscomm.com: that name is NPM's proxy host for Home Assistant.
|
||||
KVITT_PUBLIC_HOST=CHANGE_ME_KVITT_PUBLIC_HOST
|
||||
# LAN IPv4 address of the Home Assistant Pi, where Nginx Proxy Manager terminates TLS and forwards to this Pi's port
|
||||
# 8080 (D-120). The server trusts X-Forwarded-For only from this address (KVITT_TRUSTED_PROXY), so it must be the
|
||||
# address NPM's connections come from: an IP literal such as 192.168.1.20, not a host name (install.sh checks it).
|
||||
# Kept the name HA_HOST (the proxy host); it no longer has the SSH settings HA_PORT, HA_USER, HA_CERT_DIR of cert-sync.
|
||||
HA_HOST=CHANGE_ME_HA_HOST
|
||||
# rclone remote of the Drive backups, as named in rclone.conf.
|
||||
RCLONE_REMOTE=gdrive:
|
||||
# Google Drive folder id the backups go to (OQ-4, answered).
|
||||
# compose.yaml: image tag of both services; deploy or roll back = change it, then pull and up -d.
|
||||
KVITT_TAG=v1.0.0
|
||||
# compose.yaml: the server's KVITT_PUBLIC_URL=https://<this>, used in invite links and the landing page (D-124).
|
||||
KVITT_PUBLIC_HOST=kvitt.suorra.duckdns.org
|
||||
# compose.yaml: the server's KVITT_TRUSTED_PROXY, the Home Assistant Pi's (NPM's) LAN IPv4 address, e.g. 192.168.1.20.
|
||||
HA_HOST=CHANGE_ME
|
||||
# compose.yaml: uid both containers run as, `id -u` of the user that owns ~/kvitt on the Pi.
|
||||
KVITT_UID=1000
|
||||
# compose.yaml: gid both containers run as, `id -g` of the same user.
|
||||
KVITT_GID=1000
|
||||
# compose.yaml: the backup's Google Drive folder id (OQ-4).
|
||||
RCLONE_FOLDER_ID=1jDp1dGE5R3H2DBcXCqIPN1iWzQnPkdlV
|
||||
|
||||
# --- Read by install.sh only (never by the containers) ------------------------------------------------------------
|
||||
|
||||
# Gitea user that may pull the images; leave empty when the Gitea packages are public (anonymous pull).
|
||||
KVITT_REGISTRY_USER=
|
||||
# Gitea access token of that user with read:package scope; leave empty for anonymous pull. Never commit a filled .env.
|
||||
KVITT_REGISTRY_TOKEN=
|
||||
|
||||
+3
-3
@@ -1,5 +1,5 @@
|
||||
# Filled on the Mac or the Pi, never committed: host addresses and registry credentials.
|
||||
# Filled on the Mac, never committed: the Pi's .env (host address) and the secrets staged for the copy to the Pi.
|
||||
.env
|
||||
# publish-apk.sh output (APKs, latest.json, R8 mappings) and secrets staged for the copy to the Pi.
|
||||
out/
|
||||
secrets/
|
||||
# publish-apk.sh output (APKs, latest.json, R8 mappings).
|
||||
out/
|
||||
|
||||
@@ -1,19 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run the backup once on the Kvitt Pi (SPEC §10.9) and wait for it: /opt/kvitt/backup-now.sh
|
||||
# Succeeds only when backup.sh succeeded and /backups/last-success changed: the snapshot passed integrity_check,
|
||||
# both files are confirmed on Drive and their local copies are deleted (D-002). Lists the newest files on Drive.
|
||||
set -euo pipefail
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=lib.sh
|
||||
. "$here/lib.sh"
|
||||
|
||||
before=$(compose exec -T backup cat /backups/last-success 2>/dev/null || true)
|
||||
compose exec -T backup /backup.sh || die "backup.sh failed (lines above)"
|
||||
after=$(compose exec -T backup cat /backups/last-success 2>/dev/null || true)
|
||||
[[ -n "$after" && "$after" != "$before" ]] || die "last-success did not change (was '${before:-none}', is '${after:-none}')"
|
||||
say "last-success: $after"
|
||||
remote=$(env_value RCLONE_REMOTE); folder=$(env_value RCLONE_FOLDER_ID)
|
||||
say "Newest files in the Drive folder:"
|
||||
# shellcheck disable=SC2016 # the script runs inside the backup container, with its own \$1 and \$2
|
||||
compose exec -T backup sh -c 'c=$(mktemp); cp /config/rclone/rclone.conf "$c"; RCLONE_CONFIG=$c rclone lsf "$1" --drive-root-folder-id "$2" --files-only --log-level ERROR; s=$?; rm -f "$c"; exit $s' \
|
||||
sh "${remote:-gdrive:}" "$folder" | sort | tail -n 4
|
||||
@@ -1,17 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# First admin invite on the Kvitt Pi (SPEC FR-1, §10.4). Run ON the Pi once, after install.sh: /opt/kvitt/bootstrap.sh
|
||||
# Prints the invite code, link and expiry. Exit 2 from the server means the database is already initialised.
|
||||
set -euo pipefail
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=lib.sh
|
||||
. "$here/lib.sh"
|
||||
|
||||
set +e
|
||||
compose exec -T server kvitt-server bootstrap
|
||||
status=$?
|
||||
set -e
|
||||
case "$status" in
|
||||
0) say "Open the link on the owner's phone (or enter the code on the pairing screen)." ;;
|
||||
2) say "Already initialised: no new invite. Extra devices: docker compose exec server kvitt-server pair-device --user <user_id>" ;;
|
||||
*) die "bootstrap exited $status (docker compose logs server)" ;;
|
||||
esac
|
||||
+45
-30
@@ -1,9 +1,11 @@
|
||||
# Kvitt production stack on the Kvitt Pi (SPEC §10.2). Deployed by install.sh to /opt/kvitt/compose.yaml together
|
||||
# with /opt/kvitt/.env (from .env.example). Run every command from /opt/kvitt.
|
||||
# TLS terminates in Nginx Proxy Manager on the Home Assistant Pi (SPEC §10.5, D-120), which forwards plain HTTP over
|
||||
# the LAN to the server's published port 8080. Only the server publishes a port; backup is reachable only on the
|
||||
# internal network `kvitt` (SPEC §8.9). The router forwards the internet only to NPM, never to this Pi.
|
||||
# Image names: ${KVITT_REGISTRY}/kvitt-<name>, the names the Gitea Actions workflow pushes (D-104).
|
||||
# Kvitt on the Kvitt Pi (SPEC §10.2, D-125): a self-contained compose project in /home/jacobhagstedt/kvitt.
|
||||
# That folder holds this file, .env (from .env.example), secrets/ and the bind-mounted folders data/, backups/
|
||||
# and import/; nothing outside it is used. Run every command there (infra/README.md "Pi: what the owner does").
|
||||
# Both containers run as the owner's uid/gid (KVITT_UID, KVITT_GID in .env), so every file they write in data/ and
|
||||
# backups/ belongs to the owner. Create data/ (and data/apk), backups/ and import/ as the owner before the first
|
||||
# start: a folder Docker creates itself belongs to root and the containers cannot write to it.
|
||||
# TLS terminates in Nginx Proxy Manager on the Home Assistant Pi (SPEC §10.5, D-120), which forwards plain HTTP
|
||||
# over the LAN to the server's published port 8080, the only published port.
|
||||
|
||||
name: kvitt
|
||||
|
||||
@@ -18,13 +20,14 @@ services:
|
||||
image: ${KVITT_REGISTRY:?set KVITT_REGISTRY in .env}/kvitt-server:${KVITT_TAG:?set KVITT_TAG in .env}
|
||||
command: ["serve"]
|
||||
restart: unless-stopped
|
||||
user: "10001:10001"
|
||||
user: "${KVITT_UID:?set KVITT_UID in .env}:${KVITT_GID:?set KVITT_GID in .env}"
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp:size=16m
|
||||
cap_drop: [ALL]
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
mem_limit: 160m
|
||||
# On all interfaces of the Pi: NPM on the Home Assistant Pi connects to this Pi's LAN address, port 8080.
|
||||
ports:
|
||||
- "8080:8080"
|
||||
environment:
|
||||
@@ -32,14 +35,23 @@ services:
|
||||
KVITT_DATA_DIR: /data
|
||||
KVITT_PUBLIC_URL: https://${KVITT_PUBLIC_HOST:?set KVITT_PUBLIC_HOST in .env}
|
||||
# NPM's LAN address: only a request whose TCP peer is this address may set X-Forwarded-For, whose last entry
|
||||
# is then the client IP of the rate limits (SPEC §8.5, §8.6, D-120). An IP literal (install.sh checks it).
|
||||
# is then the client IP of the rate limits (SPEC §8.5, §8.6, D-120). An IPv4 address, not a host name.
|
||||
KVITT_TRUSTED_PROXY: ${HA_HOST:?set HA_HOST in .env}
|
||||
# The remaining server settings keep their SPEC §10.4 defaults (KVITT_FCM_PROJECT_ID kvitt-1337, ...).
|
||||
# The other server settings keep their SPEC §10.4 defaults (KVITT_FCM_CREDENTIALS
|
||||
# /secrets/fcm-service-account.json, KVITT_FCM_PROJECT_ID kvitt-1337, ...).
|
||||
volumes:
|
||||
- data:/data
|
||||
- /opt/kvitt/secrets/fcm-service-account.json:/secrets/fcm-service-account.json:ro
|
||||
# SQLite database, pictures and APKs (data/apk/).
|
||||
- ./data:/data
|
||||
# Only the FCM key, not rclone.conf (D-117). create_host_path false: a missing file stops the start with
|
||||
# "bind source path does not exist" instead of Docker creating a root-owned directory in its place.
|
||||
- type: bind
|
||||
source: ./secrets/fcm-service-account.json
|
||||
target: /secrets/fcm-service-account.json
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
# Empty except during the one-off Splitwise import (SPEC §10.8, D-025).
|
||||
- /opt/kvitt/import:/import:ro
|
||||
- ./import:/import:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "kvitt-server", "healthcheck"]
|
||||
interval: 30s
|
||||
@@ -47,26 +59,38 @@ services:
|
||||
retries: 3
|
||||
start_period: 20s
|
||||
logging: *logging
|
||||
networks: [kvitt]
|
||||
|
||||
backup:
|
||||
image: ${KVITT_REGISTRY}/kvitt-backup:${KVITT_TAG}
|
||||
restart: unless-stopped
|
||||
# uid 10001 like the server, so it can open the WAL database files the server owns; rclone.conf on the Pi is
|
||||
# therefore owned by 10001 (infra/README.md "Backup uid", D-105).
|
||||
user: "10001:10001"
|
||||
# The server's uid, so sqlite3 .backup can open the WAL database files the server owns (D-105).
|
||||
user: "${KVITT_UID}:${KVITT_GID}"
|
||||
read_only: true
|
||||
# backup.sh's private copy of rclone.conf and the start marker of the healthcheck live here.
|
||||
tmpfs:
|
||||
- /tmp:size=16m
|
||||
cap_drop: [ALL]
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
mem_limit: 96m
|
||||
environment:
|
||||
TZ: Europe/Stockholm
|
||||
RCLONE_REMOTE: ${RCLONE_REMOTE:-gdrive:}
|
||||
# The owner's uid has no account in the image; rclone keeps its cache under HOME.
|
||||
HOME: /tmp
|
||||
# The [gdrive] remote in secrets/rclone.conf.
|
||||
RCLONE_REMOTE: "gdrive:"
|
||||
RCLONE_FOLDER_ID: ${RCLONE_FOLDER_ID:?set RCLONE_FOLDER_ID in .env}
|
||||
volumes:
|
||||
- data:/data
|
||||
- backups:/backups
|
||||
- /opt/kvitt/secrets/rclone.conf:/config/rclone/rclone.conf:ro
|
||||
- ./data:/data
|
||||
- ./backups:/backups
|
||||
# Only rclone.conf (Drive token), read-only; backup.sh copies it to /tmp because rclone writes refreshed
|
||||
# tokens back (D-105).
|
||||
- type: bind
|
||||
source: ./secrets/rclone.conf
|
||||
target: /config/rclone/rclone.conf
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
healthcheck:
|
||||
test: ["CMD", "/healthcheck.sh"]
|
||||
interval: 5m
|
||||
@@ -74,12 +98,3 @@ services:
|
||||
retries: 1
|
||||
start_period: 30s
|
||||
logging: *logging
|
||||
networks: [kvitt]
|
||||
|
||||
volumes:
|
||||
data:
|
||||
backups:
|
||||
|
||||
networks:
|
||||
kvitt:
|
||||
driver: bridge
|
||||
|
||||
@@ -1,61 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Deploy an image tag on the Kvitt Pi (SPEC §10.3, §10.10). Run ON the Pi: /opt/kvitt/deploy.sh <tag>
|
||||
# 1. backup first (backup-now.sh: Drive upload confirmed, last-success updated), because a tag may carry a
|
||||
# migration and the Pi cannot tell beforehand (the rule "backup before a deploy with a migration" is applied
|
||||
# to every deploy); skipped when no database exists yet, or with --skip-backup (rollback while Drive is down)
|
||||
# 2. sets KVITT_TAG in /opt/kvitt/.env (the previous tag is kept in /opt/kvitt/previous-tag)
|
||||
# 3. docker compose pull, prints the pulled image digests and appends them to /opt/kvitt/deployed-digests (D-117),
|
||||
# docker compose up -d, waits for server to be healthy, prints GET /health on port 8080 and through Nginx
|
||||
# Proxy Manager (D-120; a failure through NPM is a warning, the deploy itself succeeded)
|
||||
# Rollback after a failed migration: /opt/kvitt/deploy.sh "$(cat /opt/kvitt/previous-tag)"; if a migration had
|
||||
# partially applied, restore the pre-deploy backup (infra/RESTORE.md).
|
||||
set -euo pipefail
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=lib.sh
|
||||
. "$here/lib.sh"
|
||||
|
||||
skip_backup=0
|
||||
tag=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--skip-backup) skip_backup=1 ;;
|
||||
-h|--help) sed -n '2,11p' "$0"; exit 0 ;;
|
||||
-*) die "unknown option $arg" ;;
|
||||
*) [[ -z "$tag" ]] || die "usage: $0 <tag> [--skip-backup]"; tag=$arg ;;
|
||||
esac
|
||||
done
|
||||
[[ -n "$tag" ]] || die "usage: $0 <tag> [--skip-backup]"
|
||||
[[ "$tag" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || die "not a valid image tag: $tag"
|
||||
[[ -f "$KVITT_DIR/.env" ]] || die "$KVITT_DIR/.env is missing: run install.sh first"
|
||||
|
||||
current=$(env_value KVITT_TAG)
|
||||
say "Deploying $tag (running: ${current:-none})"
|
||||
|
||||
# 1. Backup before the deploy.
|
||||
if (( skip_backup == 1 )); then
|
||||
say "Backup skipped (--skip-backup)"
|
||||
elif [[ "$(service_health server)" == none ]] || ! compose exec -T backup test -f /data/kvitt.sqlite 2>/dev/null; then
|
||||
say "No running server with a database: no backup needed"
|
||||
else
|
||||
"$here/backup-now.sh" || die "the pre-deploy backup failed; nothing deployed (fix the backup, or use --skip-backup knowingly)"
|
||||
fi
|
||||
|
||||
# 2. Tag.
|
||||
[[ -n "$current" ]] && printf '%s\n' "$current" > "$KVITT_DIR/previous-tag"
|
||||
env_set KVITT_TAG "$tag"
|
||||
|
||||
# 3. Pull and start.
|
||||
if ! compose pull; then
|
||||
[[ -n "$current" ]] && env_set KVITT_TAG "$current"
|
||||
die "pull of $tag failed; KVITT_TAG left at ${current:-its old value}"
|
||||
fi
|
||||
record_digests "$tag"
|
||||
compose up -d --remove-orphans
|
||||
if ! wait_healthy server 180; then
|
||||
compose logs --tail 50 server || true
|
||||
die "server is not healthy on $tag. Roll back: $KVITT_DIR/deploy.sh ${current:-<previous tag>} --skip-backup"
|
||||
fi
|
||||
compose ps
|
||||
print_health || die "server is healthy but GET /health on port 8080 failed (docker compose logs server)"
|
||||
print_proxy_health || say "WARNING: /health through Nginx Proxy Manager failed: Kvitt is not reachable from outside (README, Troubleshooting)"
|
||||
say "Deployed $tag."
|
||||
@@ -1,42 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Put the APKs that infra/publish-apk.sh made on the Mac into the server's data volume (SPEC §10.6).
|
||||
# The owner copies infra/pi/out/apk/ from the Mac to /opt/kvitt/apk/ on the Pi, then runs ON the Pi:
|
||||
# /opt/kvitt/install-apk.sh
|
||||
# Copies every kvitt-<version_code>.apk from /opt/kvitt/apk into /data/apk, then latest.json last and atomically
|
||||
# (temp file, rename), so the server never announces a file it does not have; keeps the newest 5 APKs in /data/apk.
|
||||
# R8 mapping files (kvitt-<version_code>.mapping.txt) stay in /opt/kvitt/apk; the server never needs them.
|
||||
set -euo pipefail
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=lib.sh
|
||||
. "$here/lib.sh"
|
||||
|
||||
src="$KVITT_DIR/apk"
|
||||
[[ -f "$src/latest.json" ]] || die "no $src/latest.json: copy infra/pi/out/apk/ from the Mac first (README, Publishing an APK)"
|
||||
file_name=$(sed -n 's/.*"file_name": *"\([^"]*\)".*/\1/p' "$src/latest.json")
|
||||
[[ "$file_name" =~ ^kvitt-[0-9]+\.apk$ ]] || die "latest.json has no valid file_name"
|
||||
[[ -f "$src/$file_name" ]] || die "latest.json names $file_name, which is not in $src"
|
||||
sha=$(sed -n 's/.*"sha256": *"\([0-9a-f]\{64\}\)".*/\1/p' "$src/latest.json")
|
||||
[[ "$(sha256sum "$src/$file_name" | cut -d' ' -f1)" == "$sha" ]] || die "$file_name does not match the sha256 in latest.json"
|
||||
|
||||
# backup runs as uid 10001, the owner of /data, and has a shell; the server image has none.
|
||||
# shellcheck disable=SC2016 # the script runs inside the backup container
|
||||
compose run --rm --no-deps -T -v "$src:/incoming:ro" --entrypoint /bin/sh backup -c '
|
||||
set -eu
|
||||
mkdir -p /data/apk
|
||||
for f in /incoming/kvitt-*.apk; do
|
||||
[ -f "$f" ] || continue
|
||||
n=$(basename "$f")
|
||||
[ -f "/data/apk/$n" ] && cmp -s "$f" "/data/apk/$n" && continue
|
||||
cp "$f" "/data/apk/.$n.tmp" && mv -f "/data/apk/.$n.tmp" "/data/apk/$n"
|
||||
echo "copied $n"
|
||||
done
|
||||
cp /incoming/latest.json /data/apk/.latest.json.tmp
|
||||
mv -f /data/apk/.latest.json.tmp /data/apk/latest.json
|
||||
# keep the newest 5 APKs by version code
|
||||
ls /data/apk | sed -n "s/^kvitt-\([0-9]*\)\.apk$/\1/p" | sort -n -r | tail -n +6 | while read -r v; do
|
||||
rm -f "/data/apk/kvitt-$v.apk"; echo "removed kvitt-$v.apk"
|
||||
done
|
||||
echo "latest.json installed"'
|
||||
say "GET $SERVER_URL/apk/latest:"
|
||||
curl -fsS --max-time 15 "$SERVER_URL/apk/latest" || die "GET /apk/latest failed"
|
||||
echo
|
||||
@@ -1,171 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Kvitt Pi install (infra/README.md "Pi: what the owner does"). Run ON the Kvitt Pi, as the owner's user, from the
|
||||
# bundle folder copied from the Mac (infra/pi/): cd ~/kvitt-pi && ./install.sh
|
||||
# Idempotent: safe to run again after changing .env, the secrets or compose.yaml.
|
||||
# 1. checks Docker Engine and the compose plugin
|
||||
# 2. checks .env (no CHANGE_ME left in the keys the stack reads; KVITT_PUBLIC_HOST a host name, HA_HOST an IPv4
|
||||
# address)
|
||||
# 3. creates /opt/kvitt/{secrets,import,apk}, installs the secret files from ./secrets/ with their modes and
|
||||
# removes what the Caddy and cert-sync stack of before D-120 left in /opt/kvitt
|
||||
# 4. trusts the plain-HTTP registry in /etc/docker/daemon.json (insecure-registries) and logs in when credentials are set
|
||||
# 5. copies compose.yaml, .env and the operation scripts to /opt/kvitt; a KVITT_TAG that deploy.sh set
|
||||
# in /opt/kvitt/.env is kept, so re-running an older bundle never rolls the server back to the bundle's tag
|
||||
# 6. pulls the images (digests recorded in /opt/kvitt/deployed-digests), gives the named volumes their owners,
|
||||
# starts the stack, waits for /health on port 8080, then tries /health through Nginx Proxy Manager
|
||||
# Uses sudo for /opt/kvitt, the secret files and the Docker daemon configuration.
|
||||
set -euo pipefail
|
||||
bundle="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=lib.sh
|
||||
. "$bundle/lib.sh"
|
||||
|
||||
# 1. Docker
|
||||
command -v docker >/dev/null || die "docker is not installed (README step 2)"
|
||||
docker compose version >/dev/null 2>&1 || die "the docker compose plugin is missing (README step 2)"
|
||||
docker info >/dev/null 2>&1 || die "cannot talk to Docker as $(id -un): add yourself to the docker group (sudo usermod -aG docker $(id -un)), log out and in"
|
||||
command -v curl >/dev/null || die "curl is missing: sudo apt-get install -y curl"
|
||||
command -v python3 >/dev/null || die "python3 is missing (step 4 edits daemon.json with it): sudo apt-get install -y python3"
|
||||
|
||||
# 2. .env
|
||||
env_src="$bundle/.env"
|
||||
[[ -f "$env_src" ]] || die "no .env next to install.sh: cp .env.example .env and fill it (README step 4)"
|
||||
for key in KVITT_REGISTRY KVITT_TAG KVITT_PUBLIC_HOST HA_HOST RCLONE_REMOTE RCLONE_FOLDER_ID; do
|
||||
value=$(env_value "$key" "$env_src")
|
||||
[[ -n "$value" ]] || die ".env: $key is empty"
|
||||
[[ "$value" != CHANGE_ME* ]] || die ".env: $key is still $value"
|
||||
done
|
||||
# The server compares the TCP peer with KVITT_TRUSTED_PROXY (= HA_HOST). A host name would have to resolve inside
|
||||
# the container to exactly NPM's LAN address (mDNS names such as homeassistant.local do not), so require the address.
|
||||
ha_host=$(env_value HA_HOST "$env_src")
|
||||
[[ "$ha_host" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]] \
|
||||
|| die ".env: HA_HOST must be the Home Assistant Pi's LAN IPv4 address (for example 192.168.1.20), not $ha_host"
|
||||
public_host=$(env_value KVITT_PUBLIC_HOST "$env_src")
|
||||
[[ "$public_host" =~ ^[A-Za-z0-9]([A-Za-z0-9.-]*[A-Za-z0-9])?\.[A-Za-z]{2,}$ ]] \
|
||||
|| die ".env: KVITT_PUBLIC_HOST must be a host name such as kvitt.example.org (no scheme, no port), not $public_host"
|
||||
[[ "$public_host" != icehack.asuscomm.com ]] \
|
||||
|| die ".env: KVITT_PUBLIC_HOST cannot be icehack.asuscomm.com, NPM's proxy host for Home Assistant (D-120)"
|
||||
registry=$(env_value KVITT_REGISTRY "$env_src")
|
||||
registry_host=${registry%%/*}
|
||||
registry_user=$(env_value KVITT_REGISTRY_USER "$env_src")
|
||||
registry_token=$(env_value KVITT_REGISTRY_TOKEN "$env_src")
|
||||
|
||||
# 3. Directories and secrets. uid 10001 = server and backup (SPEC §10.2, §10.7, D-105).
|
||||
say "Creating $KVITT_DIR"
|
||||
sudo install -d -m 0755 -o "$(id -u)" -g "$(id -g)" "$KVITT_DIR" "$KVITT_DIR/import" "$KVITT_DIR/apk"
|
||||
sudo install -d -m 0700 -o root -g root "$KVITT_DIR/secrets"
|
||||
# name : owner uid : group : mode
|
||||
secret_specs=(
|
||||
"fcm-service-account.json:0:10001:0440"
|
||||
"rclone.conf:10001:10001:0600"
|
||||
)
|
||||
missing=()
|
||||
for spec in "${secret_specs[@]}"; do
|
||||
IFS=: read -r name uid gid mode <<< "$spec"
|
||||
src="$bundle/secrets/$name"
|
||||
dst="$KVITT_DIR/secrets/$name"
|
||||
if [[ -f "$src" ]]; then
|
||||
sudo install -m "$mode" -o "$uid" -g "$gid" "$src" "$dst"
|
||||
say " secrets/$name installed (mode $mode, owner $uid:$gid)"
|
||||
elif sudo test -f "$dst"; then
|
||||
sudo chown "$uid:$gid" "$dst"
|
||||
sudo chmod "$mode" "$dst"
|
||||
say " secrets/$name already in place (mode $mode, owner $uid:$gid)"
|
||||
else
|
||||
missing+=("$name")
|
||||
fi
|
||||
if sudo test -d "$dst"; then
|
||||
die "$dst is a directory (Docker creates one when a file was missing at start): sudo rmdir $dst, then run again"
|
||||
fi
|
||||
done
|
||||
(( ${#missing[@]} == 0 )) || die "missing secret files: ${missing[*]} (put them in $bundle/secrets/, README step 5)"
|
||||
if [[ -d "$bundle/secrets" ]]; then
|
||||
say " The copies in $bundle/secrets/ are no longer needed: rm -rf $bundle/secrets"
|
||||
fi
|
||||
# Left by the Caddy and cert-sync stack of before D-120: the Home Assistant SSH key must not stay on this Pi.
|
||||
for old in "$KVITT_DIR/secrets/ha_ssh_key" "$KVITT_DIR/secrets/ha_known_hosts" "$KVITT_DIR/Caddyfile"; do
|
||||
if sudo test -e "$old"; then
|
||||
sudo rm -rf "$old"
|
||||
say " removed $old (not used since D-120)"
|
||||
fi
|
||||
done
|
||||
|
||||
# 4. Registry trust (plain HTTP on the home Wi-Fi, SPEC §10.3) and optional login.
|
||||
daemon_json=/etc/docker/daemon.json
|
||||
changed=$(sudo python3 - "$daemon_json" "$registry_host" <<'PY'
|
||||
import json, os, sys
|
||||
path, host = sys.argv[1], sys.argv[2]
|
||||
cfg = {}
|
||||
if os.path.exists(path):
|
||||
with open(path) as f:
|
||||
text = f.read().strip()
|
||||
cfg = json.loads(text) if text else {}
|
||||
regs = cfg.setdefault("insecure-registries", [])
|
||||
if host in regs:
|
||||
print("no")
|
||||
else:
|
||||
regs.append(host)
|
||||
tmp = path + ".kvitt-tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(cfg, f, indent=2)
|
||||
f.write("\n")
|
||||
os.replace(tmp, path)
|
||||
print("yes")
|
||||
PY
|
||||
)
|
||||
if [[ "$changed" == yes ]]; then
|
||||
say "Added $registry_host to insecure-registries in $daemon_json; restarting Docker"
|
||||
sudo systemctl restart docker
|
||||
else
|
||||
say "$registry_host is already in insecure-registries"
|
||||
fi
|
||||
if [[ -n "$registry_user" && -n "$registry_token" ]]; then
|
||||
printf '%s' "$registry_token" | docker login "$registry_host" -u "$registry_user" --password-stdin >/dev/null
|
||||
say "Logged in to $registry_host as $registry_user"
|
||||
else
|
||||
say "No registry credentials in .env: pulling anonymously"
|
||||
fi
|
||||
|
||||
# 5. Stack files and operation scripts.
|
||||
installed_tag=""
|
||||
[[ -f "$KVITT_DIR/.env" ]] && installed_tag=$(env_value KVITT_TAG)
|
||||
bundle_tag=$(env_value KVITT_TAG "$env_src")
|
||||
install -m 0644 "$bundle/compose.yaml" "$KVITT_DIR/compose.yaml"
|
||||
install -m 0600 "$env_src" "$KVITT_DIR/.env"
|
||||
if [[ -n "$installed_tag" && "$installed_tag" != "$bundle_tag" ]]; then
|
||||
env_set KVITT_TAG "$installed_tag"
|
||||
say "KVITT_TAG stays at $installed_tag (set by deploy.sh; the bundle's .env says $bundle_tag). Other tag: $KVITT_DIR/deploy.sh <tag>"
|
||||
fi
|
||||
for script in lib.sh deploy.sh bootstrap.sh backup-now.sh install-apk.sh; do
|
||||
install -m 0755 "$bundle/$script" "$KVITT_DIR/$script"
|
||||
done
|
||||
say "Installed compose.yaml, .env and the scripts in $KVITT_DIR"
|
||||
|
||||
# 6. Pull, volume owners, start.
|
||||
compose config --quiet
|
||||
compose pull
|
||||
record_digests "$(env_value KVITT_TAG)"
|
||||
compose create
|
||||
# Named volumes start owned by root; give each its container's uid (the server image has no shell, so the
|
||||
# backup image, which mounts the same data volume, sets /data). Runs as root once per install, then exits.
|
||||
compose run --rm --no-deps -T --user 0:0 --cap-add CHOWN --entrypoint /bin/sh backup \
|
||||
-c 'chown 10001:10001 /data /backups && mkdir -p /data/apk && chown 10001:10001 /data/apk'
|
||||
# --remove-orphans also removes the caddy and cert-sync containers of a stack installed before D-120.
|
||||
compose up -d --remove-orphans
|
||||
# Their volumes: certs held a copy of the certificate's private key.
|
||||
for volume in kvitt_certs kvitt_caddy_data; do
|
||||
if docker volume inspect "$volume" >/dev/null 2>&1; then
|
||||
if docker volume rm "$volume" >/dev/null; then
|
||||
say "Removed the volume $volume (not used since D-120)"
|
||||
else
|
||||
say "Could not remove the volume $volume (still in use?): docker volume rm $volume"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
ok=1
|
||||
wait_healthy server 180 || ok=0
|
||||
compose ps
|
||||
print_health || ok=0
|
||||
(( ok == 1 )) || die "the stack is not fully up yet; see the lines above"
|
||||
if ! print_proxy_health; then
|
||||
say "Not reachable through Nginx Proxy Manager yet: expected until its proxy host is set up (README step 6)."
|
||||
fi
|
||||
say "Kvitt is running. Next: README step 6 (NPM proxy host), then $KVITT_DIR/bootstrap.sh (first install only)."
|
||||
-119
@@ -1,119 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared helpers for the Kvitt Pi scripts (install.sh, deploy.sh, bootstrap.sh, backup-now.sh, install-apk.sh).
|
||||
# Sourced, not run. These scripts run ON the Kvitt Pi as the owner's user (a member of the docker group).
|
||||
# shellcheck disable=SC2034 # variables are used by the scripts that source this file
|
||||
|
||||
KVITT_DIR=${KVITT_DIR:-/opt/kvitt}
|
||||
# The server's port, published on the Pi's LAN interfaces; Nginx Proxy Manager forwards to it (D-120).
|
||||
SERVER_URL=http://localhost:8080
|
||||
# Nginx Proxy Manager's HTTPS port on the Home Assistant Pi, the default HTTPS port that Kvitt's public URL uses
|
||||
# (the router's existing external 443 forward reaches it, D-124).
|
||||
NPM_HTTPS_PORT=443
|
||||
|
||||
say() { printf '%s\n' "$*"; }
|
||||
die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
# Value of KEY in an env file (default /opt/kvitt/.env); empty when unset.
|
||||
env_value() {
|
||||
local file=${2:-$KVITT_DIR/.env}
|
||||
sed -n "s/^$1=//p" "$file" | tail -n 1
|
||||
}
|
||||
|
||||
# Set KEY=VALUE in an env file in place (adds the line when missing). Keeps the file's mode.
|
||||
env_set() {
|
||||
local key=$1 value=$2 file=${3:-$KVITT_DIR/.env} tmp
|
||||
tmp=$(mktemp "$file.XXXXXX")
|
||||
if grep -q "^$key=" "$file"; then
|
||||
awk -v k="$key" -v v="$value" 'index($0, k "=") == 1 { print k "=" v; next } { print }' "$file" > "$tmp" \
|
||||
|| { rm -f "$tmp"; return 1; }
|
||||
else
|
||||
{ cat "$file" && printf '%s=%s\n' "$key" "$value"; } > "$tmp" || { rm -f "$tmp"; return 1; }
|
||||
fi
|
||||
if ! { chmod --reference="$file" "$tmp" && mv -f "$tmp" "$file"; }; then rm -f "$tmp"; return 1; fi
|
||||
}
|
||||
|
||||
compose() {
|
||||
docker compose --project-directory "$KVITT_DIR" -f "$KVITT_DIR/compose.yaml" --env-file "$KVITT_DIR/.env" "$@"
|
||||
}
|
||||
|
||||
# Health status of a compose service's container: healthy, unhealthy, starting, or none.
|
||||
service_health() {
|
||||
local id
|
||||
id=$(compose ps -q "$1" 2>/dev/null | head -n 1)
|
||||
[[ -n "$id" ]] || { echo none; return; }
|
||||
docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$id" 2>/dev/null || echo none
|
||||
}
|
||||
|
||||
# Wait until a service is healthy (default 180 s). Returns 1 on timeout.
|
||||
wait_healthy() {
|
||||
local service=$1 timeout=${2:-180} waited=0 status
|
||||
while :; do
|
||||
status=$(service_health "$service")
|
||||
[[ "$status" == healthy ]] && { say "$service: healthy"; return 0; }
|
||||
if (( waited >= timeout )); then
|
||||
say "$service: $status after ${timeout}s"
|
||||
return 1
|
||||
fi
|
||||
sleep 5
|
||||
waited=$((waited + 5))
|
||||
done
|
||||
}
|
||||
|
||||
# GET /health from the server's published port on this Pi.
|
||||
print_health() {
|
||||
local out
|
||||
if out=$(curl -fsS --max-time 15 "$SERVER_URL/health" 2>&1); then
|
||||
say "GET $SERVER_URL/health: $out"
|
||||
return 0
|
||||
fi
|
||||
say "GET $SERVER_URL/health failed: $out"
|
||||
return 1
|
||||
}
|
||||
|
||||
# GET /health through Nginx Proxy Manager on the Home Assistant Pi (HA_HOST in .env) for KVITT_PUBLIC_HOST, with
|
||||
# the real certificate check (no -k): proves NPM's proxy host, its certificate and the LAN path to this Pi's port
|
||||
# 8080. The router forward is not on this path.
|
||||
print_proxy_health() {
|
||||
local out ha_host host
|
||||
ha_host=$(env_value HA_HOST)
|
||||
host=$(env_value KVITT_PUBLIC_HOST)
|
||||
if out=$(curl -fsS --max-time 15 --resolve "$host:$NPM_HTTPS_PORT:$ha_host" "https://$host/health" 2>&1); then
|
||||
say "GET https://$host/health (via NPM at $ha_host:$NPM_HTTPS_PORT): $out"
|
||||
return 0
|
||||
fi
|
||||
say "GET https://$host/health (via NPM at $ha_host:$NPM_HTTPS_PORT) failed: $out"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Print the registry digest of every image the stack runs and append one line per image to
|
||||
# $KVITT_DIR/deployed-digests: "<UTC time> <KVITT_TAG> <image reference> <digest>" (D-117). The registry is plain
|
||||
# HTTP on the LAN (D-001), so a swapped image would otherwise go unnoticed: a tag that was recorded before with a
|
||||
# different digest prints a WARNING (expected only for a tag like `latest` that is meant to move).
|
||||
record_digests() {
|
||||
local tag=$1 log=$KVITT_DIR/deployed-digests stamp ref repo digest prev d
|
||||
local -a refs repo_digests
|
||||
stamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
mapfile -t refs < <(compose config --images)
|
||||
touch "$log"
|
||||
say "Image digests (also in $log):"
|
||||
for ref in "${refs[@]}"; do
|
||||
[[ -n "$ref" ]] || continue
|
||||
digest=""
|
||||
if [[ "$ref" == *@sha256:* ]]; then
|
||||
digest=${ref#*@}
|
||||
else
|
||||
repo=${ref%:*}
|
||||
read -r -a repo_digests <<< "$(docker image inspect --format '{{range .RepoDigests}}{{.}} {{end}}' "$ref" 2>/dev/null || true)"
|
||||
for d in "${repo_digests[@]}"; do
|
||||
if [[ "$d" == "$repo@"* ]]; then digest=${d#*@}; break; fi
|
||||
done
|
||||
fi
|
||||
[[ -n "$digest" ]] || digest=unknown
|
||||
prev=$(awk -v r="$ref" '$3 == r { d = $4 } END { print d }' "$log")
|
||||
if [[ -n "$prev" && "$prev" != "$digest" ]]; then
|
||||
say "WARNING: $ref was $prev at an earlier deploy and is $digest now: the image behind this tag changed"
|
||||
fi
|
||||
say " $ref $digest"
|
||||
printf '%s %s %s %s\n' "$stamp" "$tag" "$ref" "$digest" >> "$log"
|
||||
done
|
||||
}
|
||||
@@ -10,7 +10,7 @@
|
||||
# 5. writes into infra/pi/out/apk/: kvitt-<version_code>.apk, kvitt-<version_code>.mapping.txt (R8 mapping, for
|
||||
# reading stack traces; never served), and latest.json (written to a temp file, then renamed)
|
||||
# 6. keeps the newest 5 APKs (and their mappings) in infra/pi/out/apk/
|
||||
# Then the owner copies the folder to the Pi and runs /opt/kvitt/install-apk.sh there (infra/README.md).
|
||||
# Then the owner copies the APK and then latest.json into ~/kvitt/data/apk/ on the Pi with scp (infra/README.md).
|
||||
# Refuses an unsigned build: without the kvitt.release.* keys in android/local.properties the APK cannot be
|
||||
# installed over an earlier release (D-024, D-101).
|
||||
set -euo pipefail
|
||||
@@ -24,7 +24,7 @@ short_sha="$(git rev-parse --short=7 HEAD)"
|
||||
commit_day="$(TZ=Europe/Stockholm git show -s --date=format-local:%Y.%m.%d --format=%cd HEAD)"
|
||||
version_name="$commit_day-$short_sha"
|
||||
|
||||
# The public host the app is built for (D-121), the same value and checks as infra/pi/install.sh (D-120).
|
||||
# The public host the app is built for (D-121), the value the Pi's compose.yaml reads from the same .env (D-120).
|
||||
env_file="$repo_root/infra/pi/.env"
|
||||
[[ -f "$env_file" ]] || { echo "publish-apk.sh: no infra/pi/.env; cp infra/pi/.env.example infra/pi/.env and set KVITT_PUBLIC_HOST." >&2; exit 1; }
|
||||
KVITT_PUBLIC_HOST="$(sed -n 's/^KVITT_PUBLIC_HOST=//p' "$env_file" | tail -n 1)"
|
||||
@@ -95,6 +95,6 @@ done
|
||||
|
||||
echo "Staged in infra/pi/out/apk/: $file_name ($size_bytes bytes, sha256 $sha256) and latest.json."
|
||||
echo "Built for the release server https://$KVITT_PUBLIC_HOST (KVITT_PUBLIC_HOST in infra/pi/.env)."
|
||||
echo "Copy to the Pi and install (infra/README.md, Publishing an APK):"
|
||||
echo " scp infra/pi/out/apk/latest.json infra/pi/out/apk/$file_name infra/pi/out/apk/kvitt-$version_code.mapping.txt <ssh user>@<Kvitt Pi address>:/opt/kvitt/apk/"
|
||||
echo " ssh <ssh user>@<Kvitt Pi address> /opt/kvitt/install-apk.sh"
|
||||
echo "Copy to the Pi, the APK first and latest.json last (infra/README.md, Publishing an APK):"
|
||||
echo " scp infra/pi/out/apk/$file_name jacobhagstedt@<kvitt-pi-address>:kvitt/data/apk/"
|
||||
echo " scp infra/pi/out/apk/latest.json jacobhagstedt@<kvitt-pi-address>:kvitt/data/apk/"
|
||||
|
||||
Reference in New Issue
Block a user