Infra security fixes: Caddy log redaction, headers and non-root container; dev mounts only the FCM key; deploy records image digests (D-117)
- Caddyfile: access and error logs replace the /i/<code> segment and every query string with REDACTED (regexp filter on request>uri); Authorization stays redacted by default. Deferred response headers: Content-Security-Policy, X-Content-Type-Options, X-Frame-Options, Referrer-Policy. - Pi compose: caddy as uid 10002 with group 1000, read-only root, cap_drop ALL plus NET_BIND_SERVICE (needed to exec the image's file-capability binary), tmpfs /config and /tmp; install.sh sets /certs 0750 1000:1000 and Caddy's /data to 10002. - deploy.sh and install.sh print the pulled image digests and append them to /opt/kvitt/deployed-digests, warning when a recorded tag changed digest; the workflow prints the pushed digests; kvitt_cicd needs only write:package. - Dev stack: up.sh stages only fcm-service-account.json into infra/dev/secrets/ (git-ignored), mounted read-only at /secrets. - README: restricted cert-sync key required (plain root key not acceptable), remote_ip and log checks, APK download note, owner decisions (backup encryption, APK limit, key form, token scope), overhaul lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TQd3pEP9UeN2uifbHknQ7B
This commit is contained in:
+83
-24
@@ -17,6 +17,7 @@ runs every Pi command from the checklist in "Pi: what the owner does".
|
||||
| `dev/backup.sh` | Runs the backup job once and checks the snapshot with `PRAGMA integrity_check` |
|
||||
| `dev/copy-google-services.sh` | Copies `secrets/google-services.json` to `android/app/google-services.json` |
|
||||
| `dev/lib.sh` | Helpers sourced by the scripts |
|
||||
| `dev/secrets/` | Git-ignored: only `fcm-service-account.json`, copied there by `dev/up.sh` and mounted read-only at `/secrets` |
|
||||
| `backup/Dockerfile`, `backup/backup.sh`, `backup/healthcheck.sh` | Backup image and job of SPEC §10.9 (shared with the Pi stack); Drive upload and D-002 deletion |
|
||||
| `cert-sync/Dockerfile`, `cert-sync/sync.sh`, `cert-sync/entrypoint.sh` | Certificate sync image of SPEC §10.5.1 |
|
||||
| `pi/` | **The Pi bundle** (copied to the Kvitt Pi): `compose.yaml`, `Caddyfile`, `.env.example`, `install.sh`, `deploy.sh`, `bootstrap.sh`, `backup-now.sh`, `install-apk.sh`, `lib.sh`; git-ignored `pi/.env`, `pi/secrets/`, `pi/out/` |
|
||||
@@ -34,8 +35,12 @@ All Mac commands below run from the repository root of the main checkout (`/User
|
||||
publishes its own port (default 8080) on all of the Mac's interfaces. The server keeps the hardening of SPEC §8.9
|
||||
(uid 10001, read-only root, no capabilities, `no-new-privileges`), the 160 MiB limit, `json-file` logging
|
||||
(10 MB x 3) and `TZ=Europe/Stockholm`. Data lives in the bind-mounted folder `infra/dev/data/` (git-ignored) so it
|
||||
is easy to inspect and reset. The secrets folder is mounted read-only at `/secrets`; if
|
||||
`fcm-service-account.json` is missing the server runs with push disabled (§10.4).
|
||||
is easy to inspect and reset. Only the FCM key reaches the container (D-117): `up.sh` copies
|
||||
`${KVITT_SECRETS_DIR}/fcm-service-account.json` (default `infra/secrets/`) into `infra/dev/secrets/` (git-ignored,
|
||||
nothing else in it) and that folder is mounted read-only at `/secrets`, so the server never sees the Home Assistant
|
||||
SSH key, `rclone.conf`, the registry credentials or `google-services.json`. If the key is missing the folder is
|
||||
empty and the server runs with push disabled (§10.4). Run `up.sh` once before the bare `docker compose` commands
|
||||
below (they mount the same folder); after rotating the key, run `up.sh` again.
|
||||
|
||||
On this Mac `docker` is Podman (`docker compose` runs `podman-compose`); the files work with Docker Desktop as well.
|
||||
|
||||
@@ -43,7 +48,7 @@ On this Mac `docker` is Podman (`docker compose` runs `podman-compose`); the fil
|
||||
|
||||
```bash
|
||||
infra/dev/build.sh # both images
|
||||
infra/dev/build.sh server # only kvitt-server:dev; needs server/Dockerfile (milestone M2a)
|
||||
infra/dev/build.sh server # only kvitt-server:dev; needs server/Dockerfile
|
||||
infra/dev/build.sh backup # only kvitt-backup:dev
|
||||
```
|
||||
|
||||
@@ -97,7 +102,8 @@ Runs `infra/backup/backup.sh` once in the `backup` image (compose profile `backu
|
||||
`RCLONE_FOLDER_ID` empty: `.backup` snapshot, `PRAGMA integrity_check` must print `ok`, then
|
||||
`infra/dev/backups/kvitt-<UTC stamp>.sqlite.gz`, `pictures-<stamp>.tar.gz` and `last-success`; the Drive upload
|
||||
is skipped with a log line and the files are kept (SPEC §10.9, D-001). The wrapper then checks the new snapshot
|
||||
again. To check by hand: `gunzip -c infra/dev/backups/kvitt-<stamp>.sqlite.gz > /tmp/k.sqlite && sqlite3 /tmp/k.sqlite "PRAGMA integrity_check"`.
|
||||
again. The dev env file has no rclone settings (`RCLONE_REMOTE` is no longer in `dev/.env.example`): only the Pi's
|
||||
`.env` names the remote. To check by hand: `gunzip -c infra/dev/backups/kvitt-<stamp>.sqlite.gz > /tmp/k.sqlite && sqlite3 /tmp/k.sqlite "PRAGMA integrity_check"`.
|
||||
|
||||
### Resetting the dev data
|
||||
|
||||
@@ -190,8 +196,16 @@ mode 0600 **owned by 10001:10001** (not 1000). cert-sync stays uid 1000 with `ha
|
||||
mode 0600 owned by 1000:1000 (ssh refuses a key not owned by its user or readable by others). rclone writes
|
||||
refreshed access tokens back to its config; the mounted file is read-only, so `backup.sh` works on a private copy
|
||||
in `/tmp` each run. `install.sh` gives the named volumes their owners once (`/data`, `/data/apk`, `/backups` to
|
||||
10001, `/certs` to 1000) with a one-off root container (`docker compose run --user 0:0 --cap-add CHOWN`). Caddy
|
||||
runs as root (official image) and reads the 0640 certificate files.
|
||||
10001, `/certs` to 1000 with mode 0750, Caddy's `/data` to 10002) with one-off root containers
|
||||
(`docker compose run --user 0:0 --cap-add CHOWN`, plus `FOWNER` for the `chmod`).
|
||||
|
||||
**Caddy uid (D-117).** Caddy runs as **uid 10002** with the supplementary group **1000** (cert-sync's group): the
|
||||
certificate files are 0640 `1000:1000` and `/certs` is 0750 `1000:1000`, so cert-sync writes and Caddy only reads.
|
||||
Read-only root filesystem, `no-new-privileges`, `cap_drop: [ALL]` with `cap_add: [NET_BIND_SERVICE]`, `/config`
|
||||
and `/tmp` as tmpfs, `caddy_data` writable by 10002. Port 8443 needs no capability, but the official image's
|
||||
`/usr/bin/caddy` has the file capability `cap_net_bind_service=ep`, and the kernel refuses to exec it when that
|
||||
capability is outside the bounding set ("exec container process /usr/bin/caddy: Operation not permitted" with
|
||||
`cap_drop: [ALL]` alone, checked 2026-10-18), so it is the one capability kept.
|
||||
|
||||
### What was checked on the Mac (2026-10-18)
|
||||
|
||||
@@ -208,6 +222,16 @@ runs as root (official image) and reads the 0640 certificate files.
|
||||
* Backup rehearsal with a local rclone remote: upload, `lsf` confirmation by name and size, local deletion only
|
||||
after both files were confirmed (D-002); files kept and `last-success` unchanged when the upload failed; the
|
||||
leftovers uploaded by the next good run; healthcheck green on a fresh start and after a run, red at 26 h.
|
||||
* Caddy hardening rehearsal (D-117, 2026-10-18, Podman, pinned Caddy image, `pi/Caddyfile`, a self-signed pair in
|
||||
a `certs` volume set up as `install.sh` and cert-sync do, a stand-in upstream named `server`): Caddy runs as
|
||||
`uid=10002 gid=10002 groups=1000,10002`, `NoNewPrivs: 1`, capability sets `0x400` (NET_BIND_SERVICE only), the root
|
||||
filesystem refuses writes, the healthcheck command succeeds, memory about 12 MB. `GET /i/<code>?from=<name>` with a
|
||||
bearer header answered 200 with `Content-Security-Policy`, `X-Content-Type-Options: nosniff`,
|
||||
`X-Frame-Options: DENY` and one `Referrer-Policy: no-referrer` (the upstream sent it too). The access log showed
|
||||
`"uri":"/i/REDACTED?REDACTED"` and `"/sync/pull?REDACTED"`, `"Authorization":["REDACTED"]`; with the upstream
|
||||
stopped (502) Caddy's error log carried `/i/REDACTED?REDACTED` too; the code, the name, `kvt_` and the query value
|
||||
occurred 0 times in either log. A reload through the admin API from another container (`POST /load`, as
|
||||
cert-sync does) succeeded. TLS 1.1 refused.
|
||||
* Not checked (needs the real hosts): the Home Assistant add-on's handling of `authorized_keys` options, the
|
||||
Gitea workflow (not yet run), pulls from `gitea.lan`, the router forward, Drive.
|
||||
|
||||
@@ -218,6 +242,15 @@ Same setup as the owner's working example: `runs-on: host`, `sh` steps, `apk add
|
||||
a buildx builder whose `buildkitd.toml` sets `http = true` for `gitea.lan` (created once and reused, so its cache
|
||||
survives), `actions/checkout@v4` with `set-safe-directory` and `fetch-depth: 0`, the version from the newest
|
||||
`v1.0.*` git tag plus one, and the git tag pushed after the build. That version is what `deploy.sh` takes.
|
||||
The token reaches `docker login` through an environment variable, never as text in the script. `kvitt_cicd` needs
|
||||
only the `write:package` scope: the tag push uses the credentials `actions/checkout` stored (the job's own token).
|
||||
If the "Tag and Push Version" step fails with a permission error, the job token cannot push tags on this Gitea;
|
||||
then add `write:repository` to the token (owner decision below). The step "Print image digests" prints each
|
||||
image's `sha256:` digest, which must match what `deploy.sh` prints on the Pi (D-117).
|
||||
|
||||
The builder's `buildkitd.toml` is read only when the builder is created. After changing the registry setting in the
|
||||
workflow, remove the builder once on the runner Pi: `docker buildx rm local-http-builder` (this also drops its
|
||||
build cache, so the next server build is a full one).
|
||||
|
||||
**Risk: the runner is a Raspberry Pi 3 with 1 GB RAM.** The server image compiles about 232 Rust crates,
|
||||
aws-lc-sys's C code among them. The workflow passes `--build-arg CARGO_BUILD_JOBS=1`; it takes effect only once
|
||||
@@ -265,7 +298,7 @@ Fill these in first (in `infra/pi/.env` on the Mac; the bundle carries the file
|
||||
| `KVITT_PI_SSH_USER` | Your user on the Kvitt Pi | The user set in Raspberry Pi Imager |
|
||||
| `HA_HOST` | LAN address of the Home Assistant Pi | Router client list, or Home Assistant, Settings, System, Network |
|
||||
| `HA_PORT` | SSH port of the SSH add-on (default 22) | Home Assistant, Settings, Add-ons, the SSH add-on, Configuration, Network |
|
||||
| `HA_USER` | SSH user of the add-on (`root` for "Terminal & SSH"; the configured `username` for "Advanced SSH & Web Terminal") | Same add-on, Configuration |
|
||||
| `HA_USER` | SSH user of the add-on (`root` for "Terminal & SSH"; the configured `username` for "Advanced SSH & Web Terminal"); `root` only with the restricted key of step 3, otherwise a dedicated non-root user | Same add-on, Configuration |
|
||||
| `HA_CERT_DIR` | Directory holding `fullchain.pem` and `privkey.pem` on the Home Assistant Pi (`/ssl` on Home Assistant OS) | In the add-on's terminal: `ls -l /ssl` |
|
||||
| `KVITT_TAG` | Image tag to run: the newest `v1.0.N` the Gitea workflow pushed (or the SHA from `build-images.sh`) | Gitea, the repository, Packages, `kvitt-server` |
|
||||
| `KVITT_REGISTRY_USER`, `KVITT_REGISTRY_TOKEN` | Only if the Gitea packages are private: a Gitea user and a token with `read:package` | Gitea, Settings, Applications, Generate New Token |
|
||||
@@ -306,27 +339,25 @@ Log in again; `docker compose version` prints a version and `docker run --rm hel
|
||||
|
||||
### 3. Put the cert-sync key on the Home Assistant SSH add-on
|
||||
|
||||
Home Assistant, Settings, Add-ons, your SSH add-on, Configuration, `authorized_keys`: add **one** new entry.
|
||||
Try variant A first (replace `/ssl` if your `HA_CERT_DIR` differs):
|
||||
|
||||
Variant A (restricted, preferred):
|
||||
Home Assistant, Settings, Add-ons, your SSH add-on, Configuration, `authorized_keys`: add **one** new entry, the
|
||||
restricted key (variant A; replace `/ssl` if your `HA_CERT_DIR` differs):
|
||||
|
||||
```
|
||||
restrict,command="tar -C /ssl -cf - fullchain.pem privkey.pem" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMiXj2dU8GU0Xmdj91WxLhVpPOI1zRWvBF5h2MbE4S8V kvitt-cert-sync
|
||||
```
|
||||
|
||||
Save, restart the add-on. If the add-on refuses to save or start with it (its Log tab shows an error about the
|
||||
key), replace it with variant B:
|
||||
Optionally also limit it to the Kvitt Pi: start the line with `restrict,from="<KVITT_PI_HOST>",command=...`.
|
||||
Save, restart the add-on; its Log tab shows no error about the key.
|
||||
|
||||
Variant B (plain key):
|
||||
|
||||
```
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMiXj2dU8GU0Xmdj91WxLhVpPOI1zRWvBF5h2MbE4S8V kvitt-cert-sync
|
||||
```
|
||||
|
||||
cert-sync works with both unchanged: it always asks for exactly that `tar` command. Tell the orchestrator which
|
||||
variant the add-on accepted, so it is recorded here. The private key is
|
||||
`/Users/jhag/projects/kvitt/infra/secrets/ha_ssh_key` (git-ignored; never paste it anywhere).
|
||||
**A plain key for `root` (the old variant B) is not acceptable (D-117).** The Home Assistant SSH add-ons log in as
|
||||
root, and without `restrict,command=` whoever holds this key (a compromised cert-sync container, or a copy of the
|
||||
key on the Mac or the Pi) gets an unrestricted root shell on the Home Assistant Pi, which also owns ports 80/443
|
||||
and the house's automation. If the add-on refuses the restricted line, use a dedicated non-root user instead: a
|
||||
user with no `sudo` that can only read `HA_CERT_DIR` (set `HA_USER` to it), with the plain key line in that user's
|
||||
`authorized_keys`. If neither is possible, stop and tell the orchestrator; the fallback is Caddy's internal CA
|
||||
(SPEC §10.5.3). cert-sync works unchanged with both: it always asks for exactly that `tar` command. Tell the
|
||||
orchestrator which one the add-on accepted (restricted root key, or the dedicated user), so it is recorded here.
|
||||
The private key is `/Users/jhag/projects/kvitt/infra/secrets/ha_ssh_key` (git-ignored; never paste it anywhere).
|
||||
|
||||
### 4. Fill the `.env` and stage the secrets, on the Mac
|
||||
|
||||
@@ -373,7 +404,8 @@ root:10001; `ha_ssh_key`, `ha_known_hosts` 0600 1000:1000; `rclone.conf` 0600 10
|
||||
`Caddyfile`, `.env` (0600) and the scripts to `/opt/kvitt`, pulls, starts and waits. Expected end:
|
||||
`server: healthy`, `cert-sync: healthy`, `caddy: healthy`, a `docker compose ps` table, the line
|
||||
`GET https://icehack.asuscomm.com:8443/health (via this Pi): {"status":"ok",...}`, then "Kvitt is running.".
|
||||
Then `rm -r ~/kvitt-pi/secrets` (the script reminds you). Running `install.sh` again is safe.
|
||||
Then `rm -r ~/kvitt-pi/secrets` (the script reminds you). Running `install.sh` again is safe. Re-running
|
||||
`install.sh` keeps the `KVITT_TAG` that `deploy.sh` set; change the tag only with `deploy.sh <tag>`.
|
||||
|
||||
### 7. Router forward
|
||||
|
||||
@@ -399,7 +431,11 @@ After the Gitea workflow pushed `v1.0.N` (or `infra/build-images.sh` printed a t
|
||||
```
|
||||
|
||||
It backs up first (Drive confirmed) on every deploy, because any tag may carry a migration (SPEC §10.10), sets
|
||||
`KVITT_TAG`, pulls, starts, waits for `server: healthy` and prints `/health`. If the server does not come up:
|
||||
`KVITT_TAG`, pulls, prints the `sha256:` digest of every image and appends it to `/opt/kvitt/deployed-digests`,
|
||||
starts, waits for `server: healthy` and prints `/health`. The images come over plain HTTP from `gitea.lan`
|
||||
(D-001), so compare the printed digests with the workflow's "Print image digests" step for that run; a line
|
||||
starting `WARNING:` means a tag that was deployed before now points at a different image: do not continue, roll
|
||||
back and find out why (D-117). If the server does not come up:
|
||||
`/opt/kvitt/deploy.sh "$(cat /opt/kvitt/previous-tag)" --skip-backup`; if a migration had partially applied,
|
||||
restore the pre-deploy backup (`infra/RESTORE.md`). A newer bundle with a changed `compose.yaml` or `Caddyfile` is
|
||||
installed by copying the bundle again (step 5, without the secrets) and running `./install.sh`.
|
||||
@@ -431,8 +467,31 @@ condition from `depends_on` in `/opt/kvitt/compose.yaml`.
|
||||
| Memory | `docker stats --no-stream` (server RSS under 100 MB with both phones syncing) |
|
||||
| Disk | `df -h /` and `docker system df`; SQLite a few MB, pictures under 1 MB, APKs about 20 MB each, 5 kept |
|
||||
| No token in Caddy's log | `docker compose logs caddy \| grep -c 'kvt_'` prints `0` |
|
||||
| No invite code or name in Caddy's log | `docker compose logs caddy \| grep '"uri":"/i/' \| grep -vc '/i/REDACTED'` prints `0` |
|
||||
| Image digests deployed | `tail -n 8 /opt/kvitt/deployed-digests` |
|
||||
| Client IPs are real (once, after the router forward) | open `https://icehack.asuscomm.com:8443/health` on the phone with Wi-Fi off, then `docker compose logs --tail 20 caddy \| grep -o '"remote_ip":"[^"]*"'`: the address is your mobile network's public address, not `172.*` or `10.*`. If every request shows the Docker bridge gateway, all internet clients share one per-IP rate limit (SPEC §8.6): tell the orchestrator |
|
||||
| Restore | `infra/RESTORE.md` |
|
||||
|
||||
**APK downloads (security review L-3).** `GET /apk/files/<name>` is public (the landing page links it) and about
|
||||
11 MB per response; the only limit is the general 60 requests per minute per IP of SPEC §8.6, so one stranger can
|
||||
pull several hundred MB a minute and fill the home uplink. A lower per-IP limit for this path is a server change
|
||||
and a SPEC number that is not decided yet (owner decision below). Until then, if `docker compose logs caddy`
|
||||
shows one address fetching `/apk/files/` over and over, remove the router forward (step 7) until it stops.
|
||||
|
||||
### Owner decisions (open)
|
||||
|
||||
1. **Backup encryption (security review M-1).** The Drive backups (`kvitt-*.sqlite.gz`, `pictures-*.tar.gz`) are
|
||||
not encrypted (SPEC §8.7), and `rclone.conf` holds a token with the full `drive` scope (read and write to your
|
||||
whole Drive). Should the backups go through an rclone `crypt` remote (two passwords kept in `rclone.conf`,
|
||||
restore through the same remote) and should rclone be re-authorised with the narrower `drive.file` scope? Both
|
||||
change SPEC §8.7 and §10.9. Until you decide, the backup scripts stay as they are.
|
||||
2. **APK download limit (security review L-3).** May `/apk/files/*` get its own lower per-IP limit (for example a
|
||||
few downloads per minute per IP)? The number goes into SPEC §8.6 and the server; it is not set here.
|
||||
3. **cert-sync key (security review M-2, step 3).** Which form did the Home Assistant add-on accept: the restricted
|
||||
root key, or a dedicated non-root user?
|
||||
4. **Gitea token scope.** Does the workflow's tag push work with `kvitt_cicd` limited to `write:package`? If not,
|
||||
is adding `write:repository` acceptable?
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
* `cert-sync` unhealthy, so `caddy` does not start: `docker compose logs cert-sync`. "Host key verification
|
||||
|
||||
Reference in New Issue
Block a user