jacobh2
d9dd5848fe
- D-035 unusable mutation_id/entity_type/entity.id fails the whole push with 400; other shape errors per mutation.
- D-036 0 mutations is batch_too_large ("outside 1 to 100").
- D-037 never-synced delete runs full step-8 validation; current on every rejection; created_at kept; canonical ids.
- D-038 FCM retries: one send plus three retries (1/5/25 s), 401 drops the token, other 4xx not retried.
- D-039 pruning strictly older than 90 days; tasks::start returns AppState with the push queue, SqlAuditWriter default.
- Clarifications: step 8 order, negative since, pull read transaction, FCM JWT kid and cache, token clearing only if
unchanged, unusable credentials, scheduler clock jumps, rotate-epoch exit codes, /i/ empty code, image build needs
a C toolchain and CA roots (dev image rebuilt 2026-10-18).
- PLAN-M4 pruning wording, prompts/README status, HANDOFF at bd6f10a.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQd3pEP9UeN2uifbHknQ7B
93 lines
11 KiB
Markdown
93 lines
11 KiB
Markdown
# PLAN-M4: Server sync, audit, push, scheduler
|
|
|
|
## 1. Goal
|
|
Implement the sync protocol (pull and push), the append-only audit log with its field diff, `sync_mutations` idempotency and pruning, the FCM HTTP v1 sender, the recurring-expense scheduler and `rotate-epoch`. Every rule of §14.4 and every case of §9.5 has an integration test.
|
|
|
|
## 2. Read first
|
|
1. SPEC §14.4 Sync protocol v1: 14.4.1 Principles, 14.4.2 Pull, 14.4.3 Push, 14.4.4 Server processing of one mutation, 14.4.6 Rejection reasons, 14.4.7 Audit diff, 14.4.8 Epoch recovery and re-pairing, 14.4.9 Limits and timing (14.4.5 for the phone's expectations).
|
|
2. SPEC §14.6 Push payload v1.
|
|
3. SPEC §9.5 Recurring generation, §9.1 Splits (server check), §9.4 settlement validity, §9.6 Limits and validation, §9.10 Other edge cases.
|
|
4. SPEC §4 FR-9 Recurring expenses, FR-12 Audit log and timeline, FR-13 Push notifications, FR-15 Offline and sync.
|
|
5. SPEC §14.1 SQLite schema v1 and server-side rules; §14.3.3 Wire objects; §14.3.4 E-31, E-32.
|
|
6. SPEC §10.4 (`rotate-epoch`, `KVITT_FCM_CREDENTIALS`, `KVITT_FCM_PROJECT_ID`), §10.9 restore step 5, §6.3 (FCM auth row, Scheduler row), §8.7 Privacy and logging.
|
|
7. SPEC §11.2 (row M4), §11.3, §14.7, §14.8; §6.3 "Server seams (D-015)"; `docs/decisions.md` D-004, D-015, D-021.
|
|
8. Files: `contracts/examples/sync/*.json`, `contracts/examples/push/*.json`, `contracts/test-vectors/*.json`, the M2a code under `server/` (including the shells `src/tasks/mod.rs`, `src/audit/`, `src/push/`), M3's generation function in `src/api/recurring.rs` once M3 has landed, `CLAUDE.md`.
|
|
|
|
## 3. Already established
|
|
- Pull: 409 `cursor_ahead` when `since` > current sequence; lowest `server_version > since` across the seven tables via `UNION ALL`, tombstones included, parent with children counts as one row; `limit` default 500, max 1000; `next_since` = highest version in page or `since`; `has_more`; every array present and sorted by `server_version`; `epoch`, `server_time` (§14.4.2, §9.6).
|
|
- Push: 1 to 100 mutations (`batch_too_large`), body ≤ 1 MiB; whole request in one `BEGIN IMMEDIATE`, one `SAVEPOINT` per mutation; steps 1 to 12 of §14.4.4 in order (replay, household, shape, clock skew > now + 10 min, load, tombstone, delete always wins, validate, insert, stale, update with diff or no-op, record in `sync_mutations`); pushes enqueued after commit; results in request order (§14.4.3, §14.4.4).
|
|
- Server recomputes shares with `kvitt-domain` and rejects mismatches `validation_failed`; mode `default` bp only checked to sum to 10000 (§9.1).
|
|
- Audit diff fields, order and value types per §14.4.7; `create` lists every field with `old` null, `update` only differing fields, `delete` only `deleted_at`; `client_at` = pushed `updated_at` (delete: `deleted_at`); scheduler and importer use server now for both; entries never updated or deleted (§14.4.7, AC-12.1).
|
|
- `sync_mutations` rows older than 90 days are pruned daily at 04:00 Europe/Stockholm (DST-aware) (§14.1, §14.4.9, D-021).
|
|
- Seams (D-015): `serve` calls `tasks::start` in `src/tasks/mod.rs` (M2a's stub, now M4's); M4 starts the scheduler, the push sender and the pruning from it. M2a defined the traits `AuditWriter` (`src/audit/`) and `PushQueue` (`src/push/`) with no-op implementations; M4 replaces the implementations with the real writer and diff and the FCM queue, keeping the trait signatures (a change to them is a contract request). M3 and M9 call the traits.
|
|
- Recurring generation (D-015): the function that generates the due periods of one template is M3's, in `src/api/recurring.rs`; M4's scheduler calls it for every template with `deleted_at IS NULL` and `paused = 0`. M4 lands after M3: if M3 has not landed when the scheduler is written, write the loop against the signature in M3's report and rebase on M3 before the gate.
|
|
- Push: data-only FCM v1 message per target device, `android.priority` high, `ttl` 86400s, all values strings, types exactly the eight of §14.6, recipients per §14.6 (other user's non-revoked devices with a token; recurring to both users); more than 3 notifiable changes in one `POST /sync` → one `sync_batch`; importer sends none; no personal content (§14.6, FR-13).
|
|
- Sender: background task, bounded queue of 100 dropping the oldest with a warning; 3 attempts at 1 s, 5 s, 25 s on 429, 5xx, network errors; `UNREGISTERED` or 404 clears the device's `fcm_token`; disabled with a start-up warning when the credentials file is missing; OAuth2 via service-account JWT (RS256) at `https://oauth2.googleapis.com/token`, scope `https://www.googleapis.com/auth/firebase.messaging`, cached until 5 minutes before expiry (§14.6, §6.3).
|
|
- Scheduler: tokio task inside `kvitt-server`, runs at start and daily 06:00 Europe/Stockholm (DST-aware), and generates through M3's function (candidate periods, clamping, one transaction per candidate, split resolution, audit, `last_generated_period`, push are inside that function) (§9.5, AC-9.2 to AC-9.6).
|
|
- `rotate-epoch` sets `meta.sync_epoch` to a new `lower(hex(randomblob(16)))`, exit 0 (§10.4, §14.1).
|
|
- Environment variables only those of §10.4; the FCM endpoints are code constants, overridable only through code (constructor parameters) for the stub-server tests.
|
|
|
|
- From SPEC v0.5 (D-032): M3's rate limiter keys per-IP limits on `ConnectInfo<SocketAddr>`; M4 changes `serve` to run the router with `into_make_service_with_connect_info::<SocketAddr>()` (§11.3 note M4). Until then all clients share one IP bucket.
|
|
|
|
## 4. Owned paths
|
|
From SPEC §14.7 (v0.3): `kvitt-server`: `src/api/sync.rs`, `src/tasks/**`, `src/sync/**`, `src/audit/**`, `src/push/**`, `src/scheduler/**`, `src/cli/rotate_epoch.rs`, `tests/sync_*.rs`, `tests/scheduler_*.rs`, `tests/push_*.rs` (owner M4, shells from M2a; after landing: "later server plans").
|
|
Also from §14.7: M4 may add dependency lines to a crate's `Cargo.toml` (second to land resolves `Cargo.lock` on rebase) and one line per new module to `mod.rs` files it needs.
|
|
|
|
Edit nothing outside these paths; a needed change elsewhere is a contract request to the orchestrator.
|
|
|
|
Never edit `SPEC.md` or `docs/decisions.md`: put proposed decision entries in the report.
|
|
|
|
Resolved open points (SPEC v0.3; the start prompt may repeat them):
|
|
- (a) `serve` calls `tasks::start` (`src/tasks/mod.rs`, created by M2a, owned by M4); everything M4 runs in the background starts there (D-015).
|
|
- (b) The generation function is M3's; M4's scheduler calls it; M4 replaces the `AuditWriter` and `PushQueue` implementations that M3 calls; M4 lands after M3 (D-015).
|
|
- (c) Pruning runs daily at 04:00 Europe/Stockholm (D-021).
|
|
|
|
## 5. Deliverables
|
|
- `src/api/sync.rs`: E-31 and E-32 handlers.
|
|
- `src/sync/**`: pull query and paging, push processing of §14.4.4.
|
|
- `src/audit/**`: the real `AuditWriter` implementation and the diff of §14.4.7, used by sync, M3's generation function and the importer.
|
|
- `src/push/**`: the real `PushQueue` implementation: queue, FCM v1 sender, OAuth token cache, recipient selection, `sync_batch` aggregation.
|
|
- `src/tasks/**`: `start` launching the scheduler, the push sender and the pruning.
|
|
- `src/scheduler/**`: the daily 06:00 Europe/Stockholm tick (and one run at start) calling M3's generation function for every active template; the daily 04:00 `sync_mutations` pruning.
|
|
- `src/cli/rotate_epoch.rs`: the subcommand.
|
|
- `tests/sync_*.rs`: every rule of §14.4.2 and §14.4.4 steps 1 to 12, each rejection reason of §14.4.6, the §9.10 conflict cases, replay idempotency, batch limits, cursor_ahead, epoch change visible in pull, a 14-day/200-change offline scenario with a scripted clock (server side of AC-15.6).
|
|
- `tests/scheduler_*.rs`: the scheduler cases of §9.5: run at start, next 06:00 tick computed across both DST changes, catch-up after downtime through M3's function, paused and deleted templates skipped; the 04:00 pruning (rows with `received_at` strictly older than 90 days removed; a row exactly 90 days old and newer rows kept, D-039). The per-template generation cases are M3's tests.
|
|
- `tests/push_*.rs`: payloads for every type against `contracts/examples/push/`, recipients, `sync_batch` threshold, retries and delays, token clearing on `UNREGISTERED`/404, queue overflow, disabled without credentials, all against a local stub HTTP server.
|
|
|
|
## 6. Steps
|
|
1. Worktree. From the main checkout `/Users/jhag/projects/kvitt`: `git worktree add .claude/worktrees/agent-m4 -b agent-m4 main`, then in `.claude/worktrees/agent-m4`: `git reset --hard main`. No git-ignored files are needed (the FCM key is not used; tests use a stub). Run `tools/install-git-hooks.sh`.
|
|
2. Read §2 in order. Confirm M2a has landed; if not, stop and report.
|
|
3. Versions: crates added (reqwest with rustls, jsonwebtoken, chrono-tz, a stub HTTP server for tests if not axum itself) at latest stable from crates.io / docs.rs; confirm the FCM HTTP v1 request and error format and the OAuth2 service-account flow from Google's official Firebase documentation; record each.
|
|
4. Audit writer and diff first, with unit tests for each §14.4.7 rule.
|
|
5. Push processing (§14.4.4) step by step, a test per step and per rejection reason; then pull (§14.4.2) with paging tests.
|
|
6. Push sender with the stub server tests.
|
|
7. Scheduler with an injectable clock, calling M3's generation function; tests for the scheduler cases.
|
|
8. `rotate-epoch`; pruning at 04:00.
|
|
9. Wiring through `tasks::start` (resolved point a).
|
|
10. Run the gate (§7). Commit on `agent-m4`, `git rebase main` (resolve `Cargo.lock` if M3 landed first), rerun the gate. Never merge, push or remove the worktree.
|
|
|
|
## 7. Gate
|
|
From SPEC §11.2 (v0.3), verbatim: "Server gate; integration tests for every sync rule of §14.4 and the scheduler cases of §9.5 (run at start, daily 06:00 on DST days, catch-up through M3's generation function); FCM sender tested against a local stub HTTP server".
|
|
Command, from `server/`: `cargo fmt --check && cargo clippy -- -D warnings && cargo test`.
|
|
Green means: all three exit 0 in this session; `test result:` lines show 0 failed; the report maps each §14.4.4 step, §14.4.6 reason and §9.5 scheduler case to a test name with no gaps.
|
|
|
|
## 8. HUMAN steps
|
|
From SPEC §11.2, verbatim: "none".
|
|
|
|
## 9. Out of scope
|
|
- REST endpoints E-2 to E-30 and E-33, landing page, pictures, rate limiting, `bootstrap`, `pair-device` (M3).
|
|
- The importer (M9); real FCM sends to devices (M8, M10); `infra/`; Android.
|
|
- Changing the sync protocol or push payload (contract change).
|
|
|
|
## 10. Report
|
|
- Branch `agent-m4`, head commit hash after rebasing on `main`.
|
|
- Gate output: fmt and clippy exit status, every `test result:` line.
|
|
- The rule → test mapping (§14.4.2, §14.4.4 steps, §14.4.6 reasons, §9.5 cases, §14.6 sender rules).
|
|
- Versions of crates added, with sources; FCM documentation pages consulted.
|
|
- How resolved points (a) to (c) were applied.
|
|
- Proposed decision entries (never written to `SPEC.md` or `docs/decisions.md` by this agent) or `TODO(spec)` occurrences.
|
|
- Contract requests, open questions, spec gaps (with section numbers).
|
|
|
|
## 11. Numbering
|
|
Decision ids, spec version and schema version are assigned by the orchestrator in the start prompt; use `TODO(spec)` if none was given.
|