269 lines
20 KiB
Markdown
269 lines
20 KiB
Markdown
# Data Refresh Plan — Försäkringskassan BankID Scraper + `login` Test Harness
|
||
|
||
**Owner:** Senior developer
|
||
**Status:** Ready to implement
|
||
**Last updated:** 2026-06-12
|
||
|
||
This plan covers the **data-acquisition (scraping) part only**: a headless browser that logs into Försäkringskassan via BankID (QR code), shows the QR to the user so they can scan it, then navigates and pulls the parental-leave figures. It also covers a local **`login` test harness** (a dev web page with a "Refresh" button → QR → fetched data) so we can iterate without ceremony.
|
||
|
||
It deliberately slots into the existing architecture rather than inventing new contracts. The interfaces, entities, and endpoint stubs already exist — this plan fills in the real implementation behind them.
|
||
|
||
---
|
||
|
||
## 0. Decisions locked for this work
|
||
|
||
These were confirmed with the product owner; do not re-litigate without checking back.
|
||
|
||
1. **Scraper runtime = Playwright for Python, inside the existing backend.** The implementation lives in `backend/app/scrapers/fk_scraper.py` behind the existing `FetchDataService` / `FKScraperService` contract. We do **not** spin up a separate Node scraper service. `npm run login` is a thin dev convenience wrapper, not a second implementation.
|
||
2. **QR handling = relay Försäkringskassan's own rendered QR.** FK controls the BankID session; its login page renders the animated QR itself. We screenshot / extract that QR element and stream it to our page. We do **not** integrate with the BankID API directly, start our own BankID order, or handle order references / certificates. This keeps us out of the BankID protocol entirely.
|
||
3. **Page captures from product owner = HTML + PNG per page**, plus a one-line note of which element to click and the URL/hash at that step. See §6 for the exact capture protocol.
|
||
4. **Legal gate still applies.** Scraping ships disabled (`settings.scraper_enabled = False`) until the legal/ToS + BankID-automation review in ARCHITECTURE §2.3 clears. This plan builds the capability; it does not flip the switch for public launch. Local/dev testing runs with the flag on in a dev profile only.
|
||
|
||
---
|
||
|
||
## 1. Where this fits in the existing codebase
|
||
|
||
The scaffolding is already in place — read these before writing code:
|
||
|
||
| File | What's there now | What this plan adds |
|
||
|---|---|---|
|
||
| `backend/app/scrapers/fk_scraper.py` | `FKScraperService` stub: `start_bankid_session`, `poll_session`, `cancel_session`, `get_latest`, `list_history` all raise `NotImplementedError` behind a legal-gate message. | Real Playwright implementation of the session lifecycle + extraction. |
|
||
| `backend/app/services/fetch_data_service.py` | `FetchDataService` ABC + `ManualFetchService` (the always-works fallback). | No change to the contract. `FKScraperService` must satisfy the same ABC so the rest of the app stays decoupled from FK's DOM. |
|
||
| `backend/app/routers/fetch_data.py` | `/api/fetch-data/start|poll|cancel` return **501** when `scraper_enabled` is false. Manual endpoints are live. | Wire these three to the real `FKScraperService`; keep the 501 guard for when the flag is off. |
|
||
| `backend/app/entities/parental_leave.py` | `ParentalLeaveFetch` + `TierData` — the **stable contract** the scraper must produce. | Scraper outputs this exact shape with `source='scraped'`. |
|
||
| `backend/app/config.py` | `scraper_enabled: bool = False`. | Add the scraper config block in §7 (timeouts, headless flag, dev profile). |
|
||
| `backend/app/models/parental_leave.py` | Pydantic request/response schemas incl. field bounds (`total_days` 0–480, `dubbeldagar_left` 0–60, etc.). | Reuse for validating scraped output before persisting. |
|
||
|
||
**Golden rule (from ARCHITECTURE §2.3):** every FK-specific selector and flow detail lives *only* inside `fk_scraper.py`. The rest of the app depends on `ParentalLeaveFetch`, never on FK's markup. If FK changes their site, exactly one file should need editing.
|
||
|
||
---
|
||
|
||
## 2. The data we must extract (target contract)
|
||
|
||
The scraper's job is to populate a `ParentalLeaveFetch` (see `entities/parental_leave.py`). These are the fields and where they come from on FK's "Ansök och planera föräldrapenning" pages. Use the captures (§6) to confirm exact Swedish labels and the page each lives on.
|
||
|
||
| Field | Meaning | Notes |
|
||
|---|---|---|
|
||
| `total_days` | Total föräldrapenning days for the child | Per parent; whole days only (no fractional). |
|
||
| `days_used` | Days already taken | |
|
||
| `days_left` | Derived = `total_days - days_used` | Don't trust a displayed "left" blindly — recompute and cross-check; flag mismatch. |
|
||
| `reserved_days_total` | Reserved (non-transferable) days, default 90 | |
|
||
| `reserved_days_left` | Reserved days remaining | Must be ≤ `reserved_days_total`. |
|
||
| `transferable_days_left` | Transferable days remaining | |
|
||
| `dubbeldagar_left` | Double-days remaining, max 60 (first 15 months only) | |
|
||
| `tier_data.sjukpenning_total/used/left` | Sjukpenningnivå tier (the ~390 income-based days) | `left = total - used`. |
|
||
| `tier_data.lagsta_niva_total/used/left` | Lägstanivå tier (the 90 flat-rate days) | `left = total - used`. |
|
||
| `source` | `'scraped'` | Set by scraper. Manual entry uses `'manual'`. |
|
||
| `fetched_at` | Timestamp of this fetch | |
|
||
|
||
**Validation before persisting:** reuse the bounds in `ManualFetchRequest` and the invariants `ManualFetchService.save_manual_entry` already enforces (`days_left >= 0`, `reserved_days_left <= reserved_days_total`). If FK's numbers violate an invariant, do **not** silently persist — return an `error` status with a clear message so we can fix the parser. Scraping a wrong number into the planner is worse than failing loudly.
|
||
|
||
---
|
||
|
||
## 3. Session lifecycle / state machine
|
||
|
||
Because FK owns the BankID session, our flow is a long-lived, polled session that mirrors what FK shows. One headless browser context per session.
|
||
|
||
```
|
||
start ──▶ launching ──▶ awaiting_scan ──▶ authenticating ──▶ fetching ──▶ done
|
||
│ │ │
|
||
└──────────────────┴────────────────┴────▶ error / cancelled / timeout
|
||
```
|
||
|
||
**Statuses returned to the client (`poll`):**
|
||
|
||
- `launching` — browser starting, navigating to the login URL.
|
||
- `awaiting_scan` — QR is being shown. `poll` returns the **current** QR image (base64 PNG). The QR animates (BankID refreshes it roughly once per second), so each `poll` re-captures it.
|
||
- `authenticating` — user scanned; BankID/FK is completing auth.
|
||
- `fetching` — authenticated; scraper is navigating FK pages and reading values.
|
||
- `done` — `result` contains the `ParentalLeaveFetchResponse`. Browser context is torn down and cookies/session discarded.
|
||
- `error` — `message` explains what broke (selector miss, timeout, validation failure). Browser torn down.
|
||
- `cancelled` — user/cancel endpoint stopped it. Browser torn down.
|
||
|
||
**Endpoint contracts** (already stubbed in `routers/fetch_data.py`):
|
||
|
||
```
|
||
POST /api/fetch-data/start
|
||
body: { workspace_id, personal_data_uuid, child_id }
|
||
→ 200 { session_id, status: "launching" }
|
||
→ 501 if scraper_enabled is false
|
||
|
||
GET /api/fetch-data/poll?session_id=...
|
||
→ 200 { status, qr_data?: base64png, result?: ParentalLeaveFetchResponse, message?: string }
|
||
|
||
POST /api/fetch-data/cancel
|
||
body: { session_id }
|
||
→ 200 { status: "cancelled" }
|
||
```
|
||
|
||
**Session store:** in-memory dict `session_id -> SessionState` (Playwright context handle + status + latest QR bytes + result + created_at). This is fine because the scraper runs in a single sandboxed worker and sessions are ephemeral. **Never persist** the context, cookies, or QR to disk or DB.
|
||
|
||
**Timeouts:** auto-cancel any session that sits in `awaiting_scan`/`authenticating` longer than a configurable TTL (default 180s) and tear down the browser. Belt-and-suspenders against orphaned headless Chromiums.
|
||
|
||
**Concurrency for v1:** one active scrape session at a time is acceptable for the test harness. Note it as a known limitation; a session pool/queue is a later concern.
|
||
|
||
---
|
||
|
||
## 4. Playwright login + navigation flow
|
||
|
||
Implement in `fk_scraper.py`. Pseudocode of the happy path — exact selectors come from the §6 captures.
|
||
|
||
```python
|
||
# 1. Launch
|
||
browser = await playwright.chromium.launch(headless=settings.scraper_headless)
|
||
context = await browser.new_context(locale="sv-SE") # Swedish UI for stable labels
|
||
page = await context.new_page()
|
||
|
||
# 2. Go to login, choose BankID med QR-kod
|
||
await page.goto(LOGIN_URL, wait_until="networkidle") # the goto= deep link from §5
|
||
await page.get_by_role("button", name="BankID med QR-kod").click() # confirm via capture
|
||
|
||
# 3. Relay the QR (loop while status == awaiting_scan)
|
||
qr = page.locator(QR_SELECTOR) # the <img>/<canvas> FK renders — confirm via capture
|
||
png = await qr.screenshot() # re-capture each poll; QR animates
|
||
session.qr_data = base64(png)
|
||
|
||
# 4. Detect auth success — wait for the post-login landing element to appear
|
||
await page.wait_for_selector(POST_LOGIN_MARKER, timeout=...) # transitions us to "fetching"
|
||
|
||
# 5. Navigate to the parental-leave figures and read them
|
||
# (FK is a SPA using hash routing — prefer in-app nav/clicks over raw goto where possible,
|
||
# and wait_for_selector on the data element, not arbitrary sleeps)
|
||
data = await extract_parental_leave(page) # returns dict mapped to ParentalLeaveFetch
|
||
|
||
# 6. Tear down — always, even on error
|
||
await context.close(); await browser.close()
|
||
```
|
||
|
||
**SPA notes (important):** the login URL uses hash routing (`.../logga-in#/?goto=...`), so FK is a single-page app (React/Angular-style). Two consequences:
|
||
- Don't rely on full-page loads between steps. Use `wait_for_selector` / `expect(...).to_be_visible()` on the specific element you need, never `sleep(n)`.
|
||
- Re-navigation by changing the hash may not trigger what you expect; prefer clicking the in-app navigation elements the user identifies in the captures.
|
||
|
||
**QR selector:** confirm from the capture whether FK renders the QR as an `<img src="data:image/png;base64,...">`, a `<canvas>`, or an `<svg>`. Screenshotting the element handles all three uniformly and is the most robust (decision §0.2), so default to `locator.screenshot()` rather than reading the src.
|
||
|
||
---
|
||
|
||
## 5. The login URL
|
||
|
||
Target deep link (from product owner):
|
||
|
||
```
|
||
https://www.forsakringskassan.se/privatperson/logga-in#/?goto=%2Fprivatperson%2Fmina-sidor%2Falla-e-tjanster%2Fansok-och-planera-foraldrapenning
|
||
```
|
||
|
||
The `goto` param routes the user, after BankID auth, straight to "Ansök och planera föräldrapenning." Keep this URL as a single constant at the top of `fk_scraper.py`. If FK changes the routing, this plus the selectors are the only things to update.
|
||
|
||
---
|
||
|
||
## 6. What I need from you (page-capture protocol)
|
||
|
||
This is the highest-leverage thing you can give me to make the scraper succeed first try. For **each page in the flow**, capture all three of:
|
||
|
||
1. **HTML** — the full rendered DOM at that moment. Best method: open DevTools → Elements → right-click `<html>` (or the relevant container) → **Copy → Copy outerHTML**, paste into a `.html` file. (Plain "Save page as → Webpage, Complete" also works but DevTools outerHTML reflects the live SPA state better.) This is what I write selectors from — **prefer pages where I can see `data-testid`, `id`, `name`, or `aria-label` attributes**.
|
||
2. **PNG** — a screenshot of the same page (full page if possible).
|
||
3. **A one-line note** with: (a) the **URL/hash** in the address bar, (b) **which element to click** to get to the next step (describe it: visible text + roughly where), and (c) anything conditional ("this dialog only appears the first time").
|
||
|
||
**Naming convention** so I can order them:
|
||
|
||
```
|
||
captures/
|
||
01_login_landing.html / .png
|
||
02_after_bankid_qr_button.html / .png ← the QR screen
|
||
03_after_bankid_qr_scanned.html / .png
|
||
04_post_login_landing.html / .png ← where total/used/left live including sjukpenningnivå vs lägstanivå
|
||
05_foraldrapenning_overview.html / .png ← where total/used/left live including sjukpenningnivå vs lägstanivå including for user vs total for child (to know what the user has vs the partner has)
|
||
```
|
||
|
||
**Pages I specifically need** (map to the data in §2): the login landing (so I find the "BankID med QR-kod" button), the QR screen (so I find the QR element), the post-login landing (so I know auth succeeded), and every page that shows: total/used/remaining days, the reserved-vs-transferable split, dubbeldagar remaining, and the sjukpenningnivå-vs-lägstanivå tier breakdown.
|
||
|
||
**Privacy when capturing:** these are *your own* real pages, so **redact before sharing** — black out / overwrite your personnummer, name, and the child's identifiers in both the PNG and HTML. I only need the **structure and the Swedish labels**, not your actual values. (If it's easier, replace digits with `0`s — keep the surrounding markup intact.) Do not commit captures with real personal data to git; keep `captures/` git-ignored.
|
||
|
||
---
|
||
|
||
## 7. The `login` test harness (`npm run login`)
|
||
|
||
Goal: one command spins up everything, you open a dev page, click **Refresh**, the **QR appears**, you scan, and the **fetched data renders**. Used for iterating on the scraper, not a shipped feature.
|
||
|
||
**Command** (in `frontend/package.json`):
|
||
|
||
```jsonc
|
||
"scripts": {
|
||
"login": "concurrently -k -n backend,web \"npm:login:backend\" \"npm:login:web\"",
|
||
"login:backend": "cd ../backend && SCRAPER_ENABLED=true SCRAPER_HEADLESS=true uv run uvicorn app.main:app --reload --port 8000",
|
||
"login:web": "vite --open /dev/fetch"
|
||
}
|
||
```
|
||
|
||
Add `concurrently` to frontend devDependencies. (If you'd rather not add it, a tiny `scripts/login.mjs` that spawns both processes works too — implementer's choice.)
|
||
|
||
**Dev-only page `/dev/fetch`:**
|
||
- A single route, mounted **only when `import.meta.env.DEV`** (never bundled into production builds).
|
||
- A **Refresh** button → `POST /api/fetch-data/start` → then polls `GET /api/fetch-data/poll?session_id=...` every ~1s.
|
||
- Renders by status: `awaiting_scan` → show `<img src="data:image/png;base64,{qr_data}">` (refreshes each poll as the QR animates); `authenticating`/`fetching` → spinner + status text; `done` → pretty-print the `ParentalLeaveFetchResponse` JSON in a `<pre>`; `error` → show `message`.
|
||
- A **Cancel** button → `POST /api/fetch-data/cancel`.
|
||
- Keep it dumb and dependency-free — it's a harness. This is also the seed for the eventual real `QRCodeModal.tsx` (referenced in ARCHITECTURE §UI), but don't gold-plate it yet.
|
||
|
||
**Dev profile note:** `SCRAPER_ENABLED=true` here only flips the local dev backend. The committed default in `config.py` stays `False`. Document in the harness page header that this is running the *real* FK scraper against the *real* site with *your* BankID — so only run it intentionally.
|
||
|
||
**Auth:** the scraper endpoints require `CurrentUserDep`. The dev page needs a logged-in app session (the app's own auth, not BankID) to call them. Either log in through the normal app flow first, or add a dev-only token shortcut — note which you chose.
|
||
|
||
---
|
||
|
||
## 8. Security & legal guardrails (non-negotiable, from §2.3)
|
||
|
||
- **Never store BankID credentials or the FK session.** No cookies, tokens, or QR payloads to disk or DB. The browser context is discarded the moment the fetch completes or fails.
|
||
- **Act only on the authenticated user's own data, with active consent each session.** No background/scheduled scraping. A session starts only from an explicit user action.
|
||
- **Sandbox the browser.** Run Playwright/Chromium in a dedicated, locked-down container in any non-local environment (own network egress rules, no access to other services, ephemeral filesystem). Document the container setup as part of this work.
|
||
- **Log no personal data.** Logs may record status transitions and timings, never names, personnummer, day counts, or page content. Scrub before logging.
|
||
- **Keep the flag off for public launch** until the legal/ToS + BankID-automation review clears (§2.3, open question §1372 in ARCHITECTURE). This work delivers the *capability*; the go/no-go is a separate product+legal decision.
|
||
- **Manual fallback stays first-class.** The planner must remain fully usable with `ManualFetchService` alone if scraping is disabled, blocked, or broken. Don't introduce any hard dependency on the scraper.
|
||
|
||
---
|
||
|
||
## 9. Robustness & maintainability
|
||
|
||
- **One file owns FK's DOM.** All selectors/flow in `fk_scraper.py`. Consider a small `_selectors.py` constant block at the top so a site change is a one-place edit.
|
||
- **Selector priority:** role/label/test-id > stable `id`/`name` > structural CSS/XPath. Avoid brittle nth-child chains and auto-generated class hashes.
|
||
- **No fixed sleeps.** Always `wait_for_selector` / `expect().to_be_visible()` with sane timeouts.
|
||
- **Fail loud, fail clean.** On any selector miss or invariant violation, set status `error` with a specific message and tear down. Never persist partial/guessed data.
|
||
- **Swedish locale** on the context so labels match the captures.
|
||
- **Idempotent teardown** — closing the context/browser must run in a `finally` so a crash never leaks a Chromium process.
|
||
|
||
---
|
||
|
||
## 10. Testing strategy
|
||
|
||
- **Unit-test the extractor against saved HTML fixtures.** Take the captured `.html` files (redacted), load them into a Playwright page via `set_content` (or parse with the same selectors), and assert the extractor produces the right `ParentalLeaveFetch`. This is the test that protects us from FK markup changes and runs in CI with **no network and no BankID**.
|
||
- **Unit-test the validation layer** — feed out-of-bounds / inconsistent numbers, assert it raises rather than persists.
|
||
- **Unit-test the session state machine** — transitions, timeout auto-cancel, double-cancel safety — with the browser mocked.
|
||
- **Manual E2E via `npm run login`** — the only step that needs a real BankID; done by hand, not in CI.
|
||
- Keep the existing `ManualFetchService` tests green; the contract must not change.
|
||
|
||
---
|
||
|
||
## 11. Build order / milestones
|
||
|
||
1. **Add config** (`scraper_headless`, `scraper_session_ttl_seconds`, dev profile) to `config.py`; add Playwright to `backend/pyproject.toml` dev deps and document `playwright install chromium`.
|
||
2. **Session state machine + endpoints** wired to `FKScraperService`, with the QR-relay and poll contract — using a **mocked/stub page** first so the harness works end-to-end before real FK selectors exist.
|
||
3. **`npm run login` harness + `/dev/fetch` page** — prove the start→poll→QR→result loop against the stub.
|
||
4. **Receive captures (§6)** → implement the real login + QR-relay against FK.
|
||
5. **Implement the extractor** for each data field (§2) → fixture tests from the captures.
|
||
6. **Real E2E** via the harness with a live BankID scan; iterate on selectors.
|
||
7. **Security/sandbox container** setup + the §8 review checklist; confirm flag stays off for prod.
|
||
|
||
Milestones 2–3 are unblocked **today** (they don't need the captures). Milestones 4–6 are blocked on the §6 captures — that's the critical path, so prioritise producing those.
|
||
|
||
---
|
||
|
||
## 12. Open questions to resolve as we go
|
||
|
||
- Exact Swedish labels and which page each datapoint sits on — answered by the captures.
|
||
- Does FK gate "Ansök och planera föräldrapenning" behind an extra consent/child-selection step? If so, capture that page too (likely `03b_...`).
|
||
- Multiple children: does the overview list per-child, requiring a child selection that we map to `child_id`? Capture the selector if so.
|
||
- Whether the QR is `<img>`/`<canvas>`/`<svg>` — we default to element screenshot regardless, but good to know.
|
||
|
||
---
|
||
|
||
### TL;DR for the implementer
|
||
Build the real `FKScraperService` with Playwright-Python behind the existing `FetchDataService` contract, driving FK's own BankID-QR login and **relaying FK's rendered QR** (no BankID API). Expose it through the already-stubbed `start/poll/cancel` endpoints as a polled session state machine. Add a dev-only `npm run login` harness (`/dev/fetch` page: Refresh → QR → JSON result) so we can iterate. Keep every FK selector in one file, never persist credentials/session, fail loud on bad data, and keep the manual fallback first-class. Ships disabled until legal sign-off. Milestones 2–3 start now; 4–6 wait on the page captures (§6) — that's the critical path.
|