jacobh2/F-r-ldraledigplannerare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d556ce0f6f486a8ba5fde5218052f0137e554305
F-r-ldraledigplannerare/data_refresh_plan.md
T
jacobh2 772d853e37 MVP frontend+backend
2026-06-14 09:58:30 +02:00

20 KiB
Raw Blame History

Data Refresh Plan — Försäkringskassan BankID Scraper + login Test Harness

Owner: Senior developer Status: Ready to implement Last updated: 2026-06-12

This plan covers the data-acquisition (scraping) part only: a headless browser that logs into Försäkringskassan via BankID (QR code), shows the QR to the user so they can scan it, then navigates and pulls the parental-leave figures. It also covers a local login test harness (a dev web page with a "Refresh" button → QR → fetched data) so we can iterate without ceremony.

It deliberately slots into the existing architecture rather than inventing new contracts. The interfaces, entities, and endpoint stubs already exist — this plan fills in the real implementation behind them.

0. Decisions locked for this work

These were confirmed with the product owner; do not re-litigate without checking back.

  1. Scraper runtime = Playwright for Python, inside the existing backend. The implementation lives in backend/app/scrapers/fk_scraper.py behind the existing FetchDataService / FKScraperService contract. We do not spin up a separate Node scraper service. npm run login is a thin dev convenience wrapper, not a second implementation.
  2. QR handling = relay Försäkringskassan's own rendered QR. FK controls the BankID session; its login page renders the animated QR itself. We screenshot / extract that QR element and stream it to our page. We do not integrate with the BankID API directly, start our own BankID order, or handle order references / certificates. This keeps us out of the BankID protocol entirely.
  3. Page captures from product owner = HTML + PNG per page, plus a one-line note of which element to click and the URL/hash at that step. See §6 for the exact capture protocol.
  4. Legal gate still applies. Scraping ships disabled (settings.scraper_enabled = False) until the legal/ToS + BankID-automation review in ARCHITECTURE §2.3 clears. This plan builds the capability; it does not flip the switch for public launch. Local/dev testing runs with the flag on in a dev profile only.

1. Where this fits in the existing codebase

The scaffolding is already in place — read these before writing code:

File What's there now What this plan adds
backend/app/scrapers/fk_scraper.py FKScraperService stub: start_bankid_session, poll_session, cancel_session, get_latest, list_history all raise NotImplementedError behind a legal-gate message. Real Playwright implementation of the session lifecycle + extraction.
backend/app/services/fetch_data_service.py FetchDataService ABC + ManualFetchService (the always-works fallback). No change to the contract. FKScraperService must satisfy the same ABC so the rest of the app stays decoupled from FK's DOM.
backend/app/routers/fetch_data.py `/api/fetch-data/start poll
backend/app/entities/parental_leave.py ParentalLeaveFetch + TierData — the stable contract the scraper must produce. Scraper outputs this exact shape with source='scraped'.
backend/app/config.py scraper_enabled: bool = False. Add the scraper config block in §7 (timeouts, headless flag, dev profile).
backend/app/models/parental_leave.py Pydantic request/response schemas incl. field bounds (total_days 0–480, dubbeldagar_left 0–60, etc.). Reuse for validating scraped output before persisting.

Golden rule (from ARCHITECTURE §2.3): every FK-specific selector and flow detail lives only inside fk_scraper.py. The rest of the app depends on ParentalLeaveFetch, never on FK's markup. If FK changes their site, exactly one file should need editing.

2. The data we must extract (target contract)

The scraper's job is to populate a ParentalLeaveFetch (see entities/parental_leave.py). These are the fields and where they come from on FK's "Ansök och planera föräldrapenning" pages. Use the captures (§6) to confirm exact Swedish labels and the page each lives on.

Field Meaning Notes
total_days Total föräldrapenning days for the child Per parent; whole days only (no fractional).
days_used Days already taken
days_left Derived = total_days - days_used Don't trust a displayed "left" blindly — recompute and cross-check; flag mismatch.
reserved_days_total Reserved (non-transferable) days, default 90
reserved_days_left Reserved days remaining Must be ≤ reserved_days_total.
transferable_days_left Transferable days remaining
dubbeldagar_left Double-days remaining, max 60 (first 15 months only)
tier_data.sjukpenning_total/used/left Sjukpenningnivå tier (the ~390 income-based days) left = total - used.
tier_data.lagsta_niva_total/used/left Lägstanivå tier (the 90 flat-rate days) left = total - used.
source 'scraped' Set by scraper. Manual entry uses 'manual'.
fetched_at Timestamp of this fetch

Validation before persisting: reuse the bounds in ManualFetchRequest and the invariants ManualFetchService.save_manual_entry already enforces (days_left >= 0, reserved_days_left <= reserved_days_total). If FK's numbers violate an invariant, do not silently persist — return an error status with a clear message so we can fix the parser. Scraping a wrong number into the planner is worse than failing loudly.

3. Session lifecycle / state machine

Because FK owns the BankID session, our flow is a long-lived, polled session that mirrors what FK shows. One headless browser context per session.

start ──▶ launching ──▶ awaiting_scan ──▶ authenticating ──▶ fetching ──▶ done
                              │                  │                │
                              └──────────────────┴────────────────┴────▶ error / cancelled / timeout

Statuses returned to the client (poll):

  • launching — browser starting, navigating to the login URL.
  • awaiting_scan — QR is being shown. poll returns the current QR image (base64 PNG). The QR animates (BankID refreshes it roughly once per second), so each poll re-captures it.
  • authenticating — user scanned; BankID/FK is completing auth.
  • fetching — authenticated; scraper is navigating FK pages and reading values.
  • done — result contains the ParentalLeaveFetchResponse. Browser context is torn down and cookies/session discarded.
  • error — message explains what broke (selector miss, timeout, validation failure). Browser torn down.
  • cancelled — user/cancel endpoint stopped it. Browser torn down.

Endpoint contracts (already stubbed in routers/fetch_data.py):

POST /api/fetch-data/start
  body: { workspace_id, personal_data_uuid, child_id }
  → 200 { session_id, status: "launching" }
  → 501 if scraper_enabled is false

GET  /api/fetch-data/poll?session_id=...
  → 200 { status, qr_data?: base64png, result?: ParentalLeaveFetchResponse, message?: string }

POST /api/fetch-data/cancel
  body: { session_id }
  → 200 { status: "cancelled" }

Session store: in-memory dict session_id -> SessionState (Playwright context handle + status + latest QR bytes + result + created_at). This is fine because the scraper runs in a single sandboxed worker and sessions are ephemeral. Never persist the context, cookies, or QR to disk or DB.

Timeouts: auto-cancel any session that sits in awaiting_scan/authenticating longer than a configurable TTL (default 180s) and tear down the browser. Belt-and-suspenders against orphaned headless Chromiums.

Concurrency for v1: one active scrape session at a time is acceptable for the test harness. Note it as a known limitation; a session pool/queue is a later concern.

4. Playwright login + navigation flow

Implement in fk_scraper.py. Pseudocode of the happy path — exact selectors come from the §6 captures.

# 1. Launch
browser = await playwright.chromium.launch(headless=settings.scraper_headless)
context = await browser.new_context(locale="sv-SE")   # Swedish UI for stable labels
page = await context.new_page()

# 2. Go to login, choose BankID med QR-kod
await page.goto(LOGIN_URL, wait_until="networkidle")   # the goto= deep link from §5
await page.get_by_role("button", name="BankID med QR-kod").click()   # confirm via capture

# 3. Relay the QR (loop while status == awaiting_scan)
qr = page.locator(QR_SELECTOR)            # the <img>/<canvas> FK renders — confirm via capture
png = await qr.screenshot()               # re-capture each poll; QR animates
session.qr_data = base64(png)

# 4. Detect auth success — wait for the post-login landing element to appear
await page.wait_for_selector(POST_LOGIN_MARKER, timeout=...)   # transitions us to "fetching"

# 5. Navigate to the parental-leave figures and read them
#    (FK is a SPA using hash routing — prefer in-app nav/clicks over raw goto where possible,
#     and wait_for_selector on the data element, not arbitrary sleeps)
data = await extract_parental_leave(page)   # returns dict mapped to ParentalLeaveFetch

# 6. Tear down — always, even on error
await context.close(); await browser.close()

SPA notes (important): the login URL uses hash routing (.../logga-in#/?goto=...), so FK is a single-page app (React/Angular-style). Two consequences:

  • Don't rely on full-page loads between steps. Use wait_for_selector / expect(...).to_be_visible() on the specific element you need, never sleep(n).
  • Re-navigation by changing the hash may not trigger what you expect; prefer clicking the in-app navigation elements the user identifies in the captures.

QR selector: confirm from the capture whether FK renders the QR as an <img src="data:image/png;base64,...">, a <canvas>, or an <svg>. Screenshotting the element handles all three uniformly and is the most robust (decision §0.2), so default to locator.screenshot() rather than reading the src.

5. The login URL

Target deep link (from product owner):

https://www.forsakringskassan.se/privatperson/logga-in#/?goto=%2Fprivatperson%2Fmina-sidor%2Falla-e-tjanster%2Fansok-och-planera-foraldrapenning

The goto param routes the user, after BankID auth, straight to "Ansök och planera föräldrapenning." Keep this URL as a single constant at the top of fk_scraper.py. If FK changes the routing, this plus the selectors are the only things to update.

6. What I need from you (page-capture protocol)

This is the highest-leverage thing you can give me to make the scraper succeed first try. For each page in the flow, capture all three of:

  1. HTML — the full rendered DOM at that moment. Best method: open DevTools → Elements → right-click <html> (or the relevant container) → Copy → Copy outerHTML, paste into a .html file. (Plain "Save page as → Webpage, Complete" also works but DevTools outerHTML reflects the live SPA state better.) This is what I write selectors from — prefer pages where I can see data-testid, id, name, or aria-label attributes.
  2. PNG — a screenshot of the same page (full page if possible).
  3. A one-line note with: (a) the URL/hash in the address bar, (b) which element to click to get to the next step (describe it: visible text + roughly where), and (c) anything conditional ("this dialog only appears the first time").

Naming convention so I can order them:

captures/
  01_login_landing.html / .png
  02_after_bankid_qr_button.html / .png   ← the QR screen
  03_after_bankid_qr_scanned.html / .png
  04_post_login_landing.html / .png ← where total/used/left live including sjukpenningnivå vs lägstanivå
  05_foraldrapenning_overview.html / .png ← where total/used/left live including sjukpenningnivå vs lägstanivå including for user vs total for child (to know what the user has vs the partner has)

Pages I specifically need (map to the data in §2): the login landing (so I find the "BankID med QR-kod" button), the QR screen (so I find the QR element), the post-login landing (so I know auth succeeded), and every page that shows: total/used/remaining days, the reserved-vs-transferable split, dubbeldagar remaining, and the sjukpenningnivå-vs-lägstanivå tier breakdown.

Privacy when capturing: these are your own real pages, so redact before sharing — black out / overwrite your personnummer, name, and the child's identifiers in both the PNG and HTML. I only need the structure and the Swedish labels, not your actual values. (If it's easier, replace digits with 0s — keep the surrounding markup intact.) Do not commit captures with real personal data to git; keep captures/ git-ignored.

7. The login test harness (npm run login)

Goal: one command spins up everything, you open a dev page, click Refresh, the QR appears, you scan, and the fetched data renders. Used for iterating on the scraper, not a shipped feature.

Command (in frontend/package.json):

"scripts": {
  "login": "concurrently -k -n backend,web \"npm:login:backend\" \"npm:login:web\"",
  "login:backend": "cd ../backend && SCRAPER_ENABLED=true SCRAPER_HEADLESS=true uv run uvicorn app.main:app --reload --port 8000",
  "login:web": "vite --open /dev/fetch"
}

Add concurrently to frontend devDependencies. (If you'd rather not add it, a tiny scripts/login.mjs that spawns both processes works too — implementer's choice.)

Dev-only page /dev/fetch:

  • A single route, mounted only when import.meta.env.DEV (never bundled into production builds).
  • A Refresh button → POST /api/fetch-data/start → then polls GET /api/fetch-data/poll?session_id=... every ~1s.
  • Renders by status: awaiting_scan → show <img src="data:image/png;base64,{qr_data}"> (refreshes each poll as the QR animates); authenticating/fetching → spinner + status text; done → pretty-print the ParentalLeaveFetchResponse JSON in a <pre>; error → show message.
  • A Cancel button → POST /api/fetch-data/cancel.
  • Keep it dumb and dependency-free — it's a harness. This is also the seed for the eventual real QRCodeModal.tsx (referenced in ARCHITECTURE §UI), but don't gold-plate it yet.

Dev profile note: SCRAPER_ENABLED=true here only flips the local dev backend. The committed default in config.py stays False. Document in the harness page header that this is running the real FK scraper against the real site with your BankID — so only run it intentionally.

Auth: the scraper endpoints require CurrentUserDep. The dev page needs a logged-in app session (the app's own auth, not BankID) to call them. Either log in through the normal app flow first, or add a dev-only token shortcut — note which you chose.

8. Security & legal guardrails (non-negotiable, from §2.3)

  • Never store BankID credentials or the FK session. No cookies, tokens, or QR payloads to disk or DB. The browser context is discarded the moment the fetch completes or fails.
  • Act only on the authenticated user's own data, with active consent each session. No background/scheduled scraping. A session starts only from an explicit user action.
  • Sandbox the browser. Run Playwright/Chromium in a dedicated, locked-down container in any non-local environment (own network egress rules, no access to other services, ephemeral filesystem). Document the container setup as part of this work.
  • Log no personal data. Logs may record status transitions and timings, never names, personnummer, day counts, or page content. Scrub before logging.
  • Keep the flag off for public launch until the legal/ToS + BankID-automation review clears (§2.3, open question §1372 in ARCHITECTURE). This work delivers the capability; the go/no-go is a separate product+legal decision.
  • Manual fallback stays first-class. The planner must remain fully usable with ManualFetchService alone if scraping is disabled, blocked, or broken. Don't introduce any hard dependency on the scraper.

9. Robustness & maintainability

  • One file owns FK's DOM. All selectors/flow in fk_scraper.py. Consider a small _selectors.py constant block at the top so a site change is a one-place edit.
  • Selector priority: role/label/test-id > stable id/name > structural CSS/XPath. Avoid brittle nth-child chains and auto-generated class hashes.
  • No fixed sleeps. Always wait_for_selector / expect().to_be_visible() with sane timeouts.
  • Fail loud, fail clean. On any selector miss or invariant violation, set status error with a specific message and tear down. Never persist partial/guessed data.
  • Swedish locale on the context so labels match the captures.
  • Idempotent teardown — closing the context/browser must run in a finally so a crash never leaks a Chromium process.

10. Testing strategy

  • Unit-test the extractor against saved HTML fixtures. Take the captured .html files (redacted), load them into a Playwright page via set_content (or parse with the same selectors), and assert the extractor produces the right ParentalLeaveFetch. This is the test that protects us from FK markup changes and runs in CI with no network and no BankID.
  • Unit-test the validation layer — feed out-of-bounds / inconsistent numbers, assert it raises rather than persists.
  • Unit-test the session state machine — transitions, timeout auto-cancel, double-cancel safety — with the browser mocked.
  • Manual E2E via npm run login — the only step that needs a real BankID; done by hand, not in CI.
  • Keep the existing ManualFetchService tests green; the contract must not change.

11. Build order / milestones

  1. Add config (scraper_headless, scraper_session_ttl_seconds, dev profile) to config.py; add Playwright to backend/pyproject.toml dev deps and document playwright install chromium.
  2. Session state machine + endpoints wired to FKScraperService, with the QR-relay and poll contract — using a mocked/stub page first so the harness works end-to-end before real FK selectors exist.
  3. npm run login harness + /dev/fetch page — prove the start→poll→QR→result loop against the stub.
  4. Receive captures (§6) → implement the real login + QR-relay against FK.
  5. Implement the extractor for each data field (§2) → fixture tests from the captures.
  6. Real E2E via the harness with a live BankID scan; iterate on selectors.
  7. Security/sandbox container setup + the §8 review checklist; confirm flag stays off for prod.

Milestones 2–3 are unblocked today (they don't need the captures). Milestones 4–6 are blocked on the §6 captures — that's the critical path, so prioritise producing those.

12. Open questions to resolve as we go

  • Exact Swedish labels and which page each datapoint sits on — answered by the captures.
  • Does FK gate "Ansök och planera föräldrapenning" behind an extra consent/child-selection step? If so, capture that page too (likely 03b_...).
  • Multiple children: does the overview list per-child, requiring a child selection that we map to child_id? Capture the selector if so.
  • Whether the QR is <img>/<canvas>/<svg> — we default to element screenshot regardless, but good to know.

TL;DR for the implementer

Build the real FKScraperService with Playwright-Python behind the existing FetchDataService contract, driving FK's own BankID-QR login and relaying FK's rendered QR (no BankID API). Expose it through the already-stubbed start/poll/cancel endpoints as a polled session state machine. Add a dev-only npm run login harness (/dev/fetch page: Refresh → QR → JSON result) so we can iterate. Keep every FK selector in one file, never persist credentials/session, fail loud on bad data, and keep the manual fallback first-class. Ships disabled until legal sign-off. Milestones 2–3 start now; 4–6 wait on the page captures (§6) — that's the critical path.

Reference in New Issue View Git Blame Copy Permalink