jacobh2/F-r-ldraledigplannerare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Parse transferable days from FK instead of deriving them

Browse Source 
The scraper never read "överlåtbara dagar" from Försäkringskassan. It only
visited ansok-och-planera-foraldrapenning#/valjdagar and derived the
reserved/transferable split in build_snapshot, assuming transferable days
are spent before reserved ones. On real data that gave 49 for the user and
0 for the partner where FK states 129 and 71.

The assumption cannot be repaired: reserved-first, transferable-first,
proportional and cap-based models were all checked against real figures and
none reproduces FK's accounting (reserved-first fits the partner but is 10
days off for the user). FK publishes the real numbers per parent per tier on
the min-foraldrapenning page, which the scraper had never visited.

- Scrape the "Dagar ni kan föra över" panel after the valjdagar extraction,
  summing sjukpenningnivå + lägstanivå per parent. The totals are not
  literals on the page. Parse only direct-child <p> elements: the first
  group embeds a tooltip with decoy "45 dagar"/"90 dagar" text.
- build_snapshot accepts the reported value and derives reserved days from
  it; the old arithmetic remains only as a fallback. An implausible FK value
  degrades to that fallback rather than failing the fetch.
- Record provenance in transferable_days_reported (migration 006) and badge
  derived values as estimates in the dashboard breakdown. Drop the footnote's
  claim that at most 150 days are transferable — real data contradicts it as
  a per-family figure.
- Select the child's section by name when FK lists several; never guess.

Verified against a live BankID scrape: scraped 240/101/139 with 129
transferable, deduced partner 240/169/71 with 71, both flagged as reported.

Incidental fixes found along the way:

- The fetch router never forwarded the partner's transferable figure to
  persist_partner_deduction, so the partner's stored snapshot would have kept
  the flawed split even once the parse landed.
- 'deduced' snapshots were labelled "Manually entered" by a two-way ternary
  over a three-value union. Replaced with a compiler-checked exhaustive map.
- Alembic resolved its database from alembic.ini alone, ignoring
  APP_SQLITE_PATH, so migrations silently targeted the developer's real
  database even when pointed elsewhere. env.py now resolves it, preferring
  -x sqlalchemy.url, then a real APP_SQLITE_PATH, then the ini value.
- backend/captures/ was not ignored: the root pattern was anchored as
  /captures/ despite a comment saying scraper dumps must never be tracked.
- CLAUDE.md quoted test counts and coverage percentages that had gone stale,
  which made a correct verification report look suspect. Gate thresholds kept.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DXPBfr1v7F71t6QcbHCaaT
This commit is contained in:
Jacob Hagstedt
2026-09-20 09:11:53 +02:00
parent 3eb8511eb6
commit 97a7200c1a
29 changed files with 1422 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/scrapers/fk_scraper.py
+276 -1
View File
@@ -33,7 +33,7 @@ import logging
import re
import uuid
from collections.abc import Awaitable, Callable
from dataclasses import asdict, dataclass, field
from dataclasses import asdict, dataclass, field, replace
from datetime import datetime
from pathlib import Path
from typing import Literal
@@ -43,6 +43,7 @@ from app.config import settings
from app.entities.parental_leave import ParentalLeaveFetch
from app.services.fetch_data_service import FetchDataService
from app.services.leave_snapshot import build_snapshot
from app.utils.errors import ValidationError
logger = logging.getLogger(__name__)
@@ -61,6 +62,14 @@ POST_LOGIN_URL_FRAGMENT = "ansok-och-planera-foraldrapenning#/"
# The data page we navigate to after clicking Ansök
DATA_PAGE_URL_FRAGMENT = "ansok-och-planera-foraldrapenning#/valjdagar"
# The "Min föräldrapenning" e-service page. Reports the transferable-days
# split ("Dagar ni kan föra över") directly, per-tier, for both parents —
# figures that cannot be reconstructed from #/valjdagar alone (see
# `_extract_transferable_days`'s docstring).
MIN_FP_URL = (
"https://www.forsakringskassan.se/privatperson/mina-sidor/alla-e-tjanster/min-foraldrapenning"
)
class _SEL:
# Login page
@@ -100,6 +109,15 @@ class _SEL:
FAMILY_DISCRETE = FAMILY + " .visa-dagar__discrete"
FAMILY_NIVA_STATISTIK = FAMILY + " .visa-dagar__desktop .niva-statistik__dagar span"
# Min föräldrapenning page — "Dagar ni kan föra över" panel (per child).
# See `MIN_FP_URL`'s comment and `_extract_transferable_days`.
MIN_FP_BARN_SEKTION = ".min-fp .barn-sektion"
BARN_RUBRIK = ".barn-rubrik strong"
PANEL_FORA_OVER_DAGAR = ".panel-fora-over-dagar"
FORA_OVER_GROUP = ".fora-over-group"
MINFP_LABEL = "h4.minfp-label"
EXPANDABLE_BUTTON = ".expandable-paragraph__button"
# ---------------------------------------------------------------------------
# Family-panel summary and partner deduction
@@ -125,6 +143,33 @@ class PartnerDeduction:
sjukpenning_left: int
lagsta_niva_left: int
dubbeldagar_left: int
transferable_days_left: int | None = None
@dataclass(frozen=True)
class TransferableDays:
"""FK's own reported "Dagar ni kan föra över" split, per tier and parent.
Read from the `min-foraldrapenning` e-service page (see `MIN_FP_URL`),
not derived — this is the source of truth `_apply_transferable` feeds
back into `build_snapshot` as `transferable_days_left`. Partner fields
are `None` for a solo parent (no "Annan förälder" group on the page).
"""
mine_sjukpenning: int
mine_lagsta_niva: int
partner_sjukpenning: int | None = None
partner_lagsta_niva: int | None = None
@property
def mine_total(self) -> int:
return self.mine_sjukpenning + self.mine_lagsta_niva
@property
def partner_total(self) -> int | None:
if self.partner_sjukpenning is None or self.partner_lagsta_niva is None:
return None
return self.partner_sjukpenning + self.partner_lagsta_niva
# ---------------------------------------------------------------------------
@@ -146,6 +191,13 @@ class _SessionState:
# treat a mismatched owner exactly like an unknown session_id — never a
# 403, so a foreign session_id can't be distinguished from a made-up one.
owner_user_id: UUID
# The child's display name, when the caller had it loaded already (see
# `start_session`). Used only to pick the right `.barn-sektion` on the
# min-foraldrapenning page when a workspace has more than one child —
# see `_extract_transferable_days`. `None` disables that feature for
# multi-child workspaces (never guesses a section) but never affects
# the caller's own #/valjdagar extraction.
child_name: str | None = None
status: SessionStatus = "launching"
qr_data: bytes | None = None # raw PNG bytes; never persisted to disk/DB
result: ParentalLeaveFetch | None = None
@@ -210,6 +262,7 @@ class FKScraperService(FetchDataService):
persist_callback: (
Callable[[ParentalLeaveFetch, PartnerDeduction | None], Awaitable[None]] | None
) = None,
child_name: str | None = None,
) -> str:
"""Launch a new headless browser session. Returns the session_id.
@@ -218,6 +271,11 @@ class FKScraperService(FetchDataService):
`owner_user_id` binds the session to the caller that started it — see
`_SessionState.owner_user_id`.
`child_name`, if the caller already has the `Child` loaded, is used
only to pick the right section on the min-foraldrapenning page when
a workspace has more than one child — see `_extract_transferable_days`.
The scraper never looks it up itself (no repository access from here).
At most one live (non-terminal) session per `owner_user_id` — starting
a new one cancels any session already in flight for the same caller
(see F5: unbounded growth of `_sessions` per owner).
@@ -237,6 +295,7 @@ class FKScraperService(FetchDataService):
personal_data_uuid=personal_data_uuid,
child_id=child_id,
owner_user_id=owner_user_id,
child_name=child_name,
status="launching",
persist_callback=persist_callback,
)
@@ -572,6 +631,24 @@ async def _do_extract(page, state: _SessionState) -> None:
state.family = None
state.partner_deduction = None
# "Dagar ni kan föra över" panel on the min-foraldrapenning page -> FK's
# own reported transferable-days split, replacing the transferable-first
# assumption in `data`/`state.partner_deduction`. Best-effort only, and
# must run after the family-panel block above: the family panel only
# exists on #/valjdagar, and this navigates away from it.
try:
transferable = await _extract_transferable_days(page, state, state.child_name)
if transferable is not None:
data = _apply_transferable(data, transferable)
if state.partner_deduction is not None and transferable.partner_total is not None:
state.partner_deduction = replace(
state.partner_deduction, transferable_days_left=transferable.partner_total
)
except Exception as exc:
logger.info(
"[%s] transferable-days parse skipped: %s", state.session_id, type(exc).__name__
)
if state.persist_callback is not None:
try:
await state.persist_callback(data, state.partner_deduction)
@@ -795,3 +872,201 @@ def _deduce_partner(mine: ParentalLeaveFetch, family: FamilySummary) -> PartnerD
lagsta_niva_left=lagsta_left,
dubbeldagar_left=mine.dubbeldagar_left,
)
# ---------------------------------------------------------------------------
# Transferable days ("Dagar ni kan föra över") — FK's own reported split
# ---------------------------------------------------------------------------
_TRANSFERABLE_RE = re.compile(r"(\d+)\s+dagar?\s+på\s+(sjukpenningnivå|lägstanivå)")
async def _extract_transferable_days(
page, state: _SessionState, child_name: str | None
) -> TransferableDays | None:
"""Read the "Dagar ni kan föra över" panel on the min-foraldrapenning
e-service page and return FK's own reported transferable-days split.
This figure cannot be reconstructed from #/valjdagar (see
`app.services.leave_snapshot.build_snapshot`'s module docstring) — FK's
own accounting there is not reproducible by any plausible model, so it
must be read off this second page directly.
Fully defensive by design, mirroring `_extract_family_summary`: any
missing element, unmatched child, or unexpected text is "no transferable
data available" — logged at INFO and returned as None — never raised.
The caller's own `data` (already validated) must be unaffected either way.
"""
try:
logger.debug("[%s] navigating to min-foraldrapenning page", state.session_id)
await page.goto(MIN_FP_URL, wait_until="domcontentloaded")
logger.debug("[%s] waiting for transfer panel", state.session_id)
await page.wait_for_selector(_SEL.PANEL_FORA_OVER_DAGAR, timeout=30_000)
# The panel mounts before the async fetch lands — wait for real
# per-tier figures rather than trusting first paint (mirrors the
# #/valjdagar hydration guard in `_do_extract`).
logger.debug("[%s] waiting for transfer panel data to hydrate", state.session_id)
await page.wait_for_function(
"""(sel) => {
const els = document.querySelectorAll(sel);
for (const el of els) {
if (/\\d+\\s+dagar?\\s+på/.test(el.textContent)) return true;
}
return false;
}""",
arg=_SEL.FORA_OVER_GROUP + " p",
timeout=30_000,
)
logger.debug("[%s] transfer panel data hydrated", state.session_id)
sections = page.locator(_SEL.MIN_FP_BARN_SEKTION)
section_count = await sections.count()
if section_count == 0:
logger.info(
"[%s] no barn-sektion found on min-foraldrapenning page; skipping "
"transferable-days parse",
state.session_id,
)
return None
if section_count == 1:
chosen = sections.first
else:
if not child_name or not child_name.strip():
logger.info(
"[%s] multiple children on min-foraldrapenning page and no "
"child_name to match; skipping transferable-days parse",
state.session_id,
)
return None
target = child_name.strip().split()[0].casefold()
chosen = None
for i in range(section_count):
section = sections.nth(i)
rubrik_loc = section.locator(_SEL.BARN_RUBRIK)
if await rubrik_loc.count() == 0:
continue
rubrik_text = (await rubrik_loc.first.inner_text(timeout=2_000)).strip().casefold()
if rubrik_text == target:
chosen = section
break
if chosen is None:
logger.info(
"[%s] no barn-sektion matched child_name; skipping transferable-days parse",
state.session_id,
)
return None
panel = chosen.locator(_SEL.PANEL_FORA_OVER_DAGAR)
if await panel.count() == 0:
logger.info(
"[%s] chosen barn-sektion has no transfer panel; skipping transferable-days parse",
state.session_id,
)
return None
# Content is present in the DOM even when the accordion is collapsed,
# so text reads work as-is. Still click through as insurance against
# FK switching to lazy rendering — a same-page toggle, never a dialog.
button = panel.locator(_SEL.EXPANDABLE_BUTTON)
if await button.count() > 0:
aria_expanded = await button.first.get_attribute("aria-expanded")
if aria_expanded == "false":
logger.debug("[%s] expanding collapsed transfer panel", state.session_id)
await button.first.click(timeout=5_000)
groups = panel.locator(_SEL.FORA_OVER_GROUP)
group_count = await groups.count()
mine_sjuk: int | None = None
mine_lagsta: int | None = None
partner_sjuk: int | None = None
partner_lagsta: int | None = None
for i in range(group_count):
group = groups.nth(i)
label_loc = group.locator(_SEL.MINFP_LABEL)
if await label_loc.count() == 0:
continue
label_text = (await label_loc.first.inner_text(timeout=2_000)).strip()
# Direct children only — the tooltip inside "Du kan föra över"
# holds decoy "45 dagar"/"90 dagar" text that a descendant
# selector would also pick up.
p_texts = await group.locator(":scope > p").all_inner_texts()
sjuk: int | None = None
lagsta: int | None = None
for t in p_texts:
m = _TRANSFERABLE_RE.search(t)
if not m:
continue
value = int(m.group(1))
if m.group(2) == "sjukpenningnivå":
sjuk = value
else:
lagsta = value
if label_text.startswith("Du kan"):
mine_sjuk, mine_lagsta = sjuk, lagsta
elif label_text.startswith("Annan förälder"):
partner_sjuk, partner_lagsta = sjuk, lagsta
if mine_sjuk is None or mine_lagsta is None:
logger.info(
"[%s] 'Du kan föra över' group missing or incomplete; skipping "
"transferable-days parse",
state.session_id,
)
return None
return TransferableDays(
mine_sjukpenning=mine_sjuk,
mine_lagsta_niva=mine_lagsta,
partner_sjukpenning=partner_sjuk,
partner_lagsta_niva=partner_lagsta,
)
except Exception as exc:
logger.info(
"[%s] transferable-days panel parsing failed (%s); skipping",
state.session_id,
type(exc).__name__,
)
return None
def _apply_transferable(
data: ParentalLeaveFetch, transferable: TransferableDays
) -> ParentalLeaveFetch:
"""Re-run `build_snapshot` with FK's own reported transferable-days
figure in place of the transferable-first assumption used when `data`
was first built.
An FK-reported figure that fails `build_snapshot`'s validation (should
never happen, but FK's markup is not a contract) degrades to the
original `data` unchanged rather than failing the whole fetch.
"""
try:
return build_snapshot(
personal_data_uuid=data.personal_data_uuid,
workspace_id=data.workspace_id,
child_id=data.child_id,
source=data.source,
total_days=data.total_days,
sjukpenning_left=data.tier_data.sjukpenning_left,
lagsta_niva_left=data.tier_data.lagsta_niva_left,
dubbeldagar_left=data.dubbeldagar_left,
transferable_days_left=transferable.mine_total,
fetched_at=data.fetched_at,
)
except ValidationError as exc:
logger.warning(
"FK-reported transferable_days_left=%d implausible against derived "
"days_left=%d (child_id=%s); keeping derived reserved/transferable split: %s",
transferable.mine_total,
data.days_left,
data.child_id,
exc,
)
return data