97a7200c1a
The scraper never read "överlåtbara dagar" from Försäkringskassan. It only visited ansok-och-planera-foraldrapenning#/valjdagar and derived the reserved/transferable split in build_snapshot, assuming transferable days are spent before reserved ones. On real data that gave 49 for the user and 0 for the partner where FK states 129 and 71. The assumption cannot be repaired: reserved-first, transferable-first, proportional and cap-based models were all checked against real figures and none reproduces FK's accounting (reserved-first fits the partner but is 10 days off for the user). FK publishes the real numbers per parent per tier on the min-foraldrapenning page, which the scraper had never visited. - Scrape the "Dagar ni kan föra över" panel after the valjdagar extraction, summing sjukpenningnivå + lägstanivå per parent. The totals are not literals on the page. Parse only direct-child <p> elements: the first group embeds a tooltip with decoy "45 dagar"/"90 dagar" text. - build_snapshot accepts the reported value and derives reserved days from it; the old arithmetic remains only as a fallback. An implausible FK value degrades to that fallback rather than failing the fetch. - Record provenance in transferable_days_reported (migration 006) and badge derived values as estimates in the dashboard breakdown. Drop the footnote's claim that at most 150 days are transferable — real data contradicts it as a per-family figure. - Select the child's section by name when FK lists several; never guess. Verified against a live BankID scrape: scraped 240/101/139 with 129 transferable, deduced partner 240/169/71 with 71, both flagged as reported. Incidental fixes found along the way: - The fetch router never forwarded the partner's transferable figure to persist_partner_deduction, so the partner's stored snapshot would have kept the flawed split even once the parse landed. - 'deduced' snapshots were labelled "Manually entered" by a two-way ternary over a three-value union. Replaced with a compiler-checked exhaustive map. - Alembic resolved its database from alembic.ini alone, ignoring APP_SQLITE_PATH, so migrations silently targeted the developer's real database even when pointed elsewhere. env.py now resolves it, preferring -x sqlalchemy.url, then a real APP_SQLITE_PATH, then the ini value. - backend/captures/ was not ignored: the root pattern was anchored as /captures/ despite a comment saying scraper dumps must never be tracked. - CLAUDE.md quoted test counts and coverage percentages that had gone stale, which made a correct verification report look suspect. Gate thresholds kept. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DXPBfr1v7F71t6QcbHCaaT
172 lines
6.8 KiB
Python
172 lines
6.8 KiB
Python
"""FetchDataService — stable contract for parental-leave data acquisition.
|
|
|
|
All features (analytics, tips, planning) depend on this interface, never on
|
|
FK's DOM or a specific scraper. Two concrete implementations:
|
|
- ManualFetchService — always ships, always works (manual user entry)
|
|
- FKScraperService — behind settings.scraper_enabled guard (pending legal review)
|
|
|
|
See ARCHITECTURE.md §2.3 for the legal/ToS gate before shipping scraping.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from abc import ABC, abstractmethod
|
|
from uuid import UUID
|
|
|
|
from app.config import settings
|
|
from app.entities.parental_leave import ParentalLeaveFetch
|
|
from app.repositories.parental_leave_repository import ParentalLeaveRepository
|
|
from app.repositories.personal_data_repository import PersonalDataRepository
|
|
from app.services.leave_snapshot import build_snapshot
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
class FetchDataService(ABC):
|
|
@abstractmethod
|
|
async def get_latest(
|
|
self, personal_data_uuid: UUID, child_id: UUID
|
|
) -> ParentalLeaveFetch | None:
|
|
"""Return the most recent snapshot for this parent+child, or None."""
|
|
|
|
@abstractmethod
|
|
async def list_history(
|
|
self, personal_data_uuid: UUID, child_id: UUID
|
|
) -> list[ParentalLeaveFetch]:
|
|
"""Return all historical snapshots, newest first."""
|
|
|
|
|
|
class ManualFetchService(FetchDataService):
|
|
"""Persist user-entered FK data as a snapshot with source='manual'/'deduced'."""
|
|
|
|
def __init__(
|
|
self,
|
|
repo: ParentalLeaveRepository,
|
|
personal_data_repo: PersonalDataRepository | None = None,
|
|
) -> None:
|
|
self._repo = repo
|
|
# Optional: only needed to default `total_days` in `save_manual_entry`
|
|
# when the caller omits it. Existing direct-construction call sites
|
|
# (scripts/seed_demo.py, tests) that always pass an explicit
|
|
# `total_days` don't need to wire one up.
|
|
self._personal_data_repo = personal_data_repo
|
|
|
|
async def save_manual_entry(
|
|
self,
|
|
personal_data_uuid: UUID,
|
|
workspace_id: UUID,
|
|
child_id: UUID,
|
|
sjukpenning_left: int,
|
|
lagsta_niva_left: int,
|
|
dubbeldagar_left: int,
|
|
reserved_days_left: int | None = None,
|
|
total_days: int | None = None,
|
|
source: str = "manual",
|
|
) -> ParentalLeaveFetch:
|
|
"""Validate and persist a manual (or deduced) data-entry snapshot.
|
|
|
|
When `total_days` is omitted, it defaults to
|
|
`rules.total_days_per_child // parent_count` (480 for a sole parent,
|
|
240 with a partner) — the number of *workspace members*, not just
|
|
this one parent, since the pool is split per how many parents the
|
|
child actually has. Forwards to the one shared derivation,
|
|
`build_snapshot`, so a manual snapshot is derived identically to a
|
|
scraped one.
|
|
"""
|
|
if total_days is None:
|
|
if self._personal_data_repo is None:
|
|
raise ValueError(
|
|
"total_days was omitted but ManualFetchService has no "
|
|
"personal_data_repo to resolve a parent count from"
|
|
)
|
|
members = await self._personal_data_repo.list_by_workspace(workspace_id)
|
|
parent_count = max(len(members), 1)
|
|
total_days = settings.rules.total_days_per_child // parent_count
|
|
fetch = build_snapshot(
|
|
personal_data_uuid=personal_data_uuid,
|
|
workspace_id=workspace_id,
|
|
child_id=child_id,
|
|
source=source,
|
|
total_days=total_days,
|
|
sjukpenning_left=sjukpenning_left,
|
|
lagsta_niva_left=lagsta_niva_left,
|
|
dubbeldagar_left=dubbeldagar_left,
|
|
reserved_days_left=reserved_days_left,
|
|
)
|
|
return await self._repo.create(fetch)
|
|
|
|
async def get_latest(
|
|
self, personal_data_uuid: UUID, child_id: UUID
|
|
) -> ParentalLeaveFetch | None:
|
|
return await self._repo.get_latest_by_parent(personal_data_uuid, child_id)
|
|
|
|
async def list_history(
|
|
self, personal_data_uuid: UUID, child_id: UUID
|
|
) -> list[ParentalLeaveFetch]:
|
|
return await self._repo.list_history(personal_data_uuid, child_id)
|
|
|
|
|
|
async def persist_partner_deduction(
|
|
*,
|
|
personal_data_repo: PersonalDataRepository,
|
|
parental_leave_repo: ParentalLeaveRepository,
|
|
workspace_id: UUID,
|
|
child_id: UUID,
|
|
caller_personal_data_uuid: UUID,
|
|
total_days: int,
|
|
sjukpenning_left: int,
|
|
lagsta_niva_left: int,
|
|
dubbeldagar_left: int,
|
|
transferable_days_left: int | None = None,
|
|
) -> None:
|
|
"""Persist a 'deduced' snapshot for the caller's partner, if there is one.
|
|
|
|
Called by `app.routers.fetch_data.start_fetch`'s `_persist` after the FK
|
|
scraper deduces the other parent's balance from the family panel (see
|
|
`app.scrapers.fk_scraper._deduce_partner`). Takes plain fields rather
|
|
than the scraper's `PartnerDeduction` dataclass to avoid a router/scraper
|
|
import cycle (`fk_scraper` already imports from this module) — the
|
|
router unpacks it before calling in. `transferable_days_left`, when
|
|
known (FK's own reported figure for the partner), is forwarded to
|
|
`build_snapshot` so the partner's reserved/transferable split doesn't
|
|
fall back to the transferable-first assumption either.
|
|
|
|
A no-op (logged, never raised) unless the workspace has exactly one
|
|
other non-deleted member: with zero, there's no one to persist for; with
|
|
two or more, which member is "the partner" is ambiguous and guessing
|
|
wrong would silently corrupt someone else's data. Any persistence
|
|
failure here is logged and swallowed — the caller's own snapshot is
|
|
already saved by the time this runs, and a failure here must not fail
|
|
the scraper session.
|
|
"""
|
|
try:
|
|
members = await personal_data_repo.list_by_workspace(workspace_id)
|
|
others = [m for m in members if m.uuid != caller_personal_data_uuid]
|
|
if len(others) != 1:
|
|
logger.info(
|
|
"skipping partner-deduction persist: workspace has %d other member(s)",
|
|
len(others),
|
|
)
|
|
return
|
|
partner = others[0]
|
|
fetch = build_snapshot(
|
|
personal_data_uuid=partner.uuid,
|
|
workspace_id=workspace_id,
|
|
child_id=child_id,
|
|
source="deduced",
|
|
total_days=total_days,
|
|
sjukpenning_left=sjukpenning_left,
|
|
lagsta_niva_left=lagsta_niva_left,
|
|
dubbeldagar_left=dubbeldagar_left,
|
|
transferable_days_left=transferable_days_left,
|
|
)
|
|
await parental_leave_repo.create(fetch)
|
|
except Exception as exc:
|
|
logger.warning(
|
|
"failed to persist partner-deduction snapshot for workspace=%s child=%s: %s",
|
|
workspace_id,
|
|
child_id,
|
|
exc,
|
|
)
|