jacobh2/F-r-ldraledigplannerare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden the API and the data layer after a security and performance audit

Browse Source 
Security
- Every route now requires a login and refuses ids outside the caller's
  workspace with 404: plan, leave-wishes, day-overrides and vacation-days
  were fully anonymous; every other router trusted client-supplied
  workspace/child/parent ids. Scraper sessions are bound to their owner.
- Invitations can no longer be created for a foreign workspace, and a
  redeemer who is already a member (or a full workspace) is refused.
- Production refuses the default JWT secret; API docs are hidden in
  production; a failed migration aborts startup instead of serving.
- Login and register no longer reveal whether an email exists; inactive
  accounts are rejected; /auth/refresh is rate limited.
- The FK scraper writes debug artefacts only when explicitly enabled,
  logs no scraped values or page text, clears the BankID QR as soon as it
  is scanned and purges finished sessions after five minutes.
- Real FK page captures are removed from the tree (history rewrite
  follows); the parser tests use a synthetic fixture.

Performance and correctness
- Plan apply/sync run inside one locked transaction with batched writes
  instead of one commit per row, so a failure mid-batch leaves nothing
  half-applied and concurrent syncs no longer corrupt each other.
- SQLite runs in WAL mode with a busy timeout; 14 indexes cover the
  filters that were full scans; argon2 hashing runs off the event loop;
  responses are gzipped.
- Frontend: a 401 now refreshes the token once and retries, so sessions
  no longer die silently after 15 minutes; every data hook discards
  stale responses; logout clears state even when the server call fails.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CsXCCcDfmV4Sps4ZQSvNEY
This commit is contained in:
Jacob Hagstedt
2026-09-18 07:54:21 +02:00
parent ae16d71970
commit 02db5430eb
81 changed files with 3616 additions and 600 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/scrapers/fk_scraper.py
+109 -33
View File
@@ -4,10 +4,17 @@ Legal/ToS review completed (2026-06). Legal has cleared automated read-only
access for users who have given explicit per-session consent. Operational
guardrails must be maintained (see §2.3):
- Sandboxed browser context, torn down after each session.
- No cookies, session tokens, or QR images ever persisted to disk or DB.
- No personal data (names, personnummer, day counts) in logs.
- No cookies, session tokens, or QR images ever persisted to disk or DB by
default. Debug artifacts (HTML + screenshot of the logged-in page) are
OFF unless both scraper_debug_enabled and scraper_debug_dir are set —
they contain personal data and are for local debugging only.
- No personal data (names, personnummer, day counts, raw page text) in
logs above DEBUG; DEBUG lines log shapes/counts, never scraped values.
- Per-session user consent required before starting a session.
- Only the extracted day-count numbers are persisted (see persist_callback).
- QR frames and results are retained in RAM only for
scraper_result_retention_seconds after a session ends, then purged; at
most one live session per user.
Enable via APP_SCRAPER_ENABLED=true in the environment or backend/.env.
@@ -94,6 +101,10 @@ class _SessionState:
workspace_id: UUID
personal_data_uuid: UUID
child_id: UUID
# The User.id that started this session. `poll_session`/`cancel_session`
# treat a mismatched owner exactly like an unknown session_id — never a
# 403, so a foreign session_id can't be distinguished from a made-up one.
owner_user_id: UUID
status: SessionStatus = "launching"
qr_data: bytes | None = None # raw PNG bytes; never persisted to disk/DB
result: ParentalLeaveFetch | None = None
@@ -146,19 +157,35 @@ class FKScraperService(FetchDataService):
workspace_id: UUID,
personal_data_uuid: UUID,
child_id: UUID,
owner_user_id: UUID,
persist_callback: Callable[[ParentalLeaveFetch], Awaitable[None]] | None = None,
) -> str:
"""Launch a new headless browser session. Returns the session_id.
persist_callback, if provided, is called with the validated ParentalLeaveFetch
once extraction succeeds. Failure in the callback sets the session to 'error'.
`owner_user_id` binds the session to the caller that started it — see
`_SessionState.owner_user_id`.
At most one live (non-terminal) session per `owner_user_id` — starting
a new one cancels any session already in flight for the same caller
(see F5: unbounded growth of `_sessions` per owner).
"""
for existing_id, existing_state in list(_sessions.items()):
if existing_state.owner_user_id == owner_user_id and existing_state.status not in (
"done",
"error",
"cancelled",
):
await self.cancel_session(existing_id, owner_user_id)
session_id = str(uuid.uuid4())
state = _SessionState(
session_id=session_id,
workspace_id=workspace_id,
personal_data_uuid=personal_data_uuid,
child_id=child_id,
owner_user_id=owner_user_id,
status="launching",
persist_callback=persist_callback,
)
@@ -172,10 +199,15 @@ class FKScraperService(FetchDataService):
logger.info("scraper session started [id=%s]", session_id)
return session_id
async def poll_session(self, session_id: str) -> dict:
"""Return current status, QR bytes (base64), or result."""
async def poll_session(self, session_id: str, owner_user_id: UUID) -> dict:
"""Return current status, QR bytes (base64), or result.
A `session_id` owned by a different caller is treated exactly like
an unknown one — never a 403, so a foreign session_id can't be
distinguished from a made-up one.
"""
state = _sessions.get(session_id)
if state is None:
if state is None or state.owner_user_id != owner_user_id:
return {"status": "error", "message": "Unknown session_id"}
response: dict = {"status": state.status}
@@ -187,12 +219,17 @@ class FKScraperService(FetchDataService):
response["message"] = state.message
return response
async def cancel_session(self, session_id: str) -> None:
"""Cancel a running session and tear down the browser."""
async def cancel_session(self, session_id: str, owner_user_id: UUID) -> None:
"""Cancel a running session and tear down the browser.
A `session_id` owned by a different caller is a no-op, exactly like
an unknown one — never a 403 (see `poll_session`).
"""
state = _sessions.get(session_id)
if state is None:
if state is None or state.owner_user_id != owner_user_id:
return
_set_status(state, "cancelled")
state.qr_data = None
if state._task and not state._task.done():
state._task.cancel()
logger.info("scraper session cancelled [id=%s]", session_id)
@@ -209,15 +246,50 @@ def _set_status(state: _SessionState, status: SessionStatus, message: str | None
state.message = message
def _generic_error_message(exc: Exception) -> str:
"""Map an internal exception to a fixed, generic message safe to return
to the poll client (see F4). Never embeds `str(exc)` — a `ValueError`
raised while parsing the FK page may otherwise carry raw page text.
Logs carry the exception type name only; full detail is never emitted
above DEBUG and is not persisted unless debug artifacts are explicitly
enabled (`scraper_debug_enabled`).
"""
if isinstance(exc, ValueError):
return "Scraper error: could not read data from the FK page. Please try again."
if type(exc).__name__ == "TimeoutError":
return "Scraper error: timed out waiting for the FK page. Please try again."
return "Scraper error: an unexpected error occurred. Please try again."
async def _ttl_watchdog(session_id: str) -> None:
"""Auto-cancel sessions that sit in awaiting_scan/authenticating too long."""
"""Auto-cancel sessions that sit in awaiting_scan/authenticating too long.
Also acts as a secondary safety net (F5): if the session has already
reached a terminal state by the time this fires, purge it immediately
rather than rely solely on `_purge_later`.
"""
await asyncio.sleep(settings.scraper_session_ttl_seconds)
state = _sessions.get(session_id)
if state and state.status in ("awaiting_scan", "authenticating", "launching"):
if state is None:
return
if state.status in ("awaiting_scan", "authenticating", "launching"):
_set_status(state, "error", "Session timed out waiting for BankID scan")
state.qr_data = None
if state._task and not state._task.done():
state._task.cancel()
logger.info("scraper session timed out [id=%s]", session_id)
elif state.status in ("done", "error", "cancelled"):
_sessions.pop(session_id, None)
async def _purge_later(session_id: str) -> None:
"""Remove a finished session from `_sessions` after the retention window.
Bounds how long a scraped result or a stale (already-cleared) QR frame
slot can be read back via `/poll` after the session ended (see F5).
"""
await asyncio.sleep(settings.scraper_result_retention_seconds)
_sessions.pop(session_id, None)
async def _check_maintenance(page) -> str | None:
@@ -243,9 +315,14 @@ async def _check_maintenance(page) -> str | None:
async def _save_debug_artifacts(page, session_id: str, console_messages: list[str]) -> None:
"""Save HTML, screenshot, and console log to scraper_debug_dir on error."""
"""Save HTML, screenshot, and console log to scraper_debug_dir on error.
OFF by default and requires both `scraper_debug_enabled` and
`scraper_debug_dir` to be set — these artifacts contain personal data
(name, balances, payout amounts) from the logged-in FK page.
"""
debug_dir = settings.scraper_debug_dir
if not debug_dir:
if not settings.scraper_debug_enabled or not debug_dir:
return
try:
out = Path(debug_dir)
@@ -289,8 +366,10 @@ async def _run_session(state: _SessionState) -> None:
except asyncio.CancelledError:
_set_status(state, "cancelled")
except Exception as exc:
_set_status(state, "error", f"Scraper error: {type(exc).__name__}: {exc}")
logger.error("scraper error [id=%s]: %s", state.session_id, exc, exc_info=False)
_set_status(state, "error", _generic_error_message(exc))
logger.error(
"scraper error [id=%s]: %s", state.session_id, type(exc).__name__, exc_info=False
)
await _save_debug_artifacts(page, state.session_id, console_messages)
finally:
try:
@@ -299,6 +378,10 @@ async def _run_session(state: _SessionState) -> None:
except Exception:
pass # already torn down
# Session has reached a terminal state (done/error/cancelled) — schedule
# its removal from `_sessions` after the retention window (see F5).
asyncio.create_task(_purge_later(state.session_id))
async def _do_login(page, state: _SessionState) -> None:
"""Navigate to FK login, handle cookie consent, click BankID QR, relay QR."""
@@ -372,6 +455,10 @@ async def _do_login(page, state: _SessionState) -> None:
logger.debug("[%s] QR element gone; BankID app likely confirming", state.session_id)
break
# Leaving awaiting_scan (scanned, cancelled, or timed out) — the last QR
# frame is no longer needed and must not linger in RAM (see F5).
state.qr_data = None
if state.status == "cancelled":
return
@@ -440,14 +527,14 @@ async def _extract_parental_leave(page, state: _SessionState) -> ParentalLeaveFe
logger.debug("[%s] reading days_left (FOR_DIG_DAGAR_KVAR)", state.session_id)
days_left_text = await page.locator(_SEL.FOR_DIG_DAGAR_KVAR).first.inner_text(timeout=5_000)
days_left = int(days_left_text.strip())
logger.debug("[%s] days_left=%d", state.session_id, days_left)
logger.debug("[%s] days_left parsed", state.session_id)
# -- total days from "av 240 dagar kvar efter ansökan" --
logger.debug("[%s] reading total days (FOR_DIG_DISCRETE)", state.session_id)
discrete_text = await page.locator(_SEL.FOR_DIG_DISCRETE).first.inner_text(timeout=5_000)
total_match = re.search(r"av\s+(\d+)\s+dagar", discrete_text)
if not total_match:
raise ValueError(f"Could not parse total days from: {discrete_text!r}")
raise ValueError("Could not parse total days from FK page text")
total_days = int(total_match.group(1))
if total_days == 0:
raise ValueError(
@@ -455,14 +542,14 @@ async def _extract_parental_leave(page, state: _SessionState) -> ParentalLeaveFe
"(placeholder state). The hydration wait in _do_extract should prevent this."
)
days_used = total_days - days_left
logger.debug("[%s] total_days=%d days_used=%d", state.session_id, total_days, days_used)
logger.debug("[%s] total/used parsed", state.session_id)
# -- sjukpenningnivå and lägstanivå remaining (desktop niva-statistik) --
# all_inner_texts() does not accept a timeout argument — wait for the first element first.
logger.debug("[%s] reading tier breakdown (NIVA_STATISTIK_DESKTOP)", state.session_id)
await page.locator(_SEL.NIVA_STATISTIK_DESKTOP).first.wait_for(timeout=5_000)
niva_texts = await page.locator(_SEL.NIVA_STATISTIK_DESKTOP).all_inner_texts()
logger.debug("[%s] niva_texts raw=%r", state.session_id, niva_texts)
logger.debug("[%s] niva items=%d", state.session_id, len(niva_texts))
# Each item is like "(119 dagar)" — filter to desktop section (first two)
niva_numbers = []
for t in niva_texts:
@@ -473,31 +560,20 @@ async def _extract_parental_leave(page, state: _SessionState) -> ParentalLeaveFe
break
if len(niva_numbers) < 2:
raise ValueError(f"Expected 2 tier values, got {niva_numbers} from {niva_texts!r}")
raise ValueError(f"Expected 2 tier values, got {len(niva_numbers)}")
sjukpenning_left, lagsta_niva_left = niva_numbers[0], niva_numbers[1]
logger.debug(
"[%s] sjukpenning_left=%d lagsta_niva_left=%d",
state.session_id,
sjukpenning_left,
lagsta_niva_left,
)
logger.debug("[%s] tier breakdown parsed", state.session_id)
# Cross-check: sjukpenning_left + lagsta_niva_left should equal days_left
if sjukpenning_left + lagsta_niva_left != days_left:
logger.warning(
"Tier sum mismatch: sjukpenning=%d + lagstaniva=%d != days_left=%d [id=%s]",
sjukpenning_left,
lagsta_niva_left,
days_left,
state.session_id,
)
logger.warning("Tier sum mismatch [id=%s]", state.session_id)
# -- dubbeldagar remaining --
logger.debug("[%s] reading dubbeldagar (DUBBELDAG_KVAR)", state.session_id)
dubbeldag_text = await page.locator(_SEL.DUBBELDAG_KVAR).first.inner_text(timeout=5_000)
dubbeldagar_left = int(dubbeldag_text.strip())
logger.debug("[%s] dubbeldagar_left=%d", state.session_id, dubbeldagar_left)
logger.debug("[%s] dubbeldagar parsed", state.session_id)
# -- Derive tier totals (standard FK allocation per parent) --
# Standard: 195 sjukpenningnivå + 45 lägstanivå = 240 per parent.