From 4c1a4b5a9d95f0e4684eec2505cc923684b04410 Mon Sep 17 00:00:00 2001 From: Jacob Hagstedt Date: Sun, 18 Oct 2026 07:38:39 +0200 Subject: [PATCH] UX: category suggestion rule and evidence from the Splitwise export; importer mapping gap Co-Authored-By: Claude Opus 5.5 --- docs/ux/category-suggestion.md | 106 +++++++++++++++++++++++++++++++++ docs/ux/proposed-decisions.md | 21 ++++--- 2 files changed, 120 insertions(+), 7 deletions(-) create mode 100644 docs/ux/category-suggestion.md diff --git a/docs/ux/category-suggestion.md b/docs/ux/category-suggestion.md new file mode 100644 index 0000000..72964e7 --- /dev/null +++ b/docs/ux/category-suggestion.md @@ -0,0 +1,106 @@ +# Category suggestion in the expense editor + +Proposal, 2026-10-18. The owner decided that the expense editor (S-7) fills in the category from the description while the user types, and that the user can always change it. This file is the proposed rule as spec text, the evidence from the household's Splitwise export behind it, and the open points. The built-in keyword table is `category-keywords.csv` next to this file. Nothing here is in SPEC.md yet; if accepted it becomes an addition to AC-8.6 with a decision entry. + +## Proposed rule (spec text) + +**CS-1 Scope.** The suggestion runs only in the editor for a **new** expense. It never runs when editing an existing expense, and never changes the category of a saved expense. + +**CS-2 Manual choice wins.** The editor keeps a flag "category picked by hand", false when the editor opens. Picking any category in the category grid sets it to true, also when the user picks the category that is already shown. While the flag is true, the suggestion never changes the category for the rest of this editor session (until the editor closes). Nothing resets the flag. + +**CS-3 When it runs.** While the flag is false, the category is recomputed on every change of the description and on every change of the account. It is a local computation over Room data; no network. + +**CS-4 Normalisation.** Descriptions (typed and stored) and keywords are compared after this normalisation: +1. Unicode NFC, then lowercase without locale rules (Kotlin `lowercase()`, `Locale.ROOT`). +2. Delete `'`, `’` and `:` (so "Willy:s" becomes "willys", "McDonald's" becomes "mcdonalds"). +3. Replace every remaining character that is not a Unicode letter, a Unicode digit, whitespace or `&` with a space. +4. Collapse runs of whitespace to one space and trim. + +Diacritics are kept (å, ä, ö and é stay distinct; the keyword table lists both spellings where it matters). Digits are kept. The **tokens** are the normalised description split on spaces; the **first token** is the first of them. An empty normalised description gives no history or keyword match. + +**CS-5 Order.** The suggested category is the first that applies of: +1. **History, exact:** among the non-deleted expenses on the phone (every account, either user, including imported and recurring ones) whose normalised description equals the normalised typed description, the most recent one's category. +2. **History, first token:** among the non-deleted expenses whose first token equals the typed description's first token, the most recent one's category. +3. **Keywords:** the category of the winning entry of the built-in keyword table (CS-6). +4. **Default:** the AC-8.6 default (the category last used in the selected account on this phone if still active, else `other`). + +"Most recent" is the expense-list order of AC-8.10: `expense_date` descending, then `created_at` descending. A history candidate whose category is archived or deleted is skipped and the next most recent candidate of the same step is tried; when none is left, the next step applies. + +**CS-6 Keyword table.** Each entry has `keyword` (already normalised), `match` and `default_key`. An entry matches when: +* `token`: some token equals the keyword; +* `prefix`: some token starts with the keyword; +* `phrase`: the keyword's tokens occur consecutively in the tokens. + +When several entries match, the longest keyword (in code points, spaces included) wins; on equal length, the entry earlier in the table wins (the table is sorted by `default_key`, then `keyword`). The winning entry gives the category with that `default_key` (renamed defaults included, since the key stays). If that category is archived, the next matching entry in the same order is tried; when none is left, step 4 applies. The table is fixed data shipped with the app; users do not edit it. + +**CS-7 What the user sees.** The suggestion replaces the category shown in the Kategori field and needs no confirmation. While a category comes from step 1, 2 or 3 and the flag of CS-2 is false, a one-line caption under the category field says "Vald utifrån beskrivningen" / "Chosen from the description" (string `expense_editor_category_suggested`); it disappears when the user picks a category by hand or when step 4 applies. Clearing the description returns the field to the step-4 default. + +## Evidence + +Source: the household's two Splitwise exports, read-only on the owner's Mac. 1 775 expense rows from 2020-11 to 2026-10 (payment rows, blank lines and the total row excluded), mapped to Kvitt categories with the §10.8 table. True categories: groceries 662, home 315, restaurants 301, other 171, entertainment 111, transport 89, utilities 58, travel 35, health 17, shopping 16. 21 rows (1.2 %) had a Swedish Splitwise category name missing from the §10.8 table and count as `other` (see open points). + +Method: chronological simulation. Rows are walked in date order; each row's category is predicted from earlier rows only and compared with its true category. The step-4 default is approximated as the previous row's category in the same file (one file is one account), `other` for the first row. + +**History variants alone** (coverage = share of rows with a prediction; accuracy among those): + +| Variant | Coverage | Accuracy (most recent) | Accuracy (most frequent) | +|---|---|---|---| +| Exact normalised description | 56.2 % | 94.9 % | 95.9 % | +| First two tokens | 57.2 % | 94.8 % | 95.8 % | +| First token | 71.5 % | 92.0 % | 93.0 % | +| Exact, then first token (chosen) | 71.5 % | 92.3 % | 93.2 % | +| Any shared token of ≥ 3 letters | 79.0 % | 86.2 % | 89.4 % | +| Any shared token of ≥ 4 letters | 63.7 % | 85.4 % | 88.3 % | + +Exact-then-first-token keeps the owner's "matching description" literally as step 1 and adds the first token for coverage. "Most frequent" is about one point better but harder to explain; the owner chose most recent. Dropping digit-only tokens changed nothing. + +**Full rule (CS-5, chosen variant, keyword table as committed):** + +| Stage | Share of rows | Accuracy | +|---|---|---| +| History (exact 998 rows at 94.9 %, first token 272 rows at 82.7 %) | 71.5 % | 92.3 % | +| Keywords | 7.8 % | 94.2 % | +| Default (approximated) | 20.7 % | 30.0 % | +| **Overall** | 100 % | **79.5 %** | + +For comparison: the AC-8.6 default alone 31.9 %; history and default without keywords 74.6 %. Putting keywords before history would give 80.7 %, exact history then keywords then first token 79.9 %; the differences are small, so the owner's order stands. + +**Cold start** (keywords and default only, no history): first 10 % of rows (177): keywords cover 78.0 % at 96.4 % accuracy, 85.3 % overall. Over all rows: 72.2 % coverage at 96.7 %. These are in-sample for the `data` entries. An out-of-sample check (keep a `data` entry only if it reaches ≥ 2 rows and ≥ 80 % precision in the first half; test on the second half) gives 58.1 % coverage at 98.4 % accuracy. + +**Most common wrong suggestions** (suggested → true, full rule, 363 wrong of 1 775; 257 of them at the default step): groceries → restaurants 55, groceries → other 38, restaurants → groceries 35, groceries → home 23, home → other 18, restaurants → other 14, other → groceries 13, other → home 12. History errors are mostly the groceries/restaurants pair on a shared first token such as "mat". Keyword errors (8 in total) are mostly `apotek` (the household filed pharmacy purchases under other as often as under health) and `ikea` (the IKEA restaurant). + +**Keyword table:** 149 entries, 76 `data` and 73 `curated`; 119 `token`, 24 `prefix`, 6 `phrase`. + +| `default_key` | data | curated | +|---|---|---| +| groceries | 20 | 4 | +| restaurants | 17 | 11 | +| home | 14 | 6 | +| utilities | 1 | 7 | +| transport | 6 | 15 | +| travel | 4 | 5 | +| entertainment | 7 | 9 | +| health | 1 | 7 | +| shopping | 1 | 9 | +| other | 5 | 0 | + +`data` entries predicted one category with high precision in the export (most ≥ 90 %, `ikea` 79 %, `apotek` 55 % kept because the meaning is unambiguous). `curated` entries are well-known Swedish chains and services and unambiguous words, absent or rare in the export. + +## Excluded on purpose + +Kinds of word left out of the table even when they predicted well: +* person names and nicknames, addresses, street names, workplaces, private places and anything that could identify a person; +* town, city and country names (they predict a trip, not a category); +* names of single local restaurants, cafés and shops whose identity was unclear; +* function words and abbreviations (och, till, för, på, mm, etc, o, &); +* words the household used against their dictionary meaning: `el` (always part of "hyra och el", filed under home), `kläder` (filed under other); and words split between categories: `frukost`, `vin`, `öl`, `blommor`, `brunch`, `cykel`. + +## Open points + +1. **`other` entries.** The five `data` entries for `other` (insurance words, ränteåterbäring, förskola) are kept because they beat the default; the brief said the curated set never targets `other`. Drop them if keyword hits should never select Övrigt. +2. **Missing Swedish Splitwise names in §10.8.** Nine Swedish category names in the export are not in the mapping table (21 rows): Städning, Verktyg - Övrigt, Tjänster, Cykel, Livet - Övrigt, Flyg, Spel, Kläder, Utbildning. By their English equivalents they would map to utilities, utilities ("Utilities - Other"), home, transport, other, travel, entertainment, shopping and other. The importer would count them as unmapped; the history step inherits whatever the import stores. +3. **Imported history.** After M11 the whole Splitwise history is in Room, so for this household history decides most descriptions from day one; the keyword table matters for new words and after a fresh install. +4. **Recurring template editor.** Not covered; the same rule could apply to new templates. +5. **Recency key.** CS-5 uses the list order (`expense_date`). Using `updated_at` instead would let a re-categorised old expense teach the rule; not measurable from the export. +6. **Where the table lives.** Proposed: `:domain` as data plus a unit test that loads `docs/ux/category-keywords.csv` (or its successor) and checks every keyword is already normalised and unique. +7. **English keywords.** The table is Swedish plus brand names; English words are not covered. diff --git a/docs/ux/proposed-decisions.md b/docs/ux/proposed-decisions.md index 194d054..3b8d6b5 100644 --- a/docs/ux/proposed-decisions.md +++ b/docs/ux/proposed-decisions.md @@ -29,13 +29,20 @@ Every entry has: the owner's answer or the finding, the proposed decision text f * **Decision (owner, 2026-10-18):** for a new expense, the editor selects a category from the description as the user types, while the user has not picked a category by hand in this editor session: (1) history match, (2) built-in keyword list, (3) the AC-8.6 default. The user can always change it. Editing an existing expense never changes its category automatically. * **Why:** most expenses repeat the same shops; fewer taps in the ten-second flow (§13.1). -* **Spec changes (to be completed from `category-suggestion.md` when the analysis finishes):** - * AC-8.6: add the suggestion as the first source of the category default. - * New rule section (proposed §9.11 "Category suggestion"): normalisation, history match and tie-break, keyword matching, skipped archived/deleted categories, when the suggestion stops. - * New contract file: the keyword list (proposed path `contracts/category-keywords.csv`, columns `keyword,match,default_key,source`; draft in `docs/ux/category-keywords.csv`), versioned like other contract files. - * §14.2.4 contract change: `ExpenseRepository` gains `suspend fun suggestCategory(description: String): CategoryId?` (computed in `:data` from Room history and the keyword list, with a pure function in `:domain`); the editor calls it on description changes. - * §14.5: the pure function and its test vectors (Kotlin only, like analytics). +* **Evidence** (chronological replay of the household's 1 775 Splitwise expenses, aggregates only, `category-suggestion.md`): history step right 92.3 % of the time and used for 71.5 % of rows; keyword step 94.2 % on 7.8 %; overall 79.5 % correct versus 31.9 % for today's AC-8.6 default alone. At cold start (no history) keywords fire on 78 % of rows and are right 96 % of the time. +* **Spec changes:** + * AC-8.6: the category default becomes "the suggestion of §9.11, which falls back to the current rule". + * New §9.11 "Category suggestion": the text CS-1 to CS-7 of `category-suggestion.md`. + * New contract file `contracts/category-keywords.csv` (columns `keyword,match,default_key,source`; draft `docs/ux/category-keywords.csv`, 149 entries), listed in §14.8. + * §14.2.4: `ExpenseRepository` gains `suspend fun suggestCategory(accountId: AccountId?, description: String): CategoryId?` (computed in `:data` from Room and the keyword table; the editor calls it on description and account changes while the category was not picked by hand). + * §14.5: a pure Kotlin function `suggestCategory(description, history, keywords, activeCategories, fallback)` in `:domain` with test vectors (Kotlin only, like analytics). * `strings.csv`: `expense_editor_category_suggested` "Vald utifrån beskrivningen" / "Chosen from the description". +* **Open points** (owner, low stakes; defaults in brackets): keep the five keywords that pick Övrigt, e.g. insurance words [keep]; "most recent" vs "most frequent" history match, the latter about one point better [most recent, as chosen]. + +## UX-7 §10.8 misses nine Swedish Splitwise category names + +* **Finding** (from the export analysis): 21 rows use Swedish Splitwise category names that the §10.8 mapping table lacks: Utbildning (4), Tjänster (3), Livet - Övrigt (3), Kläder (3), Städning (2), Flyg (2), Spel (2), Verktyg - Övrigt (1), Cykel (1). They would import as `other` with a warning. +* **Proposed decision:** add them to the §10.8 table and `server/crates/kvitt-import/data/splitwise_categories.csv` by their English equivalents: Utbildning → other, Tjänster → home, Livet - Övrigt → other, Kläder → shopping, Städning → utilities, Flyg → travel, Spel → entertainment, Verktyg - Övrigt → utilities, Cykel → transport. (Needed by M9.) ## UX-4 S-1 asks for the display name @@ -62,5 +69,5 @@ Every entry has: the owner's answer or the finding, the proposed decision text f ## Status * U1: delivered (`d27f92e` rebased on `main`), direction picked. -* In progress: muted Kalas revision; keyword list and suggestion analysis from the Splitwise export. +* Done: muted Kalas revision (`directions/3-kalas/`, awaiting the owner's look); keyword list and suggestion analysis (`category-keywords.csv`, `category-suggestion.md`). * Next: owner approves the muted palette → U2 `design-spec.md`, renamed `strings.csv`, per-screen PNGs, and the final values for UX-3 and UX-5.