motovaultpro/.claude/skills/mvp-failure-archaeology/SKILL.md

name, description

name

description

mvp-failure-archaeology

Chronicle of every major MotoVaultPro incident, dead end, and revert - symptom, root cause, evidence (commit SHAs, issue numbers), status. Load this skill BEFORE re-investigating any symptom that smells like a past battle, proposing to add caching, "simplifying" a mapper or date handler, or reverting an odd-looking line. Raw symptom triage starts in mvp-debugging-playbook; this is the history behind its entries. Distinguishing triggers - "has this been tried before", "Invalid time value", mobile login redirect loop, Auth0 callback stripped, removeChild/NotFoundError from Google Maps, Gemini/google-genai pydantic validation error, Stripe subscription has no payment method, staging deploy "Take N", "why is there no VIN cache", OCR returns no VIN pattern. Also load before deleting code that looks vestigial - it may be a settled battle.

MotoVaultPro Failure Archaeology

The project's institutional memory. 706 commits (2025-07-15 to 2026-06-20), restarted three times: "Fresh Start" 2e8816d (2025-08-07), "Initial Commit" a052040 (2025-09-17), "Initial Commit" b611b56 (2025-12-17). The modern, issue-driven era begins 2025-12-17. Every fact below was re-verified against the repo on 2026-07-07.

When to use / When NOT to use

Use this skill when:

• A symptom matches anything in the chronicle table - check here BEFORE debugging from scratch.
• You are about to change code listed under "Settled battles" - the odd-looking line is load-bearing.
• You are tempted to add a cache, a new Date() call on a DATE column, or an SDK config option to the Gemini path.
• You finished a significant investigation - the maintenance rule at the bottom says append it here.

Do NOT use this skill for:

• Live triage steps and discriminating experiments - that is mvp-debugging-playbook (this skill is the history behind those entries).
• The rules and invariants themselves with full rationale - mvp-architecture-contract.
• How to classify/gate/ship a fix - mvp-change-control.
• Domain math (VIN check digits, fuel efficiency, DATE semantics theory) - mvp-vehicle-domain-reference.
• The Gemini/OCR subsystem's current design - mvp-ocr-gemini-pipeline.
• Evidence standards for calling something "root cause" - mvp-proof-and-analysis-toolkit.

Reading protocol for an AI session: scan the chronicle table for your symptom, read that row's status, then check "Settled battles" before writing any code.

1. Incident chronicle

Chronological. "Evidence" = commit SHAs and issue numbers you can git show yourself.

#	Symptom	Root cause	Evidence	Status
1	Blank Stations page; React crashes (removeChild/NotFoundError) from Google Maps (2025-11-04, 8-commit day)	(a) Data fetched before auth initialized; (b) Maps async-loading callback misused; (c) Google Maps mutates DOM nodes React owns	0e8d8e7, f1dd488, def2494 (auth polling), 4151f58, 8e52f3f (async loading), 715250d, 9a01ebd (suppression), 45fea0f "Finally Working"	Resolved, but the DOM-conflict errors are SUPPRESSED, not eliminated: frontend/src/App.tsx (~line 409) has a global error + unhandledrejection handler that swallows removeChild-class messages. If Maps errors "disappear", that handler is why.
2	Prod Traefik/CI broken during GitHub-to-Gitea migration (2025-12-29 to 31)	Configs not checked out on prod; acme.json permissions; runner config; blue-green deploy path bug serving stale content	83d79da, eaab8c0, dfff25e, 9b22c5e, da3ad51, 17d868a, 3464fad "Traefik won't start. Take 12", e5837df, 9c0fafd, then 13abbc1 + c57a05d (same message twice - the FIRST carries the auto-rollback.sh/health-check.sh fixes plus DEPLOY_PATH/verification changes; the second is a follow-up touching switch-traffic.sh and the prod workflow only)	Resolved. scripts/ci/ blue-green scripts are battle-scarred; treat with respect. See mvp-run-and-operate.
3	Fuel Logs API 500 "Invalid time value" (#47, 2026-01-17)	"Enhanced" repo methods returned camelCase mapRow() output but the service's toEnhancedResponse() expected snake_case raw rows	574acf3 "return raw rows from enhanced repository methods (refs #47)"	"Fixed" by making enhanced methods return RAW rows - a decision that planted the latent bug fixed four months later as #244 (incident 12). Canonical example of a fix that violates a convention creating future debt.
4	Fuel values display wrong (#49, same day as #47)	node-postgres returns DECIMAL/NUMERIC columns as strings	5c62b6a - Number() coercion in service layer	Band-aid at the wrong layer. Theme recurred as #239/#241/#243/#244 (incident 12). Rule now: coerce in repository mappers.
5	VIN OCR fails "No VIN pattern found" on all images (#113, 2026-02-06, 12+ commits in one day)	Tesseract preprocessing: wrong channel selection, char whitelist incompatible with LSTM engine, naive first-17-chars trim of noisy text	6a4c213, a07ec32, 63c027a, ae5221c, 432b3bd, final e4336ce (sliding window + char deletion with check-digit validation)	Resolved, then obsoleted: Tesseract deleted in b9fe222. Do not study Tesseract preprocessing - the engine is gone.
6	PaddleOCR migration breakage (#115, PR #122, 2026-02-07)	New engine API differences, timeouts, crop-box regression	639ca11, dab4a3b, 9d2d4e5, fcffb0b, 3adbb10 "OCR Timout still", 0499c90, 9a2b12c, 3c1a090 (crop regression, refs #120)	Resolved; migration completed. Later superseded again by Vision/Gemini path (#127, #129). Every OCR engine swap has cost 5-10 follow-up fix commits.
7	Mobile login redirects to homepage, Auth0 page never shown (#188/#189/#190, 2026-02-14/15, 9 commits)	TWO independent root causes: (a) IndexedDB token cache used store.getAll() which returns raw values not key-value entries, so the in-memory cache was always empty after reload; (b) App.tsx URL-sync effect ran history.replaceState() before Auth0Provider's effect, stripping ?code=&state= from /callback	Symptom-chasing chain a195fa9, 6e493e9, 38debaa, b5b82db; root cause (a) da59168; root cause (b) 850f713	Resolved. The guard survives at frontend/src/App.tsx lines ~374/386: URL-sync effects early-return on /callback, /signup, /verify-email, /guide, /guide/. Settled battle - see section 2.
8	Stripe live wiring failures (2026-02-15/16)	Wrong price/product IDs (live vs sandbox), payment method never attached to customer, subscription period dates read from the wrong object level	ddae397, c1e8807, bc0be75, 15956a8 "take 2", 07c3d85 "take 3", then correct fixes 72e5573 (attach payment method first), 8a73352 (charge immediately, item-level period dates); properly refactored via #205/#209/#210 in 93e79d1 (ensureStripeCustomer)	Resolved via refactor after take-3 flailing. ensureStripeCustomer lives in backend/src/features/subscriptions/domain/subscriptions.service.ts (~line 174).
9	VIN decode saga: NHTSA replaced by Gemini, then reliability chase (#223-#230, 2026-02-18 to 02-28)	NHTSA decode replaced with Gemini via the OCR service (a75f7b5, 3cd6125, 5cbf9c7, f590421, d967367, all 2026-02-18); then Gemini cold-start timeouts, a VIN-cache race + fuzzy-match misses, and wrong model years	3b5b847 (timeout to 60s, refs #229/#230), 361f58d (cache race, refs #229), 7d90f4b (year-code table in prompt), then 283ba6b (2026-02-20) "Remove VIN Cache" - 93 lines deleted, cache NOT fixed - with a final year-fix straggler 0055d9f (2026-02-28)	Settled: there is NO VIN decode cache, by design. Note the vin_cache TABLE still exists in backend/src/features/vehicles/migrations/001_create_vehicles_tables.sql and the vehicles README still claims caching - both are stale; the code (no cache reads/writes) wins.
10	google-genai SDK migration breaks VIN decode on staging (#231-#236, 2026-02-28 to 03-01)	Vertex AI SDK replaced by google-genai (398d673, b7f472b, 9f51e62, 1464a0e); then AutomaticFunctionCallingConfig(max_remote_calls=3) passed to a google-genai version that rejects it with a pydantic validation error	936753f (timeouts + logic; raised frontend VIN timeout 60s to 120s), 1add6c8 (remove param), 56df5d4 (revert AFC entirely + diagnostic logging)	Resolved. AFC config is confirmed absent from ocr/app/engines/gemini_engine.py (2026-07-07). Migration complete; Vertex AI SDK gone.
11	Dates off by one day (#237, 2026-03-23 - four fixes in ONE day)	Three distinct UTC traps: (a) pg parses DATE columns to local-midnight Date, then toISOString() shifts a day; (b) frontend new Date("YYYY-MM-DD") parses as UTC midnight, toLocaleDateString() shifts back; (c) OCR parser used toISOString().split('T')[0]	c05e33e and f0fc427 (both "fix: Date picker bug"; f0fc427 added types.setTypeParser(1082, val => val) to backend/src/core/config/database.ts), 1e056f0 (dayjs for display/sort, refs #237), 087f7b9 (OCR parser local-time formatting, refs #237)	All three fixed. Settled: DATE columns flow as plain YYYY-MM-DD strings end-to-end; display via dayjs; sort lexicographically. See section 2 and mvp-vehicle-domain-reference.
12	Maintenance cost blank on vehicle summary; pg-numeric bug train returns (#239, #241, #243, #244, 2026-05-15/16)	Same root cause as #49 four months earlier: pg returns NUMERIC as strings and mappers were inconsistent. The audit found three features affected. #244 specifically: fuel-logs "enhanced" methods had returned raw pg rows since 574acf3 (incident 3)	a49f419 + 55b8b67 (refs #239), fdc34ae (refs #241 - coerce in maintenance + ownership-costs mappers, revert frontend Number() workarounds), 0d90829 (refs #244 - fuel-logs enhanced methods finally coerce). #243 (stations prices) has no dedicated fix commit; stations mappers already coerce via parseFloat (verified 2026-07-07)	Resolved, with the architectural rule now in root CLAUDE.md: repository mappers coerce numerics; never return raw rows to the API layer.
13	UUID identity migration and its fallout (#206 + subs #211-#217, then #220, 2026-02-16/17)	auth0_sub VARCHAR identity replaced by user_profiles.id UUID across 17 features; big-bang migration left stragglers	Migration fixes 7fc80ab, 28165e4 (refs #206); fallout batch: dd3b58e (remaining controllers), e909313 (remaining auth0_sub references), both refs #220	Completed after two mop-up commits. Lesson: a cross-cutting identity migration WILL leave stragglers; grep for the old identifier after "done".

2. Settled battles - DO NOT RE-FIGHT

Each of these looks like a candidate for "cleanup". Each one re-fights an incident above. Verify the evidence before touching any of them.

Rule	Evidence	If you break it
Keep types.setTypeParser(1082, (val: string) => val) at backend/src/core/config/database.ts line 12	f0fc427, incident 11	Every DATE column shifts one day for users west of UTC
Repository mappers MUST coerce NUMERIC/DECIMAL columns (Number() or parseFloat); never return raw pg rows to services/API	574acf3 created the debt; fdc34ae, 0d90829 paid it (incidents 3, 4, 12)	Strings leak into arithmetic; NaN/blank UI four months later
NO VIN decode cache. Deleted deliberately, not lost	283ba6b (incident 9). Stale remnants: vin_cache table in migration 001 and the vehicles README caching claim - docs/schema are stale, code wins	You re-inherit the cache race + fuzzy-match wrong-vehicle bugs 361f58d fought
NO AutomaticFunctionCallingConfig (or max_remote_calls) on google-genai calls in ocr/app/engines/gemini_engine.py	1add6c8, 56df5d4 (incident 10)	Pydantic validation error; VIN decode dies on staging
App.tsx URL-sync effects must early-return on /callback, /signup, /verify-email, /guide, /guide/ (lines ~374, ~386)	850f713 (incident 7), /guide added via #203	history.replaceState() strips Auth0 ?code=&state=; login breaks, worst on mobile
VIN decode frontend timeout is 120s (timeout: 120000 in frontend/src/features/vehicles/api/vehicles.api.ts line ~90)	Raised 60s to 120s in 936753f for Gemini + Search grounding. The "60s" figure floats around in older notes - it is STALE	Gemini cold start + grounding exceeds 60s; decode aborts client-side while server still working
Stripe: attach payment method to customer BEFORE creating subscription; charge immediately; read period dates from the subscription ITEM level; go through ensureStripeCustomer	72e5573, 8a73352, 93e79d1 (incident 8)	Subscriptions with no payment method; wrong renewal dates; the take-2/take-3 flailing repeats
Log Fuel field order and decimal keypad are deliberate UX (#246, 2221819, 2026-06-19)	Most-used fields first on mobile; inputMode decimal keypad	"Alphabetizing" or "grouping logically" reverts a shipped UX decision
nginx try_files $uri /index.html; in frontend/nginx.conf deliberately omits $uri/	4927b66 (refs #203): $uri/ matched the static /guide/ screenshot directory and 301-redirected, bypassing SPA routing	/guide route breaks with a trailing-slash redirect loop
App.tsx global error handler suppressing removeChild-class Google Maps errors stays	715250d, 9a01ebd (incident 1)	Users see crash overlays for harmless Maps DOM churn. But be honest in reviews: errors are suppressed, not fixed

3. Dead ends and abandoned arcs

Do not resurrect these; do not spend time investigating their remnants.

• Kubernetes: a k8s-style redesign was pursued mid-2025 (040da4c "k8s redesign complete", 17d27f4) - in reality a "Kubernetes-like Docker Compose" restructure; no k8s manifests were ever committed (see mvp-architecture-contract 1.1) - and abandoned by the 2025-12-17 restart. The project is docker-compose only. Any k8s reference you find in old history is dead.
• Legacy vehicle ETL: removed bfb0c23 (2025-12-27) after a "Possible working ETL" dead end. Gone on purpose.
• Per-vehicle TCO fields: removed from vehicle forms in #37 (9644010, merged 913e084, 2026-01-15). The ownership-costs capsule SURVIVES as its own feature - the removal was of duplicate per-vehicle fields, not the concept. Do not add TCO fields back to vehicle forms.
• Local branch list: an accident (points at 2221819, identical to the merged issue-246-reorder-log-fuel-fields tip; likely a mistyped git branch list). Safe to delete; carries no work.
• Committed backup tarball: cd7f8f5 (2026-03-22, "fix: Missing packages") accidentally committed motovaultpro_backup_2026-03-22T15-00-02.tar.gz (9,570,741 bytes); 068bb75 (same message) removed it. The blob still bloats git history. Do not "fix" this with a history rewrite without owner sign-off - it would invalidate every clone and open PR.
• Stale GitHub remote: github/main is frozen (last activity 2026-02-04); github/issue-37-remove-tco-fields was merged and never pruned. Gitea (gitea/main) is the only live remote.

4. Recurring themes, ranked by total cost

Recognize the family on contact - the third occurrence of a theme should take minutes, not a day.

1. pg type-contract drift (NUMERIC-as-string, raw rows) - 5+ incidents over 4 months (#47, #49, #239, #241, #243, #244). Highest total cost because incident 3's fix CREATED incident 12. Coercion styles are still hand-rolled per repository (Number() vs parseFloat, no shared helper) - the pattern can recur in any NEW repository method. On contact: check the mapper first.
2. UTC/timezone date shifting - dayjs picker migration (0e85cf4, 2025-12-18), then four fixes in one day (incident 11). On contact with any off-by-one-day report: it is one of the three traps in incident 11; do not invent a fourth theory first.
3. OCR/AI engine churn - Tesseract to PaddleOCR to Vision/Gemini to google-genai SDK (incidents 5, 6, 9, 10). Each swap cost 5-10 follow-up fixes but each was COMPLETED with old code deleted. Budget the follow-up tail into any future engine/SDK change proposal.
4. CI/CD deployment fragility - "Take 12" Traefik saga, blue-green path bug committed twice (incident 2). Mostly pre-2026; the remaining structural risk (zero test gates, :latest clobbering) is the subject of mvp-deploy-safety-campaign.
5. Auth-state races - blank Stations page (incident 1), mobile login (incident 7): effects and fetches racing auth initialization. On contact with any blank-page-after-login or callback symptom: check effect ordering against Auth0Provider before anything else.
6. Big-bang migration stragglers - UUID identity (incident 13), google-genai (incident 10): the migration "completes", then 1-2 mop-up commits chase remaining references. After any cross-cutting rename: grep for the old symbol before declaring done.

Maintenance rule

Every new SIGNIFICANT investigation (multi-commit debugging chain, root-cause reversal, deliberate deletion, migration fallout) gets appended to this chronicle AT PR TIME, in the same row format: symptom, root cause, evidence (SHAs + issue numbers), status. If a fix overturns a settled battle above, update that row in the same PR - do not leave this file contradicting the code. Where this file and the code disagree, the code wins; fix this file.

Provenance and maintenance

Authored 2026-07-07 by direct repo inspection (git log/show, file reads); issue titles cross-checked against the Gitea tracker (zero open issues as of 2026-07-07). Volatile facts and their re-verification commands:

Fact (as of 2026-07-07)	Re-verify with
706 commits; three restarts (2e8816d, a052040, b611b56)	git rev-list --count HEAD and git log --oneline --grep='Initial Commit|Fresh Start' -i
setTypeParser(1082, ...) present at database.ts line 12	grep -n "setTypeParser" backend/src/core/config/database.ts
VIN frontend timeout 120000	grep -n "timeout" frontend/src/features/vehicles/api/vehicles.api.ts
No AFC config in gemini_engine.py	grep -in "AutomaticFunctionCalling|max_remote_calls" ocr/app/engines/gemini_engine.py (expect no hits)
App.tsx callback-skip guard and Maps error suppression	grep -n "callback|removeChild" frontend/src/App.tsx
No VIN cache in code; stale vin_cache remnants in migration 001 + vehicles README	grep -rn "vin_cache" backend/src/
nginx try_files omits $uri/	grep -n "try_files" frontend/nginx.conf
ensureStripeCustomer in subscriptions.service.ts	grep -rn "ensureStripeCustomer" backend/src/features/subscriptions/
Stations mappers coerce with parseFloat (why #243 has no fix commit)	grep -n "parseFloat" backend/src/features/stations/data/*.repository.ts
Local branch list still exists and equals 2221819	git rev-parse list 2>/dev/null
Tarball blob still in history (9,570,741 bytes)	git show --stat cd7f8f5 | grep tar.gz
Any commit SHA or message quoted above	git log -1 --format='%h %ad %s' --date=short <sha>