Files
motovaultpro/.claude/skills/mvp-debugging-playbook/SKILL.md
T
Eric GullicksonandClaude Fable 5 c239bb9347
Deploy to Staging / Build Images (push) Successful in 5m32s
Deploy to Staging / Deploy to Staging (push) Successful in 44s
Deploy to Staging / Verify Staging (push) Successful in 5s
Deploy to Staging / Notify Staging Ready (push) Successful in 4s
Deploy to Staging / Notify Staging Failure (push) Has been skipped
chore: replace AI skill library with 16 verified mvp-* skills
Removes the old planner/decision-critic/role-agents/domain-agents system
(step-injector scripts, stale scopes, sprint-era workflow) and replaces it
with 16 ground-truth-verified skills under .claude/skills/: change control,
debugging playbook, failure archaeology, architecture contract, domain
reference, OCR/Gemini pipeline, config and secrets, build and env, run and
operate, diagnostics (with tested scripts), validation and QA, docs and
writing, launch readiness, deploy-safety campaign, proof and analysis
toolkit, and research frontier. RULE 0/1/2, the temporal-contamination rule,
and the decision stress-test protocol are carried forward into the new
skills; the retired content remains in git history.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-09 20:21:42 -05:00

247 lines
27 KiB
Markdown

---
name: mvp-debugging-playbook
description: Symptom-to-cause triage for MotoVaultPro. Load when debugging any of - numbers display wrong, .toFixed is not a function, values arrive as strings, dates off by one day, unexpected 403 (EMAIL_NOT_VERIFIED, TIER_REQUIRED, Unauthorized), stale data after create/update/delete, deleted record still shows, blank page on load, API calls never fire, bug only on mobile or only on desktop, route unexpectedly public or unprotected, VIN decode timeout or failure, OCR/receipt extraction returns empty fields, migration did not run, missing table or column, container unhealthy, staging deploy failed, health check failing, jest hangs or never exits. Gives the discriminating check and fix pattern for each known failure mode.
---
# MotoVaultPro Debugging Playbook
Known failure modes of this codebase, each with a discriminating check (an observation that separates this cause from lookalikes) and the sanctioned fix pattern. All file paths are repo-relative. All line numbers verified 2026-07-07 and may drift; the "Provenance and maintenance" section at the end has re-verification commands.
**When to use:** you have a symptom (bug report, failing staging deploy, weird API response) and need to know where to look and what the likely cause is.
**When NOT to use — route instead:**
- Investigation smells like a past battle (date bugs, numeric-string bugs, OCR engine behavior, auth races, "why is there no VIN cache") -> read `mvp-failure-archaeology` FIRST so you do not re-fight a settled war or reintroduce a reverted fix.
- You need LogQL recipes, Grafana access, health-check anatomy, or container debugging technique -> `mvp-diagnostics-and-logging`.
- You need to set up an environment or run tests at all -> `mvp-build-and-env`.
- You are deciding how to ship the fix (issue/branch/PR, review rules) -> `mvp-change-control`.
- You need rigorous root-cause methodology or benchmark discipline -> `mvp-proof-and-analysis-toolkit`.
- Deploy/rollback mechanics on staging or prod -> `mvp-run-and-operate`.
**Environment reality (2026-07-07):** development is done by AI sessions; end-to-end verification happens on STAGING via the PR deploy pipeline (every PR push builds images and redeploys staging). There is no fully working local dev loop. CI gates nothing beyond build + boot (canonical statement: `mvp-validation-and-qa` section 1) — do not assume "CI passed" means anything about correctness.
**Owner non-negotiables — never route around them while debugging:**
1. No destructive database operations without a fresh backup. This includes `make clean` (destroys volumes), `scripts/import-database.sh --drop-existing`, schema migrations on staging/prod, and the backend integration tests, which `DROP TABLE ... CASCADE` on the shared dev database (e.g. `backend/src/features/vehicles/tests/integration/vehicles.integration.test.ts:38`).
2. Never hand-edit files on the staging/prod servers — `rsync --delete` on every deploy reverts them. All server changes flow through the repo and workflows.
## First five minutes checklist
Before forming any theory, gather these observations (cross-ref `mvp-diagnostics-and-logging` for the full cookbook):
1. **Health endpoints.** `curl -s https://staging.motovaultpro.com/api/health | jq .` — must be `status: "healthy"` with a `features` array (20 features listed in `backend/src/app.ts`; the staging verify job requires a 13-feature subset, `.gitea/workflows/staging.yaml`).
2. **Container state** (on the server, `/opt/motovaultpro`): `docker compose -f docker-compose.yml -f docker-compose.staging.yml ps` — look for `unhealthy` or restart loops. Container names on staging carry a `-staging` suffix (`mvp-backend-staging`); dev/base names are `mvp-backend`, `mvp-ocr`, etc.
3. **Backend request logs in Grafana** (logs.staging.motovaultpro.com, RFC1918-only access):
`{container="mvp-backend-staging"} | json | msg="Request processed" | status >= 400`
Fields available: `level, requestId, method, path, status, duration`. `msg="Request processed"` is the request-log filter key.
4. **Errors across all containers:** `{container=~"mvp-.*"} | json | level="error"`.
5. **OCR container logs** if the symptom involves VIN/receipts/manuals: `docker logs --tail 200 mvp-ocr-staging` on staging (dev/base name is `mvp-ocr`; prod also uses the un-suffixed `mvp-ocr` — the blue-green overlay defines no OCR variant, it is a single shared instance there).
6. **Frontend errors: browser devtools console only.** Frontend logs never reach Loki.
7. **Did a deploy just happen?** Every PR push redeploys staging with brief full downtime; mid-deploy 502s are expected noise.
Then, before touching code: state a hypothesis that predicts a specific observation, and check the triage table below.
## Triage table
| # | Symptom | Likely cause | Jump to |
|---|---------|--------------|---------|
| 1 | Numbers wrong, `.toFixed is not a function`, values are strings | pg NUMERIC returned as string; mapper missed coercion | 1 |
| 2 | Dates off by one day | One of the three UTC traps reintroduced | 2 |
| 3 | Unexpected 403 | EMAIL_NOT_VERIFIED vs TIER_REQUIRED vs ownership | 3 |
| 4 | Stale data after mutation; deleted record still visible | Redis cache invalidation miss | 4 |
| 5 | Blank page; API calls never fire on load | Auth-gate race / bypassed apiClient | 5 |
| 6 | Bug only on mobile (or page missing on mobile) | 768px app fork; a registration-checklist step missed | 6 |
| 7 | Route unexpectedly public / unguarded | Missing per-route preHandler; tier guard fails open | 7 |
| 8 | VIN decode failure or timeout | Gemini path: timeout stack, event-loop blocking, SDK drift | 8 |
| 9 | OCR/receipt extraction returns empty fields | Silent Gemini fallback; Vision monthly cap; engine config | 9 |
| 10 | Migration did not run; missing table/column | Not in MIGRATION_ORDER, or image not rebuilt | 10 |
| 11 | Container unhealthy; staging deploy failed | Health-check anatomy; start_period; verify job | 11 |
| 12 | Jest hangs locally / frontend jest crashes | Open pg/redis handles; broken tdd-guard reporter config | 12 |
### 1. Numbers display wrong / `.toFixed` crashes / values are strings
**Likely cause.** The pg driver returns PostgreSQL `NUMERIC`/`DECIMAL` (type OID 1700) as JavaScript strings. The only global type-parser override in `backend/src/core/config/database.ts` is for `DATE` (OID 1082, line 12) — there is deliberately NO global numeric override. Every repository mapper must coerce numerics manually with `Number()`/`parseFloat()`. This has caused 5 historical incidents (#47, #49, #239, #241, #244); it is statistically the most likely bug when adding or changing any repository method.
**Discriminating check.** Hit the API and inspect the JSON type: a numeric field arriving as `"42.50"` (string) instead of `42.5` is this bug. Or in the failing frontend code, `typeof value === 'string'` where a number is expected.
```bash
grep -n "parseFloat\|Number(" backend/src/features/<feature>/data/<feature>.repository.ts
```
**Fix pattern.** Coerce in the repository mapper (`mapRow()` or equivalent), never in the frontend and never by returning raw rows. Beware the fuel-logs exception: `mapEnhancedRow` in `backend/src/features/fuel-logs/data/fuel-logs.repository.ts` deliberately returns coerced snake_case rows and the service's `toEnhancedResponse` does the camelCase mapping — do not "normalize" this without reading `mvp-failure-archaeology` (incident #47 -> #244 chain). Do NOT add a global OID 1700 parser as a drive-by fix; that is an architectural decision (see `mvp-architecture-contract`).
### 2. Dates off by one day
**Likely cause.** One of three settled UTC traps has been reintroduced:
(a) backend: removing/bypassing the DATE parser override `types.setTypeParser(1082, ...)` in `backend/src/core/config/database.ts:12` — pg would return DATE as a local-midnight `Date` object that shifts a day under `toISOString()`;
(b) frontend: `new Date("YYYY-MM-DD")` parses as UTC midnight, then `toLocaleDateString()` shifts a day back west of UTC;
(c) OCR/backend: `toISOString().split('T')[0]` on a Date built from a date-only value.
**Discriminating check.** Trace where the value first deviates: check the raw API response (should be a plain `"YYYY-MM-DD"` string for DATE columns). If the API string is right, the bug is frontend display; if wrong, backend/OCR.
**Fix pattern (the settled rule — do not innovate).** DATE columns flow as plain `YYYY-MM-DD` strings end to end; the full handling rules (dayjs display, lexicographic sort, when `new Date` is legitimate) are canonical in `mvp-vehicle-domain-reference` section 3. See `mvp-failure-archaeology` (four fixes in one day, 2026-03-23) before touching any date code.
### 3. Unexpected 403 responses
Three distinct producers — identify which by the response body:
| Body | Producer | Where |
|------|----------|-------|
| `code: "EMAIL_NOT_VERIFIED"` | Auth plugin email-verification guard | `backend/src/core/plugins/auth.plugin.ts` (~line 231); exempt route prefixes `/api/auth/`, `/api/onboarding/`, `/api/health`, `/health` (lines 18-23) |
| `error: "TIER_REQUIRED"` with `requiredTier`/`currentTier` | Tier gating (two parallel mechanisms) | `backend/src/core/plugins/tier-guard.plugin.ts` (~line 93) or `backend/src/core/middleware/require-tier.ts` (~line 50) |
| Generic 403, message containing "Unauthorized" | Ownership failure string-matched in the controller | e.g. `backend/src/features/fuel-logs/api/fuel-logs.controller.ts` — controllers catch and map `error.message.includes('not found')` -> 404, `includes('Unauthorized')` -> 403, else 500. There are no typed error classes. |
**Discriminating checks.**
- EMAIL_NOT_VERIFIED: does the user's Auth0 account show verified? Guard runs on every authenticated route not in the exempt list.
- TIER_REQUIRED: is the route tier-gated? (Current key catalog and limits: `mvp-config-and-secrets` section 3.) What is `user_profiles.subscription_tier`?
- Ownership: does the row's `user_id` match the caller's internal UUID? Note `request.userContext.userId` is the internal `user_profiles` UUID, NOT the Auth0 `sub`.
**Fix pattern.** For ownership bugs, check the service's ownership comparison and that the correct user UUID reached it. When adding error paths, match the existing string-matching contract exactly ("not found", "Unauthorized") or the controller will map your error to 500.
### 4. Stale data after mutation
**Likely cause.** Redis cache invalidation miss. Cache facts: keys prefixed `mvp:` (`backend/src/core/config/redis.ts:22`); fuel-logs caches per `{userId}:{unitSystem}` with TTL 300s (`fuel-logs.service.ts:18`); platform dropdown caches TTL 6h (`backend/src/features/platform/domain/platform-cache.service.ts`). Cache errors are swallowed (cacheService returns null on failure) — a broken Redis never 500s, it just serves stale or uncached.
**KNOWN LIVE BUG.** `deleteFuelLog` invalidates only the `imperial` cache keys, so a metric-preference user sees deleted fuel logs for up to 300s. If you are debugging "deleted log still shows" for a metric user, this is it — the full record (file:line evidence and status) is canonical in `mvp-launch-readiness` section 1, gap 4; file/fix the bug rather than hunting elsewhere.
**Discriminating check.** Does the stale window self-heal at the TTL (300s fuel-logs, 6h platform)? If yes, invalidation miss. Inspect keys directly:
```bash
docker exec mvp-redis redis-cli --scan --pattern 'mvp:fuel-logs:*'
```
**Fix pattern.** Invalidate BOTH unit systems (or `deletePattern`) on every mutation path; check create/update/delete all invalidate symmetrically.
### 5. Blank page / API calls never fire on load
**Likely cause.** Auth-gate race. All axios calls through the shared `apiClient` (`frontend/src/core/api/client.ts`) are queued until Auth0 init completes (`frontend/src/core/auth/auth-gate.ts`; `setAuthReady` flipped by `Auth0Provider`). Historical incidents: blank Stations page (2025-11), dashboard auth-gate (#45), mobile login IndexedDB/callback saga (#188/#190).
**Discriminating check.** Devtools Network tab: are the requests pending/queued (never sent) or sent and failing? Queued-forever means auth never initialized — check console for Auth0 errors. Sent-and-401 pre-auth means something bypassed `apiClient` (raw `axios`/`fetch` import).
**Fix pattern.** Always use `apiClient` from `core/api/client.ts` — never raw axios. Do not add data fetches that race Auth0 init. Also settled: URL-sync effects in `App.tsx` must skip `/callback`, `/signup`, `/verify-email` (child effects fire before parent effects; stripping `?code=&state=` breaks Auth0 login).
### 6. Mobile-only bugs / page missing on mobile
**Likely cause.** The app hard-forks at `window.innerWidth <= 768` plus a UA regex in `frontend/src/App.tsx` (~lines 360-398). Desktop renders react-router `<Routes>`; mobile renders a Zustand-driven screen switcher (`useNavigationStore().activeScreen`, NO react-router). A page can work perfectly on desktop and be entirely absent on mobile.
**Discriminating check.** Shrink the window below 768px (or use device emulation). Then walk the CANONICAL registration checklist in `mvp-architecture-contract` invariant 2.7 (7 steps: desktop route, `MobileScreen` union, both nav maps, lazy import, mobile render block, navigation entry point) and find the missing step.
**Fix pattern.** Complete every step of that checklist — a screen registered in the maps but missing its lazy import or a navigation entry point is silently unreachable. Note the breakpoint systems are NOT uniform: app fork = 768px, MUI component checks = `sm` (600) / `md` (900). Mobile+desktop is a hard project requirement — verify both on staging.
### 7. Route unexpectedly public or unguarded
**Likely cause.** Auth is per-route: every protected route must list `preHandler: [fastify.authenticate]` explicitly — there is NO global auth hook. Intentional exceptions exist (webhooks, signup, health, `/auth/verify`, feature-tiers config): the canonical list lives in the header comment of `.claude/skills/mvp-diagnostics-and-logging/scripts/check-route-auth.sh` — check a suspect route against it before "fixing" a deliberately public endpoint. Forgetting the preHandler ships an open route, and CI will not catch it.
**Compounding traps (all verified):**
- The tier guard FAILS OPEN on unknown feature keys: `canAccessFeature` in `backend/src/core/config/feature-tiers.ts` (~lines 57-61) returns `true` for unregistered keys. A typo in a `featureKey` silently ungates a plugin-guarded route.
- Two parallel tier mechanisms exist with different failure behavior — the middleware 500s on unknown keys, the plugin decorator fails open (mechanics canonical in `mvp-config-and-secrets` section 3).
- `backend/src/features/backup/api/backup.routes.ts` uses untyped `(fastify as any).requireAdmin` 16 times (from line 32); a misspelled guard there compiles clean and ships an unguarded admin endpoint. Do not copy that file as a route template.
**Discriminating check.** `curl` the route with no Authorization header — a 200 on a should-be-protected route confirms. Audit:
```bash
grep -L "authenticate" backend/src/features/*/api/*.routes.ts
grep -rn "featureKey" backend/src/features/*/api/*.routes.ts # then match each key against FEATURE_TIERS
```
**Fix pattern.** Add the explicit preHandler; verify the featureKey exists in `FEATURE_TIERS`; prefer the typed `fastify.requireAdmin`/`fastify.requireTier` decorators.
### 8. VIN decode failures / timeouts
**The chain:** frontend `POST /vehicles/decode-vin` (axios timeout 120s, `frontend/src/features/vehicles/api/vehicles.api.ts:90`) -> `vehicles.controller.ts` (maps OCR 503/422 -> 502, timeout/abort -> 504) -> `backend/src/features/ocr/external/ocr-client.ts` (`OCR_TIMEOUT_MS = 120000`, line 9, hardcoded) -> Python OCR container `POST /decode/vin` -> Gemini with Google Search grounding.
**Likely causes, in order:**
1. **Timeout race:** frontend 120s == backend-client 120s with no headroom, and there is NO service-side timeout on the Gemini call itself. Search-grounded decodes have been observed >60s.
2. **Event-loop blocking:** `ocr/app/routers/decode.py` calls the synchronous `_gemini_engine.decode_vin(vin)` inside an `async def` handler — a slow call blocks the whole uvicorn event loop including `/health`. The OCR healthcheck (docker-compose.yml: interval 5s, timeout 5s, retries 3) can then mark the container unhealthy and Docker can restart it MID-REQUEST.
3. **SDK drift:** `google-genai>=1.0.0` is unpinned in `ocr/requirements.txt`; new config params have broken staging before (pydantic validation error from `AutomaticFunctionCallingConfig` — reverted; never re-add AFC config).
**Discriminating check.** `docker logs mvp-ocr-staging` (on staging; base/prod name `mvp-ocr`) around the failure time. Diagnostic raw-JSON logging exists: `gemini_engine.py` logs `"Gemini decoded VIN %s (confidence=...) raw=%s"` (~line 401) with the full raw Gemini response. A container restart in `docker ps` uptime plus a 504 upstream = the healthcheck-kill scenario. A pydantic ValidationError in logs = SDK drift.
**Fix pattern.** Do not "fix" by raising the frontend timeout again (already raised 30->60->120s; the fix is service-side). Settled: there is NO VIN decode cache by design (deleted after race/staleness bugs — see `mvp-failure-archaeology`); do not re-add one. Model year is computed deterministically from VIN positions 7/10, never trusted from the LLM.
### 9. OCR / receipt extraction returns empty results
**Likely cause.** In `ocr/app/extractors/maintenance_receipt_extractor.py` (~lines 129-133), any Gemini failure is caught and silently falls back to OCR-only with `gemini_fields = {}` — the user sees "no fields extracted" with no error. Also check:
- **Vision monthly cap:** `VISION_MONTHLY_LIMIT` is 1000 requests/month; compose sets `OCR_PRIMARY_ENGINE: google_vision`, `OCR_FALLBACK_ENGINE: paddleocr` (docker-compose.yml ~lines 201-206, overriding the code defaults in `ocr/app/config.py`). Cap exhaustion degrades to PaddleOCR quality.
- **Model config:** compose deploys `GEMINI_MODEL: gemini-3-flash-preview` (line 210) — a preview model; the code default is `gemini-2.5-flash`. Extraction-quality regressions can be model-side.
- Email-ingestion amplification: unconfident classification runs BOTH fuel and maintenance OCR endpoints per attachment, burning the Vision cap faster.
**Discriminating check.** `docker logs mvp-ocr-staging | grep -i "Gemini extraction failed"` (base/prod container name is `mvp-ocr`) — the fallback logs a warning with the exception. WIF/auth failures log "Gemini authentication failed". Verify effective engine config: `docker exec mvp-ocr-staging env | grep -E "OCR_|GEMINI|VISION"`.
**Fix pattern.** Fix the upstream Gemini/auth failure; do not paper over with confidence-threshold tweaks. The WIF credential chain (Auth0 M2M -> `google-wif-config.json` at `/run/secrets/`) is duplicated in `gemini_engine.py` AND `maintenance_receipt_extractor.py` `_get_client()` — auth fixes must be applied in both.
### 10. Migration did not run / missing table or column
**Likely cause.** Migrations are feature-owned SQL run in the hard-coded `MIGRATION_ORDER` array in `backend/src/_system/migrations/run-all.ts` (line 17). A new feature's `migrations/` directory is SILENTLY skipped unless the feature is appended to that array. Migrations run automatically on backend container start (`backend/Dockerfile:84`: `node dist/_system/migrations/run-all.js && npm start`), which means they are packaged into the image — a migration added to source requires an IMAGE REBUILD and redeploy to run on staging/prod.
**Discriminating check.**
```bash
grep -n "features/<your-feature>" backend/src/_system/migrations/run-all.ts # is it in MIGRATION_ORDER?
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "SELECT * FROM _migrations ORDER BY executed_at DESC LIMIT 20;"
docker logs mvp-backend 2>&1 | grep -i migrat
```
(Adjust container names/DB credentials per environment; on staging they carry `-staging`.)
**Fix pattern.** Append to `MIGRATION_ORDER` respecting dependencies (`update_updated_at_column()` is defined in `features/vehicles`, which must run first; `core/identity-migration` runs last). NON-NEGOTIABLE: schema migrations on staging/prod require a fresh backup first (`scripts/ci/maintenance-migrate.sh backup`, or `make db-backup`).
### 11. Container unhealthy / staging deploy failed
**Health-check anatomy (docker-compose.yml, verified):** backend has `start_period: 180s` (line 159) specifically because auto-migrations run on start — a backend "starting" for 2 minutes is normal, not a failure. OCR: interval 5s / timeout 5s / start_period 15s (and see symptom 8 for why OCR can flap under load). The `verify-staging` job waits up to 4 minutes (48x5s) for Docker health, then curls `https://staging.motovaultpro.com/api/health` and requires `status=healthy` plus a 13-feature required list (`.gitea/workflows/staging.yaml`, `REQUIRED_FEATURES`).
**Discriminating check, in order:**
1. Which job failed? Build failure = code/Dockerfile problem. Deploy failure = server-side (disk, secrets). Verify failure = app boot problem.
2. `docker inspect --format '{{json .State.Health}}' mvp-backend-staging | jq .` — see the actual failing probe output.
3. `docker logs mvp-backend-staging --tail 100` — migration errors are the most common boot killer.
4. Disk: `df -h /` on the staging runner — the 29G root fills with per-commit images; a full disk fails builds and deploys in confusing ways (`docker system df`; a daily prune cron exists out-of-band).
5. Missing secret trap: Docker bind-mounting a missing secret file creates a DIRECTORY; the workflows contain `rm -rf` cleanup for exactly this. An empty directory where a secret file should be breaks the consumer at read time.
**Fix pattern.** Fix in the repo and redeploy via the workflow. Never hand-edit on the server (non-negotiable; rsync --delete reverts it). Full deploy/rollback procedure: `mvp-run-and-operate`.
### 12. Jest hangs locally / frontend jest broken
**Backend:** `cd backend && npm test` runs tests and then never exits — pg pool and ioredis connect eagerly at module import (`backend/src/core/config/database.ts`, `redis.ts``new Redis(...)` with no `lazyConnect`) and `backend/jest.config.js` sets no `forceExit`. Run:
```bash
cd backend && npx jest --forceExit # or: npm test -- --forceExit
```
Also remember: backend INTEGRATION tests `DROP TABLE ... CASCADE` on the database they point at — never aim them at a database you care about without a fresh backup (non-negotiable).
**Frontend:** `cd frontend && npm test` is broken on the host: `frontend/jest.config.ts` registers the `tdd-guard-jest` reporter (line 31) with a hardcoded `projectRoot: '/home/egullickson/motovaultpro'` (line 33) — a Linux path that does not exist on this macOS checkout. Discriminating check: the failure mentions tdd-guard or that path. A verified working fallback exists — `npx jest --reporters=default` — with its traps and the known-red baseline documented in `mvp-build-and-env` section 2 (the single home for frontend-jest mechanics). Note there is no in-container run either (nginx-only image, tests dockerignored); end-to-end verification is still staging (see `mvp-validation-and-qa`).
## Discriminating-experiment discipline
Cheap rules that prevent the historical "fix symptoms until root cause found" chains (the mobile-login saga took 9 commits; the Tesseract VIN saga took 12 in one day):
1. **Predict before you look.** Write the hypothesis as "if X is the cause, then I will observe exactly Y at Z" — then look. If you find yourself explaining an observation after the fact, it did not discriminate. Full protocol: `mvp-proof-and-analysis-toolkit`.
2. **One variable per experiment.** Especially on staging, where every PR push redeploys the whole stack: change one thing, push, observe. Two changes per push means an ambiguous result.
3. **Locate the first deviation, not the last symptom.** Trace the value/request through the chain (DB -> repository mapper -> service -> controller -> API JSON -> frontend) and find the FIRST point it is wrong. Most historical band-aids (frontend `Number()` wrappers, timeout bumps) patched the last symptom.
4. **Distrust green CI.** CI runs zero tests and zero lint; "CI passed" only means images built and staging booted. Evidence standards: `mvp-validation-and-qa`.
5. **Check the archaeology first.** If the bug involves dates, numeric strings, VIN caching, auth races, Stripe IDs, or OCR engines, someone already fought it. `mvp-failure-archaeology` lists the settled outcomes and the reverts you must not resurrect.
## Provenance and maintenance
Authored 2026-07-07 by direct repo inspection; discovery-report claims were independently re-verified against code. Line numbers drift — re-verify before relying on them:
| Volatile fact | Re-verification command |
|---------------|------------------------|
| DATE-only parser override (OID 1082), no NUMERIC override | `grep -n "setTypeParser" backend/src/core/config/database.ts` |
| deleteFuelLog imperial-only invalidation (live bug) | `grep -n "invalidateCaches" backend/src/features/fuel-logs/domain/fuel-logs.service.ts` |
| Cache prefix `mvp:`, fuel-logs TTL 300s, platform TTL 6h | `grep -n "prefix" backend/src/core/config/redis.ts; grep -n "cacheTTL" backend/src/features/fuel-logs/domain/fuel-logs.service.ts; grep -n "6 \* 3600" backend/src/features/platform/domain/platform-cache.service.ts` |
| EMAIL_NOT_VERIFIED guard + exempt routes | `grep -n "EMAIL_NOT_VERIFIED\|VERIFICATION_EXEMPT" backend/src/core/plugins/auth.plugin.ts` |
| Tier guard fails open on unknown keys | `grep -n -A3 "fail open" backend/src/core/config/feature-tiers.ts` |
| Controller string-matched error mapping | `grep -n "includes('not found')\|includes('Unauthorized')" backend/src/features/fuel-logs/api/fuel-logs.controller.ts` |
| Untyped requireAdmin in backup routes | `grep -c "(fastify as any).requireAdmin" backend/src/features/backup/api/backup.routes.ts` |
| 120s timeouts (frontend + backend OCR client) | `grep -n "OCR_TIMEOUT_MS" backend/src/features/ocr/external/ocr-client.ts; grep -n "timeout" frontend/src/features/vehicles/api/vehicles.api.ts` |
| Sync Gemini call in async decode handler | `grep -n "decode_vin(vin)" ocr/app/routers/decode.py` |
| Gemini raw-JSON diagnostic logging | `grep -n "raw=%s" ocr/app/engines/gemini_engine.py` |
| Silent empty-fields fallback | `grep -n -B1 "gemini_fields = {}" ocr/app/extractors/maintenance_receipt_extractor.py` |
| Compose OCR engine/model config | `grep -n "OCR_PRIMARY_ENGINE\|GEMINI_MODEL\|VISION_MONTHLY_LIMIT" docker-compose.yml` |
| MIGRATION_ORDER + migrate-on-start | `grep -n "MIGRATION_ORDER" backend/src/_system/migrations/run-all.ts; grep -n "run-all" backend/Dockerfile` |
| Backend start_period 180s; OCR healthcheck 5s/5s | `grep -n "start_period" docker-compose.yml` |
| 13-feature staging verify list | `grep -n "REQUIRED_FEATURES" .gitea/workflows/staging.yaml` |
| 768px fork + page-registration checklist (canonical: `mvp-architecture-contract` 2.7) | `grep -n "innerWidth <= 768" frontend/src/App.tsx; grep -n "routeToScreen\|screenToRoute" frontend/src/core/store/navigation.ts` |
| Auth-gate request queue | `grep -n "authReady\|queueRequest" frontend/src/core/api/client.ts` |
| Frontend jest hardcoded projectRoot | `grep -n "projectRoot" frontend/jest.config.ts` |
| Integration tests DROP TABLE CASCADE | `grep -rn "DROP TABLE" backend/src/features/*/tests/integration/` |