chore: replace AI skill library with 16 verified mvp-* skills

Removes the old planner/decision-critic/role-agents/domain-agents system
(step-injector scripts, stale scopes, sprint-era workflow) and replaces it
with 16 ground-truth-verified skills under .claude/skills/: change control,
debugging playbook, failure archaeology, architecture contract, domain
reference, OCR/Gemini pipeline, config and secrets, build and env, run and
operate, diagnostics (with tested scripts), validation and QA, docs and
writing, launch readiness, deploy-safety campaign, proof and analysis
toolkit, and research frontier. RULE 0/1/2, the temporal-contamination rule,
and the decision stress-test protocol are carried forward into the new
skills; the retired content remains in git history.

Co-Authored-By: Claude Fable 5 <[email protected]>

.claude/skills/mvp-run-and-operate/SKILL.md

@@ -0,0 +1,185 @@
+---
+name: mvp-run-and-operate
+description: Operate MotoVaultPro environments - load when deploying to staging or production, rolling back a bad deploy, switching blue/green traffic, backing up or restoring the database, refreshing staging data from prod, running a breaking migration, or diagnosing "staging is down after my PR", "prod deployed the wrong code", "state.json says blue but green is serving", "config change did not apply to postgres/redis/grafana", "OCR blipped during deploy", "staging runner disk full", or "cert renewal failed". Covers compose file stacking, container names, the staging and production Gitea workflows, switch-traffic/health-check/auto-rollback scripts, and the legacy rollback.sh hazard.
+---
+
+# Run and Operate MotoVaultPro
+
+**When to use:** deploying, rolling back, switching traffic, backing up/restoring the database, refreshing staging data, running breaking migrations, or debugging a broken environment (staging or prod down, wrong version serving, config not applied).
+
+**When NOT to use:**
+- Getting a dev machine working, local unit tests, `make setup` -> `mvp-build-and-env`
+- Reading logs, Grafana/Loki queries, health-endpoint internals -> `mvp-diagnostics-and-logging`
+- What CI actually gates (spoiler: no tests, no lint) and PR/issue workflow -> `mvp-change-control` and `mvp-validation-and-qa`
+- Fixing the systemic deploy-safety hole (PR builds clobbering `:latest`, prod defaulting to `latest`) -> `mvp-deploy-safety-campaign`
+- Why the architecture is shaped this way -> `mvp-architecture-contract`
+- Past incidents and reverts -> `mvp-failure-archaeology`
+
+Reality check for this project (2026-07-07): development is done by AI sessions; the human owner reviews PRs; end-to-end verification happens on STAGING via the PR deploy pipeline because there is no fully working local dev loop. Deploys are how changes get verified. Treat this file as the runbook.
+
+## 1. Environment matrix
+
+Verified against `docker-compose*.yml` and `.gitea/workflows/{staging,production}.yaml` (2026-07-07).
+
+| | Dev (local) | Staging | Production |
+|---|---|---|---|
+| Compose stacking | `docker-compose.yml` alone (base, builds images locally) | base + `docker-compose.staging.yml` | base + `docker-compose.blue-green.yml` + `docker-compose.prod.yml` |
+| Server path | your machine | `/opt/motovaultpro` on `mvp-staging` (also the CI runner, label `stage`) | `/opt/motovaultpro` on prod server (internal `172.30.1.36`; runner label `prod`) |
+| Container names | `mvp-traefik`, `mvp-frontend`, `mvp-backend`, `mvp-ocr`, `mvp-postgres`, `mvp-redis`, `mvp-loki`, `mvp-alloy`, `mvp-grafana` | app containers get a `-staging` suffix (`mvp-traefik/frontend/backend/ocr/postgres/redis-staging`); the logging trio keeps base names (`mvp-loki`, `mvp-alloy`, `mvp-grafana` - no container_name override in the staging overlay) | shared: `mvp-traefik`, `mvp-ocr`, `mvp-postgres`, `mvp-redis`, `mvp-loki`, `mvp-alloy`, `mvp-grafana`; paired: `mvp-frontend-blue/green`, `mvp-backend-blue/green` |
+| URLs | https://motovaultpro.com via hosts file + self-signed cert | https://staging.motovaultpro.com, https://logs.staging.motovaultpro.com, https://traefik.staging.motovaultpro.com | https://motovaultpro.com, https://logs.motovaultpro.com |
+| Images | built from source (`build:` in base compose) | `git.motovaultpro.com/egullickson/{backend,frontend,ocr}:<7-char-sha>` via `BACKEND_IMAGE`/`FRONTEND_IMAGE`/`OCR_IMAGE` env | same registry; tag = `workflow_dispatch` input `image_tag`, DEFAULT `latest` (dangerous, see traps) |
+| Volumes | `mvp_postgres_data`, `mvp_redis_data`; host ports 5432/6379 exposed | `mvp_postgres_staging_data`, `mvp_redis_staging_data` | base volume names; `docker-compose.prod.yml` sets `ports: []` on postgres/redis (no host exposure) |
+| LOG_LEVEL | debug (compose defaults) | DEBUG (`staging.yaml` env) | INFO (`production.yaml` env) |
+| Who deploys | you, manually | CI, automatically: **every push to main AND every PR open/sync/reopen**. Full `down`/`up` = brief outage every deploy; concurrent PRs mean **last PR wins** the shared environment | human, manually: `workflow_dispatch` on `production.yaml` with `image_tag` input |
+
+Compose inheritance detail (prod): the merged prod config STILL contains the base single-stack `mvp-frontend`/`mvp-backend` services. The workflow avoids them by always naming services in `up -d`. **Never run a bare `up -d` with the prod file stack** -- it would start the base containers alongside blue/green.
+
+## 2. Staging deploy anatomy
+
+Source of truth: `.gitea/workflows/staging.yaml`. Triggers: push to `main` and every PR sync. All jobs run on the staging host itself.
+
+1. **build**: docker login to `git.motovaultpro.com`; builds backend, frontend (Vite build args from Gitea vars, including `VITE_STRIPE_PUBLISHABLE_KEY`), and ocr; tags each `:<short-sha>` AND `:latest`; pushes all six tags. This is where PR builds clobber `:latest`.
+2. **deploy-staging**:
+   - `rsync -av --delete` of `config/` and `scripts/` into `/opt/motovaultpro` (wipes any on-server edits -- this is why hand-editing server files is a non-negotiable "never")
+   - copies compose files and `secrets/app/google-wif-config.json`
+   - generates `.env` (Stripe price IDs, `GRAFANA_ADMIN_PASSWORD`, then `scripts/ci/generate-log-config.sh DEBUG >> .env` which fans one LOG_LEVEL out to backend/traefik/postgres/redis settings)
+   - `scripts/inject-secrets.sh` writes the real secret files from Gitea Actions secrets
+   - init data dirs: `data/backups` and `data/documents` chown 1001:1001; `data/traefik/acme.json` chmod 600
+   - pulls images, then `compose down --timeout 30` followed by `up -d` -- **staging takes full downtime on every deploy**
+3. **verify-staging**:
+   - asserts the 5 app containers are running, then polls Docker healthchecks up to 48 x 5s = 4 minutes per service. The backend has `start_period: 180s` because auto-migrations run on container start (`node dist/_system/migrations/run-all.js`) -- a slow migration eats into this budget.
+   - in-container `curl localhost:3001/health`
+   - external check: `https://staging.motovaultpro.com/api/health` must return `status: healthy` with all **13 required features**: `admin, auth, onboarding, vehicles, documents, fuel-logs, stations, maintenance, platform, notifications, user-profile, user-preferences, user-export`
+4. **notify** success/failure via `scripts/ci/notify.sh` (Resend email).
+
+This pipeline is also the only PR gate: green means "3 images built and staging boots healthy", nothing more (no tests, no lint -- see `mvp-validation-and-qa`).
+
+## 3. Production deploy anatomy (blue-green)
+
+Source of truth: `.gitea/workflows/production.yaml`. Manual `workflow_dispatch` only. Blue-green = two parallel frontend+backend pairs behind Traefik weighted routing; only one pair ("stack") receives traffic at a time.
+
+1. **validate**: reads `/opt/motovaultpro/config/deployment/state.json` `.active_stack` (defaults to blue if missing) and targets the OPPOSITE color.
+2. **deploy-prod**:
+   - sparse checkout; `rsync --delete` of `config/` and `scripts/` (this RESETS `state.json` and the Traefik weights file to repo defaults -- see traps)
+   - `.env` generation + `inject-secrets.sh` + data-dir init, same as staging
+   - pull images for the chosen tag, record image IDs
+   - `up -d --no-recreate mvp-postgres mvp-redis mvp-loki mvp-alloy mvp-grafana` -- shared services persist untouched across deploys (config changes to them do NOT apply; see traps)
+   - wait for postgres/redis healthy
+   - `up -d --force-recreate mvp-ocr mvp-frontend-$TARGET mvp-backend-$TARGET` -- note `mvp-ocr` is a single shared instance and is force-recreated **every** prod deploy (brief OCR outage each time)
+   - verify running image IDs match what was pulled
+   - `scripts/ci/health-check.sh $TARGET 240` (container health + HTTP endpoints, 240s budget)
+   - start Traefik, then `scripts/ci/switch-traffic.sh $TARGET instant` -- awk/sed-edits weights in `config/traefik/dynamic/blue-green.yml`; Traefik's file watcher applies it live; state.json updated
+3. **verify-prod**: container/health waits, in-container health, external `https://motovaultpro.com/api/health` with the same 13-feature check, then greps `blue-green.yml` to confirm the target stack weight is 100.
+4. **rollback job** (on failure): runs `scripts/ci/auto-rollback.sh` automatically. Emails on both paths.
+
+Migrations: non-breaking migrations run automatically on backend container start. Breaking migrations use `.gitea/workflows/maintenance.yaml` (manual, `create_backup` input default `yes`) which runs `scripts/ci/maintenance-migrate.sh [backup]`: sets ALL four Traefik weights to 0 (site returns 503), optionally `pg_dump`s to `data/backups/`, runs the migration, restores traffic.
+
+### Operator commands (on the servers, at `/opt/motovaultpro`)
+
+```bash
+# Staging stack
+docker compose -f docker-compose.yml -f docker-compose.staging.yml ps
+docker compose -f docker-compose.yml -f docker-compose.staging.yml logs -f
+docker logs -f mvp-backend-staging
+
+# Production stack (ALWAYS name services on up -d; never bare)
+docker compose -f docker-compose.yml -f docker-compose.blue-green.yml -f docker-compose.prod.yml ps
+cat config/deployment/state.json | jq .          # which stack is active (but see trap 2)
+
+# Health / traffic (DEPLOY_PATH is required so the scripts edit the live config, not a checkout)
+DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/health-check.sh blue|green [timeout_s]   # default 60s
+DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/switch-traffic.sh blue|green [instant|gradual]
+DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/auto-rollback.sh "reason"
+```
+
+`gradual` mode steps weights 25 -> 50 -> 75 -> 100 with 3s intervals. Emergency both-stacks-down recovery procedure: `docs/CICD-DEPLOY.md`, "Emergency Recovery" section (down everything, restart shared services, start one stack by name, switch traffic).
+
+## 4. Rollback
+
+**Automatic**: the production workflow's `rollback` job runs `scripts/ci/auto-rollback.sh` on verification failure. It reads `state.json` for the current/inactive stacks, health-checks the OLD (inactive) stack for 30s, switches traffic back to it, updates state, and emails. If the old stack is also unhealthy it sends a `rollback_failed` email and exits nonzero -- that is your both-stacks-down scenario (see emergency recovery above).
+
+**Manual** (on the prod server):
+
+```bash
+cd /opt/motovaultpro
+DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/health-check.sh blue 60     # confirm the target is healthy FIRST
+DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/switch-traffic.sh blue      # instant; or append "gradual"
+```
+
+```
++----------------------------------------------------------------------------+
+| WARNING: scripts/rollback.sh (repo root scripts/ dir) is LEGACY AND        |
+| DANGEROUS ON PROD. Verified 2026-07-07: it composes only base + prod       |
+| (no blue-green file), does a full `down` of the site, git-checkouts an     |
+| old commit and REBUILDS IMAGES FROM SOURCE on the prod box, then health-   |
+| checks containers named mvp-frontend/mvp-backend which DO NOT EXIST under  |
+| the blue-green topology. Running it takes prod down and cannot verify      |
+| recovery. Never use it. Use scripts/ci/switch-traffic.sh or                |
+| scripts/ci/auto-rollback.sh instead.                                       |
++----------------------------------------------------------------------------+
+```
+
+Rolling back to an older BUILD (not just the other color): re-run the production workflow with `image_tag` set to the known-good 7-char commit SHA. That deploys the old images to the inactive color and switches to it.
+
+## 5. Backup, restore, and data
+
+**Non-negotiable (owner, 2026-07-07): take a fresh backup before ANY destructive database operation.** That includes `make clean` (runs `docker compose down -v --rmi all` -- destroys DB volumes wherever you run it), `import-database.sh --drop-existing`, schema migrations on staging/prod, and the backend integration tests (which `DROP TABLE ... CASCADE` on the shared dev database).
+
+Layers, all verified 2026-07-07:
+
+- **App-level backup feature** (`backend/src/features/backup/`): admin API at `/api/admin/backups*` (list, create, get, download, restore) plus scheduled jobs -- `backup-scheduled.job.ts` and `backup-cleanup.job.ts` (daily retention cleanup at 4 AM). Writes to `./data/backups` (bind-mounted into the backend, owner 1001).
+- **Manual export**: `scripts/export-database.sh` -- interactive, or `--env production|staging` (selects container `mvp-postgres` vs `mvp-postgres-staging`), `--format sql|custom|directory` (default sql), gzips, writes to `database-exports/` with a metadata JSON and import instructions. Make shortcuts: `make db-export`, `make db-backup`.
+- **Restore**: `scripts/import-database.sh [--env ...] <file>` -- auto-detects format, **backs up the existing database to `database-exports/` first by default** (skip with `--no-backup`; do not skip), supports `--drop-existing --create-db` gated by a typed `yes` confirmation.
+- **Prod -> staging copy**: `scripts/refresh-staging-db.sh` -- runs ON the staging server as `act_runner` (SSH to `[email protected]` is a prerequisite, setup steps in the script header). Backs up staging first (unless `--skip-backup`), streams a prod `pg_dump` over SSH, stops `mvp-backend-staging`, imports, restarts. **Always run `--dry-run` first.**
+- **Pre-migration backup**: `scripts/ci/maintenance-migrate.sh backup` / the maintenance workflow's `create_backup: yes` input.
+
+**`data/documents` is the ONLY copy of user-uploaded documents** (bind mount on the server, on both staging and prod backends). It is not in any database dump; it is covered only by the app-level backup feature. Never delete or re-provision that directory without a copy elsewhere.
+
+```bash
+# Copy-paste safety sequence before anything destructive
+./scripts/export-database.sh --env production --format custom     # or --env staging
+ls -lh database-exports/ | tail -3                                # confirm the file exists and is non-trivial in size
+```
+
+## 6. Operational traps
+
+Each verified against workflows/scripts/compose, 2026-07-07.
+
+| # | Trap | Consequence |
+|---|------|-------------|
+| 1 | Staging builds push `:latest` on EVERY PR sync, and the prod workflow's `image_tag` DEFAULTS to `latest` | Deploying prod with the default tag can ship an unmerged PR build to production. **Always deploy prod by 7-char SHA tag.** The systemic fix is the subject of `mvp-deploy-safety-campaign`. |
+| 2 | `rsync --delete config/` on every deploy resets `config/deployment/state.json` (repo default: active=blue) and `config/traefik/dynamic/blue-green.yml` (repo default: blue=100/green=0) | While green is active, a prod deploy has a window between rsync and switch-traffic where weights point at blue; `state.json` read mid-deploy lies; deployment history is erased every deploy. Never trust mid-deploy state.json; never hand-edit these files on the server (the next deploy reverts them anyway). |
+| 3 | Shared services start with `--no-recreate` on prod deploys | Config changes to postgres, redis, loki, alloy, or grafana do NOT take effect from a normal prod deploy. You must manually recreate the specific service on the prod server: `docker compose -f docker-compose.yml -f docker-compose.blue-green.yml -f docker-compose.prod.yml up -d --force-recreate mvp-grafana` (name only the service you changed). |
+| 4 | `mvp-ocr` is a single shared instance and is `--force-recreate`d on every prod deploy | Brief OCR outage during every prod deploy; in-flight OCR jobs at switch time fail. Not blue-greened. |
+| 5 | Staging runner root disk is 29G and every commit AND every PR sync builds 3 SHA-tagged images on it (containerd snapshotter) | Disk fills; builds and deploys start failing. A daily `docker system prune` cron exists on the box but is OUT-OF-BAND (deliberately not in the repo -- the standing exception to "no hand-edited server state"). Re-running `ansible/deploy-staging-runner.yml` or rebuilding the box DROPS it -- re-install the prune cron afterward. That playbook also REMOVES a "legacy Docker cleanup" cron that used to destroy volumes -- do not resurrect that one. Registry-side cleanup: `scripts/ci/purge-container-images.sh --token=PAT --dry-run` first. |
+| 6 | TLS certs renew via Let's Encrypt Cloudflare DNS challenge (`config/traefik/traefik.yml`; token from `secrets/app/cloudflare-dns-token.txt`, injected from the `CF_DNS_API_TOKEN` Gitea secret) | If the Cloudflare token rots, renewals fail silently until the cert expires. ACME state lives in `data/traefik/acme.json` and must stay chmod 600 (workflows enforce it; Traefik refuses looser perms). |
+| 7 | Traefik dashboard basicauth on prod is a placeholder hash (`admin:$2y$10$foobar` in `docker-compose.prod.yml`) | There is no working dashboard credential; the firewall is the actual access control. Known-weak point -- do not treat the basicauth label as protection (also flagged in `mvp-architecture-contract`). |
+| 8 | Grafana admin password defaults to `admin` (`GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin}` in base compose) unless the `GRAFANA_ADMIN_PASSWORD` Gitea secret is set | admin/admin on the logs UI. Mitigated by the `grafana-ipwhitelist` middleware (RFC1918 source ranges only), but verify the secret is set. |
+| 9 | Breaking migrations need the maintenance flow (`maintenance.yaml` -> `maintenance-migrate.sh`) | Skipping it and relying on start-time auto-migrations for a breaking change leaves the OLD stack running against the NEW schema mid-deploy. The maintenance flow zeroes all traffic weights (503s), optionally backs up, migrates, restores traffic -- accept the downtime. |
+| 10 | Every PR sync fully redeploys the single shared staging environment (`down` then `up`) | Staging blips on every PR push, and with two open PRs the last one to sync owns staging -- your "staging verification" may be verifying someone else's build. Check which run deployed last before trusting staging state. |
+
+## Provenance and maintenance
+
+Authored 2026-07-07 against the repo at commit e729d42. Everything above was verified by direct inspection of `.gitea/workflows/*.yaml`, `docker-compose*.yml`, `scripts/`, `scripts/ci/`, `config/`, and `backend/src/features/backup/`. Where older docs disagree (e.g. `docs/CICD-DEPLOY.md` runner-label names), the workflows/scripts win.
+
+Volatile facts and how to re-verify each:
+
+- Staging triggers (push to main + every PR sync): `grep -n -A6 "^on:" .gitea/workflows/staging.yaml`
+- Prod `image_tag` default `latest`: `grep -n -B2 -A3 "image_tag:" .gitea/workflows/production.yaml`
+- `:latest` clobber on every build: `grep -n "latest" .gitea/workflows/staging.yaml`
+- 13-feature health list: `grep -n "REQUIRED_FEATURES" .gitea/workflows/staging.yaml .gitea/workflows/production.yaml`
+- Healthcheck budget (48x5s) and backend `start_period: 180s`: `grep -n "seq 1 48" .gitea/workflows/staging.yaml; grep -n "start_period" docker-compose.yml docker-compose.blue-green.yml`
+- `--no-recreate` / `--force-recreate` service lists: `grep -n -A3 "no-recreate\|force-recreate" .gitea/workflows/production.yaml`
+- rsync `--delete` state reset: `grep -n "rsync" .gitea/workflows/*.yaml; cat config/deployment/state.json`
+- Legacy rollback.sh hazards: `grep -n "build\|mvp-frontend \|DOCKER_COMPOSE" scripts/rollback.sh`
+- Switch-traffic gradual mode and DEPLOY_PATH contract: `sed -n '1,25p' scripts/ci/switch-traffic.sh`
+- Prod health-check timeout 240s: `grep -n "HEALTH_CHECK_TIMEOUT" .gitea/workflows/production.yaml`
+- Backup API routes and jobs: `grep -rn "admin/backups" backend/src/features/backup/api/; ls backend/src/features/backup/jobs/`
+- Export/import flags: `./scripts/export-database.sh --help; ./scripts/import-database.sh --help`
+- refresh-staging-db prerequisites and `--dry-run`: `sed -n '1,50p' scripts/refresh-staging-db.sh`
+- LOG_LEVEL per env: `grep -n "LOG_LEVEL" .gitea/workflows/staging.yaml .gitea/workflows/production.yaml`
+- Placeholder basicauth hash: `grep -n "basicauth" docker-compose.prod.yml`
+- Grafana password default: `grep -n "GRAFANA_ADMIN_PASSWORD" docker-compose.yml`
+- Cloudflare DNS challenge + acme.json perms: `grep -n -A5 "acme" config/traefik/traefik.yml; grep -n "acme.json" .gitea/workflows/*.yaml`
+- Ansible removes the legacy cleanup cron (and does not install the prune cron): `grep -n -B1 -A5 "cron" ansible/deploy-staging-runner.yml`
+- `make clean` destroys volumes: `grep -n -A3 "^clean:" Makefile`