motovaultpro/.claude/skills/mvp-run-and-operate/SKILL.md

name, description

name

description

mvp-run-and-operate

Operate MotoVaultPro environments - load when deploying to staging or production, rolling back a bad deploy, switching blue/green traffic, backing up or restoring the database, refreshing staging data from prod, running a breaking migration, or diagnosing "staging is down after my PR", "prod deployed the wrong code", "state.json says blue but green is serving", "config change did not apply to postgres/redis/grafana", "OCR blipped during deploy", "staging runner disk full", or "cert renewal failed". Covers compose file stacking, container names, the staging and production Gitea workflows, switch-traffic/health-check/auto-rollback scripts, and the legacy rollback.sh hazard.

Run and Operate MotoVaultPro

When to use: deploying, rolling back, switching traffic, backing up/restoring the database, refreshing staging data, running breaking migrations, or debugging a broken environment (staging or prod down, wrong version serving, config not applied).

When NOT to use:

• Getting a dev machine working, local unit tests, make setup -> mvp-build-and-env
• Reading logs, Grafana/Loki queries, health-endpoint internals -> mvp-diagnostics-and-logging
• What CI actually gates (spoiler: no tests, no lint) and PR/issue workflow -> mvp-change-control and mvp-validation-and-qa
• Fixing the systemic deploy-safety hole (PR builds clobbering :latest, prod defaulting to latest) -> mvp-deploy-safety-campaign
• Why the architecture is shaped this way -> mvp-architecture-contract
• Past incidents and reverts -> mvp-failure-archaeology

Reality check for this project (2026-07-07): development is done by AI sessions; the human owner reviews PRs; end-to-end verification happens on STAGING via the PR deploy pipeline because there is no fully working local dev loop. Deploys are how changes get verified. Treat this file as the runbook.

1. Environment matrix

Verified against docker-compose*.yml and .gitea/workflows/{staging,production}.yaml (2026-07-07).

	Dev (local)	Staging	Production
Compose stacking	docker-compose.yml alone (base, builds images locally)	base + docker-compose.staging.yml	base + docker-compose.blue-green.yml + docker-compose.prod.yml
Server path	your machine	/opt/motovaultpro on mvp-staging (also the CI runner, label stage)	/opt/motovaultpro on prod server (internal 172.30.1.36; runner label prod)
Container names	mvp-traefik, mvp-frontend, mvp-backend, mvp-ocr, mvp-postgres, mvp-redis, mvp-loki, mvp-alloy, mvp-grafana	app containers get a -staging suffix (mvp-traefik/frontend/backend/ocr/postgres/redis-staging); the logging trio keeps base names (mvp-loki, mvp-alloy, mvp-grafana - no container_name override in the staging overlay)	shared: mvp-traefik, mvp-ocr, mvp-postgres, mvp-redis, mvp-loki, mvp-alloy, mvp-grafana; paired: mvp-frontend-blue/green, mvp-backend-blue/green
URLs	https://motovaultpro.com via hosts file + self-signed cert	https://staging.motovaultpro.com, https://logs.staging.motovaultpro.com, https://traefik.staging.motovaultpro.com	https://motovaultpro.com, https://logs.motovaultpro.com
Images	built from source (build: in base compose)	git.motovaultpro.com/egullickson/{backend,frontend,ocr}:<7-char-sha> via BACKEND_IMAGE/FRONTEND_IMAGE/OCR_IMAGE env	same registry; tag = workflow_dispatch input image_tag, DEFAULT latest (dangerous, see traps)
Volumes	mvp_postgres_data, mvp_redis_data; host ports 5432/6379 exposed	mvp_postgres_staging_data, mvp_redis_staging_data	base volume names; docker-compose.prod.yml sets ports: [] on postgres/redis (no host exposure)
LOG_LEVEL	debug (compose defaults)	DEBUG (staging.yaml env)	INFO (production.yaml env)
Who deploys	you, manually	CI, automatically: every push to main AND every PR open/sync/reopen. Full down/up = brief outage every deploy; concurrent PRs mean last PR wins the shared environment	human, manually: workflow_dispatch on production.yaml with image_tag input

Compose inheritance detail (prod): the merged prod config STILL contains the base single-stack mvp-frontend/mvp-backend services. The workflow avoids them by always naming services in up -d. Never run a bare up -d with the prod file stack -- it would start the base containers alongside blue/green.

2. Staging deploy anatomy

Source of truth: .gitea/workflows/staging.yaml. Triggers: push to main and every PR sync. All jobs run on the staging host itself.

1. build: docker login to git.motovaultpro.com; builds backend, frontend (Vite build args from Gitea vars, including VITE_STRIPE_PUBLISHABLE_KEY), and ocr; tags each :<short-sha> AND :latest; pushes all six tags. This is where PR builds clobber :latest.
2. deploy-staging:
   • rsync -av --delete of config/ and scripts/ into /opt/motovaultpro (wipes any on-server edits -- this is why hand-editing server files is a non-negotiable "never")
   • copies compose files and secrets/app/google-wif-config.json
   • generates .env (Stripe price IDs, GRAFANA_ADMIN_PASSWORD, then scripts/ci/generate-log-config.sh DEBUG >> .env which fans one LOG_LEVEL out to backend/traefik/postgres/redis settings)
   • scripts/inject-secrets.sh writes the real secret files from Gitea Actions secrets
   • init data dirs: data/backups and data/documents chown 1001:1001; data/traefik/acme.json chmod 600
   • pulls images, then compose down --timeout 30 followed by up -d -- staging takes full downtime on every deploy
3. verify-staging:
   • asserts the 5 app containers are running, then polls Docker healthchecks up to 48 x 5s = 4 minutes per service. The backend has start_period: 180s because auto-migrations run on container start (node dist/_system/migrations/run-all.js) -- a slow migration eats into this budget.
   • in-container curl localhost:3001/health
   • external check: https://staging.motovaultpro.com/api/health must return status: healthy with all 13 required features: admin, auth, onboarding, vehicles, documents, fuel-logs, stations, maintenance, platform, notifications, user-profile, user-preferences, user-export
4. notify success/failure via scripts/ci/notify.sh (Resend email).

This pipeline is also the only PR gate: green means "3 images built and staging boots healthy", nothing more (no tests, no lint -- see mvp-validation-and-qa).

3. Production deploy anatomy (blue-green)

Source of truth: .gitea/workflows/production.yaml. Manual workflow_dispatch only. Blue-green = two parallel frontend+backend pairs behind Traefik weighted routing; only one pair ("stack") receives traffic at a time.

1. validate: reads /opt/motovaultpro/config/deployment/state.json .active_stack (defaults to blue if missing) and targets the OPPOSITE color.
2. deploy-prod:
   • sparse checkout; rsync --delete of config/ and scripts/ (this RESETS state.json and the Traefik weights file to repo defaults -- see traps)
   • .env generation + inject-secrets.sh + data-dir init, same as staging
   • pull images for the chosen tag, record image IDs
   • up -d --no-recreate mvp-postgres mvp-redis mvp-loki mvp-alloy mvp-grafana -- shared services persist untouched across deploys (config changes to them do NOT apply; see traps)
   • wait for postgres/redis healthy
   • up -d --force-recreate mvp-ocr mvp-frontend-$TARGET mvp-backend-$TARGET -- note mvp-ocr is a single shared instance and is force-recreated every prod deploy (brief OCR outage each time)
   • verify running image IDs match what was pulled
   • scripts/ci/health-check.sh $TARGET 240 (container health + HTTP endpoints, 240s budget)
   • start Traefik, then scripts/ci/switch-traffic.sh $TARGET instant -- awk/sed-edits weights in config/traefik/dynamic/blue-green.yml; Traefik's file watcher applies it live; state.json updated
3. verify-prod: container/health waits, in-container health, external https://motovaultpro.com/api/health with the same 13-feature check, then greps blue-green.yml to confirm the target stack weight is 100.
4. rollback job (on failure): runs scripts/ci/auto-rollback.sh automatically. Emails on both paths.

Migrations: non-breaking migrations run automatically on backend container start. Breaking migrations use .gitea/workflows/maintenance.yaml (manual, create_backup input default yes) which runs scripts/ci/maintenance-migrate.sh [backup]: sets ALL four Traefik weights to 0 (site returns 503), optionally pg_dumps to data/backups/, runs the migration, restores traffic.

Operator commands (on the servers, at /opt/motovaultpro)

# Staging stack
docker compose -f docker-compose.yml -f docker-compose.staging.yml ps
docker compose -f docker-compose.yml -f docker-compose.staging.yml logs -f
docker logs -f mvp-backend-staging

# Production stack (ALWAYS name services on up -d; never bare)
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml -f docker-compose.prod.yml ps
cat config/deployment/state.json | jq .          # which stack is active (but see trap 2)

# Health / traffic (DEPLOY_PATH is required so the scripts edit the live config, not a checkout)
DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/health-check.sh blue|green [timeout_s]   # default 60s
DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/switch-traffic.sh blue|green [instant|gradual]
DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/auto-rollback.sh "reason"

gradual mode steps weights 25 -> 50 -> 75 -> 100 with 3s intervals. Emergency both-stacks-down recovery procedure: docs/CICD-DEPLOY.md, "Emergency Recovery" section (down everything, restart shared services, start one stack by name, switch traffic).

4. Rollback

Automatic: the production workflow's rollback job runs scripts/ci/auto-rollback.sh on verification failure. It reads state.json for the current/inactive stacks, health-checks the OLD (inactive) stack for 30s, switches traffic back to it, updates state, and emails. If the old stack is also unhealthy it sends a rollback_failed email and exits nonzero -- that is your both-stacks-down scenario (see emergency recovery above).

Manual (on the prod server):

cd /opt/motovaultpro
DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/health-check.sh blue 60     # confirm the target is healthy FIRST
DEPLOY_PATH=/opt/motovaultpro ./scripts/ci/switch-traffic.sh blue      # instant; or append "gradual"

+----------------------------------------------------------------------------+
| WARNING: scripts/rollback.sh (repo root scripts/ dir) is LEGACY AND        |
| DANGEROUS ON PROD. Verified 2026-07-07: it composes only base + prod       |
| (no blue-green file), does a full `down` of the site, git-checkouts an     |
| old commit and REBUILDS IMAGES FROM SOURCE on the prod box, then health-   |
| checks containers named mvp-frontend/mvp-backend which DO NOT EXIST under  |
| the blue-green topology. Running it takes prod down and cannot verify      |
| recovery. Never use it. Use scripts/ci/switch-traffic.sh or                |
| scripts/ci/auto-rollback.sh instead.                                       |
+----------------------------------------------------------------------------+

Rolling back to an older BUILD (not just the other color): re-run the production workflow with image_tag set to the known-good 7-char commit SHA. That deploys the old images to the inactive color and switches to it.

5. Backup, restore, and data

Non-negotiable (owner, 2026-07-07): take a fresh backup before ANY destructive database operation. That includes make clean (runs docker compose down -v --rmi all -- destroys DB volumes wherever you run it), import-database.sh --drop-existing, schema migrations on staging/prod, and the backend integration tests (which DROP TABLE ... CASCADE on the shared dev database).

Layers, all verified 2026-07-07:

• App-level backup feature (backend/src/features/backup/): admin API at /api/admin/backups* (list, create, get, download, restore) plus scheduled jobs -- backup-scheduled.job.ts and backup-cleanup.job.ts (daily retention cleanup at 4 AM). Writes to ./data/backups (bind-mounted into the backend, owner 1001).
• Manual export: scripts/export-database.sh -- interactive, or --env production|staging (selects container mvp-postgres vs mvp-postgres-staging), --format sql|custom|directory (default sql), gzips, writes to database-exports/ with a metadata JSON and import instructions. Make shortcuts: make db-export, make db-backup.
• Restore: scripts/import-database.sh [--env ...] <file> -- auto-detects format, backs up the existing database to database-exports/ first by default (skip with --no-backup; do not skip), supports --drop-existing --create-db gated by a typed yes confirmation.
• Prod -> staging copy: scripts/refresh-staging-db.sh -- runs ON the staging server as act_runner (SSH to [email protected] is a prerequisite, setup steps in the script header). Backs up staging first (unless --skip-backup), streams a prod pg_dump over SSH, stops mvp-backend-staging, imports, restarts. Always run --dry-run first.
• Pre-migration backup: scripts/ci/maintenance-migrate.sh backup / the maintenance workflow's create_backup: yes input.

data/documents is the ONLY copy of user-uploaded documents (bind mount on the server, on both staging and prod backends). It is not in any database dump; it is covered only by the app-level backup feature. Never delete or re-provision that directory without a copy elsewhere.

# Copy-paste safety sequence before anything destructive
./scripts/export-database.sh --env production --format custom     # or --env staging
ls -lh database-exports/ | tail -3                                # confirm the file exists and is non-trivial in size

6. Operational traps

Each verified against workflows/scripts/compose, 2026-07-07.

#	Trap	Consequence
1	Staging builds push :latest on EVERY PR sync, and the prod workflow's image_tag DEFAULTS to latest	Deploying prod with the default tag can ship an unmerged PR build to production. Always deploy prod by 7-char SHA tag. The systemic fix is the subject of mvp-deploy-safety-campaign.
2	rsync --delete config/ on every deploy resets config/deployment/state.json (repo default: active=blue) and config/traefik/dynamic/blue-green.yml (repo default: blue=100/green=0)	While green is active, a prod deploy has a window between rsync and switch-traffic where weights point at blue; state.json read mid-deploy lies; deployment history is erased every deploy. Never trust mid-deploy state.json; never hand-edit these files on the server (the next deploy reverts them anyway).
3	Shared services start with --no-recreate on prod deploys	Config changes to postgres, redis, loki, alloy, or grafana do NOT take effect from a normal prod deploy. You must manually recreate the specific service on the prod server: docker compose -f docker-compose.yml -f docker-compose.blue-green.yml -f docker-compose.prod.yml up -d --force-recreate mvp-grafana (name only the service you changed).
4	mvp-ocr is a single shared instance and is --force-recreated on every prod deploy	Brief OCR outage during every prod deploy; in-flight OCR jobs at switch time fail. Not blue-greened.
5	Staging runner root disk is 29G and every commit AND every PR sync builds 3 SHA-tagged images on it (containerd snapshotter)	Disk fills; builds and deploys start failing. A daily docker system prune cron exists on the box but is OUT-OF-BAND (deliberately not in the repo -- the standing exception to "no hand-edited server state"). Re-running ansible/deploy-staging-runner.yml or rebuilding the box DROPS it -- re-install the prune cron afterward. That playbook also REMOVES a "legacy Docker cleanup" cron that used to destroy volumes -- do not resurrect that one. Registry-side cleanup: scripts/ci/purge-container-images.sh --token=PAT --dry-run first.
6	TLS certs renew via Let's Encrypt Cloudflare DNS challenge (config/traefik/traefik.yml; token from secrets/app/cloudflare-dns-token.txt, injected from the CF_DNS_API_TOKEN Gitea secret)	If the Cloudflare token rots, renewals fail silently until the cert expires. ACME state lives in data/traefik/acme.json and must stay chmod 600 (workflows enforce it; Traefik refuses looser perms).
7	Traefik dashboard basicauth on prod is a placeholder hash (admin:$2y$10$foobar in docker-compose.prod.yml)	There is no working dashboard credential; the firewall is the actual access control. Known-weak point -- do not treat the basicauth label as protection (also flagged in mvp-architecture-contract).
8	Grafana admin password defaults to admin (GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin} in base compose) unless the GRAFANA_ADMIN_PASSWORD Gitea secret is set	admin/admin on the logs UI. Mitigated by the grafana-ipwhitelist middleware (RFC1918 source ranges only), but verify the secret is set.
9	Breaking migrations need the maintenance flow (maintenance.yaml -> maintenance-migrate.sh)	Skipping it and relying on start-time auto-migrations for a breaking change leaves the OLD stack running against the NEW schema mid-deploy. The maintenance flow zeroes all traffic weights (503s), optionally backs up, migrates, restores traffic -- accept the downtime.
10	Every PR sync fully redeploys the single shared staging environment (down then up)	Staging blips on every PR push, and with two open PRs the last one to sync owns staging -- your "staging verification" may be verifying someone else's build. Check which run deployed last before trusting staging state.

Provenance and maintenance

Authored 2026-07-07 against the repo at commit e729d42. Everything above was verified by direct inspection of .gitea/workflows/*.yaml, docker-compose*.yml, scripts/, scripts/ci/, config/, and backend/src/features/backup/. Where older docs disagree (e.g. docs/CICD-DEPLOY.md runner-label names), the workflows/scripts win.

Volatile facts and how to re-verify each:

• Staging triggers (push to main + every PR sync): grep -n -A6 "^on:" .gitea/workflows/staging.yaml
• Prod image_tag default latest: grep -n -B2 -A3 "image_tag:" .gitea/workflows/production.yaml
• :latest clobber on every build: grep -n "latest" .gitea/workflows/staging.yaml
• 13-feature health list: grep -n "REQUIRED_FEATURES" .gitea/workflows/staging.yaml .gitea/workflows/production.yaml
• Healthcheck budget (48x5s) and backend start_period: 180s: grep -n "seq 1 48" .gitea/workflows/staging.yaml; grep -n "start_period" docker-compose.yml docker-compose.blue-green.yml
• --no-recreate / --force-recreate service lists: grep -n -A3 "no-recreate\|force-recreate" .gitea/workflows/production.yaml
• rsync --delete state reset: grep -n "rsync" .gitea/workflows/*.yaml; cat config/deployment/state.json
• Legacy rollback.sh hazards: grep -n "build\|mvp-frontend \|DOCKER_COMPOSE" scripts/rollback.sh
• Switch-traffic gradual mode and DEPLOY_PATH contract: sed -n '1,25p' scripts/ci/switch-traffic.sh
• Prod health-check timeout 240s: grep -n "HEALTH_CHECK_TIMEOUT" .gitea/workflows/production.yaml
• Backup API routes and jobs: grep -rn "admin/backups" backend/src/features/backup/api/; ls backend/src/features/backup/jobs/
• Export/import flags: ./scripts/export-database.sh --help; ./scripts/import-database.sh --help
• refresh-staging-db prerequisites and --dry-run: sed -n '1,50p' scripts/refresh-staging-db.sh
• LOG_LEVEL per env: grep -n "LOG_LEVEL" .gitea/workflows/staging.yaml .gitea/workflows/production.yaml
• Placeholder basicauth hash: grep -n "basicauth" docker-compose.prod.yml
• Grafana password default: grep -n "GRAFANA_ADMIN_PASSWORD" docker-compose.yml
• Cloudflare DNS challenge + acme.json perms: grep -n -A5 "acme" config/traefik/traefik.yml; grep -n "acme.json" .gitea/workflows/*.yaml
• Ansible removes the legacy cleanup cron (and does not install the prune cron): grep -n -B1 -A5 "cron" ansible/deploy-staging-runner.yml
• make clean destroys volumes: grep -n -A3 "^clean:" Makefile