chore: replace AI skill library with 16 verified mvp-* skills
Deploy to Staging / Build Images (push) Successful in 5m32s
Deploy to Staging / Deploy to Staging (push) Successful in 44s
Deploy to Staging / Verify Staging (push) Successful in 5s
Deploy to Staging / Notify Staging Ready (push) Successful in 4s
Deploy to Staging / Notify Staging Failure (push) Has been skipped
Deploy to Staging / Build Images (push) Successful in 5m32s
Deploy to Staging / Deploy to Staging (push) Successful in 44s
Deploy to Staging / Verify Staging (push) Successful in 5s
Deploy to Staging / Notify Staging Ready (push) Successful in 4s
Deploy to Staging / Notify Staging Failure (push) Has been skipped
Removes the old planner/decision-critic/role-agents/domain-agents system (step-injector scripts, stale scopes, sprint-era workflow) and replaces it with 16 ground-truth-verified skills under .claude/skills/: change control, debugging playbook, failure archaeology, architecture contract, domain reference, OCR/Gemini pipeline, config and secrets, build and env, run and operate, diagnostics (with tested scripts), validation and QA, docs and writing, launch readiness, deploy-safety campaign, proof and analysis toolkit, and research frontier. RULE 0/1/2, the temporal-contamination rule, and the decision stress-test protocol are carried forward into the new skills; the retired content remains in git history. Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5
parent
e729d425fd
commit
c239bb9347
@@ -0,0 +1,470 @@
|
||||
---
|
||||
name: mvp-diagnostics-and-logging
|
||||
description: >-
|
||||
Load when you need to observe or measure MotoVaultPro instead of guessing - querying
|
||||
logs in Grafana/Loki, writing LogQL, tracing a request by requestId/X-Request-Id,
|
||||
checking /health or /api/health, cache inspection via redis-cli, PostgreSQL activity,
|
||||
or running the shipped diagnostic scripts (check-numeric-coercion.sh,
|
||||
check-route-auth.sh, local-gate.sh). This is the HOW-to-observe toolbox - load it when
|
||||
mvp-debugging-playbook (the symptom-triage hub) or a task needs measurements.
|
||||
Distinguishing triggers - "logs not appearing", Grafana unreachable, Loki curl fails,
|
||||
no logs from a container, alert firing, request timing, "slow requests", "5xx spike".
|
||||
---
|
||||
|
||||
# MotoVaultPro Diagnostics and Logging
|
||||
|
||||
Measuring instead of eyeballing: where logs live, how to query them, how to
|
||||
check health and state, and three shipped read-only diagnostic scripts.
|
||||
|
||||
## When to use / When NOT to use
|
||||
|
||||
Use this skill when you need to OBSERVE the system: query logs, trace a
|
||||
request, verify health, measure latency, inspect cache or database activity,
|
||||
or run the pre-push diagnostic scripts.
|
||||
|
||||
Do NOT use this skill for:
|
||||
- Symptom-to-root-cause triage of known failure modes: `mvp-debugging-playbook`
|
||||
- Deploying, rolling back, blue-green mechanics, backups: `mvp-run-and-operate`
|
||||
- What counts as test evidence / definition of done: `mvp-validation-and-qa`
|
||||
- Historical incidents and settled dead ends: `mvp-failure-archaeology`
|
||||
- Getting a working local environment at all: `mvp-build-and-env`
|
||||
- Changing log config or adding config axes: `mvp-config-and-secrets`
|
||||
|
||||
Environment reality (2026-07-07): development is done by AI sessions in this
|
||||
repo; end-to-end verification happens on STAGING via the PR pipeline. There
|
||||
is no fully working local dev loop, and CI gates nothing beyond build + boot
|
||||
(canonical statement: `mvp-validation-and-qa` section 1). The scripts in
|
||||
this skill exist because of that gap.
|
||||
|
||||
## 1. Logging topology
|
||||
|
||||
One pipeline, file-provisioned end to end. Frontend is the exception: its
|
||||
logs go to the browser console ONLY (`frontend/src/utils/logger.ts`) and
|
||||
never reach Loki.
|
||||
|
||||
```
|
||||
mvp-traefik mvp-frontend mvp-backend mvp-ocr mvp-postgres mvp-redis
|
||||
| | | | | |
|
||||
+------------+------+------+-----------+----------+------------+
|
||||
| (frontend: container
|
||||
v stdout only; app logs
|
||||
Docker json-file driver (max-size 10m, stay in the browser)
|
||||
max-file 3 - set on every service in
|
||||
docker-compose.yml)
|
||||
|
|
||||
v
|
||||
mvp-alloy (config/alloy/config.alloy)
|
||||
- discovers containers via /var/run/docker.sock
|
||||
- labels: container = container name, service = compose service
|
||||
|
|
||||
v
|
||||
mvp-loki (config/loki/config.yml)
|
||||
- TSDB schema v13, filesystem storage
|
||||
- retention_period: 720h (30 days)
|
||||
- auth disabled; distroless image, healthcheck disabled
|
||||
|
|
||||
v
|
||||
mvp-grafana (Grafana 12.4.0)
|
||||
- https://logs.motovaultpro.com (prod)
|
||||
- https://logs.staging.motovaultpro.com (staging)
|
||||
- 4 provisioned dashboards + 5 provisioned alert rules
|
||||
```
|
||||
|
||||
Grafana access: the Traefik router applies `grafana-ipwhitelist@file`
|
||||
(`config/traefik/dynamic/grafana.yml` and `dynamic-staging/grafana.yml`),
|
||||
which allows only RFC1918 source ranges (10/8, 172.16/12, 192.168/16).
|
||||
Grafana is unreachable from the public internet by design - reach it from
|
||||
inside the network (the servers themselves, or VPN). Admin password comes
|
||||
from `GRAFANA_ADMIN_PASSWORD` (default `admin`).
|
||||
|
||||
Container name matrix (label `container` in LogQL):
|
||||
|
||||
| Service | Dev | Staging | Production |
|
||||
|----------|----------------|------------------------|---------------------------------|
|
||||
| backend | `mvp-backend` | `mvp-backend-staging` | `mvp-backend-blue` / `-green` |
|
||||
| frontend | `mvp-frontend` | `mvp-frontend-staging` | `mvp-frontend-blue` / `-green` |
|
||||
| traefik | `mvp-traefik` | `mvp-traefik-staging` | `mvp-traefik` |
|
||||
| ocr | `mvp-ocr` | `mvp-ocr-staging` | `mvp-ocr` |
|
||||
| postgres | `mvp-postgres` | `mvp-postgres-staging` | `mvp-postgres` |
|
||||
| redis | `mvp-redis` | `mvp-redis-staging` | `mvp-redis` |
|
||||
| loki/alloy/grafana | `mvp-loki` / `mvp-alloy` / `mvp-grafana` in ALL environments (not renamed by overlays) ||||
|
||||
|
||||
KNOWN GAP (verified 2026-07-07): the provisioned dashboards and the
|
||||
backend-related alert rules select `{container=~"mvp-backend(-staging)?"}`.
|
||||
Loki regex matchers are fully anchored, so this does NOT match the
|
||||
production blue-green names `mvp-backend-blue`/`-green`. On production,
|
||||
backend panels are blind and the "Container Silence: mvp-backend" alert
|
||||
(noDataState: Alerting) fires from no-data. Use
|
||||
`{container=~"mvp-backend-(blue|green)"}` for prod queries.
|
||||
|
||||
## 2. LogQL cookbook
|
||||
|
||||
Field names below are verified against
|
||||
`backend/src/core/plugins/logging.plugin.ts`: every completed request logs
|
||||
`msg="Request processed"` with `requestId`, `method`, `path`, `status`,
|
||||
`duration` (ms, integer), `ip`; Pino adds `level` and `time`. Substitute the
|
||||
right container name from the matrix above.
|
||||
|
||||
Request logs (the workhorse):
|
||||
|
||||
```logql
|
||||
{container="mvp-backend-staging"} | json | msg="Request processed"
|
||||
```
|
||||
|
||||
Error sweep across all containers:
|
||||
|
||||
```logql
|
||||
{container=~"mvp-.*"} | json | level="error"
|
||||
```
|
||||
|
||||
Errors per container over time:
|
||||
|
||||
```logql
|
||||
sum by (container) (count_over_time({container=~"mvp-.*"} | json | level="error" [5m]))
|
||||
```
|
||||
|
||||
PostgreSQL errors (postgres logs are not JSON - use line filter):
|
||||
|
||||
```logql
|
||||
{container="mvp-postgres"} |~ "ERROR|FATAL|PANIC"
|
||||
```
|
||||
|
||||
OCR (Python) errors:
|
||||
|
||||
```logql
|
||||
{container=~"mvp-ocr(-staging)?"} |~ "ERROR|Exception|Traceback"
|
||||
```
|
||||
|
||||
Slow requests, p95 latency. The `| __error__=""` after `unwrap` is
|
||||
REQUIRED - it drops lines where `duration` failed to parse; without it the
|
||||
quantile returns errors or garbage:
|
||||
|
||||
```logql
|
||||
quantile_over_time(0.95,
|
||||
{container="mvp-backend-staging"} | json | msg="Request processed"
|
||||
| unwrap duration | __error__="" [5m])
|
||||
```
|
||||
|
||||
Individual requests slower than 500 ms:
|
||||
|
||||
```logql
|
||||
{container="mvp-backend-staging"} | json | msg="Request processed" | duration > 500
|
||||
```
|
||||
|
||||
All 5xx responses:
|
||||
|
||||
```logql
|
||||
{container="mvp-backend-staging"} | json | msg="Request processed" | status >= 500
|
||||
```
|
||||
|
||||
Correlate one request across services. The backend takes `X-Request-Id`
|
||||
from the incoming request or generates a UUID; Traefik access logs (JSON)
|
||||
keep the header. The backend does not echo it in the response, so find the
|
||||
ID in the request log first, then sweep:
|
||||
|
||||
```logql
|
||||
{container=~"mvp-.*"} |= "550e8400-e29b-41d4-a716-446655440000"
|
||||
```
|
||||
|
||||
To force a known ID end-to-end, send one (CORS-allowed header):
|
||||
|
||||
```bash
|
||||
curl -H "X-Request-Id: trace-$(date +%s)" https://staging.motovaultpro.com/api/health
|
||||
```
|
||||
|
||||
### Provisioned alerts
|
||||
|
||||
Defined in `config/grafana/alerting/alert-rules.yml`; evaluated every 1m,
|
||||
must hold for 5m. Contact point is a webhook placeholder - alerts are
|
||||
visible in the Grafana UI but notify nobody (2026-07-07).
|
||||
|
||||
| Alert | Severity | Condition |
|
||||
|-------|----------|-----------|
|
||||
| Error Rate Spike | critical | error-level logs > 5% of all logs over 5m |
|
||||
| Container Silence: mvp-backend / mvp-postgres / mvp-redis | warning | no logs for 5m (noDataState: Alerting) |
|
||||
| 5xx Response Spike | critical | > 10 HTTP 5xx from backend in 5m |
|
||||
|
||||
### Log-level control is deploy-time
|
||||
|
||||
One `LOG_LEVEL` fans out to all containers via
|
||||
`scripts/ci/generate-log-config.sh <DEBUG|INFO|WARN|ERROR>`, whose output
|
||||
the CI workflows append to `.env` on the server (NOT `.env.logging` -
|
||||
docs/LOGGING.md says `.env.logging`; the code in
|
||||
`.gitea/workflows/staging.yaml` appends to `.env`, and code wins). It sets
|
||||
`BACKEND_LOG_LEVEL`, `TRAEFIK_LOG_LEVEL`, `POSTGRES_LOG_STATEMENT`,
|
||||
`POSTGRES_LOG_MIN_DURATION`, `REDIS_LOGLEVEL`. Staging runs DEBUG, prod
|
||||
runs INFO. Caveat: the postgres leg of this fan-out is dead - the
|
||||
`POSTGRES_LOG_*` variables are set on the container but never applied (see
|
||||
the known gap under "PostgreSQL activity" in section 4); the backend,
|
||||
traefik, and redis legs are wired. There is no runtime toggle: changing
|
||||
verbosity means redeploying.
|
||||
Never hand-edit `.env` on a server - the next deploy's rsync/regeneration
|
||||
reverts it (owner non-negotiable).
|
||||
|
||||
## 3. Health and state checks
|
||||
|
||||
### /health vs /api/health
|
||||
|
||||
Both defined in `backend/src/app.ts`, both unauthenticated, both return
|
||||
`status: "healthy"` plus a `features` array (currently 20 entries in code).
|
||||
|
||||
- `GET /health` (port 3001, in-container) - used by the Docker healthcheck
|
||||
and CI's in-container curl. No `/api` prefix; not routed by Traefik.
|
||||
- `GET /api/health` - routed through Traefik; adds `scope: "api"`. This is
|
||||
the external verification target.
|
||||
|
||||
The CI contract (`.gitea/workflows/staging.yaml`, REQUIRED_FEATURES) checks
|
||||
that 13 specific features are present in the array: admin, auth, onboarding,
|
||||
vehicles, documents, fuel-logs, stations, maintenance, platform,
|
||||
notifications, user-profile, user-preferences, user-export. The endpoint
|
||||
returns more than 13; the contract is a subset check. If you add a feature
|
||||
capsule, adding it to the app.ts arrays is cosmetic; adding it to
|
||||
REQUIRED_FEATURES makes it deploy-gating.
|
||||
|
||||
```bash
|
||||
curl -s https://staging.motovaultpro.com/api/health | jq '.status, (.features | length)'
|
||||
```
|
||||
|
||||
### Container state per environment
|
||||
|
||||
The `-f` stack matters - a bare `docker compose ps` on staging/prod uses
|
||||
only the base file and shows wrong/partial state:
|
||||
|
||||
```bash
|
||||
# Dev (local machine)
|
||||
docker compose ps
|
||||
make health-check-all # ps table + Traefik service discovery counts
|
||||
|
||||
# Staging (on mvp-staging, /opt/motovaultpro)
|
||||
docker compose -f docker-compose.yml -f docker-compose.staging.yml ps
|
||||
|
||||
# Production (on prod server, /opt/motovaultpro)
|
||||
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml -f docker-compose.prod.yml ps
|
||||
```
|
||||
|
||||
### config/deployment/state.json - and why it lies mid-deploy
|
||||
|
||||
On the prod server, `config/deployment/state.json` records
|
||||
`active_stack` (blue/green), last deployment, and per-stack health. Read it
|
||||
with `cat /opt/motovaultpro/config/deployment/state.json | jq .`
|
||||
|
||||
It lies in two windows:
|
||||
1. Every deploy rsyncs `config/` with `--delete`, overwriting state.json
|
||||
with the repo default (`active_stack: blue`, all-null history). The
|
||||
workflow reads the real state BEFORE the rsync and re-stamps it at the
|
||||
end - but any inspection between rsync and the final stamp shows the
|
||||
repo default, not reality.
|
||||
2. Traefik weights in `config/traefik/dynamic/blue-green.yml` are the
|
||||
actual routing truth; state.json is a record of intent. When in doubt,
|
||||
read the weights file.
|
||||
|
||||
### When the log pipeline itself is broken
|
||||
|
||||
1. `docker logs mvp-alloy` (same name in every environment) - Alloy is the
|
||||
collector; discovery or push errors appear here.
|
||||
2. You CANNOT `curl` Loki from the host or exec into it: Loki 3.x is a
|
||||
distroless image (no shell, no wget/curl inside; the compose healthcheck
|
||||
is explicitly disabled for this reason - see the comment in
|
||||
docker-compose.yml). Verify Loki through Grafana: Connections >
|
||||
Data sources > Loki > Test, or from any container on the `backend`
|
||||
network, e.g.
|
||||
`docker exec mvp-backend-staging wget -qO- http://mvp-loki:3100/ready`.
|
||||
3. `docker logs mvp-grafana` for provisioning errors (bad dashboard JSON or
|
||||
alert YAML shows up here at startup).
|
||||
4. Docker json-file logs still exist even if the pipeline is down:
|
||||
`docker logs --since 10m mvp-backend-staging`.
|
||||
|
||||
## 4. Measuring instead of eyeballing
|
||||
|
||||
### Request timings
|
||||
|
||||
Timings live in the `duration` field (ms) of `msg="Request processed"`
|
||||
lines - see the quantile queries above. The API Performance dashboard
|
||||
(p50/p95/p99, slowest endpoints by avg duration) is provisioned from
|
||||
`config/grafana/dashboards/api-performance.json`. For a one-off check
|
||||
without Grafana:
|
||||
|
||||
```bash
|
||||
docker logs --since 5m mvp-backend-staging 2>&1 |
|
||||
grep '"msg":"Request processed"' |
|
||||
awk -F'"duration":' '{split($2,a,","); gsub(/[^0-9]/,"",a[1]); print a[1]}' | sort -n | tail -5
|
||||
```
|
||||
|
||||
### Cache behavior (Redis)
|
||||
|
||||
Backend uses DB 0 with key prefix `mvp:` (`backend/src/core/config/redis.ts`;
|
||||
DB from `config/app/*.yml` `redis.db: 0`). The OCR service uses DB 1
|
||||
(`REDIS_DB: 1` in docker-compose.yml) for job state. Locks use `mvp:lock:`.
|
||||
|
||||
```bash
|
||||
# Staging names shown; drop -staging for dev/prod
|
||||
docker exec mvp-redis-staging redis-cli -n 0 --scan --pattern 'mvp:*' | head -50
|
||||
docker exec mvp-redis-staging redis-cli -n 0 TTL 'mvp:some-key'
|
||||
docker exec mvp-redis-staging redis-cli -n 1 --scan --pattern '*' | head # OCR jobs
|
||||
docker exec mvp-redis-staging redis-cli INFO keyspace
|
||||
```
|
||||
|
||||
Cache reads swallow errors and return null (cache failure never breaks a
|
||||
request), so a dead Redis looks like a 100% miss rate, not errors. Check
|
||||
`docker logs mvp-redis-staging` and hit rates via `INFO stats`.
|
||||
|
||||
### PostgreSQL activity
|
||||
|
||||
Dev shell: `make db-shell-app` (wraps
|
||||
`docker compose exec mvp-postgres psql -U postgres -d motovaultpro`).
|
||||
Staging/prod equivalent:
|
||||
|
||||
```bash
|
||||
docker exec -it mvp-postgres-staging psql -U postgres -d motovaultpro
|
||||
```
|
||||
|
||||
Useful read-only checks inside psql:
|
||||
|
||||
```sql
|
||||
SELECT pid, state, now() - query_start AS age, left(query, 80)
|
||||
FROM pg_stat_activity WHERE state <> 'idle' ORDER BY age DESC;
|
||||
SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 15;
|
||||
```
|
||||
|
||||
KNOWN GAP: the `POSTGRES_LOG_STATEMENT` / `POSTGRES_LOG_MIN_DURATION_STATEMENT`
|
||||
values set in compose (`docker-compose.yml:245,247`) are INERT - they are plain
|
||||
environment variables that the official postgres image does not read, and
|
||||
nothing in the repo applies them (no `command: postgres -c log_statement=...`,
|
||||
no custom postgresql.conf, no initdb script). Postgres runs with defaults
|
||||
(`log_statement=none`, `log_min_duration_statement=-1`) at every LOG_LEVEL, so
|
||||
per-statement and `duration:` lines never reach Loki; a
|
||||
`{container="mvp-postgres-staging"} |~ "duration:"` hunt finds nothing. Until
|
||||
compose passes the values via `command: postgres -c ...`, slow-query hunting
|
||||
must use `pg_stat_activity` (above) instead.
|
||||
|
||||
### End-to-end request trace, step by step
|
||||
|
||||
1. Send the request with a known `X-Request-Id` (see cookbook above).
|
||||
2. Traefik: `{container="mvp-traefik-staging"} | json |= "<id>"` - confirms
|
||||
arrival, router matched, status returned to client.
|
||||
3. Backend: `{container="mvp-backend-staging"} |= "<id>"` - all application
|
||||
log lines plus the final `Request processed` line with duration.
|
||||
4. If OCR involved: `{container="mvp-ocr-staging"} |= "<id>"`.
|
||||
5. Postgres/Redis logs are not request-tagged; correlate by timestamp.
|
||||
|
||||
## 5. Shipped diagnostic scripts
|
||||
|
||||
All in `.claude/skills/mvp-diagnostics-and-logging/scripts/`, all
|
||||
executable, all safe read-only checks (no network, no writes outside temp
|
||||
logs), portable bash (macOS bash 3.2). Run them from anywhere inside the
|
||||
repo. Each was run against the repo on 2026-07-07.
|
||||
|
||||
### check-numeric-coercion.sh - the #1 recurring bug class
|
||||
|
||||
node-postgres returns NUMERIC/DECIMAL columns as STRINGS.
|
||||
`backend/src/core/config/database.ts` overrides only the DATE parser, so
|
||||
every repository mapper must coerce manually with `Number()`/`parseFloat()`.
|
||||
Missed coercion caused issues #239, #241, #244 (symptoms: MPG shows NaN,
|
||||
string concatenation instead of addition, sort by cost is alphabetical).
|
||||
|
||||
```bash
|
||||
./.claude/skills/mvp-diagnostics-and-logging/scripts/check-numeric-coercion.sh
|
||||
```
|
||||
|
||||
What it does: extracts every NUMERIC/DECIMAL column name from all migration
|
||||
SQL (column defs, ADD COLUMN, ALTER COLUMN TYPE), then (a) flags any
|
||||
`row.<col>` read in a `*.repository.ts` file whose line lacks
|
||||
`Number(`/`parseFloat(`, and (b) flags raw-row returns
|
||||
(`return res.rows...`) in features whose migrations define numeric columns -
|
||||
the exact shape of bug #244, which bypassed the mapper entirely.
|
||||
|
||||
Interpretation: exit 0 = clean. Exit 1 = each SUSPECT line needs coercion
|
||||
(`row.x != null ? parseFloat(row.x) : null` is the house pattern) - or is a
|
||||
false positive if coercion happens on another line or in the service layer;
|
||||
read before fixing. Run after ANY repository or migration change that
|
||||
touches numeric columns.
|
||||
|
||||
### check-route-auth.sh - unlisted-public routes
|
||||
|
||||
There is no global auth hook; every route opts in via
|
||||
`preHandler: [fastify.authenticate]` (or `requireAdmin`/`requireTier`, or
|
||||
the common alias `const requireAuth = fastify.authenticate.bind(fastify)`).
|
||||
A forgotten preHandler ships a public endpoint silently - pre-launch, this
|
||||
is a RULE 0 severity gap.
|
||||
|
||||
```bash
|
||||
./.claude/skills/mvp-diagnostics-and-logging/scripts/check-route-auth.sh
|
||||
```
|
||||
|
||||
What it does: parses every registration block in
|
||||
`backend/src/features/*/api/*.routes.ts` (199 blocks as of 2026-07-07) and
|
||||
prints any block containing no recognized guard. Known-intentional public
|
||||
routes are allowlisted in the script and printed as OK: `/webhooks/stripe`,
|
||||
`/webhooks/resend/inbound` (provider signature auth), `/auth/signup`,
|
||||
`/auth/resend-verification-public` (pre-auth flows).
|
||||
|
||||
Interpretation: exit 0 = only allowlisted routes are public AMONG the
|
||||
`features/*/api` route files the script scans. Routes registered in core
|
||||
code are OUT OF SCOPE and pass silently - including the intentionally
|
||||
public `GET /api/config/feature-tiers` (`backend/src/core/config/config.routes.ts`,
|
||||
no preHandler) and the `app.ts` registrations (`/health`, `/api/health`,
|
||||
`/auth/verify`); the script's header comment carries the canonical
|
||||
intentional-public list covering both scopes. A future unguarded route
|
||||
added outside `features/*/api/` would also pass silently - exit 0 is not a
|
||||
whole-backend guarantee. Exit 1 = an UNGUARDED route: either add a guard
|
||||
or, if genuinely public, add it to KNOWN_PUBLIC in the script with a
|
||||
justification comment. Heuristic caveats:
|
||||
a guard aliased under a new name reads as UNGUARDED (safe direction); a
|
||||
comment containing a guard word inside a block could mask a gap (verify by
|
||||
reading the code). Run after adding or reshaping any route file.
|
||||
|
||||
### local-gate.sh - the pre-push gate CI does not provide
|
||||
|
||||
CI runs zero tests and zero lint. This script is the substitute:
|
||||
|
||||
```bash
|
||||
./.claude/skills/mvp-diagnostics-and-logging/scripts/local-gate.sh
|
||||
```
|
||||
|
||||
Runs, with a PASS/FAIL table and per-step logs in `$TMPDIR`:
|
||||
backend lint, backend type-check, backend UNIT tests
|
||||
(`--testPathPattern='(tests/unit|src/core)' --testPathIgnorePatterns=integration --forceExit`),
|
||||
frontend lint, frontend type-check. Requires `npm install` in `backend/`
|
||||
and `frontend/` first.
|
||||
|
||||
Deliberate exclusions: backend integration tests need a live database and
|
||||
some DROP TABLE CASCADE (owner non-negotiable: no destructive DB operations
|
||||
without a fresh backup - never point them at a shared DB casually); the
|
||||
audit-log `__tests__` also require a live database and are excluded by
|
||||
pattern; frontend jest is broken outside the container (2026-07-07).
|
||||
`--forceExit` is required because open pg/redis handles otherwise hang jest.
|
||||
|
||||
Interpretation: all PASS = safe to push (integration behavior still
|
||||
unverified until staging). Any FAIL = fix first; CI will not catch it.
|
||||
KNOWN STATE (2026-07-07): backend unit tests FAIL on main - 12 of 22 suites
|
||||
have pre-existing ts-jest compile errors (stale test mocks vs current
|
||||
types, e.g. `new AuthService(...)` missing the termsData argument) plus 2
|
||||
failing assertions. This is untriaged rot from CI running nothing; treat a
|
||||
FAIL here as "no worse than main" only after diffing against a main-branch
|
||||
run, and see `mvp-validation-and-qa` for the evidence bar.
|
||||
|
||||
## Provenance and maintenance
|
||||
|
||||
Authored 2026-07-07 by direct inspection of the repo (all commands, paths,
|
||||
field names, and line-anchored claims verified against code; where
|
||||
docs/LOGGING.md disagrees with code - `.env.logging` vs `.env` - code wins).
|
||||
|
||||
Volatile facts and re-verification commands:
|
||||
|
||||
| Fact (as of 2026-07-07) | Re-verify with |
|
||||
|---|---|
|
||||
| Request log fields (requestId/method/path/status/duration/ip) | `grep -A8 "Request processed" backend/src/core/plugins/logging.plugin.ts` |
|
||||
| Loki 30-day retention, TSDB v13 | `grep -E "retention_period\|schema:" config/loki/config.yml` |
|
||||
| Alloy labels container/service | `grep target_label config/alloy/config.alloy` |
|
||||
| json-file 10m x 3 on all services | `grep -c "max-size" docker-compose.yml` |
|
||||
| Grafana RFC1918 whitelist | `cat config/traefik/dynamic/grafana.yml config/traefik/dynamic-staging/grafana.yml` |
|
||||
| Alert rules (5%/5m, silence, 5xx>10) | `grep -E "title\|for:\|noDataState" config/grafana/alerting/alert-rules.yml` |
|
||||
| Dashboard/alert regex missing prod blue-green | `grep -c "mvp-backend(-staging)?" config/grafana/dashboards/*.json config/grafana/alerting/alert-rules.yml` vs `grep container_name docker-compose.blue-green.yml` |
|
||||
| 13-feature CI health contract | `grep REQUIRED_FEATURES .gitea/workflows/staging.yaml` |
|
||||
| /health features array (20 entries) | `grep -n "features:" backend/src/app.ts` (read the two arrays) |
|
||||
| Log config appended to .env | `grep -n "generate-log-config" .gitea/workflows/*.yaml` |
|
||||
| Redis DB 0 backend / DB 1 OCR, prefix mvp: | `grep "db:" config/app/*.yml; grep REDIS_DB docker-compose.yml; grep "prefix = " backend/src/core/config/redis.ts` |
|
||||
| Known-public routes allowlist | `./.claude/skills/mvp-diagnostics-and-logging/scripts/check-route-auth.sh` |
|
||||
| Numeric coercion clean state | `./.claude/skills/mvp-diagnostics-and-logging/scripts/check-numeric-coercion.sh` |
|
||||
| Backend unit tests failing on main | `cd backend && npm test -- --testPathPattern='tests/unit' --forceExit` |
|
||||
| Container names per env | `grep container_name docker-compose*.yml` |
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
#!/bin/bash
|
||||
# check-numeric-coercion.sh - Find NUMERIC/DECIMAL columns returned from
|
||||
# repositories without Number()/parseFloat() coercion.
|
||||
#
|
||||
# Why: node-postgres returns NUMERIC/DECIMAL (OID 1700) as *strings*.
|
||||
# backend/src/core/config/database.ts overrides only the DATE parser, so
|
||||
# every repository mapper must coerce manually. Missed coercion is this
|
||||
# project's #1 recurring bug class (issues #239, #241, #244).
|
||||
#
|
||||
# Usage: ./check-numeric-coercion.sh (read-only; run from anywhere in repo)
|
||||
# Exit: 0 = no suspects, 1 = suspects found, 2 = setup error
|
||||
set -u
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null)"
|
||||
[ -z "$ROOT" ] && { echo "ERROR: not inside a git repo" >&2; exit 2; }
|
||||
SRC="$ROOT/backend/src"
|
||||
[ -d "$SRC" ] && cd "$ROOT" || { echo "ERROR: $SRC missing" >&2; exit 2; }
|
||||
|
||||
# 1. Collect numeric/decimal column names from all migration SQL.
|
||||
# Three shapes: column definition lines, ADD COLUMN, ALTER COLUMN ... TYPE.
|
||||
COLS=$(
|
||||
{
|
||||
grep -rhiE '^[[:space:]]*"?[a-z_]+"?[[:space:]]+(numeric|decimal)[[:space:](]' \
|
||||
--include='*.sql' "$SRC" | awk '{gsub(/"/,""); print tolower($1)}'
|
||||
grep -rhioE 'add column (if not exists )?"?[a-z_]+"? +(numeric|decimal)' \
|
||||
--include='*.sql' "$SRC" | awk '{gsub(/"/,""); print tolower($(NF-1))}'
|
||||
grep -rhioE 'alter column "?[a-z_]+"? type +(numeric|decimal)' \
|
||||
--include='*.sql' "$SRC" | awk '{gsub(/"/,""); print tolower($(NF-2))}'
|
||||
} | sort -u
|
||||
)
|
||||
[ -z "$COLS" ] && { echo "ERROR: no numeric columns found (wrong dir?)" >&2; exit 2; }
|
||||
|
||||
echo "Numeric/decimal columns found in migrations:"
|
||||
echo "$COLS" | tr '\n' ' '; echo; echo
|
||||
|
||||
# 2. In every repository file, flag lines that read row.<col> without
|
||||
# Number( or parseFloat( on the same line.
|
||||
SUSPECTS=0
|
||||
for col in $COLS; do
|
||||
HITS=$(find "$SRC" -name '*.repository.ts' -exec \
|
||||
grep -nE "row\.${col}([^a-zA-Z0-9_]|\$)" {} + 2>/dev/null |
|
||||
grep -v 'parseFloat(' | grep -v 'Number(')
|
||||
if [ -n "$HITS" ]; then
|
||||
echo "SUSPECT column '$col' (string from pg, no coercion on line):"
|
||||
echo "$HITS" | sed "s|$ROOT/||; s/^/ /"
|
||||
SUSPECTS=$((SUSPECTS + $(echo "$HITS" | wc -l | tr -d ' ')))
|
||||
fi
|
||||
done
|
||||
|
||||
# 3. Flag raw-row returns (the actual shape of bug #244: "return res.rows[0]"
|
||||
# bypasses the mapper entirely) in features whose migrations define
|
||||
# numeric columns. Scalar returns (.count/.exists/.length) are filtered.
|
||||
for repo in $(find "$SRC/features" -name '*.repository.ts'); do
|
||||
feat=$(echo "$repo" | sed "s|$SRC/features/||; s|/.*||")
|
||||
featcols=$(grep -rhiE "(numeric|decimal)" --include='*.sql' \
|
||||
"$SRC/features/$feat/migrations" 2>/dev/null | head -1)
|
||||
[ -z "$featcols" ] && continue
|
||||
RAW=$(grep -nE 'return (res|result)\.rows' "$repo" |
|
||||
grep -vE 'map[A-Za-z]*\(|\.map\(|\?\.|\.length|\.count|\.exists')
|
||||
if [ -n "$RAW" ]; then
|
||||
echo "SUSPECT raw-row return in numeric-bearing feature '$feat':"
|
||||
echo "$RAW" | sed "s|^| ${repo#"$ROOT"/}:|"
|
||||
SUSPECTS=$((SUSPECTS + $(echo "$RAW" | wc -l | tr -d ' ')))
|
||||
fi
|
||||
done
|
||||
|
||||
echo
|
||||
if [ "$SUSPECTS" -eq 0 ]; then
|
||||
echo "PASS: no uncoerced numeric column reads or raw-row returns found."
|
||||
exit 0
|
||||
else
|
||||
echo "FAIL: $SUSPECTS suspect line(s). Each must wrap the value in Number()"
|
||||
echo "or parseFloat() (nullable: 'row.x != null ? parseFloat(row.x) : null')."
|
||||
echo "Limitations: same-line heuristic; coercion done on a different line or"
|
||||
echo "in the service layer will still be flagged - verify before fixing."
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,90 @@
|
||||
#!/bin/bash
|
||||
# check-route-auth.sh - List backend routes with NO auth guard in their
|
||||
# registration block, so unintentionally-public endpoints are visible.
|
||||
#
|
||||
# Why: there is no global auth hook. Every route must opt in via
|
||||
# preHandler: [fastify.authenticate] / requireAdmin / requireTier
|
||||
# (backend/src/core/plugins/*.plugin.ts). A forgotten preHandler ships a
|
||||
# public endpoint silently.
|
||||
#
|
||||
# Usage: ./check-route-auth.sh (read-only; run from anywhere in repo)
|
||||
# Exit: 0 = only known-public routes unguarded, 1 = unexpected unguarded route
|
||||
set -u
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null)"
|
||||
[ -z "$ROOT" ] && { echo "ERROR: not inside a git repo" >&2; exit 2; }
|
||||
cd "$ROOT" || exit 2
|
||||
|
||||
# CANONICAL intentional-public-route list (verified 2026-07-09). This comment
|
||||
# is the ONE home for it; mvp-architecture-contract invariant 2.1 and
|
||||
# mvp-debugging-playbook section 7 point here instead of keeping their own lists.
|
||||
#
|
||||
# Within this script's scan scope (backend/src/features/*/api/*.routes.ts),
|
||||
# allowlisted via KNOWN_PUBLIC below:
|
||||
# /webhooks/stripe, /webhooks/resend/inbound - provider signature auth
|
||||
# /auth/signup, /auth/resend-verification-public - pre-auth flows
|
||||
# (comments in auth.routes.ts: "public, no JWT required")
|
||||
#
|
||||
# OUTSIDE this script's scan scope (registered in core code, also intentionally
|
||||
# public - the script can neither flag nor clear them):
|
||||
# /health, /api/health - backend/src/app.ts health endpoints
|
||||
# /auth/verify - backend/src/app.ts (Traefik forward-auth)
|
||||
# GET /api/config/feature-tiers - backend/src/core/config/config.routes.ts
|
||||
# (no preHandler; config is not sensitive)
|
||||
KNOWN_PUBLIC='/webhooks/stripe /webhooks/resend/inbound /auth/signup /auth/resend-verification-public'
|
||||
|
||||
OUT=$(awk '
|
||||
function flush() {
|
||||
if (inblock && !guarded) printf "%s:%d %s %s\n", f, startline, method, path
|
||||
inblock = 0
|
||||
}
|
||||
FNR == 1 { flush() }
|
||||
/fastify\.(get|post|put|patch|delete)[<(]/ {
|
||||
flush()
|
||||
inblock = 1; guarded = 0; startline = FNR; path = "?"; f = FILENAME
|
||||
match($0, /fastify\.(get|post|put|patch|delete)/)
|
||||
method = toupper(substr($0, RSTART + 8, RLENGTH - 8))
|
||||
}
|
||||
inblock && path == "?" {
|
||||
if (match($0, "\047/[^\047]*\047")) path = substr($0, RSTART + 1, RLENGTH - 2)
|
||||
}
|
||||
# requireAuth is the common local alias: const requireAuth = fastify.authenticate.bind(fastify)
|
||||
inblock && /[ .\[](authenticate|requireAuth|requireAdmin|requireTier)[ ,\]\}\)\(]/ { guarded = 1 }
|
||||
END { flush() }
|
||||
' backend/src/features/*/api/*.routes.ts)
|
||||
|
||||
TOTAL=$(grep -c 'fastify\.\(get\|post\|put\|patch\|delete\)[<(]' \
|
||||
backend/src/features/*/api/*.routes.ts | awk -F: '{s+=$2} END {print s}')
|
||||
|
||||
echo "Route registrations scanned: $TOTAL"
|
||||
echo
|
||||
|
||||
FAIL=0
|
||||
if [ -z "$OUT" ]; then
|
||||
echo "PASS: every route block contains an auth guard."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Routes with NO authenticate/requireAdmin/requireTier in their block:"
|
||||
echo "$OUT" | while read -r line; do
|
||||
p=$(echo "$line" | awk '{print $3}')
|
||||
case " $KNOWN_PUBLIC " in
|
||||
*" $p "*) echo " OK (known-public, allowlisted): $line" ;;
|
||||
*) echo " UNGUARDED: $line" ;;
|
||||
esac
|
||||
done
|
||||
# Re-count outside the pipe subshell (bash 3.2: while-in-pipe loses vars)
|
||||
FAIL=$(echo "$OUT" | awk -v kp=" $KNOWN_PUBLIC " '{ if (index(kp, " " $3 " ") == 0) n++ } END {print n+0}')
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "PASS: all unguarded routes are on the known-public allowlist."
|
||||
exit 0
|
||||
else
|
||||
echo "FAIL: $FAIL route(s) unguarded and not on the allowlist."
|
||||
echo "Either add a preHandler guard or, if intentionally public, add the"
|
||||
echo "path to KNOWN_PUBLIC in this script with a justification comment."
|
||||
echo "Limitation: block-scope heuristic; a guard mentioned in a comment"
|
||||
echo "inside the block can mask a real gap - verify hits by reading code."
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/bin/bash
|
||||
# local-gate.sh - Run every quality check that works on a dev machine and
|
||||
# print a PASS/FAIL table.
|
||||
#
|
||||
# Why: CI runs ZERO tests and ZERO lint (the only PR gate is that images
|
||||
# build and staging boots healthy). This script IS the pre-push gate.
|
||||
#
|
||||
# Scope (deliberate):
|
||||
# - backend lint, type-check, UNIT tests only (--forceExit; open pg/redis
|
||||
# handles otherwise hang jest). Integration tests are EXCLUDED: they
|
||||
# need a live database and some DROP TABLE CASCADE (owner non-negotiable:
|
||||
# never without a fresh backup). audit-log __tests__ also need a DB and
|
||||
# are excluded via path pattern.
|
||||
# - frontend lint, type-check. Frontend jest is broken outside the
|
||||
# container (2026-07-07), so no frontend tests here.
|
||||
#
|
||||
# Usage: ./local-gate.sh (needs node_modules in backend/ and frontend/:
|
||||
# run "npm install" in each first)
|
||||
# Exit: 0 = all pass, 1 = at least one failure
|
||||
set -u
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null)"
|
||||
[ -z "$ROOT" ] && { echo "ERROR: not inside a git repo" >&2; exit 2; }
|
||||
LOGDIR="${TMPDIR:-/tmp}/mvp-local-gate.$$"
|
||||
mkdir -p "$LOGDIR"
|
||||
|
||||
RESULTS=""
|
||||
OVERALL=0
|
||||
|
||||
run_step() { # run_step <label> <dir> <command...>
|
||||
label="$1"; dir="$2"; shift 2
|
||||
log="$LOGDIR/$(echo "$label" | tr ' /' '__').log"
|
||||
printf '%-28s ' "$label..."
|
||||
if (cd "$ROOT/$dir" && "$@") >"$log" 2>&1; then
|
||||
echo "PASS"
|
||||
RESULTS="$RESULTS$label|PASS
|
||||
"
|
||||
else
|
||||
echo "FAIL (log: $log)"
|
||||
tail -15 "$log" | sed 's/^/ /'
|
||||
RESULTS="$RESULTS$label|FAIL
|
||||
"
|
||||
OVERALL=1
|
||||
fi
|
||||
}
|
||||
|
||||
run_step "backend lint" backend npm run --silent lint
|
||||
run_step "backend type-check" backend npm run --silent type-check
|
||||
run_step "backend unit tests" backend npm test --silent -- \
|
||||
--testPathPattern='(tests/unit|src/core)' \
|
||||
--testPathIgnorePatterns='integration' --forceExit
|
||||
run_step "frontend lint" frontend npm run --silent lint
|
||||
run_step "frontend type-check" frontend npm run --silent type-check
|
||||
|
||||
echo
|
||||
echo "==================== LOCAL GATE ===================="
|
||||
printf '%-28s %s\n' "CHECK" "RESULT"
|
||||
echo "$RESULTS" | awk -F'|' 'NF { printf "%-28s %s\n", $1, $2 }'
|
||||
echo "====================================================="
|
||||
if [ "$OVERALL" -eq 0 ]; then
|
||||
echo "PASS: safe to push. (Integration tests NOT run - container-only.)"
|
||||
else
|
||||
echo "FAIL: fix before pushing. CI will NOT catch these."
|
||||
fi
|
||||
exit "$OVERALL"
|
||||
Reference in New Issue
Block a user