chore: replace AI skill library with 16 verified mvp-* skills

Removes the old planner/decision-critic/role-agents/domain-agents system (step-injector scripts, stale scopes, sprint-era workflow) and replaces it with 16 ground-truth-verified skills under .claude/skills/: change control, debugging playbook, failure archaeology, architecture contract, domain reference, OCR/Gemini pipeline, config and secrets, build and env, run and operate, diagnostics (with tested scripts), validation and QA, docs and writing, launch readiness, deploy-safety campaign, proof and analysis toolkit, and research frontier. RULE 0/1/2, the temporal-contamination rule, and the decision stress-test protocol are carried forward into the new skills; the retired content remains in git history. Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-09 20:21:42 -05:00
parent e729d425fd
commit c239bb9347
71 changed files with 4087 additions and 10221 deletions
@@ -0,0 +1,197 @@
+---
+name: mvp-ocr-gemini-pipeline
+description: The AI extraction subsystem - OCR engines, Gemini semantic extraction, VIN decode, and the WIF auth chain. Load this when working on anything under backend/src/features/ocr/, ocr/app/, or backend/src/features/email-ingestion/, or when you see symptoms like VIN decode failing/timing out, "Gemini authentication failed", "GeminiUnavailableError", pydantic validation error from google-genai, receipt scan returning empty fields, "No maintenance receipt fields could be extracted", manual extraction job stuck or expired, OCR container unhealthy, Google Vision quota, or "OCR service error 502/503/504". Also load before changing GEMINI_MODEL, GenerateContentConfig fields, or ocr/requirements.txt.
+---
+
+# mvp-ocr-gemini-pipeline
+
+The AI extraction subsystem: React frontend -> Fastify backend proxy -> Python FastAPI OCR service -> PaddleOCR / Google Vision / Gemini (Vertex AI via google-genai SDK). Authored 2026-07-07; all paths and values verified against code on that date.
+
+## When to use / When NOT to use
+
+Use this skill when you are:
+- Modifying `backend/src/features/ocr/`, `ocr/app/`, or `backend/src/features/email-ingestion/`.
+- Debugging VIN decode, receipt scan, manual extraction, or email-ingested receipts.
+- Touching Gemini config (`GEMINI_MODEL`, `GenerateContentConfig`, response schemas) or the WIF auth chain.
+
+Do NOT use this skill for:
+- General failure triage across the app -> `mvp-debugging-playbook`.
+- The full incident history (the AFC revert appears there in chronicle form) -> `mvp-failure-archaeology`.
+- Adding config knobs or secrets outside OCR -> `mvp-config-and-secrets`.
+- VIN structure theory, check digits, year-code tables as domain knowledge -> `mvp-vehicle-domain-reference` (this skill covers only how the code uses them).
+- Deploy/rollback mechanics for the OCR container -> `mvp-run-and-operate`.
+- What counts as test evidence, definition of done -> `mvp-validation-and-qa`.
+
+WARNING: `docs/ocr-pipeline-tech-stack.md` is substantially stale (describes Celery, spaCy, S3/MinIO - none exist). Code wins. Do not use that doc as a source.
+
+## 1. End-to-end flows (real paths)
+
+Three hops. Every OCR feature goes: frontend hook -> backend proxy (auth + tier gate + size/type validation) -> Python service.
+
+### Backend proxy: `backend/src/features/ocr/`
+
+- `api/ocr.routes.ts` - route registration, auth + tier preHandlers.
+- `domain/ocr.service.ts` - size caps (`MAX_SYNC_SIZE` 10MB, `MAX_ASYNC_SIZE` 200MB) and MIME allowlist (jpeg/png/heic/heif/pdf); manual jobs must be `application/pdf`.
+- `external/ocr-client.ts` - HTTP client to `OCR_SERVICE_URL` (default `http://mvp-ocr:8000`). Every call goes through `fetchWithTimeout` with hardcoded `OCR_TIMEOUT_MS = 120000` (line 9; AbortController, NOT env-configurable). `decodeVin()` posts JSON; everything else is multipart.
+
+### Endpoint map (verified 2026-07-07)
+
+| Backend route (`/api` prefix) | Tier gate | Python endpoint | Engine path |
+|---|---|---|---|
+| `POST /ocr/extract` | auth only | `POST /extract` | `services/ocr_service.py` -> engine from `engines/engine_factory.py` |
+| `POST /ocr/extract/vin` | auth only | `POST /extract/vin` | `extractors/vin_extractor.py`: preprocess -> OCR -> 17-char pattern + check digit (`validators/vin_validator.py`), I/O/Q correction |
+| `POST /ocr/extract/receipt` | pro: `fuelLog.receiptScan` | `POST /extract/receipt` | `extractors/receipt_extractor.py` + `fuel_receipt.py` (OCR + regex in `app/patterns/`) |
+| `POST /ocr/extract/maintenance-receipt` | pro: `maintenance.receiptScan` | `POST /extract/maintenance-receipt` | `extractors/maintenance_receipt_extractor.py`: OCR -> Gemini structured JSON -> regex cross-validation |
+| `POST /ocr/extract/manual` | pro: `document.scanMaintenanceSchedule` | `POST /extract/manual` (async) | Redis DB 1 job via `services/job_queue.py`, BackgroundTask -> `extractors/manual_extractor.py` -> `GeminiEngine.extract_maintenance()` (whole PDF inline) -> `patterns/service_mapping.py` (27 subtypes) |
+| `POST /ocr/jobs`, `GET /ocr/jobs/:jobId` | auth only | `POST /jobs`, `GET /jobs/{job_id}` | generic async OCR + job polling (`routers/jobs.py`) |
+| `POST /vehicles/decode-vin` (vehicles feature, not ocr) | pro: `vehicle.vinDecode` | `POST /decode/vin` | `routers/decode.py` -> shared lazy `GeminiEngine.decode_vin()` |
+
+Python routers: `ocr/app/routers/extract.py`, `decode.py`, `jobs.py`, registered in `ocr/app/main.py`. `/health` is in `main.py`.
+
+### VIN decode flow (the most fragile path)
+
+1. Frontend: `frontend/src/features/vehicles/api/vehicles.api.ts:90` - axios timeout 120000ms.
+2. Backend: `backend/src/features/vehicles/api/vehicles.controller.ts:decodeVin()` (line 384) - regex-validates `^[A-HJ-NPR-Z0-9]{17}$`, calls `ocrClient.decodeVin()`. Error mapping: OCR 503/422 -> 502 `VIN_DECODE_FAILED`; message contains "timed out"/"aborted" -> 504 `VIN_DECODE_TIMEOUT`.
+3. Python: `ocr/app/routers/decode.py` -> `GeminiEngine.decode_vin()` with Google Search grounding (`tools=[types.Tool(google_search=types.GoogleSearch())]`, `gemini_engine.py:384`).
+4. Model year is NEVER taken from the LLM: `resolve_vin_year()` (`gemini_engine.py:58`) computes it deterministically from VIN positions 7 and 10 (alphabetic pos 7 -> 2010-2039 cycle, per NHTSA FMVSS No. 115; this logic was inverted before commit `936753f` - do not "fix" it back). The result overrides whatever Gemini returns (`gemini_engine.py:409`).
+
+### Email ingestion path: `backend/src/features/email-ingestion/`
+
+`POST /api/webhooks/resend/inbound` (public, Svix-signature-verified; `api/email-ingestion.routes.ts`) -> `domain/email-ingestion.service.ts:processEmail()`:
+1. Validate sender against user profiles; fetch and filter attachments.
+2. Classify subject+body via `domain/receipt-classifier.ts` (keyword matching; confident requires >= 2 keyword matches for one type, ties are unclassified).
+3. Confident -> call the matching OCR endpoint, with the other as fallback on failure. Unconfident -> call BOTH `/extract/receipt` and `/extract/maintenance-receipt` per attachment, re-classify from returned `rawText`, then a field-count heuristic (`email-ingestion.service.ts` around lines 280-325).
+4. Vehicle association: exact match creates the `fuel_log` or `maintenance_record` directly; ambiguous match inserts a pending association (`insertPendingAssociation`) for the user to resolve later.
+
+## 2. Engine architecture
+
+Two engine categories - do not conflate them:
+
+1. **OcrEngine subclasses** (image -> text + word boxes): `ocr/app/engines/paddle_engine.py` (PaddleOCR, local), `cloud_engine.py` (Google Vision TEXT_DETECTION), `hybrid_engine.py` (primary + fallback wrapper). Built via `engine_factory.create_engine()`; registry accepts only `paddleocr` and `google_vision`. If `OCR_FALLBACK_ENGINE != "none"`, factory returns a `HybridEngine`.
+2. **GeminiEngine** (`engines/gemini_engine.py`): standalone, NOT an OcrEngine subclass. Semantic extraction (PDF -> maintenance schedule JSON) and VIN decode. Instantiated directly by `manual_extractor.py` and `routers/decode.py`, never through the factory. `maintenance_receipt_extractor.py` has its OWN copy-pasted Gemini client (see section 3).
+
+**Which engine actually runs - code defaults vs compose (compose wins in every deployed environment):**
+
+| Setting | Code default (`ocr/app/config.py`) | `docker-compose.yml` (all envs inherit) |
+|---|---|---|
+| `OCR_PRIMARY_ENGINE` | `paddleocr` | `google_vision` |
+| `OCR_FALLBACK_ENGINE` | `none` | `paddleocr` |
+
+Code defaults only apply where env vars are absent - i.e., bare `pytest` runs and ad-hoc local Python. Deployed containers run Vision-primary with PaddleOCR fallback. When reasoning about production behavior, always use the compose values.
+
+`HybridEngine` semantics (`hybrid_engine.py`): with a CLOUD primary, the Vision monthly cap (`VISION_MONTHLY_LIMIT`, 1000/calendar month, Redis counter `ocr:vision_requests:YYYY-MM` in DB 1) is checked BEFORE calling it; once exhausted, PaddleOCR is the sole engine until month end. With a LOCAL primary, fallback triggers on confidence < `OCR_FALLBACK_THRESHOLD`. `_CLOUD_TIMEOUT_SECONDS = 10.0` (raised from 5s because first-call WIF token exchange takes 6-8s, refs #182) is a discard-after-completion threshold, NOT an abort: `_run_cloud_with_cap` runs `cloud.recognize()` to completion with no timeout or cancellation, only then compares elapsed time, and throws away results slower than 10s (without incrementing the Vision counter). A hung or slow Vision call can therefore block far past 10s -- directly relevant to weak point 1 (sync calls block the event loop); do not rule out the Vision path in a hung-container investigation because of this constant.
+
+## 3. Auth chain: WIF via Auth0 (memorize this before touching credentials)
+
+WIF = Workload Identity Federation: Google Cloud accepts a third-party JWT (here, an Auth0 machine-to-machine token) instead of a service-account key file. No Google key ever exists on disk.
+
+Chain, in order:
+1. `secrets/app/google-wif-config.json` - **deliberately committed to git; it is configuration, not a secret** (it contains only the pool/provider audience and an executable pointer). Mounted at `/run/secrets/google-wif-config.json`; `GOOGLE_VISION_KEY_PATH` points at it. Do not "fix" its presence in git.
+2. Its `credential_source.executable.command` is `/app/scripts/fetch-auth0-token.sh` (repo: `ocr/scripts/fetch-auth0-token.sh`, chmod +x in `ocr/Dockerfile:48`, 30s timeout). The script reads `/run/secrets/auth0-ocr-client-id` and `/run/secrets/auth0-ocr-client-secret` (real secrets, NOT in git; compose mounts them from `secrets/app/*.txt`), requests an Auth0 M2M token, and emits Google's executable-credential JSON format.
+3. Google STS exchanges that JWT, then impersonates `[email protected]`.
+4. Client bootstrap sets, as process-wide env vars INSIDE `_get_client()` immediately before `genai.Client(vertexai=True, ...)` construction: `GOOGLE_APPLICATION_CREDENTIALS=<key_path>` and `GOOGLE_EXTERNAL_ACCOUNT_ALLOW_EXECUTABLES=1` (executables are refused by google-auth without this flag).
+
+**The bootstrap is DUPLICATED.** Identical `_get_client()` implementations exist in `ocr/app/engines/gemini_engine.py:218` and `ocr/app/extractors/maintenance_receipt_extractor.py:173` (and `cloud_engine.py:67` sets the same env vars for Vision). Any auth or client-construction fix must land in BOTH `_get_client()` copies or you fix VIN decode/manual extraction while leaving maintenance receipts broken (or vice versa).
+
+On a dev machine the `auth0-ocr-client-*` secrets typically exist only as Docker-created empty directories - the live Gemini path cannot run locally; end-to-end verification happens on staging.
+
+## 4. google-genai SDK sharp edges
+
+**The dependency is unpinned:** `ocr/requirements.txt:25` has `google-genai>=1.0.0` (floor only). Each image build resolves whatever is newest. This already caused a staging breakage:
+
+- `936753f` added `automatic_function_calling=types.AutomaticFunctionCallingConfig(max_remote_calls=3)` to the VIN decode `GenerateContentConfig`.
+- The installed SDK version rejected it with a pydantic validation error at request time - VIN decode broke on staging while the build stayed green (CI runs zero tests; the only gate is images build + containers report healthy).
+- `1add6c8` removed the parameter; `56df5d4` confirmed the revert and added diagnostic logging (full raw Gemini JSON at `gemini_engine.py:401-406`; `hasTrim/hasEngine/hasTransmission` in `vehicles.controller.ts`). Do not re-add AFC config without pinning and verifying the SDK version.
+
+**Rule: before shipping any new `GenerateContentConfig` field (or `types.*` construct), validate it against the INSTALLED SDK version**, not the docs:
+
+```bash
+# What version is actually in the running container
+docker compose exec mvp-ocr pip show google-genai | head -2
+
+# Does the field exist / validate on that version (pydantic validates at construction)
+docker compose exec mvp-ocr python3 -c "
+from google.genai import types
+print(types.GenerateContentConfig(response_mime_type='application/json', YOUR_NEW_FIELD=...))"
+```
+
+**Model config drift (known-weak, deliberate as of 2026-07-07):** code default is `gemini-2.5-flash` (`ocr/app/config.py:37`) but `docker-compose.yml` overrides `GEMINI_MODEL: gemini-3-flash-preview` - a PREVIEW model in deployed config. Preview models can be withdrawn or change behavior without notice. If Gemini responses suddenly degrade or 404, check the model name first. CLAUDE.md files under `ocr/` still say "Gemini 2.5 Flash" - compose wins.
+
+**Response schemas** use Vertex-style UPPERCASE types (`"OBJECT"`, `"STRING"`, `"NUMBER"`, nullable flags) - keep that convention when editing `_VIN_DECODE_SCHEMA` / `_RESPONSE_SCHEMA` / `_RECEIPT_RESPONSE_SCHEMA`.
+
+## 5. Config knob table (verified 2026-07-07)
+
+| Knob | Where | Default / deployed value |
+|---|---|---|
+| `GEMINI_MODEL` | `ocr/app/config.py:37` / `docker-compose.yml` | `gemini-2.5-flash` / `gemini-3-flash-preview` |
+| `VERTEX_AI_PROJECT` / `VERTEX_AI_LOCATION` | `config.py:33-36` / compose | `""` / `global` -> compose: `motovaultpro` / `global` |
+| `OCR_PRIMARY_ENGINE` / `OCR_FALLBACK_ENGINE` | `config.py:13,19` / compose | `paddleocr`/`none` -> compose: `google_vision`/`paddleocr` |
+| `OCR_CONFIDENCE_THRESHOLD` / `OCR_FALLBACK_THRESHOLD` | `config.py` | 0.6 / 0.6 |
+| `VISION_MONTHLY_LIMIT` | `config.py:28` | 1000 requests/calendar month |
+| `GOOGLE_VISION_KEY_PATH` | `config.py:23` | `/run/secrets/google-wif-config.json` (shared by Vision and Gemini) |
+| Sync size cap | `ocr.service.ts:20`, `extract.py:30` | 10MB (413 above) |
+| Manual/async size cap | `ocr.service.ts:23`, `extract.py:33` | 200MB |
+| Gemini inline PDF cap | `gemini_engine.py:20` `_MAX_PDF_BYTES` | 20MB hard limit |
+| Backend->OCR timeout | `ocr-client.ts:9` | 120000ms, hardcoded, all endpoints |
+| Frontend timeouts | `vehicles.api.ts:90` (VIN decode), `useVinOcr.ts:52`, `useManualExtraction.ts:54` | 120000ms |
+|  | `useReceiptOcr.ts:153`, `useMaintenanceReceiptOcr.ts:147` | 30000ms |
+| Hybrid cloud result-discard threshold (the call itself is unbounded) | `hybrid_engine.py:25` | 10.0s |
+| Job TTLs | `job_queue.py:19-22` | 3600s regular / 7200s manual; expired jobs -> backend 410 |
+| OCR Redis | `config.py:40-42` | `mvp-redis` DB 1 (backend uses DB 0) |
+| Gemini base field confidence | `maintenance_receipt_extractor.py:31`, `manual_extractor.py:58` | 0.85 before regex cross-validation |
+| Service-side Gemini timeout | (nowhere) | NONE - `generate_content` calls are unbounded inside the Python service |
+
+## 6. Known-weak points (owner-acknowledged, 2026-07-07; do not silently "improve" without an issue)
+
+1. **Sync Gemini calls block the uvicorn event loop.** `routers/decode.py:42` and `routers/extract.py:239,327` run synchronous OCR/Gemini work directly in `async def` handlers. A slow Search-grounded VIN decode (observed >60s) blocks ALL requests including `/health` (compose healthcheck: interval 5s, timeout 5s, retries 3 -> container flagged unhealthy within ~15s). Verified: plain compose does not auto-restart unhealthy containers, but `scripts/ci/health-check.sh` and the deploy/rollback gates treat unhealthy as failure, so a deploy landing during a long decode can fail its gate or trigger rollback. Only manual extraction uses `run_in_executor` (`extract.py:472`).
+2. **120s == 120s timeout race.** Frontend VIN decode timeout equals the backend->OCR timeout exactly; no headroom, so at the boundary the user sees a generic axios timeout instead of the backend's mapped 504. Receipt hooks are worse in a different way: frontend gives up at 30s while backend + OCR + Gemini keep burning quota for up to 120s.
+3. **No retries anywhere** for transient Gemini/Vision failures. The single exception is maintenance-receipt's silent fallback (next item).
+4. **Silent empty-fields fallback.** In `maintenance_receipt_extractor.py:129-133`, any Gemini exception is caught, logged at WARNING, and `gemini_fields = {}` - which usually surfaces as 422 "No maintenance receipt fields could be extracted". A dead Gemini auth chain therefore looks like "bad receipt photo". Check OCR container logs for "Gemini extraction failed, falling back to OCR-only" before blaming image quality.
+5. **20MB vs 200MB manual gap.** `/extract/manual` accepts up to 200MB, queues the job, then `GeminiEngine.extract_maintenance()` rejects anything over 20MB with an error suggesting GCS URIs - which are not implemented. Every 21-200MB manual is a guaranteed deferred failure.
+6. **Email-ingestion cost amplification.** Unconfident classification runs BOTH OCR endpoints per attachment (2x Vision against the 1000/month cap, plus a Gemini call for the maintenance path). A burst of ambiguous forwarded emails can drain the Vision quota.
+7. **Manual job progress race** (`manual_extractor.py:122-126`): fire-and-forget progress updates from worker threads could overwrite COMPLETED status; mitigated only by the convention "never send 100% from the extractor". Preserve that convention.
+8. **VIN decode trusts LLM+Search for specs.** Only the year is ground-truthed. Full raw Gemini JSON is logged per decode (`gemini_engine.py:401-406`) - diagnostic aid, but log volume/PII to keep in mind.
+
+## 7. How to test
+
+CI gates nothing beyond build + boot (canonical statement: `mvp-validation-and-qa` section 1). All tests below are local-only and are your responsibility to run.
+
+**Python unit tests (all Gemini/network calls mocked - safe without secrets):**
+```bash
+cd ocr && pytest
+```
+Key files: `tests/test_gemini_engine.py` (client init, 20MB cap, extraction, error paths), `test_resolve_vin_year.py` (year-cycle disambiguation - regenerated in `936753f`, do not weaken), `test_vin_decode.py`, `test_receipt_extraction.py`, `test_maintenance_patterns.py`, `test_engine_abstraction.py`, `test_service_mapping.py`. Remember bare pytest sees code defaults (paddleocr primary), not compose values.
+
+**Backend unit tests (OcrClient mocked):**
+```bash
+cd backend && npm test -- --testPathPattern="features/ocr"
+```
+Files: `backend/src/features/ocr/tests/unit/ocr-receipt.test.ts`, `ocr-manual.test.ts`. Do NOT run the backend integration tests casually - they DROP TABLE CASCADE on the shared dev database (owner non-negotiable: fresh backup first).
+
+**Live pipeline: staging only.** The WIF chain needs real `auth0-ocr-client-id/secret` files, which exist only as empty mount directories on dev machines. Push a PR, let the staging deploy run, then exercise the flow through the staging UI, or from the staging host:
+```bash
+docker compose exec mvp-backend curl -s -X POST http://mvp-ocr:8000/decode/vin \
+  -H 'Content-Type: application/json' -d '{"vin":"<17-char VIN>"}'
+```
+Watch logs: `docker logs mvp-ocr-staging --tail 100` on the staging host (dev/prod container name is `mvp-ocr`); look for "Gemini engine initialized", "VIN year resolved", "Gemini decoded VIN". Never hand-edit files on staging/prod servers - rsync --delete reverts them on the next deploy.
+
+## Provenance and maintenance
+
+Authored 2026-07-07 by direct inspection of the repo (all line numbers and values verified against code, not docs). Volatile facts and how to re-verify each:
+
+| Fact | Re-verify with |
+|---|---|
+| Backend->OCR timeout 120000ms | `grep -n OCR_TIMEOUT_MS backend/src/features/ocr/external/ocr-client.ts` |
+| Frontend timeouts (120s VIN decode, 30s receipts) | `grep -rn "timeout:" frontend/src/features/vehicles/api/vehicles.api.ts frontend/src/features/*/hooks/use*Ocr*.ts frontend/src/features/documents/hooks/useManualExtraction.ts` |
+| Deployed model `gemini-3-flash-preview`, Vision-primary engines | `grep -n "GEMINI_MODEL\|OCR_PRIMARY_ENGINE\|OCR_FALLBACK_ENGINE" docker-compose.yml` |
+| Code defaults (`gemini-2.5-flash`, paddleocr/none) | `grep -n "getenv" ocr/app/config.py` |
+| google-genai unpinned | `grep -n google-genai ocr/requirements.txt` |
+| Installed SDK version in a running container | `docker compose exec mvp-ocr pip show google-genai` |
+| Size caps 10MB/200MB/20MB | `grep -n "MAX_SYNC_SIZE\|MAX_ASYNC_SIZE\|MAX_MANUAL_SIZE\|_MAX_PDF_BYTES" backend/src/features/ocr/domain/ocr.service.ts ocr/app/routers/extract.py ocr/app/engines/gemini_engine.py` |
+| Job TTLs 3600/7200, Redis DB 1 | `grep -n "JOB_TTL\|MANUAL_JOB_TTL" ocr/app/services/job_queue.py; grep -n REDIS_DB docker-compose.yml` |
+| Duplicated `_get_client()` bootstrap | `grep -n "_get_client\|ALLOW_EXECUTABLES" ocr/app/engines/gemini_engine.py ocr/app/extractors/maintenance_receipt_extractor.py ocr/app/engines/cloud_engine.py` |
+| WIF config committed, executable helper path | `git ls-files secrets/app/ \| grep wif; grep -n executable secrets/app/google-wif-config.json` |
+| AFC incident commits | `git log --oneline 936753f 1add6c8 56df5d4 -1 --no-walk 2>/dev/null \|\| git show --stat 56df5d4` |
+| Tier gates on OCR routes | `grep -n requireTier backend/src/features/ocr/api/ocr.routes.ts backend/src/features/vehicles/api/vehicles.routes.ts` |
+| CI still runs no tests | `grep -rn "pytest\|npm test\|jest" .gitea/workflows/` (expect no test-execution hits) |
+| Event-loop-blocking sync calls | `grep -n "_gemini_engine.decode_vin\|receipt_extractor.extract\|maintenance_receipt_extractor.extract" ocr/app/routers/decode.py ocr/app/routers/extract.py` |