---
name: mvp-ocr-gemini-pipeline
description: The AI extraction subsystem - OCR engines, Gemini semantic extraction, VIN decode, and the WIF auth chain. Load this when working on anything under backend/src/features/ocr/, ocr/app/, or backend/src/features/email-ingestion/, or when you see symptoms like VIN decode failing/timing out, "Gemini authentication failed", "GeminiUnavailableError", pydantic validation error from google-genai, receipt scan returning empty fields, "No maintenance receipt fields could be extracted", manual extraction job stuck or expired, OCR container unhealthy, Google Vision quota, or "OCR service error 502/503/504". Also load before changing GEMINI_MODEL, GenerateContentConfig fields, or ocr/requirements.txt.
---

# mvp-ocr-gemini-pipeline

The AI extraction subsystem: React frontend -> Fastify backend proxy -> Python FastAPI OCR service -> PaddleOCR / Google Vision / Gemini (Vertex AI via google-genai SDK). Authored 2026-07-07; all paths and values verified against code on that date.

## When to use / When NOT to use

Use this skill when you are:
- Modifying `backend/src/features/ocr/`, `ocr/app/`, or `backend/src/features/email-ingestion/`.
- Debugging VIN decode, receipt scan, manual extraction, or email-ingested receipts.
- Touching Gemini config (`GEMINI_MODEL`, `GenerateContentConfig`, response schemas) or the WIF auth chain.

Do NOT use this skill for:
- General failure triage across the app -> `mvp-debugging-playbook`.
- The full incident history (the AFC revert appears there in chronicle form) -> `mvp-failure-archaeology`.
- Adding config knobs or secrets outside OCR -> `mvp-config-and-secrets`.
- VIN structure theory, check digits, year-code tables as domain knowledge -> `mvp-vehicle-domain-reference` (this skill covers only how the code uses them).
- Deploy/rollback mechanics for the OCR container -> `mvp-run-and-operate`.
- What counts as test evidence, definition of done -> `mvp-validation-and-qa`.

WARNING: `docs/ocr-pipeline-tech-stack.md` is substantially stale (describes Celery, spaCy, S3/MinIO - none exist). Code wins. Do not use that doc as a source.

## 1. End-to-end flows (real paths)

Three hops. Every OCR feature goes: frontend hook -> backend proxy (auth + tier gate + size/type validation) -> Python service.

### Backend proxy: `backend/src/features/ocr/`

- `api/ocr.routes.ts` - route registration, auth + tier preHandlers.
- `domain/ocr.service.ts` - size caps (`MAX_SYNC_SIZE` 10MB, `MAX_ASYNC_SIZE` 200MB) and MIME allowlist (jpeg/png/heic/heif/pdf); manual jobs must be `application/pdf`.
- `external/ocr-client.ts` - HTTP client to `OCR_SERVICE_URL` (default `http://mvp-ocr:8000`). Every call goes through `fetchWithTimeout` with hardcoded `OCR_TIMEOUT_MS = 120000` (line 9; AbortController, NOT env-configurable). `decodeVin()` posts JSON; everything else is multipart.

### Endpoint map (verified 2026-07-07)

| Backend route (`/api` prefix) | Tier gate | Python endpoint | Engine path |
|---|---|---|---|
| `POST /ocr/extract` | auth only | `POST /extract` | `services/ocr_service.py` -> engine from `engines/engine_factory.py` |
| `POST /ocr/extract/vin` | auth only | `POST /extract/vin` | `extractors/vin_extractor.py`: preprocess -> OCR -> 17-char pattern + check digit (`validators/vin_validator.py`), I/O/Q correction |
| `POST /ocr/extract/receipt` | pro: `fuelLog.receiptScan` | `POST /extract/receipt` | `extractors/receipt_extractor.py` + `fuel_receipt.py` (OCR + regex in `app/patterns/`) |
| `POST /ocr/extract/maintenance-receipt` | pro: `maintenance.receiptScan` | `POST /extract/maintenance-receipt` | `extractors/maintenance_receipt_extractor.py`: OCR -> Gemini structured JSON -> regex cross-validation |
| `POST /ocr/extract/manual` | pro: `document.scanMaintenanceSchedule` | `POST /extract/manual` (async) | Redis DB 1 job via `services/job_queue.py`, BackgroundTask -> `extractors/manual_extractor.py` -> `GeminiEngine.extract_maintenance()` (whole PDF inline) -> `patterns/service_mapping.py` (27 subtypes) |
| `POST /ocr/jobs`, `GET /ocr/jobs/:jobId` | auth only | `POST /jobs`, `GET /jobs/{job_id}` | generic async OCR + job polling (`routers/jobs.py`) |
| `POST /vehicles/decode-vin` (vehicles feature, not ocr) | pro: `vehicle.vinDecode` | `POST /decode/vin` | `routers/decode.py` -> shared lazy `GeminiEngine.decode_vin()` |

Python routers: `ocr/app/routers/extract.py`, `decode.py`, `jobs.py`, registered in `ocr/app/main.py`. `/health` is in `main.py`.

### VIN decode flow (the most fragile path)

1. Frontend: `frontend/src/features/vehicles/api/vehicles.api.ts:90` - axios timeout 120000ms.
2. Backend: `backend/src/features/vehicles/api/vehicles.controller.ts:decodeVin()` (line 384) - regex-validates `^[A-HJ-NPR-Z0-9]{17}$`, calls `ocrClient.decodeVin()`. Error mapping: OCR 503/422 -> 502 `VIN_DECODE_FAILED`; message contains "timed out"/"aborted" -> 504 `VIN_DECODE_TIMEOUT`.
3. Python: `ocr/app/routers/decode.py` -> `GeminiEngine.decode_vin()` with Google Search grounding (`tools=[types.Tool(google_search=types.GoogleSearch())]`, `gemini_engine.py:384`).
4. Model year is NEVER taken from the LLM: `resolve_vin_year()` (`gemini_engine.py:58`) computes it deterministically from VIN positions 7 and 10 (alphabetic pos 7 -> 2010-2039 cycle, per NHTSA FMVSS No. 115; this logic was inverted before commit `936753f` - do not "fix" it back). The result overrides whatever Gemini returns (`gemini_engine.py:409`).

### Email ingestion path: `backend/src/features/email-ingestion/`

`POST /api/webhooks/resend/inbound` (public, Svix-signature-verified; `api/email-ingestion.routes.ts`) -> `domain/email-ingestion.service.ts:processEmail()`:
1. Validate sender against user profiles; fetch and filter attachments.
2. Classify subject+body via `domain/receipt-classifier.ts` (keyword matching; confident requires >= 2 keyword matches for one type, ties are unclassified).
3. Confident -> call the matching OCR endpoint, with the other as fallback on failure. Unconfident -> call BOTH `/extract/receipt` and `/extract/maintenance-receipt` per attachment, re-classify from returned `rawText`, then a field-count heuristic (`email-ingestion.service.ts` around lines 280-325).
4. Vehicle association: exact match creates the `fuel_log` or `maintenance_record` directly; ambiguous match inserts a pending association (`insertPendingAssociation`) for the user to resolve later.

## 2. Engine architecture

Two engine categories - do not conflate them:

1. **OcrEngine subclasses** (image -> text + word boxes): `ocr/app/engines/paddle_engine.py` (PaddleOCR, local), `cloud_engine.py` (Google Vision TEXT_DETECTION), `hybrid_engine.py` (primary + fallback wrapper). Built via `engine_factory.create_engine()`; registry accepts only `paddleocr` and `google_vision`. If `OCR_FALLBACK_ENGINE != "none"`, factory returns a `HybridEngine`.
2. **GeminiEngine** (`engines/gemini_engine.py`): standalone, NOT an OcrEngine subclass. Semantic extraction (PDF -> maintenance schedule JSON) and VIN decode. Instantiated directly by `manual_extractor.py` and `routers/decode.py`, never through the factory. `maintenance_receipt_extractor.py` has its OWN copy-pasted Gemini client (see section 3).

**Which engine actually runs - code defaults vs compose (compose wins in every deployed environment):**

| Setting | Code default (`ocr/app/config.py`) | `docker-compose.yml` (all envs inherit) |
|---|---|---|
| `OCR_PRIMARY_ENGINE` | `paddleocr` | `google_vision` |
| `OCR_FALLBACK_ENGINE` | `none` | `paddleocr` |

Code defaults only apply where env vars are absent - i.e., bare `pytest` runs and ad-hoc local Python. Deployed containers run Vision-primary with PaddleOCR fallback. When reasoning about production behavior, always use the compose values.

`HybridEngine` semantics (`hybrid_engine.py`): with a CLOUD primary, the Vision monthly cap (`VISION_MONTHLY_LIMIT`, 1000/calendar month, Redis counter `ocr:vision_requests:YYYY-MM` in DB 1) is checked BEFORE calling it; once exhausted, PaddleOCR is the sole engine until month end. With a LOCAL primary, fallback triggers on confidence < `OCR_FALLBACK_THRESHOLD`. `_CLOUD_TIMEOUT_SECONDS = 10.0` (raised from 5s because first-call WIF token exchange takes 6-8s, refs #182) is a discard-after-completion threshold, NOT an abort: `_run_cloud_with_cap` runs `cloud.recognize()` to completion with no timeout or cancellation, only then compares elapsed time, and throws away results slower than 10s (without incrementing the Vision counter). A hung or slow Vision call can therefore block far past 10s -- directly relevant to weak point 1 (sync calls block the event loop); do not rule out the Vision path in a hung-container investigation because of this constant.

## 3. Auth chain: WIF via Auth0 (memorize this before touching credentials)

WIF = Workload Identity Federation: Google Cloud accepts a third-party JWT (here, an Auth0 machine-to-machine token) instead of a service-account key file. No Google key ever exists on disk.

Chain, in order:
1. `secrets/app/google-wif-config.json` - **deliberately committed to git; it is configuration, not a secret** (it contains only the pool/provider audience and an executable pointer). Mounted at `/run/secrets/google-wif-config.json`; `GOOGLE_VISION_KEY_PATH` points at it. Do not "fix" its presence in git.
2. Its `credential_source.executable.command` is `/app/scripts/fetch-auth0-token.sh` (repo: `ocr/scripts/fetch-auth0-token.sh`, chmod +x in `ocr/Dockerfile:48`, 30s timeout). The script reads `/run/secrets/auth0-ocr-client-id` and `/run/secrets/auth0-ocr-client-secret` (real secrets, NOT in git; compose mounts them from `secrets/app/*.txt`), requests an Auth0 M2M token, and emits Google's executable-credential JSON format.
3. Google STS exchanges that JWT, then impersonates `mvp-svc-account@motovaultpro.iam.gserviceaccount.com`.
4. Client bootstrap sets, as process-wide env vars INSIDE `_get_client()` immediately before `genai.Client(vertexai=True, ...)` construction: `GOOGLE_APPLICATION_CREDENTIALS=<key_path>` and `GOOGLE_EXTERNAL_ACCOUNT_ALLOW_EXECUTABLES=1` (executables are refused by google-auth without this flag).

**The bootstrap is DUPLICATED.** Identical `_get_client()` implementations exist in `ocr/app/engines/gemini_engine.py:218` and `ocr/app/extractors/maintenance_receipt_extractor.py:173` (and `cloud_engine.py:67` sets the same env vars for Vision). Any auth or client-construction fix must land in BOTH `_get_client()` copies or you fix VIN decode/manual extraction while leaving maintenance receipts broken (or vice versa).

On a dev machine the `auth0-ocr-client-*` secrets typically exist only as Docker-created empty directories - the live Gemini path cannot run locally; end-to-end verification happens on staging.

## 4. google-genai SDK sharp edges

**The dependency is unpinned:** `ocr/requirements.txt:25` has `google-genai>=1.0.0` (floor only). Each image build resolves whatever is newest. This already caused a staging breakage:

- `936753f` added `automatic_function_calling=types.AutomaticFunctionCallingConfig(max_remote_calls=3)` to the VIN decode `GenerateContentConfig`.
- The installed SDK version rejected it with a pydantic validation error at request time - VIN decode broke on staging while the build stayed green (CI runs zero tests; the only gate is images build + containers report healthy).
- `1add6c8` removed the parameter; `56df5d4` confirmed the revert and added diagnostic logging (full raw Gemini JSON at `gemini_engine.py:401-406`; `hasTrim/hasEngine/hasTransmission` in `vehicles.controller.ts`). Do not re-add AFC config without pinning and verifying the SDK version.

**Rule: before shipping any new `GenerateContentConfig` field (or `types.*` construct), validate it against the INSTALLED SDK version**, not the docs:

```bash
# What version is actually in the running container
docker compose exec mvp-ocr pip show google-genai | head -2

# Does the field exist / validate on that version (pydantic validates at construction)
docker compose exec mvp-ocr python3 -c "
from google.genai import types
print(types.GenerateContentConfig(response_mime_type='application/json', YOUR_NEW_FIELD=...))"
```

**Model config drift (known-weak, deliberate as of 2026-07-07):** code default is `gemini-2.5-flash` (`ocr/app/config.py:37`) but `docker-compose.yml` overrides `GEMINI_MODEL: gemini-3-flash-preview` - a PREVIEW model in deployed config. Preview models can be withdrawn or change behavior without notice. If Gemini responses suddenly degrade or 404, check the model name first. CLAUDE.md files under `ocr/` still say "Gemini 2.5 Flash" - compose wins.

**Response schemas** use Vertex-style UPPERCASE types (`"OBJECT"`, `"STRING"`, `"NUMBER"`, nullable flags) - keep that convention when editing `_VIN_DECODE_SCHEMA` / `_RESPONSE_SCHEMA` / `_RECEIPT_RESPONSE_SCHEMA`.

## 5. Config knob table (verified 2026-07-07)

| Knob | Where | Default / deployed value |
|---|---|---|
| `GEMINI_MODEL` | `ocr/app/config.py:37` / `docker-compose.yml` | `gemini-2.5-flash` / `gemini-3-flash-preview` |
| `VERTEX_AI_PROJECT` / `VERTEX_AI_LOCATION` | `config.py:33-36` / compose | `""` / `global` -> compose: `motovaultpro` / `global` |
| `OCR_PRIMARY_ENGINE` / `OCR_FALLBACK_ENGINE` | `config.py:13,19` / compose | `paddleocr`/`none` -> compose: `google_vision`/`paddleocr` |
| `OCR_CONFIDENCE_THRESHOLD` / `OCR_FALLBACK_THRESHOLD` | `config.py` | 0.6 / 0.6 |
| `VISION_MONTHLY_LIMIT` | `config.py:28` | 1000 requests/calendar month |
| `GOOGLE_VISION_KEY_PATH` | `config.py:23` | `/run/secrets/google-wif-config.json` (shared by Vision and Gemini) |
| Sync size cap | `ocr.service.ts:20`, `extract.py:30` | 10MB (413 above) |
| Manual/async size cap | `ocr.service.ts:23`, `extract.py:33` | 200MB |
| Gemini inline PDF cap | `gemini_engine.py:20` `_MAX_PDF_BYTES` | 20MB hard limit |
| Backend->OCR timeout | `ocr-client.ts:9` | 120000ms, hardcoded, all endpoints |
| Frontend timeouts | `vehicles.api.ts:90` (VIN decode), `useVinOcr.ts:52`, `useManualExtraction.ts:54` | 120000ms |
|  | `useReceiptOcr.ts:153`, `useMaintenanceReceiptOcr.ts:147` | 30000ms |
| Hybrid cloud result-discard threshold (the call itself is unbounded) | `hybrid_engine.py:25` | 10.0s |
| Job TTLs | `job_queue.py:19-22` | 3600s regular / 7200s manual; expired jobs -> backend 410 |
| OCR Redis | `config.py:40-42` | `mvp-redis` DB 1 (backend uses DB 0) |
| Gemini base field confidence | `maintenance_receipt_extractor.py:31`, `manual_extractor.py:58` | 0.85 before regex cross-validation |
| Service-side Gemini timeout | (nowhere) | NONE - `generate_content` calls are unbounded inside the Python service |

## 6. Known-weak points (owner-acknowledged, 2026-07-07; do not silently "improve" without an issue)

1. **Sync Gemini calls block the uvicorn event loop.** `routers/decode.py:42` and `routers/extract.py:239,327` run synchronous OCR/Gemini work directly in `async def` handlers. A slow Search-grounded VIN decode (observed >60s) blocks ALL requests including `/health` (compose healthcheck: interval 5s, timeout 5s, retries 3 -> container flagged unhealthy within ~15s). Verified: plain compose does not auto-restart unhealthy containers, but `scripts/ci/health-check.sh` and the deploy/rollback gates treat unhealthy as failure, so a deploy landing during a long decode can fail its gate or trigger rollback. Only manual extraction uses `run_in_executor` (`extract.py:472`).
2. **120s == 120s timeout race.** Frontend VIN decode timeout equals the backend->OCR timeout exactly; no headroom, so at the boundary the user sees a generic axios timeout instead of the backend's mapped 504. Receipt hooks are worse in a different way: frontend gives up at 30s while backend + OCR + Gemini keep burning quota for up to 120s.
3. **No retries anywhere** for transient Gemini/Vision failures. The single exception is maintenance-receipt's silent fallback (next item).
4. **Silent empty-fields fallback.** In `maintenance_receipt_extractor.py:129-133`, any Gemini exception is caught, logged at WARNING, and `gemini_fields = {}` - which usually surfaces as 422 "No maintenance receipt fields could be extracted". A dead Gemini auth chain therefore looks like "bad receipt photo". Check OCR container logs for "Gemini extraction failed, falling back to OCR-only" before blaming image quality.
5. **20MB vs 200MB manual gap.** `/extract/manual` accepts up to 200MB, queues the job, then `GeminiEngine.extract_maintenance()` rejects anything over 20MB with an error suggesting GCS URIs - which are not implemented. Every 21-200MB manual is a guaranteed deferred failure.
6. **Email-ingestion cost amplification.** Unconfident classification runs BOTH OCR endpoints per attachment (2x Vision against the 1000/month cap, plus a Gemini call for the maintenance path). A burst of ambiguous forwarded emails can drain the Vision quota.
7. **Manual job progress race** (`manual_extractor.py:122-126`): fire-and-forget progress updates from worker threads could overwrite COMPLETED status; mitigated only by the convention "never send 100% from the extractor". Preserve that convention.
8. **VIN decode trusts LLM+Search for specs.** Only the year is ground-truthed. Full raw Gemini JSON is logged per decode (`gemini_engine.py:401-406`) - diagnostic aid, but log volume/PII to keep in mind.

## 7. How to test

CI gates nothing beyond build + boot (canonical statement: `mvp-validation-and-qa` section 1). All tests below are local-only and are your responsibility to run.

**Python unit tests (all Gemini/network calls mocked - safe without secrets):**
```bash
cd ocr && pytest
```
Key files: `tests/test_gemini_engine.py` (client init, 20MB cap, extraction, error paths), `test_resolve_vin_year.py` (year-cycle disambiguation - regenerated in `936753f`, do not weaken), `test_vin_decode.py`, `test_receipt_extraction.py`, `test_maintenance_patterns.py`, `test_engine_abstraction.py`, `test_service_mapping.py`. Remember bare pytest sees code defaults (paddleocr primary), not compose values.

**Backend unit tests (OcrClient mocked):**
```bash
cd backend && npm test -- --testPathPattern="features/ocr"
```
Files: `backend/src/features/ocr/tests/unit/ocr-receipt.test.ts`, `ocr-manual.test.ts`. Do NOT run the backend integration tests casually - they DROP TABLE CASCADE on the shared dev database (owner non-negotiable: fresh backup first).

**Live pipeline: staging only.** The WIF chain needs real `auth0-ocr-client-id/secret` files, which exist only as empty mount directories on dev machines. Push a PR, let the staging deploy run, then exercise the flow through the staging UI, or from the staging host:
```bash
docker compose exec mvp-backend curl -s -X POST http://mvp-ocr:8000/decode/vin \
  -H 'Content-Type: application/json' -d '{"vin":"<17-char VIN>"}'
```
Watch logs: `docker logs mvp-ocr-staging --tail 100` on the staging host (dev/prod container name is `mvp-ocr`); look for "Gemini engine initialized", "VIN year resolved", "Gemini decoded VIN". Never hand-edit files on staging/prod servers - rsync --delete reverts them on the next deploy.

## Provenance and maintenance

Authored 2026-07-07 by direct inspection of the repo (all line numbers and values verified against code, not docs). Volatile facts and how to re-verify each:

| Fact | Re-verify with |
|---|---|
| Backend->OCR timeout 120000ms | `grep -n OCR_TIMEOUT_MS backend/src/features/ocr/external/ocr-client.ts` |
| Frontend timeouts (120s VIN decode, 30s receipts) | `grep -rn "timeout:" frontend/src/features/vehicles/api/vehicles.api.ts frontend/src/features/*/hooks/use*Ocr*.ts frontend/src/features/documents/hooks/useManualExtraction.ts` |
| Deployed model `gemini-3-flash-preview`, Vision-primary engines | `grep -n "GEMINI_MODEL\|OCR_PRIMARY_ENGINE\|OCR_FALLBACK_ENGINE" docker-compose.yml` |
| Code defaults (`gemini-2.5-flash`, paddleocr/none) | `grep -n "getenv" ocr/app/config.py` |
| google-genai unpinned | `grep -n google-genai ocr/requirements.txt` |
| Installed SDK version in a running container | `docker compose exec mvp-ocr pip show google-genai` |
| Size caps 10MB/200MB/20MB | `grep -n "MAX_SYNC_SIZE\|MAX_ASYNC_SIZE\|MAX_MANUAL_SIZE\|_MAX_PDF_BYTES" backend/src/features/ocr/domain/ocr.service.ts ocr/app/routers/extract.py ocr/app/engines/gemini_engine.py` |
| Job TTLs 3600/7200, Redis DB 1 | `grep -n "JOB_TTL\|MANUAL_JOB_TTL" ocr/app/services/job_queue.py; grep -n REDIS_DB docker-compose.yml` |
| Duplicated `_get_client()` bootstrap | `grep -n "_get_client\|ALLOW_EXECUTABLES" ocr/app/engines/gemini_engine.py ocr/app/extractors/maintenance_receipt_extractor.py ocr/app/engines/cloud_engine.py` |
| WIF config committed, executable helper path | `git ls-files secrets/app/ \| grep wif; grep -n executable secrets/app/google-wif-config.json` |
| AFC incident commits | `git log --oneline 936753f 1add6c8 56df5d4 -1 --no-walk 2>/dev/null \|\| git show --stat 56df5d4` |
| Tier gates on OCR routes | `grep -n requireTier backend/src/features/ocr/api/ocr.routes.ts backend/src/features/vehicles/api/vehicles.routes.ts` |
| CI still runs no tests | `grep -rn "pytest\|npm test\|jest" .gitea/workflows/` (expect no test-execution hits) |
| Event-loop-blocking sync calls | `grep -n "_gemini_engine.decode_vin\|receipt_extractor.extract\|maintenance_receipt_extractor.extract" ocr/app/routers/decode.py ocr/app/routers/extract.py` |