motovaultpro/.claude/skills/mvp-ocr-gemini-pipeline/SKILL.md

name, description

name

description

mvp-ocr-gemini-pipeline

The AI extraction subsystem - OCR engines, Gemini semantic extraction, VIN decode, and the WIF auth chain. Load this when working on anything under backend/src/features/ocr/, ocr/app/, or backend/src/features/email-ingestion/, or when you see symptoms like VIN decode failing/timing out, "Gemini authentication failed", "GeminiUnavailableError", pydantic validation error from google-genai, receipt scan returning empty fields, "No maintenance receipt fields could be extracted", manual extraction job stuck or expired, OCR container unhealthy, Google Vision quota, or "OCR service error 502/503/504". Also load before changing GEMINI_MODEL, GenerateContentConfig fields, or ocr/requirements.txt.

mvp-ocr-gemini-pipeline

The AI extraction subsystem: React frontend -> Fastify backend proxy -> Python FastAPI OCR service -> PaddleOCR / Google Vision / Gemini (Vertex AI via google-genai SDK). Authored 2026-07-07; all paths and values verified against code on that date.

When to use / When NOT to use

Use this skill when you are:

• Modifying backend/src/features/ocr/, ocr/app/, or backend/src/features/email-ingestion/.
• Debugging VIN decode, receipt scan, manual extraction, or email-ingested receipts.
• Touching Gemini config (GEMINI_MODEL, GenerateContentConfig, response schemas) or the WIF auth chain.

Do NOT use this skill for:

• General failure triage across the app -> mvp-debugging-playbook.
• The full incident history (the AFC revert appears there in chronicle form) -> mvp-failure-archaeology.
• Adding config knobs or secrets outside OCR -> mvp-config-and-secrets.
• VIN structure theory, check digits, year-code tables as domain knowledge -> mvp-vehicle-domain-reference (this skill covers only how the code uses them).
• Deploy/rollback mechanics for the OCR container -> mvp-run-and-operate.
• What counts as test evidence, definition of done -> mvp-validation-and-qa.

WARNING: docs/ocr-pipeline-tech-stack.md is substantially stale (describes Celery, spaCy, S3/MinIO - none exist). Code wins. Do not use that doc as a source.

1. End-to-end flows (real paths)

Three hops. Every OCR feature goes: frontend hook -> backend proxy (auth + tier gate + size/type validation) -> Python service.

Backend proxy: backend/src/features/ocr/

• api/ocr.routes.ts - route registration, auth + tier preHandlers.
• domain/ocr.service.ts - size caps (MAX_SYNC_SIZE 10MB, MAX_ASYNC_SIZE 200MB) and MIME allowlist (jpeg/png/heic/heif/pdf); manual jobs must be application/pdf.
• external/ocr-client.ts - HTTP client to OCR_SERVICE_URL (default http://mvp-ocr:8000). Every call goes through fetchWithTimeout with hardcoded OCR_TIMEOUT_MS = 120000 (line 9; AbortController, NOT env-configurable). decodeVin() posts JSON; everything else is multipart.

Endpoint map (verified 2026-07-07)

Backend route (/api prefix)	Tier gate	Python endpoint	Engine path
POST /ocr/extract	auth only	POST /extract	services/ocr_service.py -> engine from engines/engine_factory.py
POST /ocr/extract/vin	auth only	POST /extract/vin	extractors/vin_extractor.py: preprocess -> OCR -> 17-char pattern + check digit (validators/vin_validator.py), I/O/Q correction
POST /ocr/extract/receipt	pro: fuelLog.receiptScan	POST /extract/receipt	extractors/receipt_extractor.py + fuel_receipt.py (OCR + regex in app/patterns/)
POST /ocr/extract/maintenance-receipt	pro: maintenance.receiptScan	POST /extract/maintenance-receipt	extractors/maintenance_receipt_extractor.py: OCR -> Gemini structured JSON -> regex cross-validation
POST /ocr/extract/manual	pro: document.scanMaintenanceSchedule	POST /extract/manual (async)	Redis DB 1 job via services/job_queue.py, BackgroundTask -> extractors/manual_extractor.py -> GeminiEngine.extract_maintenance() (whole PDF inline) -> patterns/service_mapping.py (27 subtypes)
POST /ocr/jobs, GET /ocr/jobs/:jobId	auth only	POST /jobs, GET /jobs/{job_id}	generic async OCR + job polling (routers/jobs.py)
POST /vehicles/decode-vin (vehicles feature, not ocr)	pro: vehicle.vinDecode	POST /decode/vin	routers/decode.py -> shared lazy GeminiEngine.decode_vin()

Python routers: ocr/app/routers/extract.py, decode.py, jobs.py, registered in ocr/app/main.py. /health is in main.py.

VIN decode flow (the most fragile path)

1. Frontend: frontend/src/features/vehicles/api/vehicles.api.ts:90 - axios timeout 120000ms.
2. Backend: backend/src/features/vehicles/api/vehicles.controller.ts:decodeVin() (line 384) - regex-validates ^[A-HJ-NPR-Z0-9]{17}$, calls ocrClient.decodeVin(). Error mapping: OCR 503/422 -> 502 VIN_DECODE_FAILED; message contains "timed out"/"aborted" -> 504 VIN_DECODE_TIMEOUT.
3. Python: ocr/app/routers/decode.py -> GeminiEngine.decode_vin() with Google Search grounding (tools=[types.Tool(google_search=types.GoogleSearch())], gemini_engine.py:384).
4. Model year is NEVER taken from the LLM: resolve_vin_year() (gemini_engine.py:58) computes it deterministically from VIN positions 7 and 10 (alphabetic pos 7 -> 2010-2039 cycle, per NHTSA FMVSS No. 115; this logic was inverted before commit 936753f - do not "fix" it back). The result overrides whatever Gemini returns (gemini_engine.py:409).

Email ingestion path: backend/src/features/email-ingestion/

POST /api/webhooks/resend/inbound (public, Svix-signature-verified; api/email-ingestion.routes.ts) -> domain/email-ingestion.service.ts:processEmail():

1. Validate sender against user profiles; fetch and filter attachments.
2. Classify subject+body via domain/receipt-classifier.ts (keyword matching; confident requires >= 2 keyword matches for one type, ties are unclassified).
3. Confident -> call the matching OCR endpoint, with the other as fallback on failure. Unconfident -> call BOTH /extract/receipt and /extract/maintenance-receipt per attachment, re-classify from returned rawText, then a field-count heuristic (email-ingestion.service.ts around lines 280-325).
4. Vehicle association: exact match creates the fuel_log or maintenance_record directly; ambiguous match inserts a pending association (insertPendingAssociation) for the user to resolve later.

2. Engine architecture

Two engine categories - do not conflate them:

1. OcrEngine subclasses (image -> text + word boxes): ocr/app/engines/paddle_engine.py (PaddleOCR, local), cloud_engine.py (Google Vision TEXT_DETECTION), hybrid_engine.py (primary + fallback wrapper). Built via engine_factory.create_engine(); registry accepts only paddleocr and google_vision. If OCR_FALLBACK_ENGINE != "none", factory returns a HybridEngine.
2. GeminiEngine (engines/gemini_engine.py): standalone, NOT an OcrEngine subclass. Semantic extraction (PDF -> maintenance schedule JSON) and VIN decode. Instantiated directly by manual_extractor.py and routers/decode.py, never through the factory. maintenance_receipt_extractor.py has its OWN copy-pasted Gemini client (see section 3).

Which engine actually runs - code defaults vs compose (compose wins in every deployed environment):

Setting	Code default (ocr/app/config.py)	docker-compose.yml (all envs inherit)
OCR_PRIMARY_ENGINE	paddleocr	google_vision
OCR_FALLBACK_ENGINE	none	paddleocr

Code defaults only apply where env vars are absent - i.e., bare pytest runs and ad-hoc local Python. Deployed containers run Vision-primary with PaddleOCR fallback. When reasoning about production behavior, always use the compose values.

HybridEngine semantics (hybrid_engine.py): with a CLOUD primary, the Vision monthly cap (VISION_MONTHLY_LIMIT, 1000/calendar month, Redis counter ocr:vision_requests:YYYY-MM in DB 1) is checked BEFORE calling it; once exhausted, PaddleOCR is the sole engine until month end. With a LOCAL primary, fallback triggers on confidence < OCR_FALLBACK_THRESHOLD. _CLOUD_TIMEOUT_SECONDS = 10.0 (raised from 5s because first-call WIF token exchange takes 6-8s, refs #182) is a discard-after-completion threshold, NOT an abort: _run_cloud_with_cap runs cloud.recognize() to completion with no timeout or cancellation, only then compares elapsed time, and throws away results slower than 10s (without incrementing the Vision counter). A hung or slow Vision call can therefore block far past 10s -- directly relevant to weak point 1 (sync calls block the event loop); do not rule out the Vision path in a hung-container investigation because of this constant.

3. Auth chain: WIF via Auth0 (memorize this before touching credentials)

WIF = Workload Identity Federation: Google Cloud accepts a third-party JWT (here, an Auth0 machine-to-machine token) instead of a service-account key file. No Google key ever exists on disk.

Chain, in order:

1. secrets/app/google-wif-config.json - deliberately committed to git; it is configuration, not a secret (it contains only the pool/provider audience and an executable pointer). Mounted at /run/secrets/google-wif-config.json; GOOGLE_VISION_KEY_PATH points at it. Do not "fix" its presence in git.
2. Its credential_source.executable.command is /app/scripts/fetch-auth0-token.sh (repo: ocr/scripts/fetch-auth0-token.sh, chmod +x in ocr/Dockerfile:48, 30s timeout). The script reads /run/secrets/auth0-ocr-client-id and /run/secrets/auth0-ocr-client-secret (real secrets, NOT in git; compose mounts them from secrets/app/*.txt), requests an Auth0 M2M token, and emits Google's executable-credential JSON format.
3. Google STS exchanges that JWT, then impersonates [email protected].
4. Client bootstrap sets, as process-wide env vars INSIDE _get_client() immediately before genai.Client(vertexai=True, ...) construction: GOOGLE_APPLICATION_CREDENTIALS=<key_path> and GOOGLE_EXTERNAL_ACCOUNT_ALLOW_EXECUTABLES=1 (executables are refused by google-auth without this flag).

The bootstrap is DUPLICATED. Identical _get_client() implementations exist in ocr/app/engines/gemini_engine.py:218 and ocr/app/extractors/maintenance_receipt_extractor.py:173 (and cloud_engine.py:67 sets the same env vars for Vision). Any auth or client-construction fix must land in BOTH _get_client() copies or you fix VIN decode/manual extraction while leaving maintenance receipts broken (or vice versa).

On a dev machine the auth0-ocr-client-* secrets typically exist only as Docker-created empty directories - the live Gemini path cannot run locally; end-to-end verification happens on staging.

4. google-genai SDK sharp edges

The dependency is unpinned: ocr/requirements.txt:25 has google-genai>=1.0.0 (floor only). Each image build resolves whatever is newest. This already caused a staging breakage:

• 936753f added automatic_function_calling=types.AutomaticFunctionCallingConfig(max_remote_calls=3) to the VIN decode GenerateContentConfig.
• The installed SDK version rejected it with a pydantic validation error at request time - VIN decode broke on staging while the build stayed green (CI runs zero tests; the only gate is images build + containers report healthy).
• 1add6c8 removed the parameter; 56df5d4 confirmed the revert and added diagnostic logging (full raw Gemini JSON at gemini_engine.py:401-406; hasTrim/hasEngine/hasTransmission in vehicles.controller.ts). Do not re-add AFC config without pinning and verifying the SDK version.

Rule: before shipping any new GenerateContentConfig field (or types.* construct), validate it against the INSTALLED SDK version, not the docs:

# What version is actually in the running container
docker compose exec mvp-ocr pip show google-genai | head -2

# Does the field exist / validate on that version (pydantic validates at construction)
docker compose exec mvp-ocr python3 -c "
from google.genai import types
print(types.GenerateContentConfig(response_mime_type='application/json', YOUR_NEW_FIELD=...))"

Model config drift (known-weak, deliberate as of 2026-07-07): code default is gemini-2.5-flash (ocr/app/config.py:37) but docker-compose.yml overrides GEMINI_MODEL: gemini-3-flash-preview - a PREVIEW model in deployed config. Preview models can be withdrawn or change behavior without notice. If Gemini responses suddenly degrade or 404, check the model name first. CLAUDE.md files under ocr/ still say "Gemini 2.5 Flash" - compose wins.

Response schemas use Vertex-style UPPERCASE types ("OBJECT", "STRING", "NUMBER", nullable flags) - keep that convention when editing _VIN_DECODE_SCHEMA / _RESPONSE_SCHEMA / _RECEIPT_RESPONSE_SCHEMA.

5. Config knob table (verified 2026-07-07)

Knob	Where	Default / deployed value
GEMINI_MODEL	ocr/app/config.py:37 / docker-compose.yml	gemini-2.5-flash / gemini-3-flash-preview
VERTEX_AI_PROJECT / VERTEX_AI_LOCATION	config.py:33-36 / compose	"" / global -> compose: motovaultpro / global
OCR_PRIMARY_ENGINE / OCR_FALLBACK_ENGINE	config.py:13,19 / compose	paddleocr/none -> compose: google_vision/paddleocr
OCR_CONFIDENCE_THRESHOLD / OCR_FALLBACK_THRESHOLD	config.py	0.6 / 0.6
VISION_MONTHLY_LIMIT	config.py:28	1000 requests/calendar month
GOOGLE_VISION_KEY_PATH	config.py:23	/run/secrets/google-wif-config.json (shared by Vision and Gemini)
Sync size cap	ocr.service.ts:20, extract.py:30	10MB (413 above)
Manual/async size cap	ocr.service.ts:23, extract.py:33	200MB
Gemini inline PDF cap	gemini_engine.py:20 _MAX_PDF_BYTES	20MB hard limit
Backend->OCR timeout	ocr-client.ts:9	120000ms, hardcoded, all endpoints
Frontend timeouts	vehicles.api.ts:90 (VIN decode), useVinOcr.ts:52, useManualExtraction.ts:54	120000ms
	useReceiptOcr.ts:153, useMaintenanceReceiptOcr.ts:147	30000ms
Hybrid cloud result-discard threshold (the call itself is unbounded)	hybrid_engine.py:25	10.0s
Job TTLs	job_queue.py:19-22	3600s regular / 7200s manual; expired jobs -> backend 410
OCR Redis	config.py:40-42	mvp-redis DB 1 (backend uses DB 0)
Gemini base field confidence	maintenance_receipt_extractor.py:31, manual_extractor.py:58	0.85 before regex cross-validation
Service-side Gemini timeout	(nowhere)	NONE - generate_content calls are unbounded inside the Python service

6. Known-weak points (owner-acknowledged, 2026-07-07; do not silently "improve" without an issue)

1. Sync Gemini calls block the uvicorn event loop. routers/decode.py:42 and routers/extract.py:239,327 run synchronous OCR/Gemini work directly in async def handlers. A slow Search-grounded VIN decode (observed >60s) blocks ALL requests including /health (compose healthcheck: interval 5s, timeout 5s, retries 3 -> container flagged unhealthy within ~15s). Verified: plain compose does not auto-restart unhealthy containers, but scripts/ci/health-check.sh and the deploy/rollback gates treat unhealthy as failure, so a deploy landing during a long decode can fail its gate or trigger rollback. Only manual extraction uses run_in_executor (extract.py:472).
2. 120s == 120s timeout race. Frontend VIN decode timeout equals the backend->OCR timeout exactly; no headroom, so at the boundary the user sees a generic axios timeout instead of the backend's mapped 504. Receipt hooks are worse in a different way: frontend gives up at 30s while backend + OCR + Gemini keep burning quota for up to 120s.
3. No retries anywhere for transient Gemini/Vision failures. The single exception is maintenance-receipt's silent fallback (next item).
4. Silent empty-fields fallback. In maintenance_receipt_extractor.py:129-133, any Gemini exception is caught, logged at WARNING, and gemini_fields = {} - which usually surfaces as 422 "No maintenance receipt fields could be extracted". A dead Gemini auth chain therefore looks like "bad receipt photo". Check OCR container logs for "Gemini extraction failed, falling back to OCR-only" before blaming image quality.
5. 20MB vs 200MB manual gap. /extract/manual accepts up to 200MB, queues the job, then GeminiEngine.extract_maintenance() rejects anything over 20MB with an error suggesting GCS URIs - which are not implemented. Every 21-200MB manual is a guaranteed deferred failure.
6. Email-ingestion cost amplification. Unconfident classification runs BOTH OCR endpoints per attachment (2x Vision against the 1000/month cap, plus a Gemini call for the maintenance path). A burst of ambiguous forwarded emails can drain the Vision quota.
7. Manual job progress race (manual_extractor.py:122-126): fire-and-forget progress updates from worker threads could overwrite COMPLETED status; mitigated only by the convention "never send 100% from the extractor". Preserve that convention.
8. VIN decode trusts LLM+Search for specs. Only the year is ground-truthed. Full raw Gemini JSON is logged per decode (gemini_engine.py:401-406) - diagnostic aid, but log volume/PII to keep in mind.

7. How to test

CI gates nothing beyond build + boot (canonical statement: mvp-validation-and-qa section 1). All tests below are local-only and are your responsibility to run.

Python unit tests (all Gemini/network calls mocked - safe without secrets):

cd ocr && pytest

Key files: tests/test_gemini_engine.py (client init, 20MB cap, extraction, error paths), test_resolve_vin_year.py (year-cycle disambiguation - regenerated in 936753f, do not weaken), test_vin_decode.py, test_receipt_extraction.py, test_maintenance_patterns.py, test_engine_abstraction.py, test_service_mapping.py. Remember bare pytest sees code defaults (paddleocr primary), not compose values.

Backend unit tests (OcrClient mocked):

cd backend && npm test -- --testPathPattern="features/ocr"

Files: backend/src/features/ocr/tests/unit/ocr-receipt.test.ts, ocr-manual.test.ts. Do NOT run the backend integration tests casually - they DROP TABLE CASCADE on the shared dev database (owner non-negotiable: fresh backup first).

Live pipeline: staging only. The WIF chain needs real auth0-ocr-client-id/secret files, which exist only as empty mount directories on dev machines. Push a PR, let the staging deploy run, then exercise the flow through the staging UI, or from the staging host:

docker compose exec mvp-backend curl -s -X POST http://mvp-ocr:8000/decode/vin \
  -H 'Content-Type: application/json' -d '{"vin":"<17-char VIN>"}'

Watch logs: docker logs mvp-ocr-staging --tail 100 on the staging host (dev/prod container name is mvp-ocr); look for "Gemini engine initialized", "VIN year resolved", "Gemini decoded VIN". Never hand-edit files on staging/prod servers - rsync --delete reverts them on the next deploy.

Provenance and maintenance

Authored 2026-07-07 by direct inspection of the repo (all line numbers and values verified against code, not docs). Volatile facts and how to re-verify each:

Fact	Re-verify with
Backend->OCR timeout 120000ms	grep -n OCR_TIMEOUT_MS backend/src/features/ocr/external/ocr-client.ts
Frontend timeouts (120s VIN decode, 30s receipts)	grep -rn "timeout:" frontend/src/features/vehicles/api/vehicles.api.ts frontend/src/features/*/hooks/use*Ocr*.ts frontend/src/features/documents/hooks/useManualExtraction.ts
Deployed model gemini-3-flash-preview, Vision-primary engines	grep -n "GEMINI_MODEL|OCR_PRIMARY_ENGINE|OCR_FALLBACK_ENGINE" docker-compose.yml
Code defaults (gemini-2.5-flash, paddleocr/none)	grep -n "getenv" ocr/app/config.py
google-genai unpinned	grep -n google-genai ocr/requirements.txt
Installed SDK version in a running container	docker compose exec mvp-ocr pip show google-genai
Size caps 10MB/200MB/20MB	grep -n "MAX_SYNC_SIZE|MAX_ASYNC_SIZE|MAX_MANUAL_SIZE|_MAX_PDF_BYTES" backend/src/features/ocr/domain/ocr.service.ts ocr/app/routers/extract.py ocr/app/engines/gemini_engine.py
Job TTLs 3600/7200, Redis DB 1	grep -n "JOB_TTL|MANUAL_JOB_TTL" ocr/app/services/job_queue.py; grep -n REDIS_DB docker-compose.yml
Duplicated _get_client() bootstrap	grep -n "_get_client|ALLOW_EXECUTABLES" ocr/app/engines/gemini_engine.py ocr/app/extractors/maintenance_receipt_extractor.py ocr/app/engines/cloud_engine.py
WIF config committed, executable helper path	git ls-files secrets/app/ | grep wif; grep -n executable secrets/app/google-wif-config.json
AFC incident commits	git log --oneline 936753f 1add6c8 56df5d4 -1 --no-walk 2>/dev/null || git show --stat 56df5d4
Tier gates on OCR routes	grep -n requireTier backend/src/features/ocr/api/ocr.routes.ts backend/src/features/vehicles/api/vehicles.routes.ts
CI still runs no tests	grep -rn "pytest|npm test|jest" .gitea/workflows/ (expect no test-execution hits)
Event-loop-blocking sync calls	grep -n "_gemini_engine.decode_vin|receipt_extractor.extract|maintenance_receipt_extractor.extract" ocr/app/routers/decode.py ocr/app/routers/extract.py