300 lines
14 KiB
Markdown
300 lines
14 KiB
Markdown
# Documents Feature Plan (S3-Compatible, Phased)
|
||
|
||
This plan aligns with the current codebase: MinIO is running (`admin-minio`), object storage credentials are mounted as secrets, and `appConfig.getMinioConfig()` is available. We will implement a generic S3-compatible storage surface with a MinIO-backed adapter first, following the Docker‑first, production‑only workflow and mobile+desktop requirements.
|
||
|
||
— Read me quick —
|
||
- Storage: Start with MinIO SDK via `getMinioConfig()`. Keep the interface S3‑generic to support AWS S3 later without changing features.
|
||
- Auth/Tenant: All endpoints use `[fastify.authenticate, tenantMiddleware]`.
|
||
- Testing: Use Jest; run via containers with `make test`.
|
||
- Mobile+Desktop: Follow existing Zustand nav, React Router routes, GlassCard components, and React Query offlineFirst.
|
||
|
||
Handoff markers are provided at the end of each phase. If work pauses, pick up from the next “Done when” checklist.
|
||
|
||
## Phase 0 — Baseline Verification
|
||
|
||
Objectives
|
||
- Confirm configuration and dependencies to avoid rework.
|
||
|
||
Tasks
|
||
- Verify MinIO configuration in `config/app/production.yml` → `minio.endpoint`, `minio.port`, `minio.bucket`.
|
||
- Verify mounted secrets exist for MinIO (`secrets/app/minio-access-key.txt`, `secrets/app/minio-secret-key.txt`).
|
||
- Verify backend dependency presence:
|
||
- Present: `minio@^7.1.3`
|
||
- Missing: `@fastify/multipart` (add to `backend/package.json`)
|
||
- Rebuild and tail logs
|
||
- `make rebuild`
|
||
- `make logs`
|
||
|
||
Done when
|
||
- Containers start cleanly and backend logs show no missing module errors.
|
||
|
||
Status
|
||
- MinIO configuration verified in repo (endpoint/port/bucket present) ✓
|
||
- MinIO secrets present in repo (mounted paths defined) ✓
|
||
- Package check: `minio` present ✓, `@fastify/multipart` added to backend/package.json ✓
|
||
- Rebuild/logs runtime verification: pending (perform via `make rebuild && make logs`)
|
||
|
||
## Phase 1 — Storage Foundation (S3-Compatible, MinIO-Backed)
|
||
|
||
Objectives
|
||
- Create a generic storage façade used by features; implement first adapter using MinIO SDK.
|
||
|
||
Design
|
||
- Interface `StorageService` methods:
|
||
- `putObject(bucket, key, bodyOrStream, contentType, metadata?)`
|
||
- `getObjectStream(bucket, key)`
|
||
- `deleteObject(bucket, key)`
|
||
- `headObject(bucket, key)`
|
||
- `getSignedUrl(bucket, key, { method: 'GET'|'PUT', expiresSeconds })`
|
||
- Key scheme: `documents/{userId}/{vehicleId}/{documentId}/{version}/{uuid}.{ext}`
|
||
- Security: Private objects only; short‑lived signed URLs when needed.
|
||
|
||
Files
|
||
- `backend/src/core/storage/storage.service.ts` — façade and factory.
|
||
- `backend/src/core/storage/adapters/minio.adapter.ts` — uses MinIO SDK and `appConfig.getMinioConfig()`.
|
||
|
||
Tasks
|
||
- Implement MinIO client using endpoint/port/accessKey/secretKey/bucket from `appConfig.getMinioConfig()`.
|
||
- Ensure streaming APIs are used for uploads/downloads.
|
||
- Implement signed URL generation for downloads with short TTL (e.g., 60–300s).
|
||
|
||
Done when
|
||
- Service can put/head/get/delete and generate signed URLs against `admin-minio` bucket from inside the backend container.
|
||
|
||
Status
|
||
- Storage facade added: `backend/src/core/storage/storage.service.ts` ✓
|
||
- MinIO adapter implemented: `backend/src/core/storage/adapters/minio.adapter.ts` ✓
|
||
- Runtime validation against MinIO: pending (validate post-rebuild) ☐
|
||
|
||
## Phase 2 — Backend HTTP Foundation
|
||
|
||
Objectives
|
||
- Enable file uploads and wire security.
|
||
|
||
Tasks
|
||
- Add `@fastify/multipart` to `backend/package.json`.
|
||
- In `backend/src/app.ts`, register multipart with config‑based limits:
|
||
- `limits.fileSize` sourced from `appConfig.config.performance.max_request_size`.
|
||
- Confirm authentication plugin and tenant middleware are active (already implemented).
|
||
|
||
Done when
|
||
- Backend accepts multipart requests and enforces size limits without errors.
|
||
|
||
Status
|
||
- Dependency added: `@fastify/multipart` ✓
|
||
- Registered in `backend/src/app.ts` with byte-limit parser ✓
|
||
- Runtime verification via container: pending ☐
|
||
|
||
## Phase 3 — Documents Feature Capsule (Backend)
|
||
|
||
Objectives
|
||
- Create the feature capsule with schema, repository, service, routes, and validators, following existing patterns (see vehicles and fuel‑logs).
|
||
|
||
Structure (backend)
|
||
```
|
||
backend/src/features/documents/
|
||
├── README.md
|
||
├── index.ts
|
||
├── api/
|
||
│ ├── documents.routes.ts
|
||
│ ├── documents.controller.ts
|
||
│ └── documents.validation.ts
|
||
├── domain/
|
||
│ ├── documents.service.ts
|
||
│ └── documents.types.ts
|
||
├── data/
|
||
│ └── documents.repository.ts
|
||
├── migrations/
|
||
│ └── 001_create_documents_table.sql
|
||
└── tests/
|
||
├── unit/
|
||
└── integration/
|
||
```
|
||
|
||
Database schema
|
||
- Table `documents`:
|
||
- `id UUID PK`
|
||
- `user_id VARCHAR(255)`
|
||
- `vehicle_id UUID` FK → `vehicles(id)`
|
||
- `document_type VARCHAR(32)` CHECK IN ('insurance','registration')
|
||
- `title VARCHAR(200)`; `notes TEXT NULL`; `details JSONB`
|
||
- `storage_bucket VARCHAR(128)`; `storage_key VARCHAR(512)`
|
||
- `file_name VARCHAR(255)`; `content_type VARCHAR(128)`; `file_size BIGINT`; `file_hash VARCHAR(128) NULL`
|
||
- `issued_date DATE NULL`; `expiration_date DATE NULL`
|
||
- `created_at TIMESTAMP DEFAULT now()`; `updated_at TIMESTAMP DEFAULT now()` with `update_updated_at_column()` trigger
|
||
- `deleted_at TIMESTAMP NULL`
|
||
- Indexes: `(user_id)`, `(vehicle_id)`, `(user_id, vehicle_id)`, `(document_type)`, `(expiration_date)`; optional GIN on `details` if needed.
|
||
|
||
API endpoints
|
||
```
|
||
POST /api/documents # Create metadata (with/without file)
|
||
GET /api/documents # List (filters: vehicleId, type, expiresBefore)
|
||
GET /api/documents/:id # Get metadata
|
||
PUT /api/documents/:id # Update metadata/details
|
||
DELETE /api/documents/:id # Soft delete (and delete object)
|
||
GET /api/documents/vehicle/:vehicleId # List by vehicle
|
||
POST /api/documents/:id/upload # Upload/replace file (multipart)
|
||
GET /api/documents/:id/download # Download (proxy stream or signed URL)
|
||
```
|
||
- Pre‑handlers: `[fastify.authenticate, tenantMiddleware]` for all routes.
|
||
- Validation: Zod schemas for params/query/body in `documents.validation.ts`.
|
||
- Ownership: Validate `vehicle_id` belongs to `user_id` using vehicles pattern (like fuel‑logs).
|
||
|
||
Wire‑up
|
||
- Register in `backend/src/app.ts`:
|
||
- `import { documentsRoutes } from './features/documents/api/documents.routes'`
|
||
- `await app.register(documentsRoutes, { prefix: '/api' })`
|
||
- Health: Update `/health` feature list to include `documents`.
|
||
- Migrations: Add `'features/documents'` to `MIGRATION_ORDER` in `backend/src/_system/migrations/run-all.ts` after `'features/vehicles'`.
|
||
|
||
Done when
|
||
- CRUD + upload/download endpoints are reachable and secured; migrations run in correct order; ownership enforced.
|
||
|
||
Status
|
||
- Capsule scaffolded (api/domain/data/tests/migrations/README) ✓
|
||
- Migration added `backend/src/features/documents/migrations/001_create_documents_table.sql` ✓
|
||
- Registered routes in `backend/src/app.ts` with `/api` prefix ✓
|
||
- Health feature list updated to include `documents` ✓
|
||
- Migration order updated in `backend/src/_system/migrations/run-all.ts` ✓
|
||
- CRUD handlers for metadata implemented ✓
|
||
- Upload endpoint implemented with multipart streaming, MIME allowlist, and storage meta update ✓
|
||
- Download endpoint implemented with proxy streaming and inline/attachment disposition ✓
|
||
- Ownership validation on create via vehicles check ✓
|
||
- Runtime verification in container: pending ☐
|
||
|
||
## Phase 4 — Frontend Feature (Mobile + Desktop)
|
||
|
||
Objectives
|
||
- Implement documents UI following existing navigation, layout, and data patterns.
|
||
|
||
Structure (frontend)
|
||
```
|
||
frontend/src/features/documents/
|
||
├── pages/
|
||
├── components/
|
||
├── hooks/
|
||
└── types/
|
||
```
|
||
|
||
Navigation
|
||
- Mobile: Add “Documents” to bottom nav (Zustand store in `frontend/src/core/store/navigation.ts`).
|
||
- Desktop: Add routes in `frontend/src/App.tsx` for list/detail/upload.
|
||
- Sub‑screens (mobile): list → detail → upload; wrap content with `GlassCard`.
|
||
|
||
Upload UX
|
||
- Mobile camera/gallery: `<input type="file" accept="image/*" capture="environment" />`.
|
||
- Desktop drag‑and‑drop with progress.
|
||
- Progress tracking: React Query mutation with progress events; optimistic updates and cache invalidation.
|
||
- Offline: Use existing React Query `offlineFirst` config; queue uploads and retry on reconnect.
|
||
|
||
Viewer
|
||
- Inline image/PDF preview; `Content-Disposition` inline for images/PDF; gestures (pinch/zoom) for mobile images.
|
||
|
||
Done when
|
||
- Users can list, upload, view, and delete documents on both mobile and desktop with responsive UI and progress.
|
||
|
||
Status
|
||
- Add Documents to mobile bottom nav (Zustand): completed ✓
|
||
- Add desktop routes in `App.tsx` (list/detail/upload): completed ✓
|
||
- Scaffold pages/components/hooks structure: completed ✓
|
||
- Hook list/detail CRUD endpoints with React Query: completed ✓
|
||
- Implement upload with progress UI: completed ✓ (hooks with onUploadProgress; UI in mobile/detail)
|
||
- Optimistic updates: partial (invalidate queries on success) ◐
|
||
- Offline queuing/retry via React Query networkMode: configured via hooks ✓
|
||
- Previews: basic image/PDF preview implemented ✓ (DocumentPreview)
|
||
- Gesture-friendly viewer: pending ☐
|
||
- Desktop navigation: sidebar now defaults open and includes Documents ✓
|
||
- Build hygiene: resolved TS unused import error in frontend documents hooks ✓
|
||
|
||
## Phase 5 — Security, Validation, and Policies
|
||
|
||
Objectives
|
||
- Enforce safe file handling and consistent deletion semantics.
|
||
|
||
Tasks
|
||
- MIME allowlist: `application/pdf`, `image/jpeg`, `image/png`; reject executables.
|
||
- Upload size: Enforce via multipart limit tied to `performance.max_request_size`.
|
||
- Deletion: Soft delete DB first; delete object after. Consider retention policy later if required.
|
||
- Logging: Create/update/delete/upload/download events include `user_id`, `document_id`, `vehicle_id` (use existing logger).
|
||
- Optional rate limiting for upload route (defer dependency until needed).
|
||
|
||
Done when
|
||
- Unsafe files rejected; logs record document events; deletions are consistent.
|
||
|
||
Status
|
||
- MIME allowlist enforced for uploads (PDF, JPEG, PNG) ✓
|
||
- Upload size enforced via multipart limit (config-driven) ✓
|
||
- Deletion semantics: DB soft-delete and best-effort storage object deletion ✓
|
||
- Event logging for document actions: pending ☐
|
||
|
||
## Phase 6 — Testing (Docker-First)
|
||
|
||
Objectives
|
||
- Achieve green tests and linting across backend and frontend.
|
||
|
||
Backend tests
|
||
- Unit: repository/service/storage adapter (mock MinIO), validators.
|
||
- Integration: API with test DB + MinIO container, stream upload/download, auth/tenant checks.
|
||
|
||
Frontend tests
|
||
- Unit: components/forms, upload interactions, previews.
|
||
- Integration: hooks with mocked API; navigation flows for list/detail/upload.
|
||
|
||
Commands
|
||
- `make test` (backend + frontend)
|
||
- `make shell-backend` then `npm test -- features/documents`
|
||
- `make test-frontend`
|
||
|
||
Done when
|
||
- All tests/linters pass with zero issues; upload/download E2E verified in containers.
|
||
|
||
Status
|
||
- Backend unit tests (service/repo/storage; validators): pending ☐
|
||
- Backend integration tests (upload/download/auth/tenant): pending ☐
|
||
- Frontend unit tests (components/forms/uploads/previews): pending ☐
|
||
- Frontend integration tests (hooks + navigation flows): pending ☐
|
||
- CI via `make test` and linters green: pending ☐
|
||
|
||
## Phase 7 — Reality Checkpoints and Handoff
|
||
|
||
Checkpoints
|
||
- After each phase: `make rebuild && make logs`.
|
||
- Before moving on: Verify auth + tenant pre‑handlers, ownership checks, and mobile responsiveness.
|
||
- When interrupted: Commit current status and annotate the “Current Handoff Status” section below.
|
||
|
||
Handoff fields (update as you go)
|
||
- Storage façade: [x] implemented [ ] validated against MinIO
|
||
- Multipart plugin: [x] registered [x] enforcing limits
|
||
- Documents migrations: [x] added [ ] executed [ ] indexes verified
|
||
- Repo/service/routes: [x] implemented [x] ownership checks
|
||
- Frontend routes/nav: [x] added [x] mobile [x] desktop
|
||
- Upload/download flows: backend [x] implemented UI [x] progress [x] preview [ ] signed URLs (optional)
|
||
- Tests: [ ] unit backend [ ] int backend [ ] unit frontend [ ] int frontend
|
||
|
||
Diagnostics Notes
|
||
- Added `/api/health` endpoint in backend to validate Traefik routing to admin-backend for API paths.
|
||
- Fixed Fastify schema boot error by removing Zod schemas from documents routes (align with existing patterns). This prevented route registration and caused 404 on `/api/*` while server crashed/restarted.
|
||
|
||
## S3 Compatibility Notes
|
||
|
||
- The interface is provider‑agnostic. MinIO adapter speaks S3‑compatible API using custom endpoint and credentials from `getMinioConfig()`.
|
||
- Adding AWS S3 later: Implement `backend/src/core/storage/adapters/s3.adapter.ts` using `@aws-sdk/client-s3`, wire via a simple provider flag (e.g., `storage.provider: 'minio' | 's3'`). No feature code changes expected.
|
||
- Security parity: Keep private objects by default; consider server‑side encryption when adding AWS S3.
|
||
|
||
## Reference Pointers
|
||
|
||
- MinIO config: `backend/src/core/config/config-loader.ts` (`getMinioConfig()`) and `config/app/production.yml`.
|
||
- Auth plugin: `backend/src/core/plugins/auth.plugin.ts`.
|
||
- Tenant middleware: `backend/src/core/middleware/tenant.ts`.
|
||
- Migration runner: `backend/src/_system/migrations/run-all.ts` (edit `MIGRATION_ORDER`).
|
||
- Feature registration: `backend/src/app.ts` (register `documentsRoutes` and update `/health`).
|
||
- Frontend nav and layout: `frontend/src/App.tsx`, `frontend/src/core/store/navigation.ts`, `frontend/src/shared-minimal/components/mobile/GlassCard`.
|
||
|
||
## Success Criteria
|
||
|
||
- Documents CRUD with upload/download works on mobile and desktop.
|
||
- Ownership and tenant enforcement on every request; private object storage; safe file types.
|
||
- S3‑compatible storage layer with MinIO adapter; S3 adapter can be added without feature changes.
|
||
- All tests and linters green; migrations idempotent and ordered after vehicles.
|
||
- Build hygiene: backend TS errors fixed (unused import, override modifier, union narrowing) ✓
|