motovaultpro/docs/changes/database-20251111/database-api-update.md

# Vehicle Dropdown Database Migration - Master Overview

## Status: READY FOR IMPLEMENTATION

This document coordinates the migration from normalized ID-based vehicle dropdowns to a denormalized string-based system using ETL-generated data (1.1M+ vehicle options).

---

## Quick Context

### What Changed
- **Old System**: Normalized database (vehicles.make, vehicles.model, etc.) with ID-based API responses
- **New System**: Denormalized vehicle_options table with string-based API responses
- **Impact**: Complete replacement of vehicle dropdown system (database + backend + frontend)

### Why This Change
- **Better Data**: ETL pipeline provides 1.1M+ comprehensive vehicle configurations from 1980-2026
- **Simpler Queries**: Single denormalized table vs complex JOINs across 6+ tables
- **Real Transmissions**: 828 actual transmission types vs hardcoded ["Automatic", "Manual"]
- **Performance**: Composite indexes enable sub-50ms query times

---

## Implementation Strategy

### Parallel Work Units

This migration is broken into **8 independent work units** that can be executed by separate agents in parallel (where dependencies allow).

| Agent | Task | File | Dependencies | Can Start |
|-------|------|------|--------------|-----------|
| Agent 1 | Database Migration | `database-migration.md` | None | Immediately |
| Agent 2 | Platform Repository | `backend-platform-repository.md` | Agent 1 done | After DB |
| Agent 3 | Platform Service | `backend-platform-service.md` | Agent 2 done | After Repo |
| Agent 4 | Vehicles API | `backend-vehicles-api.md` | Agent 3 done | After Service |
| Agent 5 | Frontend API Client | `frontend-api-client.md` | Agent 4 API contract defined | After API contract |
| Agent 6 | Frontend Forms | `frontend-vehicle-form.md` | Agent 5 done | After API client |
| Agent 7 | Testing | `testing-validation.md` | Agents 1-6 done | After all |
| Agent 8 | VIN Decode (if needed) | Manual investigation | Agent 1 done | After DB |

### Critical Path
```
Database (Agent 1)
  ↓
Platform Repository (Agent 2)
  ↓
Platform Service (Agent 3)
  ↓
Vehicles API (Agent 4)
  ↓
Frontend API Client (Agent 5)
  ↓
Frontend Forms (Agent 6)
  ↓
Testing (Agent 7)
```

**Parallel opportunities:**
- Agent 8 can investigate VIN decode while Agents 2-6 work
- Agent 5 can start after Agent 4 defines API contract (doesn't need full implementation)

---

## Key Design Decisions (FINAL)

### 1. API Format: String-Based
**Decision**: Switch from `{id: number, name: string}[]` to `string[]`

**Rationale**: Aligns with denormalized database design, simpler code

**Example:**
```typescript
// OLD
GET /api/vehicles/dropdown/makes?year=2024
Response: [{id: 1, name: "Ford"}, {id: 2, name: "Honda"}]

// NEW
GET /api/vehicles/dropdown/makes?year=2024
Response: ["Ford", "Honda"]
```

### 2. Migration Strategy: Complete Replacement
**Decision**: Replace vehicles.* schema entirely (not parallel schemas)

**Rationale**: Clean architecture, no data duplication, simpler maintenance

**Impact**: Must verify VIN decode still works (Agent 8)

### 3. NULL Handling: Show with 'N/A'
**Decision**: Include electric vehicles, display 'N/A' for missing engine/transmission

**Rationale**: Ensures all vehicles available, handles 1.1% of records with NULL engine_id

**Example:**
```typescript
// Electric vehicle in dropdown
{
  engine: "N/A (Electric)",
  transmission: "Single-Speed Automatic"
}
```

### 4. Implementation Approach: Single Update
**Decision**: No feature flags, complete migration in one cycle

**Rationale**: Faster delivery, cleaner code, no dual-system complexity

**Requirement**: Thorough testing before deployment

---

## Breaking Changes

### API Changes

**Endpoint Response Format:**
```typescript
// All dropdown endpoints now return string[]
GET /api/vehicles/dropdown/years          → number[]        (unchanged)
GET /api/vehicles/dropdown/makes          → string[]        (was {id, name}[])
GET /api/vehicles/dropdown/models         → string[]        (was {id, name}[])
GET /api/vehicles/dropdown/trims          → string[]        (was {id, name}[])
GET /api/vehicles/dropdown/engines        → string[]        (was {id, name}[])
GET /api/vehicles/dropdown/transmissions  → string[]        (was {id, name}[])
```

**Query Parameters:**
```typescript
// Parameters now use string values, not IDs
GET /api/vehicles/dropdown/models?year=2024&make=Ford              (was make_id=1)
GET /api/vehicles/dropdown/trims?year=2024&make=Ford&model=F-150  (was make_id=1&model_id=42)
```

### Database Schema Changes

**Removed Tables:**
```sql
vehicles.make
vehicles.model
vehicles.model_year
vehicles.trim
vehicles.engine
vehicles.trim_engine
vehicles.transmission
vehicles.trim_transmission
```

**New Tables:**
```sql
public.engines               -- 30,066 records
public.transmissions         -- 828 records
public.vehicle_options       -- 1,122,644 records
```

**New Database Functions:**
```sql
get_makes_for_year(year INT)
get_models_for_year_make(year INT, make VARCHAR)
get_trims_for_year_make_model(year INT, make VARCHAR, model VARCHAR)
get_options_for_vehicle(year INT, make VARCHAR, model VARCHAR, trim VARCHAR)
```

### Frontend Type Changes

**Old Type:**
```typescript
interface DropdownOption {
  id: number;
  name: string;
}
```

**New Type:**
```typescript
type DropdownOption = string;
// Or simply use string[] directly
```

---

## New Database Schema Details

### Tables

**engines**
```sql
CREATE TABLE engines (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL  -- e.g., "V8 5.0L", "L4 2.0L Turbo"
);
-- 30,066 records
```

**transmissions**
```sql
CREATE TABLE transmissions (
  id SERIAL PRIMARY KEY,
  type VARCHAR(255) NOT NULL  -- e.g., "8-Speed Automatic", "6-Speed Manual"
);
-- 828 records
```

**vehicle_options**
```sql
CREATE TABLE vehicle_options (
  id SERIAL PRIMARY KEY,
  year INTEGER NOT NULL,
  make VARCHAR(100) NOT NULL,           -- "Ford", "Honda" (Title Case)
  model VARCHAR(100) NOT NULL,          -- "F-150", "Civic"
  trim VARCHAR(255),                    -- "XLT SuperCrew", "Sport Touring"
  engine_id INTEGER REFERENCES engines(id),
  transmission_id INTEGER REFERENCES transmissions(id)
);
-- 1,122,644 records
-- 1.1% have NULL engine_id (electric vehicles)

-- Composite indexes for cascade queries
CREATE INDEX idx_vehicle_year_make ON vehicle_options(year, make);
CREATE INDEX idx_vehicle_year_make_model ON vehicle_options(year, make, model);
CREATE INDEX idx_vehicle_year_make_model_trim ON vehicle_options(year, make, model, trim);
```

### Database Functions

These functions optimize common queries:

```sql
-- Returns distinct makes for a given year
get_makes_for_year(year INT) RETURNS TABLE(make VARCHAR)

-- Returns distinct models for year/make
get_models_for_year_make(year INT, make VARCHAR) RETURNS TABLE(model VARCHAR)

-- Returns distinct trims for year/make/model
get_trims_for_year_make_model(year INT, make VARCHAR, model VARCHAR)
  RETURNS TABLE(trim_name VARCHAR)

-- Returns engine/transmission options for specific vehicle
get_options_for_vehicle(year INT, make VARCHAR, model VARCHAR, trim VARCHAR)
  RETURNS TABLE(engine_name VARCHAR, transmission_type VARCHAR, ...)
```

---

## Data Flow Changes

### Old Flow (ID-Based Cascade)
```
1. User selects Year (2024)
   → GET /dropdown/makes?year=2024
   → Returns: [{id: 1, name: "Ford"}, {id: 2, name: "Honda"}]

2. User selects Make (clicks "Ford", id=1)
   → GET /dropdown/models?year=2024&make_id=1
   → Returns: [{id: 42, name: "F-150"}, {id: 43, name: "Mustang"}]

3. User selects Model (clicks "F-150", id=42)
   → GET /dropdown/trims?year=2024&make_id=1&model_id=42
   → Returns: [{id: 301, name: "XLT"}, {id: 302, name: "Lariat"}]

4. Form stores: make="Ford", model="F-150", trim="XLT" (name strings)
```

### New Flow (String-Based Cascade)
```
1. User selects Year (2024)
   → GET /dropdown/makes?year=2024
   → Returns: ["Ford", "Honda"]

2. User selects Make ("Ford")
   → GET /dropdown/models?year=2024&make=Ford
   → Returns: ["F-150", "Mustang"]

3. User selects Model ("F-150")
   → GET /dropdown/trims?year=2024&make=Ford&model=F-150
   → Returns: ["XLT", "Lariat"]

4. Form stores: make="Ford", model="F-150", trim="XLT" (same strings)
```

**Benefit**: Simpler frontend logic - no ID tracking, direct string values

---

## File Locations Reference

### Database Files
```
data/make-model-import/migrations/001_create_vehicle_database.sql
data/make-model-import/output/01_engines.sql
data/make-model-import/output/02_transmissions.sql
data/make-model-import/output/03_vehicle_options.sql
data/make-model-import/IMPLEMENTATION_SUMMARY.md (reference doc)
```

### Backend Platform Feature
```
backend/src/features/platform/domain/vehicle-data.repository.ts
backend/src/features/platform/domain/vehicle-data.service.ts
backend/src/features/platform/types/index.ts
backend/src/features/platform/cache/platform-cache.service.ts (may need updates)
```

### Backend Vehicles Feature
```
backend/src/features/vehicles/api/vehicles.controller.ts
backend/src/features/vehicles/api/vehicles.routes.ts (minimal changes)
backend/src/features/vehicles/domain/vehicles.service.ts
backend/src/features/vehicles/types/index.ts
```

### Frontend
```
frontend/src/features/vehicles/api/vehicles.api.ts
frontend/src/features/vehicles/types/vehicles.types.ts
frontend/src/features/vehicles/components/VehicleForm.tsx
```

---

## Success Criteria

### Technical Requirements
- [ ] Database migration completes successfully (3 SQL files imported)
- [ ] All dropdown endpoints return string[] format
- [ ] Query parameters use strings (not IDs)
- [ ] NULL values display as 'N/A' or appropriate label
- [ ] Transmissions show real data (not hardcoded)
- [ ] Composite indexes perform sub-50ms queries
- [ ] All backend tests pass
- [ ] All frontend tests pass
- [ ] Mobile responsiveness verified

### Functional Requirements
- [ ] Create vehicle form works end-to-end
- [ ] Edit vehicle form loads and displays correctly
- [ ] Cascading dropdowns work: Year → Make → Model → Trim → Engine/Trans
- [ ] Electric vehicles appear in results with 'N/A' engine display
- [ ] Saved vehicles store correct string values
- [ ] VIN decode feature still operational (if dependent on schema)

### Quality Requirements (per CLAUDE.md)
- [ ] All linters pass with zero issues
- [ ] All automated checks green
- [ ] No formatting errors
- [ ] Code follows project conventions
- [ ] Old code deleted (not commented out)

---

## Testing Checklist

### Backend API Tests
- [ ] GET /dropdown/years returns number[]
- [ ] GET /dropdown/makes?year=2024 returns string[]
- [ ] GET /dropdown/models?year=2024&make=Ford returns string[]
- [ ] GET /dropdown/trims with all params returns string[]
- [ ] GET /dropdown/engines returns string[] with 'N/A' for electric
- [ ] GET /dropdown/transmissions returns real data (not hardcoded)
- [ ] Invalid parameters return 400 errors
- [ ] Database errors return 500 with appropriate message

### Frontend Form Tests
- [ ] Create form: Year dropdown loads on mount
- [ ] Create form: Make dropdown loads after year selected
- [ ] Create form: Model dropdown loads after make selected
- [ ] Create form: Trim dropdown loads after model selected
- [ ] Create form: Engine dropdown loads after trim selected
- [ ] Create form: Submission saves correct string values
- [ ] Edit form: Loads existing vehicle data correctly
- [ ] Edit form: Pre-populates all dropdowns in correct order
- [ ] Edit form: Allows changing selections and cascades properly

### Mobile Testing (REQUIRED per CLAUDE.md)
- [ ] Dropdowns render correctly on mobile viewport
- [ ] Touch interactions work smoothly
- [ ] Form is usable on small screens
- [ ] No horizontal scrolling issues

### Integration Tests
- [ ] Complete flow: Select all fields → Save → Verify in DB
- [ ] Edit existing vehicle → Change year → Cascades reset correctly
- [ ] Electric vehicle selection → Engine shows 'N/A' → Saves correctly

---

## Rollback Plan

If critical issues discovered:

1. **Database Rollback**
   ```bash
   # Drop new tables
   DROP TABLE vehicle_options CASCADE;
   DROP TABLE transmissions CASCADE;
   DROP TABLE engines CASCADE;

   # Restore vehicles.* schema from backup
   # (ensure backup created before migration)
   ```

2. **Code Rollback**
   ```bash
   git revert <commit-hash>  # Revert all changes
   make rebuild              # Rebuild containers
   ```

3. **Partial Rollback (if only frontend broken)**
   - Revert frontend changes only
   - Backend can stay with new system
   - Fix frontend issues and redeploy

---

## Common Issues & Solutions

### Issue: VIN Decode Broken
**Symptom**: VIN decode endpoint returns errors after migration

**Solution**:
- Check if VIN decode queries vehicles.* tables
- If yes, need to create separate VIN decode data solution
- May need to keep minimal vehicles.* for VIN decode only

### Issue: Electric Vehicles Show No Data
**Symptom**: Electric vehicles missing from dropdowns

**Solution**:
- Verify NULL handling in repository queries
- Check that 'N/A' label generation works correctly
- Ensure frontend displays 'N/A' properly

### Issue: Slow Query Performance
**Symptom**: Dropdown queries take > 100ms

**Solution**:
- Verify composite indexes created: `\d vehicle_options`
- Check query plans: `EXPLAIN ANALYZE <query>`
- Ensure using database functions (not raw queries)
- Consider adding Redis caching

### Issue: Make Names in Wrong Case
**Symptom**: Makes show as "FORD" instead of "Ford"

**Solution**:
- Verify ETL conversion to Title Case worked
- Check database: `SELECT DISTINCT make FROM vehicle_options LIMIT 10;`
- If wrong, re-run ETL with case conversion fix

---

## Agent Communication Protocol

### Completion Signals

Each agent should:
1. Update their assigned documentation file with actual changes made
2. Post completion status in project communication channel
3. List any deviations from plan
4. Note any blockers for dependent agents

### Handoff Information

When completing work, agents should document:
- **What changed**: Specific files and functions modified
- **How to verify**: Command or test to confirm work complete
- **Breaking changes**: Any API changes affecting downstream agents
- **Blockers resolved**: Issues fixed that were blocking others

### Example Completion Message
```
Agent 2 (Platform Repository): COMPLETE

Files modified:
- backend/src/features/platform/domain/vehicle-data.repository.ts

Changes:
- getMakes() now returns string[] using get_makes_for_year()
- All 5 dropdown methods updated to query vehicle_options table
- NULL engine_id returns 'N/A (Electric)' label

Verification:
- Run: npm test -- vehicle-data.repository.test.ts
- All tests passing

API Contract for Agent 3:
- getMakes(pool, year) → Promise<string[]>
- getModels(pool, year, make) → Promise<string[]>
- getTrims(pool, year, make, model) → Promise<string[]>
- getEngines(pool, year, make, model, trim) → Promise<string[]>
- getTransmissions(pool, year, make, model) → Promise<string[]>

Agent 3 can now proceed with service layer updates.
```

---

## Detailed Task Documents

Each task has a dedicated document with implementation details:

1. **[Database Migration](./database-migration.md)** - Agent 1
2. **[Platform Repository Updates](./backend-platform-repository.md)** - Agent 2
3. **[Platform Service Updates](./backend-platform-service.md)** - Agent 3
4. **[Vehicles API Updates](./backend-vehicles-api.md)** - Agent 4
5. **[Frontend API Client Updates](./frontend-api-client.md)** - Agent 5
6. **[Frontend Form Updates](./frontend-vehicle-form.md)** - Agent 6
7. **[Testing & Validation](./testing-validation.md)** - Agent 7

---

## Questions or Issues

If you encounter issues:
1. Check this master doc for common issues
2. Review your task-specific document
3. Check database state: `docker exec mvp-postgres psql -U postgres -d motovaultpro`
4. Check logs: `make logs`
5. Consult original data docs: `data/make-model-import/IMPLEMENTATION_SUMMARY.md`

---

## Timeline Estimate

- **Agent 1** (Database): 30 minutes
- **Agent 2** (Platform Repo): 1-2 hours
- **Agent 3** (Platform Service): 1 hour
- **Agent 4** (Vehicles API): 1-2 hours
- **Agent 5** (Frontend API): 1 hour
- **Agent 6** (Frontend Forms): 2-3 hours
- **Agent 7** (Testing): 2-3 hours

**Total Sequential**: ~10-14 hours
**With Parallelization**: ~6-8 hours (agents work simultaneously where possible)

---

**Document Version**: 1.0
**Last Updated**: 2025-11-10
**Status**: Ready for Implementation