motovaultpro/docs/changes/database-20251111/database-api-update.md

Vehicle Dropdown Database Migration - Master Overview

Status: READY FOR IMPLEMENTATION

This document coordinates the migration from normalized ID-based vehicle dropdowns to a denormalized string-based system using ETL-generated data (1.1M+ vehicle options).

---

Quick Context

What Changed

• Old System: Normalized database (vehicles.make, vehicles.model, etc.) with ID-based API responses
• New System: Denormalized vehicle_options table with string-based API responses
• Impact: Complete replacement of vehicle dropdown system (database + backend + frontend)

Why This Change

• Better Data: ETL pipeline provides 1.1M+ comprehensive vehicle configurations from 1980-2026
• Simpler Queries: Single denormalized table vs complex JOINs across 6+ tables
• Real Transmissions: 828 actual transmission types vs hardcoded ["Automatic", "Manual"]
• Performance: Composite indexes enable sub-50ms query times

---

Implementation Strategy

Parallel Work Units

This migration is broken into 8 independent work units that can be executed by separate agents in parallel (where dependencies allow).

Agent	Task	File	Dependencies	Can Start
Agent 1	Database Migration	database-migration.md	None	Immediately
Agent 2	Platform Repository	backend-platform-repository.md	Agent 1 done	After DB
Agent 3	Platform Service	backend-platform-service.md	Agent 2 done	After Repo
Agent 4	Vehicles API	backend-vehicles-api.md	Agent 3 done	After Service
Agent 5	Frontend API Client	frontend-api-client.md	Agent 4 API contract defined	After API contract
Agent 6	Frontend Forms	frontend-vehicle-form.md	Agent 5 done	After API client
Agent 7	Testing	testing-validation.md	Agents 1-6 done	After all
Agent 8	VIN Decode (if needed)	Manual investigation	Agent 1 done	After DB

Critical Path

Database (Agent 1)
  ↓
Platform Repository (Agent 2)
  ↓
Platform Service (Agent 3)
  ↓
Vehicles API (Agent 4)
  ↓
Frontend API Client (Agent 5)
  ↓
Frontend Forms (Agent 6)
  ↓
Testing (Agent 7)

Parallel opportunities:

• Agent 8 can investigate VIN decode while Agents 2-6 work
• Agent 5 can start after Agent 4 defines API contract (doesn't need full implementation)

---

Key Design Decisions (FINAL)

1. API Format: String-Based

Decision: Switch from {id: number, name: string}[] to string[]

Rationale: Aligns with denormalized database design, simpler code

Example:

// OLD
GET /api/vehicles/dropdown/makes?year=2024
Response: [{id: 1, name: "Ford"}, {id: 2, name: "Honda"}]

// NEW
GET /api/vehicles/dropdown/makes?year=2024
Response: ["Ford", "Honda"]

2. Migration Strategy: Complete Replacement

Decision: Replace vehicles.* schema entirely (not parallel schemas)

Rationale: Clean architecture, no data duplication, simpler maintenance

Impact: Must verify VIN decode still works (Agent 8)

3. NULL Handling: Show with 'N/A'

Decision: Include electric vehicles, display 'N/A' for missing engine/transmission

Rationale: Ensures all vehicles available, handles 1.1% of records with NULL engine_id

Example:

// Electric vehicle in dropdown
{
  engine: "N/A (Electric)",
  transmission: "Single-Speed Automatic"
}

4. Implementation Approach: Single Update

Decision: No feature flags, complete migration in one cycle

Rationale: Faster delivery, cleaner code, no dual-system complexity

Requirement: Thorough testing before deployment

---

Breaking Changes

API Changes

Endpoint Response Format:

// All dropdown endpoints now return string[]
GET /api/vehicles/dropdown/years          → number[]        (unchanged)
GET /api/vehicles/dropdown/makes          → string[]        (was {id, name}[])
GET /api/vehicles/dropdown/models         → string[]        (was {id, name}[])
GET /api/vehicles/dropdown/trims          → string[]        (was {id, name}[])
GET /api/vehicles/dropdown/engines        → string[]        (was {id, name}[])
GET /api/vehicles/dropdown/transmissions  → string[]        (was {id, name}[])

Query Parameters:

// Parameters now use string values, not IDs
GET /api/vehicles/dropdown/models?year=2024&make=Ford              (was make_id=1)
GET /api/vehicles/dropdown/trims?year=2024&make=Ford&model=F-150  (was make_id=1&model_id=42)

Database Schema Changes

Removed Tables:

vehicles.make
vehicles.model
vehicles.model_year
vehicles.trim
vehicles.engine
vehicles.trim_engine
vehicles.transmission
vehicles.trim_transmission

New Tables:

public.engines               -- 30,066 records
public.transmissions         -- 828 records
public.vehicle_options       -- 1,122,644 records

New Database Functions:

get_makes_for_year(year INT)
get_models_for_year_make(year INT, make VARCHAR)
get_trims_for_year_make_model(year INT, make VARCHAR, model VARCHAR)
get_options_for_vehicle(year INT, make VARCHAR, model VARCHAR, trim VARCHAR)

Frontend Type Changes

Old Type:

interface DropdownOption {
  id: number;
  name: string;
}

New Type:

type DropdownOption = string;
// Or simply use string[] directly

---

New Database Schema Details

Tables

engines

CREATE TABLE engines (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL  -- e.g., "V8 5.0L", "L4 2.0L Turbo"
);
-- 30,066 records

transmissions

CREATE TABLE transmissions (
  id SERIAL PRIMARY KEY,
  type VARCHAR(255) NOT NULL  -- e.g., "8-Speed Automatic", "6-Speed Manual"
);
-- 828 records

vehicle_options

CREATE TABLE vehicle_options (
  id SERIAL PRIMARY KEY,
  year INTEGER NOT NULL,
  make VARCHAR(100) NOT NULL,           -- "Ford", "Honda" (Title Case)
  model VARCHAR(100) NOT NULL,          -- "F-150", "Civic"
  trim VARCHAR(255),                    -- "XLT SuperCrew", "Sport Touring"
  engine_id INTEGER REFERENCES engines(id),
  transmission_id INTEGER REFERENCES transmissions(id)
);
-- 1,122,644 records
-- 1.1% have NULL engine_id (electric vehicles)

-- Composite indexes for cascade queries
CREATE INDEX idx_vehicle_year_make ON vehicle_options(year, make);
CREATE INDEX idx_vehicle_year_make_model ON vehicle_options(year, make, model);
CREATE INDEX idx_vehicle_year_make_model_trim ON vehicle_options(year, make, model, trim);

Database Functions

These functions optimize common queries:

-- Returns distinct makes for a given year
get_makes_for_year(year INT) RETURNS TABLE(make VARCHAR)

-- Returns distinct models for year/make
get_models_for_year_make(year INT, make VARCHAR) RETURNS TABLE(model VARCHAR)

-- Returns distinct trims for year/make/model
get_trims_for_year_make_model(year INT, make VARCHAR, model VARCHAR)
  RETURNS TABLE(trim_name VARCHAR)

-- Returns engine/transmission options for specific vehicle
get_options_for_vehicle(year INT, make VARCHAR, model VARCHAR, trim VARCHAR)
  RETURNS TABLE(engine_name VARCHAR, transmission_type VARCHAR, ...)

---

Data Flow Changes

Old Flow (ID-Based Cascade)

1. User selects Year (2024)
   → GET /dropdown/makes?year=2024
   → Returns: [{id: 1, name: "Ford"}, {id: 2, name: "Honda"}]

2. User selects Make (clicks "Ford", id=1)
   → GET /dropdown/models?year=2024&make_id=1
   → Returns: [{id: 42, name: "F-150"}, {id: 43, name: "Mustang"}]

3. User selects Model (clicks "F-150", id=42)
   → GET /dropdown/trims?year=2024&make_id=1&model_id=42
   → Returns: [{id: 301, name: "XLT"}, {id: 302, name: "Lariat"}]

4. Form stores: make="Ford", model="F-150", trim="XLT" (name strings)

New Flow (String-Based Cascade)

1. User selects Year (2024)
   → GET /dropdown/makes?year=2024
   → Returns: ["Ford", "Honda"]

2. User selects Make ("Ford")
   → GET /dropdown/models?year=2024&make=Ford
   → Returns: ["F-150", "Mustang"]

3. User selects Model ("F-150")
   → GET /dropdown/trims?year=2024&make=Ford&model=F-150
   → Returns: ["XLT", "Lariat"]

4. Form stores: make="Ford", model="F-150", trim="XLT" (same strings)

Benefit: Simpler frontend logic - no ID tracking, direct string values

---

File Locations Reference

Database Files

data/make-model-import/migrations/001_create_vehicle_database.sql
data/make-model-import/output/01_engines.sql
data/make-model-import/output/02_transmissions.sql
data/make-model-import/output/03_vehicle_options.sql
data/make-model-import/IMPLEMENTATION_SUMMARY.md (reference doc)

Backend Platform Feature

backend/src/features/platform/domain/vehicle-data.repository.ts
backend/src/features/platform/domain/vehicle-data.service.ts
backend/src/features/platform/types/index.ts
backend/src/features/platform/cache/platform-cache.service.ts (may need updates)

Backend Vehicles Feature

backend/src/features/vehicles/api/vehicles.controller.ts
backend/src/features/vehicles/api/vehicles.routes.ts (minimal changes)
backend/src/features/vehicles/domain/vehicles.service.ts
backend/src/features/vehicles/types/index.ts

Frontend

frontend/src/features/vehicles/api/vehicles.api.ts
frontend/src/features/vehicles/types/vehicles.types.ts
frontend/src/features/vehicles/components/VehicleForm.tsx

---

Success Criteria

Technical Requirements

• Database migration completes successfully (3 SQL files imported)
• All dropdown endpoints return string[] format
• Query parameters use strings (not IDs)
• NULL values display as 'N/A' or appropriate label
• Transmissions show real data (not hardcoded)
• Composite indexes perform sub-50ms queries
• All backend tests pass
• All frontend tests pass
• Mobile responsiveness verified

Functional Requirements

• Create vehicle form works end-to-end
• Edit vehicle form loads and displays correctly
• Cascading dropdowns work: Year → Make → Model → Trim → Engine/Trans
• Electric vehicles appear in results with 'N/A' engine display
• Saved vehicles store correct string values
• VIN decode feature still operational (if dependent on schema)

Quality Requirements (per CLAUDE.md)

• All linters pass with zero issues
• All automated checks green
• No formatting errors
• Code follows project conventions
• Old code deleted (not commented out)

---

Testing Checklist

Backend API Tests

• GET /dropdown/years returns number[]
• GET /dropdown/makes?year=2024 returns string[]
• GET /dropdown/models?year=2024&make=Ford returns string[]
• GET /dropdown/trims with all params returns string[]
• GET /dropdown/engines returns string[] with 'N/A' for electric
• GET /dropdown/transmissions returns real data (not hardcoded)
• Invalid parameters return 400 errors
• Database errors return 500 with appropriate message

Frontend Form Tests

• Create form: Year dropdown loads on mount
• Create form: Make dropdown loads after year selected
• Create form: Model dropdown loads after make selected
• Create form: Trim dropdown loads after model selected
• Create form: Engine dropdown loads after trim selected
• Create form: Submission saves correct string values
• Edit form: Loads existing vehicle data correctly
• Edit form: Pre-populates all dropdowns in correct order
• Edit form: Allows changing selections and cascades properly

Mobile Testing (REQUIRED per CLAUDE.md)

• Dropdowns render correctly on mobile viewport
• Touch interactions work smoothly
• Form is usable on small screens
• No horizontal scrolling issues

Integration Tests

• Complete flow: Select all fields → Save → Verify in DB
• Edit existing vehicle → Change year → Cascades reset correctly
• Electric vehicle selection → Engine shows 'N/A' → Saves correctly

---

Rollback Plan

If critical issues discovered:

1. Database Rollback

   # Drop new tables
   DROP TABLE vehicle_options CASCADE;
   DROP TABLE transmissions CASCADE;
   DROP TABLE engines CASCADE;
   
   # Restore vehicles.* schema from backup
   # (ensure backup created before migration)
   

2. Code Rollback

   git revert <commit-hash>  # Revert all changes
   make rebuild              # Rebuild containers
   

3. Partial Rollback (if only frontend broken)

   • Revert frontend changes only
   • Backend can stay with new system
   • Fix frontend issues and redeploy

---

Common Issues & Solutions

Issue: VIN Decode Broken

Symptom: VIN decode endpoint returns errors after migration

Solution:

• Check if VIN decode queries vehicles.* tables
• If yes, need to create separate VIN decode data solution
• May need to keep minimal vehicles.* for VIN decode only

Issue: Electric Vehicles Show No Data

Symptom: Electric vehicles missing from dropdowns

Solution:

• Verify NULL handling in repository queries
• Check that 'N/A' label generation works correctly
• Ensure frontend displays 'N/A' properly

Issue: Slow Query Performance

Symptom: Dropdown queries take > 100ms

Solution:

• Verify composite indexes created: \d vehicle_options
• Check query plans: EXPLAIN ANALYZE <query>
• Ensure using database functions (not raw queries)
• Consider adding Redis caching

Issue: Make Names in Wrong Case

Symptom: Makes show as "FORD" instead of "Ford"

Solution:

• Verify ETL conversion to Title Case worked
• Check database: SELECT DISTINCT make FROM vehicle_options LIMIT 10;
• If wrong, re-run ETL with case conversion fix

---

Agent Communication Protocol

Completion Signals

Each agent should:

1. Update their assigned documentation file with actual changes made
2. Post completion status in project communication channel
3. List any deviations from plan
4. Note any blockers for dependent agents

Handoff Information

When completing work, agents should document:

• What changed: Specific files and functions modified
• How to verify: Command or test to confirm work complete
• Breaking changes: Any API changes affecting downstream agents
• Blockers resolved: Issues fixed that were blocking others

Example Completion Message

Agent 2 (Platform Repository): COMPLETE

Files modified:
- backend/src/features/platform/domain/vehicle-data.repository.ts

Changes:
- getMakes() now returns string[] using get_makes_for_year()
- All 5 dropdown methods updated to query vehicle_options table
- NULL engine_id returns 'N/A (Electric)' label

Verification:
- Run: npm test -- vehicle-data.repository.test.ts
- All tests passing

API Contract for Agent 3:
- getMakes(pool, year) → Promise<string[]>
- getModels(pool, year, make) → Promise<string[]>
- getTrims(pool, year, make, model) → Promise<string[]>
- getEngines(pool, year, make, model, trim) → Promise<string[]>
- getTransmissions(pool, year, make, model) → Promise<string[]>

Agent 3 can now proceed with service layer updates.

---

Detailed Task Documents

Each task has a dedicated document with implementation details:

1. Database Migration - Agent 1
2. Platform Repository Updates - Agent 2
3. Platform Service Updates - Agent 3
4. Vehicles API Updates - Agent 4
5. Frontend API Client Updates - Agent 5
6. Frontend Form Updates - Agent 6
7. Testing & Validation - Agent 7

---

Questions or Issues

If you encounter issues:

1. Check this master doc for common issues
2. Review your task-specific document
3. Check database state: docker exec mvp-postgres psql -U postgres -d motovaultpro
4. Check logs: make logs
5. Consult original data docs: data/make-model-import/IMPLEMENTATION_SUMMARY.md

---

Timeline Estimate

• Agent 1 (Database): 30 minutes
• Agent 2 (Platform Repo): 1-2 hours
• Agent 3 (Platform Service): 1 hour
• Agent 4 (Vehicles API): 1-2 hours
• Agent 5 (Frontend API): 1 hour
• Agent 6 (Frontend Forms): 2-3 hours
• Agent 7 (Testing): 2-3 hours

Total Sequential: ~10-14 hours With Parallelization: ~6-8 hours (agents work simultaneously where possible)

---

Document Version: 1.0 Last Updated: 2025-11-10 Status: Ready for Implementation