motovaultpro/docs/changes/vehicles-dropdown-v1/phase-01-infrastructure.md

Phase 1: Infrastructure Setup

Overview

This phase establishes the foundational infrastructure for the MVP Platform by adding three new Docker services to the main docker-compose.yml. This creates the shared services architecture pattern that future platform services will follow.

Prerequisites

• Docker and Docker Compose installed
• Main MotoVaultPro application running successfully
• Access to NHTSA vPIC database backup file (VPICList_lite_2025_07.bak)
• Understanding of existing docker-compose.yml structure

Tasks

Task 1.1: Add MVP Platform Database Service

Location: docker-compose.yml

Action: Add the following service definition to the services section:

  mvp-platform-database:
    image: postgres:15-alpine
    container_name: mvp-platform-db
    environment:
      POSTGRES_DB: mvp-platform-vehicles
      POSTGRES_USER: mvp_platform_user
      POSTGRES_PASSWORD: ${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
      POSTGRES_INITDB_ARGS: "--encoding=UTF8"
    volumes:
      - mvp_platform_data:/var/lib/postgresql/data
      - ./vehicle-etl/sql/schema:/docker-entrypoint-initdb.d
    ports:
      - "5433:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U mvp_platform_user -p 5432"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - default

Action: Add the volume definition to the volumes section:

volumes:
  postgres_data:
  redis_data:
  minio_data:
  mvp_platform_data:  # Add this line

Task 1.2: Add MSSQL Source Database Service

Location: docker-compose.yml

Action: Add the following service definition:

  mssql-source:
    image: mcr.microsoft.com/mssql/server:2019-latest
    container_name: mvp-mssql-source
    user: root
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=${MSSQL_SOURCE_PASSWORD:-Source123!}
      - MSSQL_PID=Developer
    ports:
      - "1433:1433"
    volumes:
      - mssql_source_data:/var/opt/mssql/data
      - ./vehicle-etl/volumes/mssql/backups:/backups
    healthcheck:
      test: ["CMD-SHELL", "/opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P ${MSSQL_SOURCE_PASSWORD:-Source123!} -Q 'SELECT 1'"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s
    networks:
      - default

Action: Add volume to volumes section:

volumes:
  postgres_data:
  redis_data:
  minio_data:
  mvp_platform_data:
  mssql_source_data:  # Add this line

Task 1.3: Add Scheduled ETL Service

Location: docker-compose.yml

Action: Add the following service definition:

  etl-scheduler:
    build:
      context: ./vehicle-etl
      dockerfile: docker/Dockerfile.etl
    container_name: mvp-etl-scheduler
    environment:
      # Database connections
      - MSSQL_HOST=mssql-source
      - MSSQL_PORT=1433
      - MSSQL_DATABASE=VPICList
      - MSSQL_USERNAME=sa
      - MSSQL_PASSWORD=${MSSQL_SOURCE_PASSWORD:-Source123!}
      - POSTGRES_HOST=mvp-platform-database
      - POSTGRES_PORT=5432
      - POSTGRES_DATABASE=mvp-platform-vehicles
      - POSTGRES_USERNAME=mvp_platform_user
      - POSTGRES_PASSWORD=${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      # ETL configuration
      - ETL_SCHEDULE=0 2 * * 0  # Weekly on Sunday at 2 AM
      - ETL_LOG_LEVEL=INFO
      - ETL_BATCH_SIZE=10000
      - ETL_MAX_RETRIES=3
    volumes:
      - ./vehicle-etl/logs:/app/logs
      - etl_scheduler_data:/app/data
    depends_on:
      mssql-source:
        condition: service_healthy
      mvp-platform-database:
        condition: service_healthy
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - default

Action: Add volume to volumes section:

volumes:
  postgres_data:
  redis_data:
  minio_data:
  mvp_platform_data:
  mssql_source_data:
  etl_scheduler_data:  # Add this line

Task 1.4: Update Backend Service Environment Variables

Location: docker-compose.yml

Action: Add MVP Platform database environment variables to the backend service:

  backend:
    # ... existing configuration ...
    environment:
      # ... existing environment variables ...
      # MVP Platform Database
      MVP_PLATFORM_DB_HOST: mvp-platform-database
      MVP_PLATFORM_DB_PORT: 5432
      MVP_PLATFORM_DB_NAME: mvp-platform-vehicles
      MVP_PLATFORM_DB_USER: mvp_platform_user
      MVP_PLATFORM_DB_PASSWORD: ${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
    depends_on:
      - postgres
      - redis
      - minio
      - mvp-platform-database  # Add this dependency

Task 1.5: Create Environment File Template

Location: .env.example

Action: Add the following environment variables:

# MVP Platform Database
MVP_PLATFORM_DB_PASSWORD=platform_secure_password

# ETL Source Database
MSSQL_SOURCE_PASSWORD=Source123!

# ETL Configuration
ETL_SCHEDULE=0 2 * * 0
ETL_LOG_LEVEL=INFO
ETL_BATCH_SIZE=10000
ETL_MAX_RETRIES=3

Task 1.6: Update .env File (if exists)

Location: .env

Action: If .env exists, add the above environment variables with appropriate values for your environment.

Validation Steps

Step 1: Verify Docker Compose Configuration

# Test docker-compose configuration
docker-compose config

# Should output valid YAML without errors

Step 2: Build and Start New Services

# Build the ETL scheduler container
docker-compose build etl-scheduler

# Start only the new services for testing
docker-compose up mvp-platform-database mssql-source -d

# Check service health
docker-compose ps

Step 3: Test Database Connections

# Test MVP Platform database connection
docker-compose exec mvp-platform-database psql -U mvp_platform_user -d mvp-platform-vehicles -c "SELECT version();"

# Test MSSQL source database connection
docker-compose exec mssql-source /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "Source123!" -Q "SELECT @@VERSION"

Step 4: Verify Logs Directory Creation

# Check that ETL logs directory is created
ls -la ./vehicle-etl/logs/

# Should exist and be writable

Error Handling

Common Issues and Solutions

Issue: PostgreSQL container fails to start Solution: Check port 5433 is not in use, verify password complexity requirements

Issue: MSSQL container fails health check Solution: Increase start_period, verify password meets MSSQL requirements, check available memory

Issue: ETL scheduler cannot connect to databases Solution: Verify network connectivity, check environment variable values, ensure databases are healthy

Rollback Procedure

1. Stop the new services:

   docker-compose stop mvp-platform-database mssql-source etl-scheduler
   

2. Remove the new containers:

   docker-compose rm mvp-platform-database mssql-source etl-scheduler
   

3. Remove the volume definitions from docker-compose.yml

4. Remove the service definitions from docker-compose.yml

5. Remove environment variables from backend service

Next Steps

After successful completion of Phase 1:

1. Proceed to Phase 2: Backend Migration
2. Ensure all services are running and healthy before starting backend changes
3. Take note of any performance impacts on the existing application

Dependencies for Next Phase

• MVP Platform database must be accessible and initialized
• Backend service must be able to connect to MVP Platform database
• Existing Redis service must be available for new caching patterns