motovaultpro/docs/changes/vehicles-dropdown-v1/phase-01-infrastructure.md

# Phase 1: Infrastructure Setup

## Overview

This phase establishes the foundational infrastructure for the MVP Platform by adding three new Docker services to the main `docker-compose.yml`. This creates the shared services architecture pattern that future platform services will follow.

## Prerequisites

- Docker and Docker Compose installed
- Main MotoVaultPro application running successfully
- Access to NHTSA vPIC database backup file (VPICList_lite_2025_07.bak)
- Understanding of existing docker-compose.yml structure

## Tasks

### Task 1.1: Add MVP Platform Database Service

**Location**: `docker-compose.yml`

**Action**: Add the following service definition to the services section:

```yaml
  mvp-platform-database:
    image: postgres:15-alpine
    container_name: mvp-platform-db
    environment:
      POSTGRES_DB: mvp-platform-vehicles
      POSTGRES_USER: mvp_platform_user
      POSTGRES_PASSWORD: ${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
      POSTGRES_INITDB_ARGS: "--encoding=UTF8"
    volumes:
      - mvp_platform_data:/var/lib/postgresql/data
      - ./vehicle-etl/sql/schema:/docker-entrypoint-initdb.d
    ports:
      - "5433:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U mvp_platform_user -p 5432"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - default
```

**Action**: Add the volume definition to the volumes section:

```yaml
volumes:
  postgres_data:
  redis_data:
  minio_data:
  mvp_platform_data:  # Add this line
```

### Task 1.2: Add MSSQL Source Database Service

**Location**: `docker-compose.yml`

**Action**: Add the following service definition:

```yaml
  mssql-source:
    image: mcr.microsoft.com/mssql/server:2019-latest
    container_name: mvp-mssql-source
    user: root
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=${MSSQL_SOURCE_PASSWORD:-Source123!}
      - MSSQL_PID=Developer
    ports:
      - "1433:1433"
    volumes:
      - mssql_source_data:/var/opt/mssql/data
      - ./vehicle-etl/volumes/mssql/backups:/backups
    healthcheck:
      test: ["CMD-SHELL", "/opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P ${MSSQL_SOURCE_PASSWORD:-Source123!} -Q 'SELECT 1'"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s
    networks:
      - default
```

**Action**: Add volume to volumes section:

```yaml
volumes:
  postgres_data:
  redis_data:
  minio_data:
  mvp_platform_data:
  mssql_source_data:  # Add this line
```

### Task 1.3: Add Scheduled ETL Service

**Location**: `docker-compose.yml`

**Action**: Add the following service definition:

```yaml
  etl-scheduler:
    build:
      context: ./vehicle-etl
      dockerfile: docker/Dockerfile.etl
    container_name: mvp-etl-scheduler
    environment:
      # Database connections
      - MSSQL_HOST=mssql-source
      - MSSQL_PORT=1433
      - MSSQL_DATABASE=VPICList
      - MSSQL_USERNAME=sa
      - MSSQL_PASSWORD=${MSSQL_SOURCE_PASSWORD:-Source123!}
      - POSTGRES_HOST=mvp-platform-database
      - POSTGRES_PORT=5432
      - POSTGRES_DATABASE=mvp-platform-vehicles
      - POSTGRES_USERNAME=mvp_platform_user
      - POSTGRES_PASSWORD=${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      # ETL configuration
      - ETL_SCHEDULE=0 2 * * 0  # Weekly on Sunday at 2 AM
      - ETL_LOG_LEVEL=INFO
      - ETL_BATCH_SIZE=10000
      - ETL_MAX_RETRIES=3
    volumes:
      - ./vehicle-etl/logs:/app/logs
      - etl_scheduler_data:/app/data
    depends_on:
      mssql-source:
        condition: service_healthy
      mvp-platform-database:
        condition: service_healthy
      redis:
        condition: service_healthy
    restart: unless-stopped
    networks:
      - default
```

**Action**: Add volume to volumes section:

```yaml
volumes:
  postgres_data:
  redis_data:
  minio_data:
  mvp_platform_data:
  mssql_source_data:
  etl_scheduler_data:  # Add this line
```

### Task 1.4: Update Backend Service Environment Variables

**Location**: `docker-compose.yml`

**Action**: Add MVP Platform database environment variables to the backend service:

```yaml
  backend:
    # ... existing configuration ...
    environment:
      # ... existing environment variables ...
      # MVP Platform Database
      MVP_PLATFORM_DB_HOST: mvp-platform-database
      MVP_PLATFORM_DB_PORT: 5432
      MVP_PLATFORM_DB_NAME: mvp-platform-vehicles
      MVP_PLATFORM_DB_USER: mvp_platform_user
      MVP_PLATFORM_DB_PASSWORD: ${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
    depends_on:
      - postgres
      - redis
      - minio
      - mvp-platform-database  # Add this dependency
```

### Task 1.5: Create Environment File Template

**Location**: `.env.example`

**Action**: Add the following environment variables:

```env
# MVP Platform Database
MVP_PLATFORM_DB_PASSWORD=platform_secure_password

# ETL Source Database
MSSQL_SOURCE_PASSWORD=Source123!

# ETL Configuration
ETL_SCHEDULE=0 2 * * 0
ETL_LOG_LEVEL=INFO
ETL_BATCH_SIZE=10000
ETL_MAX_RETRIES=3
```

### Task 1.6: Update .env File (if exists)

**Location**: `.env`

**Action**: If `.env` exists, add the above environment variables with appropriate values for your environment.

## Validation Steps

### Step 1: Verify Docker Compose Configuration

```bash
# Test docker-compose configuration
docker-compose config

# Should output valid YAML without errors
```

### Step 2: Build and Start New Services

```bash
# Build the ETL scheduler container
docker-compose build etl-scheduler

# Start only the new services for testing
docker-compose up mvp-platform-database mssql-source -d

# Check service health
docker-compose ps
```

### Step 3: Test Database Connections

```bash
# Test MVP Platform database connection
docker-compose exec mvp-platform-database psql -U mvp_platform_user -d mvp-platform-vehicles -c "SELECT version();"

# Test MSSQL source database connection
docker-compose exec mssql-source /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "Source123!" -Q "SELECT @@VERSION"
```

### Step 4: Verify Logs Directory Creation

```bash
# Check that ETL logs directory is created
ls -la ./vehicle-etl/logs/

# Should exist and be writable
```

## Error Handling

### Common Issues and Solutions

**Issue**: PostgreSQL container fails to start
**Solution**: Check port 5433 is not in use, verify password complexity requirements

**Issue**: MSSQL container fails health check
**Solution**: Increase start_period, verify password meets MSSQL requirements, check available memory

**Issue**: ETL scheduler cannot connect to databases
**Solution**: Verify network connectivity, check environment variable values, ensure databases are healthy

### Rollback Procedure

1. Stop the new services:
   ```bash
   docker-compose stop mvp-platform-database mssql-source etl-scheduler
   ```

2. Remove the new containers:
   ```bash
   docker-compose rm mvp-platform-database mssql-source etl-scheduler
   ```

3. Remove the volume definitions from docker-compose.yml

4. Remove the service definitions from docker-compose.yml

5. Remove environment variables from backend service

## Next Steps

After successful completion of Phase 1:

1. Proceed to [Phase 2: Backend Migration](./phase-02-backend-migration.md)
2. Ensure all services are running and healthy before starting backend changes
3. Take note of any performance impacts on the existing application

## Dependencies for Next Phase

- MVP Platform database must be accessible and initialized
- Backend service must be able to connect to MVP Platform database
- Existing Redis service must be available for new caching patterns