Initial Commit
This commit is contained in:
Eric Gullickson
290
docs/changes/vehicles-dropdown-v1/phase-01-infrastructure.md
Normal file
290
docs/changes/vehicles-dropdown-v1/phase-01-infrastructure.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Phase 1: Infrastructure Setup
|
||||
|
||||
## Overview
|
||||
|
||||
This phase establishes the foundational infrastructure for the MVP Platform by adding three new Docker services to the main `docker-compose.yml`. This creates the shared services architecture pattern that future platform services will follow.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker and Docker Compose installed
|
||||
- Main MotoVaultPro application running successfully
|
||||
- Access to NHTSA vPIC database backup file (VPICList_lite_2025_07.bak)
|
||||
- Understanding of existing docker-compose.yml structure
|
||||
|
||||
## Tasks
|
||||
|
||||
### Task 1.1: Add MVP Platform Database Service
|
||||
|
||||
**Location**: `docker-compose.yml`
|
||||
|
||||
**Action**: Add the following service definition to the services section:
|
||||
|
||||
```yaml
|
||||
mvp-platform-database:
|
||||
image: postgres:15-alpine
|
||||
container_name: mvp-platform-db
|
||||
environment:
|
||||
POSTGRES_DB: mvp-platform-vehicles
|
||||
POSTGRES_USER: mvp_platform_user
|
||||
POSTGRES_PASSWORD: ${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
|
||||
POSTGRES_INITDB_ARGS: "--encoding=UTF8"
|
||||
volumes:
|
||||
- mvp_platform_data:/var/lib/postgresql/data
|
||||
- ./vehicle-etl/sql/schema:/docker-entrypoint-initdb.d
|
||||
ports:
|
||||
- "5433:5432"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U mvp_platform_user -p 5432"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- default
|
||||
```
|
||||
|
||||
**Action**: Add the volume definition to the volumes section:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
minio_data:
|
||||
mvp_platform_data: # Add this line
|
||||
```
|
||||
|
||||
### Task 1.2: Add MSSQL Source Database Service
|
||||
|
||||
**Location**: `docker-compose.yml`
|
||||
|
||||
**Action**: Add the following service definition:
|
||||
|
||||
```yaml
|
||||
mssql-source:
|
||||
image: mcr.microsoft.com/mssql/server:2019-latest
|
||||
container_name: mvp-mssql-source
|
||||
user: root
|
||||
environment:
|
||||
- ACCEPT_EULA=Y
|
||||
- SA_PASSWORD=${MSSQL_SOURCE_PASSWORD:-Source123!}
|
||||
- MSSQL_PID=Developer
|
||||
ports:
|
||||
- "1433:1433"
|
||||
volumes:
|
||||
- mssql_source_data:/var/opt/mssql/data
|
||||
- ./vehicle-etl/volumes/mssql/backups:/backups
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "/opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P ${MSSQL_SOURCE_PASSWORD:-Source123!} -Q 'SELECT 1'"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
networks:
|
||||
- default
|
||||
```
|
||||
|
||||
**Action**: Add volume to volumes section:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
minio_data:
|
||||
mvp_platform_data:
|
||||
mssql_source_data: # Add this line
|
||||
```
|
||||
|
||||
### Task 1.3: Add Scheduled ETL Service
|
||||
|
||||
**Location**: `docker-compose.yml`
|
||||
|
||||
**Action**: Add the following service definition:
|
||||
|
||||
```yaml
|
||||
etl-scheduler:
|
||||
build:
|
||||
context: ./vehicle-etl
|
||||
dockerfile: docker/Dockerfile.etl
|
||||
container_name: mvp-etl-scheduler
|
||||
environment:
|
||||
# Database connections
|
||||
- MSSQL_HOST=mssql-source
|
||||
- MSSQL_PORT=1433
|
||||
- MSSQL_DATABASE=VPICList
|
||||
- MSSQL_USERNAME=sa
|
||||
- MSSQL_PASSWORD=${MSSQL_SOURCE_PASSWORD:-Source123!}
|
||||
- POSTGRES_HOST=mvp-platform-database
|
||||
- POSTGRES_PORT=5432
|
||||
- POSTGRES_DATABASE=mvp-platform-vehicles
|
||||
- POSTGRES_USERNAME=mvp_platform_user
|
||||
- POSTGRES_PASSWORD=${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
|
||||
- REDIS_HOST=redis
|
||||
- REDIS_PORT=6379
|
||||
# ETL configuration
|
||||
- ETL_SCHEDULE=0 2 * * 0 # Weekly on Sunday at 2 AM
|
||||
- ETL_LOG_LEVEL=INFO
|
||||
- ETL_BATCH_SIZE=10000
|
||||
- ETL_MAX_RETRIES=3
|
||||
volumes:
|
||||
- ./vehicle-etl/logs:/app/logs
|
||||
- etl_scheduler_data:/app/data
|
||||
depends_on:
|
||||
mssql-source:
|
||||
condition: service_healthy
|
||||
mvp-platform-database:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- default
|
||||
```
|
||||
|
||||
**Action**: Add volume to volumes section:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
minio_data:
|
||||
mvp_platform_data:
|
||||
mssql_source_data:
|
||||
etl_scheduler_data: # Add this line
|
||||
```
|
||||
|
||||
### Task 1.4: Update Backend Service Environment Variables
|
||||
|
||||
**Location**: `docker-compose.yml`
|
||||
|
||||
**Action**: Add MVP Platform database environment variables to the backend service:
|
||||
|
||||
```yaml
|
||||
backend:
|
||||
# ... existing configuration ...
|
||||
environment:
|
||||
# ... existing environment variables ...
|
||||
# MVP Platform Database
|
||||
MVP_PLATFORM_DB_HOST: mvp-platform-database
|
||||
MVP_PLATFORM_DB_PORT: 5432
|
||||
MVP_PLATFORM_DB_NAME: mvp-platform-vehicles
|
||||
MVP_PLATFORM_DB_USER: mvp_platform_user
|
||||
MVP_PLATFORM_DB_PASSWORD: ${MVP_PLATFORM_DB_PASSWORD:-platform_dev_password}
|
||||
depends_on:
|
||||
- postgres
|
||||
- redis
|
||||
- minio
|
||||
- mvp-platform-database # Add this dependency
|
||||
```
|
||||
|
||||
### Task 1.5: Create Environment File Template
|
||||
|
||||
**Location**: `.env.example`
|
||||
|
||||
**Action**: Add the following environment variables:
|
||||
|
||||
```env
|
||||
# MVP Platform Database
|
||||
MVP_PLATFORM_DB_PASSWORD=platform_secure_password
|
||||
|
||||
# ETL Source Database
|
||||
MSSQL_SOURCE_PASSWORD=Source123!
|
||||
|
||||
# ETL Configuration
|
||||
ETL_SCHEDULE=0 2 * * 0
|
||||
ETL_LOG_LEVEL=INFO
|
||||
ETL_BATCH_SIZE=10000
|
||||
ETL_MAX_RETRIES=3
|
||||
```
|
||||
|
||||
### Task 1.6: Update .env File (if exists)
|
||||
|
||||
**Location**: `.env`
|
||||
|
||||
**Action**: If `.env` exists, add the above environment variables with appropriate values for your environment.
|
||||
|
||||
## Validation Steps
|
||||
|
||||
### Step 1: Verify Docker Compose Configuration
|
||||
|
||||
```bash
|
||||
# Test docker-compose configuration
|
||||
docker-compose config
|
||||
|
||||
# Should output valid YAML without errors
|
||||
```
|
||||
|
||||
### Step 2: Build and Start New Services
|
||||
|
||||
```bash
|
||||
# Build the ETL scheduler container
|
||||
docker-compose build etl-scheduler
|
||||
|
||||
# Start only the new services for testing
|
||||
docker-compose up mvp-platform-database mssql-source -d
|
||||
|
||||
# Check service health
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### Step 3: Test Database Connections
|
||||
|
||||
```bash
|
||||
# Test MVP Platform database connection
|
||||
docker-compose exec mvp-platform-database psql -U mvp_platform_user -d mvp-platform-vehicles -c "SELECT version();"
|
||||
|
||||
# Test MSSQL source database connection
|
||||
docker-compose exec mssql-source /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "Source123!" -Q "SELECT @@VERSION"
|
||||
```
|
||||
|
||||
### Step 4: Verify Logs Directory Creation
|
||||
|
||||
```bash
|
||||
# Check that ETL logs directory is created
|
||||
ls -la ./vehicle-etl/logs/
|
||||
|
||||
# Should exist and be writable
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Issues and Solutions
|
||||
|
||||
**Issue**: PostgreSQL container fails to start
|
||||
**Solution**: Check port 5433 is not in use, verify password complexity requirements
|
||||
|
||||
**Issue**: MSSQL container fails health check
|
||||
**Solution**: Increase start_period, verify password meets MSSQL requirements, check available memory
|
||||
|
||||
**Issue**: ETL scheduler cannot connect to databases
|
||||
**Solution**: Verify network connectivity, check environment variable values, ensure databases are healthy
|
||||
|
||||
### Rollback Procedure
|
||||
|
||||
1. Stop the new services:
|
||||
```bash
|
||||
docker-compose stop mvp-platform-database mssql-source etl-scheduler
|
||||
```
|
||||
|
||||
2. Remove the new containers:
|
||||
```bash
|
||||
docker-compose rm mvp-platform-database mssql-source etl-scheduler
|
||||
```
|
||||
|
||||
3. Remove the volume definitions from docker-compose.yml
|
||||
|
||||
4. Remove the service definitions from docker-compose.yml
|
||||
|
||||
5. Remove environment variables from backend service
|
||||
|
||||
## Next Steps
|
||||
|
||||
After successful completion of Phase 1:
|
||||
|
||||
1. Proceed to [Phase 2: Backend Migration](./phase-02-backend-migration.md)
|
||||
2. Ensure all services are running and healthy before starting backend changes
|
||||
3. Take note of any performance impacts on the existing application
|
||||
|
||||
## Dependencies for Next Phase
|
||||
|
||||
- MVP Platform database must be accessible and initialized
|
||||
- Backend service must be able to connect to MVP Platform database
|
||||
- Existing Redis service must be available for new caching patterns
|
||||
Reference in New Issue
Block a user