465 lines
17 KiB
Markdown
465 lines
17 KiB
Markdown
# Architecture Decisions - Vehicle ETL Integration
|
|
|
|
## Overview
|
|
|
|
This document captures all architectural decisions made during the Vehicle ETL integration project. Each decision includes the context, options considered, decision made, and rationale. This serves as a reference for future AI assistants and development teams.
|
|
|
|
## Context7 Technology Validation
|
|
|
|
All technology choices were verified through Context7 for current best practices, compatibility, and production readiness:
|
|
|
|
- ✅ **Docker Compose**: Latest version with health checks and dependency management
|
|
- ✅ **PostgreSQL 15**: Stable, production-ready with excellent Docker support
|
|
- ✅ **Python 3.11**: Current stable version for FastAPI ETL processing
|
|
- ✅ **Node.js 20**: LTS version for TypeScript backend integration
|
|
- ✅ **FastAPI**: Modern async framework, perfect for ETL API endpoints
|
|
|
|
---
|
|
|
|
## Decision 1: MVP Platform Naming Convention
|
|
|
|
### Context
|
|
Need to establish a consistent naming pattern for shared services that will be used across multiple features and future platform services.
|
|
|
|
### Options Considered
|
|
1. **Generic naming**: `shared-database`, `common-db`
|
|
2. **Service-specific naming**: `vehicle-database`, `vpic-database`
|
|
3. **Platform-prefixed naming**: `mvp-platform-database`, `mvp-platform-*`
|
|
|
|
### Decision Made
|
|
**Chosen**: Platform-prefixed naming with pattern `mvp-platform-*`
|
|
|
|
### Rationale
|
|
- Establishes clear ownership and purpose
|
|
- Scales to multiple platform services
|
|
- Avoids naming conflicts with feature-specific resources
|
|
- Creates recognizable pattern for future services
|
|
- Aligns with microservices architecture principles
|
|
|
|
### Implementation
|
|
- Database service: `mvp-platform-database`
|
|
- Database name: `mvp-platform-vehicles`
|
|
- User: `mvp_platform_user`
|
|
- Cache keys: `mvp-platform:*`
|
|
|
|
---
|
|
|
|
## Decision 2: Database Separation Strategy
|
|
|
|
### Context
|
|
Need to determine how to integrate the MVP Platform database with the existing MotoVaultPro database architecture.
|
|
|
|
### Options Considered
|
|
1. **Single Database**: Add ETL tables to existing MotoVaultPro database
|
|
2. **Schema Separation**: Use separate schemas within existing database
|
|
3. **Complete Database Separation**: Separate PostgreSQL instance for platform services
|
|
|
|
### Decision Made
|
|
**Chosen**: Complete Database Separation
|
|
|
|
### Rationale
|
|
- **Service Isolation**: Platform services can be independently managed
|
|
- **Scalability**: Each service can have different performance requirements
|
|
- **Security**: Separate access controls and permissions
|
|
- **Maintenance**: Independent backup and recovery procedures
|
|
- **Future-Proofing**: Ready for microservices deployment on Kubernetes
|
|
|
|
### Implementation
|
|
- Main app database: `motovaultpro` on port 5432
|
|
- Platform database: `mvp-platform-vehicles` on port 5433
|
|
- Separate connection pools in backend service
|
|
- Independent health checks and monitoring
|
|
|
|
---
|
|
|
|
## Decision 3: ETL Processing Architecture
|
|
|
|
### Context
|
|
Need to replace external NHTSA vPIC API calls with local data while maintaining data freshness.
|
|
|
|
### Options Considered
|
|
1. **Real-time Proxy**: Cache API responses indefinitely
|
|
2. **Daily Sync**: Update local database daily
|
|
3. **Weekly Batch ETL**: Full database refresh weekly
|
|
4. **Hybrid Approach**: Local cache with periodic full refresh
|
|
|
|
### Decision Made
|
|
**Chosen**: Weekly Batch ETL with local database
|
|
|
|
### Rationale
|
|
- **Data Freshness**: Vehicle specifications change infrequently
|
|
- **Performance**: Sub-100ms response times achievable with local queries
|
|
- **Reliability**: No dependency on external API availability
|
|
- **Cost**: Reduces external API calls and rate limiting concerns
|
|
- **Control**: Complete control over data quality and availability
|
|
|
|
### Implementation
|
|
- Weekly Sunday 2 AM ETL execution
|
|
- Complete database rebuild each cycle
|
|
- Comprehensive error handling and retry logic
|
|
- Health monitoring and alerting
|
|
|
|
---
|
|
|
|
## Decision 4: Scheduled Processing Implementation
|
|
|
|
### Context
|
|
Need to implement automated ETL processing with proper scheduling, monitoring, and error handling.
|
|
|
|
### Options Considered
|
|
1. **External Cron**: Use host system cron to trigger Docker exec
|
|
2. **Container Cron**: Install cron daemon within ETL container
|
|
3. **Kubernetes CronJob**: Use K8s native job scheduling
|
|
4. **Third-party Scheduler**: Use external scheduling service
|
|
|
|
### Decision Made
|
|
**Chosen**: Container Cron with Docker Compose
|
|
|
|
### Rationale
|
|
- **Simplicity**: Maintains single Docker Compose deployment
|
|
- **Self-Contained**: No external dependencies for development
|
|
- **Kubernetes Ready**: Can be migrated to K8s CronJob later
|
|
- **Monitoring**: Container-based health checks and logging
|
|
- **Development**: Easy local testing and debugging
|
|
|
|
### Implementation
|
|
- Python 3.11 container with cron daemon
|
|
- Configurable schedule via environment variables
|
|
- Health checks and status monitoring
|
|
- Comprehensive logging and error reporting
|
|
|
|
---
|
|
|
|
## Decision 5: API Integration Pattern
|
|
|
|
### Context
|
|
Need to integrate MVP Platform database access while maintaining exact API compatibility.
|
|
|
|
### Options Considered
|
|
1. **API Gateway**: Proxy requests to separate ETL API service
|
|
2. **Direct Integration**: Query MVP Platform database directly from vehicles feature
|
|
3. **Service Layer**: Create intermediate service layer
|
|
4. **Hybrid**: Mix of direct queries and service calls
|
|
|
|
### Decision Made
|
|
**Chosen**: Direct Integration within Vehicles Feature
|
|
|
|
### Rationale
|
|
- **Performance**: Direct database queries eliminate HTTP overhead
|
|
- **Simplicity**: Reduces complexity and potential failure points
|
|
- **Maintainability**: All vehicle-related code in single feature capsule
|
|
- **Zero Breaking Changes**: Exact same API interface preserved
|
|
- **Feature Capsule Pattern**: Maintains self-contained feature architecture
|
|
|
|
### Implementation
|
|
- MVP Platform repository within vehicles feature
|
|
- Direct PostgreSQL queries using existing connection pool pattern
|
|
- Same caching strategy with Redis
|
|
- Preserve exact response formats
|
|
|
|
---
|
|
|
|
## Decision 6: VIN Decoding Algorithm Migration
|
|
|
|
### Context
|
|
Need to port complex VIN decoding logic from Python ETL to TypeScript backend.
|
|
|
|
### Options Considered
|
|
1. **Full Port**: Rewrite all VIN decoding logic in TypeScript
|
|
2. **Database Functions**: Implement logic as PostgreSQL functions
|
|
3. **API Calls**: Call Python ETL API for VIN decoding
|
|
4. **Simplified Logic**: Implement basic VIN decoding only
|
|
|
|
### Decision Made
|
|
**Chosen**: Full Port to TypeScript with Database Assist
|
|
|
|
### Rationale
|
|
- **Performance**: Avoids HTTP calls for every VIN decode
|
|
- **Consistency**: All business logic in same language/runtime
|
|
- **Maintainability**: Single codebase for vehicle logic
|
|
- **Flexibility**: Can enhance VIN logic without ETL changes
|
|
- **Testing**: Easier to test within existing test framework
|
|
|
|
### Implementation
|
|
- TypeScript VIN validation and year extraction
|
|
- Database queries for pattern matching and confidence scoring
|
|
- Comprehensive error handling and fallback logic
|
|
- Maintain exact same accuracy as original Python implementation
|
|
|
|
---
|
|
|
|
## Decision 7: Caching Strategy
|
|
|
|
### Context
|
|
Need to maintain high performance while transitioning from external API to database queries.
|
|
|
|
### Options Considered
|
|
1. **No Caching**: Direct database queries only
|
|
2. **Database-Level Caching**: PostgreSQL query caching
|
|
3. **Application Caching**: Redis with existing patterns
|
|
4. **Multi-Level Caching**: Both database and Redis caching
|
|
|
|
### Decision Made
|
|
**Chosen**: Application Caching with Updated Key Patterns
|
|
|
|
### Rationale
|
|
- **Existing Infrastructure**: Leverage existing Redis instance
|
|
- **Performance Requirements**: Meet sub-100ms response time goals
|
|
- **Cache Hit Rates**: Maintain high cache efficiency
|
|
- **TTL Strategy**: Different TTLs for different data types
|
|
- **Invalidation**: Clear invalidation strategy for data updates
|
|
|
|
### Implementation
|
|
- VIN decoding: 30-day TTL (specifications don't change)
|
|
- Dropdown data: 7-day TTL (infrequent updates)
|
|
- Cache key pattern: `mvp-platform:*` for new services
|
|
- Existing Redis instance with updated key patterns
|
|
|
|
---
|
|
|
|
## Decision 8: Error Handling and Fallback Strategy
|
|
|
|
### Context
|
|
Need to ensure system reliability when MVP Platform database is unavailable.
|
|
|
|
### Options Considered
|
|
1. **Fail Fast**: Return errors immediately when database unavailable
|
|
2. **External API Fallback**: Fall back to original NHTSA API
|
|
3. **Cached Responses**: Return stale cached data
|
|
4. **Graceful Degradation**: Provide limited functionality
|
|
|
|
### Decision Made
|
|
**Chosen**: Graceful Degradation with Cached Responses
|
|
|
|
### Rationale
|
|
- **User Experience**: Avoid complete service failure
|
|
- **Data Availability**: Cached data still valuable when fresh data unavailable
|
|
- **System Reliability**: Partial functionality better than complete failure
|
|
- **Performance**: Cached responses still meet performance requirements
|
|
- **Recovery**: System automatically recovers when database available
|
|
|
|
### Implementation
|
|
- Return cached data when database unavailable
|
|
- Appropriate HTTP status codes (503 Service Unavailable)
|
|
- Health check endpoints for monitoring
|
|
- Automatic retry logic with exponential backoff
|
|
|
|
---
|
|
|
|
## Decision 9: Authentication and Security Model
|
|
|
|
### Context
|
|
Need to maintain existing security model while adding new platform services.
|
|
|
|
### Options Considered
|
|
1. **Authenticate All**: Require authentication for all new endpoints
|
|
2. **Mixed Authentication**: Some endpoints public, some authenticated
|
|
3. **Maintain Current**: Keep dropdown endpoints unauthenticated
|
|
4. **Enhanced Security**: Add additional security layers
|
|
|
|
### Decision Made
|
|
**Chosen**: Maintain Current Security Model
|
|
|
|
### Rationale
|
|
- **Zero Breaking Changes**: Frontend requires no modifications
|
|
- **Security Analysis**: Dropdown data is public NHTSA information
|
|
- **Performance**: No authentication overhead for public data
|
|
- **Documentation**: Aligned with security.md requirements
|
|
- **Future Flexibility**: Can add authentication layers later if needed
|
|
|
|
### Implementation
|
|
- Dropdown endpoints remain unauthenticated
|
|
- CRUD endpoints still require JWT authentication
|
|
- Platform services follow same security patterns
|
|
- Comprehensive input validation and SQL injection prevention
|
|
|
|
---
|
|
|
|
## Decision 10: Testing and Validation Strategy
|
|
|
|
### Context
|
|
Need comprehensive testing to ensure zero breaking changes and meet performance requirements.
|
|
|
|
### Options Considered
|
|
1. **Unit Tests Only**: Focus on code-level testing
|
|
2. **Integration Tests**: Test API endpoints and database integration
|
|
3. **Performance Tests**: Focus on response time requirements
|
|
4. **Comprehensive Testing**: All test types with automation
|
|
|
|
### Decision Made
|
|
**Chosen**: Comprehensive Testing with Automation
|
|
|
|
### Rationale
|
|
- **Quality Assurance**: Meet all success criteria requirements
|
|
- **Risk Mitigation**: Identify issues before production deployment
|
|
- **Performance Validation**: Ensure sub-100ms response times
|
|
- **Regression Prevention**: Automated tests catch future issues
|
|
- **Documentation**: Tests serve as behavior documentation
|
|
|
|
### Implementation
|
|
- API functionality tests for response format validation
|
|
- Authentication tests for security model compliance
|
|
- Performance tests for response time requirements
|
|
- Data accuracy tests for VIN decoding validation
|
|
- ETL process tests for scheduled job functionality
|
|
- Load tests for concurrent request handling
|
|
- Error handling tests for failure scenarios
|
|
|
|
---
|
|
|
|
## Decision 11: Deployment and Infrastructure Strategy
|
|
|
|
### Context
|
|
Need to determine deployment approach that supports both development and production.
|
|
|
|
### Options Considered
|
|
1. **Docker Compose Only**: Single deployment method
|
|
2. **Kubernetes Only**: Production-focused deployment
|
|
3. **Hybrid Approach**: Docker Compose for dev, Kubernetes for prod
|
|
4. **Multiple Options**: Support multiple deployment methods
|
|
|
|
### Decision Made
|
|
**Chosen**: Hybrid Approach (Docker Compose → Kubernetes)
|
|
|
|
### Rationale
|
|
- **Development Efficiency**: Docker Compose simpler for local development
|
|
- **Production Scalability**: Kubernetes required for production scaling
|
|
- **Migration Path**: Clear path from development to production
|
|
- **Team Skills**: Matches team capabilities and tooling
|
|
- **Cost Efficiency**: Docker Compose sufficient for development/staging
|
|
|
|
### Implementation
|
|
- Current implementation: Docker Compose with production-ready containers
|
|
- Future migration: Kubernetes manifests for production deployment
|
|
- Container images designed for both environments
|
|
- Environment variable configuration for deployment flexibility
|
|
|
|
---
|
|
|
|
## Decision 12: Data Migration and Backwards Compatibility
|
|
|
|
### Context
|
|
Need to handle transition from external API to local database without service disruption.
|
|
|
|
### Options Considered
|
|
1. **Big Bang Migration**: Switch all at once
|
|
2. **Gradual Migration**: Migrate endpoints one by one
|
|
3. **Blue-Green Deployment**: Parallel systems with traffic switch
|
|
4. **Feature Flags**: Toggle between old and new systems
|
|
|
|
### Decision Made
|
|
**Chosen**: Big Bang Migration with Comprehensive Testing
|
|
|
|
### Rationale
|
|
- **Simplicity**: Single transition point reduces complexity
|
|
- **Testing**: Comprehensive test suite validates entire system
|
|
- **Rollback**: Clear rollback path if issues discovered
|
|
- **MVP Scope**: Limited scope makes big bang migration feasible
|
|
- **Zero Downtime**: Migration can be done without service interruption
|
|
|
|
### Implementation
|
|
- Complete testing in development environment
|
|
- Staging deployment for validation
|
|
- Production deployment during low-traffic window
|
|
- Immediate rollback capability if issues detected
|
|
- Monitoring and alerting for post-deployment validation
|
|
|
|
---
|
|
|
|
## MVP Platform Architecture Principles
|
|
|
|
Based on these decisions, the following principles guide MVP Platform development:
|
|
|
|
### 1. Service Isolation
|
|
- Each platform service has its own database
|
|
- Independent deployment and scaling
|
|
- Clear service boundaries and responsibilities
|
|
|
|
### 2. Standardized Naming
|
|
- All platform services use `mvp-platform-*` prefix
|
|
- Consistent naming across databases, containers, and cache keys
|
|
- Predictable patterns for future services
|
|
|
|
### 3. Performance First
|
|
- Sub-100ms response times for all public endpoints
|
|
- Aggressive caching with appropriate TTLs
|
|
- Database optimization and connection pooling
|
|
|
|
### 4. Zero Breaking Changes
|
|
- Existing API contracts never change
|
|
- Frontend requires no modifications
|
|
- Backward compatibility maintained across all changes
|
|
|
|
### 5. Comprehensive Testing
|
|
- Automated test suites for all changes
|
|
- Performance validation requirements
|
|
- Error handling and edge case coverage
|
|
|
|
### 6. Graceful Degradation
|
|
- Systems continue operating with reduced functionality
|
|
- Appropriate error responses and status codes
|
|
- Automatic recovery when services restore
|
|
|
|
### 7. Observability Ready
|
|
- Health check endpoints for all services
|
|
- Comprehensive logging and monitoring
|
|
- Alerting for critical failures
|
|
|
|
### 8. Future-Proof Architecture
|
|
- Designed for Kubernetes migration
|
|
- Microservices-ready patterns
|
|
- Extensible for additional platform services
|
|
|
|
---
|
|
|
|
## Future Architecture Evolution
|
|
|
|
### Next Platform Services
|
|
Following this pattern, future platform services will include:
|
|
|
|
1. **mvp-platform-analytics**: User behavior tracking and analysis
|
|
2. **mvp-platform-notifications**: Email, SMS, and push notifications
|
|
3. **mvp-platform-payments**: Payment processing and billing
|
|
4. **mvp-platform-documents**: File storage and document management
|
|
5. **mvp-platform-search**: Full-text search and indexing
|
|
|
|
### Kubernetes Migration Plan
|
|
When ready for production scaling:
|
|
|
|
1. **Container Compatibility**: All containers designed for Kubernetes
|
|
2. **Configuration Management**: Environment-based configuration
|
|
3. **Service Discovery**: Native Kubernetes service discovery
|
|
4. **Persistent Storage**: Kubernetes persistent volumes
|
|
5. **Auto-scaling**: Horizontal pod autoscaling
|
|
6. **Ingress**: Kubernetes ingress controllers
|
|
7. **Monitoring**: Prometheus and Grafana integration
|
|
|
|
### Microservices Evolution
|
|
Path to full microservices architecture:
|
|
|
|
1. **Service Extraction**: Extract platform services to independent deployments
|
|
2. **API Gateway**: Implement centralized API gateway
|
|
3. **Service Mesh**: Add service mesh for advanced networking
|
|
4. **Event-Driven**: Implement event-driven communication patterns
|
|
5. **CQRS**: Command Query Responsibility Segregation for complex domains
|
|
|
|
---
|
|
|
|
## Decision Review and Updates
|
|
|
|
This document should be reviewed and updated:
|
|
|
|
- **Before adding new platform services**: Ensure consistency with established patterns
|
|
- **During performance issues**: Review caching and database decisions
|
|
- **When scaling requirements change**: Evaluate deployment and infrastructure choices
|
|
- **After major technology updates**: Reassess technology choices with current best practices
|
|
|
|
All architectural decisions should be validated against:
|
|
- Performance requirements and SLAs
|
|
- Security and compliance requirements
|
|
- Team capabilities and maintenance burden
|
|
- Cost and resource constraints
|
|
- Future scalability and extensibility needs
|
|
|
|
**Document Last Updated**: [Current Date]
|
|
**Next Review Date**: [3 months from last update] |