328 lines
8.9 KiB
Markdown
328 lines
8.9 KiB
Markdown
# CLI Commands - Manual JSON ETL
|
|
|
|
## Overview
|
|
New CLI commands for processing JSON vehicle data into the PostgreSQL database.
|
|
|
|
## Primary Command: `load-manual`
|
|
|
|
### Basic Syntax
|
|
```bash
|
|
python -m etl load-manual [OPTIONS]
|
|
```
|
|
|
|
### Command Options
|
|
|
|
#### Load Mode (`--mode`)
|
|
Controls how data is handled in the database:
|
|
|
|
```bash
|
|
# Append mode (safe, default)
|
|
python -m etl load-manual --mode=append
|
|
|
|
# Clear mode (destructive - removes existing data first)
|
|
python -m etl load-manual --mode=clear
|
|
```
|
|
|
|
**Mode Details:**
|
|
- **`append`** (default): Uses `ON CONFLICT DO NOTHING` - safe for existing data
|
|
- **`clear`**: Uses `TRUNCATE CASCADE` then insert - completely replaces existing data
|
|
|
|
#### Specific Make Processing (`--make`)
|
|
Process only a specific make instead of all 55 files:
|
|
|
|
```bash
|
|
# Process only Toyota
|
|
python -m etl load-manual --make=toyota
|
|
|
|
# Process only BMW (uses filename format)
|
|
python -m etl load-manual --make=bmw
|
|
|
|
# Process Alfa Romeo (underscore format from filename)
|
|
python -m etl load-manual --make=alfa_romeo
|
|
```
|
|
|
|
#### Validation Only (`--validate-only`)
|
|
Validate JSON files without loading to database:
|
|
|
|
```bash
|
|
# Validate all JSON files
|
|
python -m etl load-manual --validate-only
|
|
|
|
# Validate specific make
|
|
python -m etl load-manual --make=tesla --validate-only
|
|
```
|
|
|
|
#### Verbose Output (`--verbose`)
|
|
Enable detailed progress output:
|
|
|
|
```bash
|
|
# Verbose processing
|
|
python -m etl load-manual --verbose
|
|
|
|
# Quiet processing (errors only)
|
|
python -m etl load-manual --quiet
|
|
```
|
|
|
|
### Complete Command Examples
|
|
|
|
```bash
|
|
# Standard usage - process all makes safely
|
|
python -m etl load-manual
|
|
|
|
# Full reload - clear and rebuild entire database
|
|
python -m etl load-manual --mode=clear --verbose
|
|
|
|
# Process specific make with validation
|
|
python -m etl load-manual --make=honda --mode=append --verbose
|
|
|
|
# Validate before processing
|
|
python -m etl load-manual --validate-only
|
|
python -m etl load-manual --mode=clear # If validation passes
|
|
```
|
|
|
|
## Secondary Command: `validate-json`
|
|
|
|
### Purpose
|
|
Standalone validation of JSON files without database operations.
|
|
|
|
### Syntax
|
|
```bash
|
|
python -m etl validate-json [OPTIONS]
|
|
```
|
|
|
|
### Options
|
|
|
|
```bash
|
|
# Validate all JSON files
|
|
python -m etl validate-json
|
|
|
|
# Validate specific make
|
|
python -m etl validate-json --make=toyota
|
|
|
|
# Generate detailed report
|
|
python -m etl validate-json --detailed-report
|
|
|
|
# Export validation results to file
|
|
python -m etl validate-json --export-report=/tmp/validation.json
|
|
```
|
|
|
|
### Validation Checks
|
|
1. **JSON structure** validation
|
|
2. **Engine parsing** validation
|
|
3. **Make name mapping** validation
|
|
4. **Data completeness** checks
|
|
5. **Cross-reference** with authoritative makes list
|
|
|
|
## Implementation Details
|
|
|
|
### CLI Command Structure
|
|
Add to `etl/main.py`:
|
|
|
|
```python
|
|
@cli.command()
|
|
@click.option('--mode', type=click.Choice(['clear', 'append']),
|
|
default='append', help='Database load mode')
|
|
@click.option('--make', help='Process specific make only (use filename format)')
|
|
@click.option('--validate-only', is_flag=True,
|
|
help='Validate JSON files without loading to database')
|
|
@click.option('--verbose', is_flag=True, help='Enable verbose output')
|
|
@click.option('--quiet', is_flag=True, help='Suppress non-error output')
|
|
def load_manual(mode, make, validate_only, verbose, quiet):
|
|
"""Load vehicle data from JSON files"""
|
|
|
|
if quiet:
|
|
logging.getLogger().setLevel(logging.ERROR)
|
|
elif verbose:
|
|
logging.getLogger().setLevel(logging.DEBUG)
|
|
|
|
try:
|
|
pipeline = ManualJsonPipeline(
|
|
sources_dir=config.JSON_SOURCES_DIR,
|
|
load_mode=LoadMode(mode.upper())
|
|
)
|
|
|
|
if validate_only:
|
|
result = pipeline.validate_all_json()
|
|
display_validation_report(result)
|
|
return
|
|
|
|
result = pipeline.run_manual_pipeline(specific_make=make)
|
|
display_pipeline_result(result)
|
|
|
|
if not result.success:
|
|
sys.exit(1)
|
|
|
|
except Exception as e:
|
|
logger.error(f"Manual load failed: {e}")
|
|
sys.exit(1)
|
|
|
|
@cli.command()
|
|
@click.option('--make', help='Validate specific make only')
|
|
@click.option('--detailed-report', is_flag=True,
|
|
help='Generate detailed validation report')
|
|
@click.option('--export-report', help='Export validation report to file')
|
|
def validate_json(make, detailed_report, export_report):
|
|
"""Validate JSON files structure and data quality"""
|
|
|
|
try:
|
|
validator = JsonValidator(sources_dir=config.JSON_SOURCES_DIR)
|
|
|
|
if make:
|
|
result = validator.validate_make(make)
|
|
else:
|
|
result = validator.validate_all_makes()
|
|
|
|
if detailed_report or export_report:
|
|
report = validator.generate_detailed_report(result)
|
|
|
|
if export_report:
|
|
with open(export_report, 'w') as f:
|
|
json.dump(report, f, indent=2)
|
|
logger.info(f"Validation report exported to {export_report}")
|
|
else:
|
|
display_detailed_report(report)
|
|
else:
|
|
display_validation_summary(result)
|
|
|
|
except Exception as e:
|
|
logger.error(f"JSON validation failed: {e}")
|
|
sys.exit(1)
|
|
```
|
|
|
|
## Output Examples
|
|
|
|
### Successful Load Output
|
|
```
|
|
$ python -m etl load-manual --mode=append --verbose
|
|
|
|
🚀 Starting manual JSON ETL pipeline...
|
|
📁 Processing 55 JSON files from sources/makes/
|
|
|
|
✅ Make normalization validation passed (55/55)
|
|
✅ Engine parsing validation passed (1,247 engines)
|
|
|
|
📊 Processing makes:
|
|
├── toyota.json → Toyota (47 models, 203 engines, 312 trims)
|
|
├── ford.json → Ford (52 models, 189 engines, 298 trims)
|
|
├── chevrolet.json → Chevrolet (48 models, 167 engines, 287 trims)
|
|
└── ... (52 more makes)
|
|
|
|
💾 Database loading:
|
|
├── Makes: 55 loaded (0 duplicates)
|
|
├── Models: 2,847 loaded (23 duplicates)
|
|
├── Model Years: 18,392 loaded (105 duplicates)
|
|
├── Engines: 1,247 loaded (45 duplicates)
|
|
└── Trims: 12,058 loaded (234 duplicates)
|
|
|
|
✅ Manual JSON ETL completed successfully in 2m 34s
|
|
```
|
|
|
|
### Validation Output
|
|
```
|
|
$ python -m etl validate-json
|
|
|
|
📋 JSON Validation Report
|
|
|
|
✅ File Structure: 55/55 files valid
|
|
✅ Make Name Mapping: 55/55 mappings valid
|
|
⚠️ Engine Parsing: 1,201/1,247 engines parsed (46 unparseable)
|
|
✅ Data Completeness: All required fields present
|
|
|
|
🔍 Issues Found:
|
|
├── Unparseable engines:
|
|
│ ├── toyota.json: "Custom Hybrid System" (1 occurrence)
|
|
│ ├── ferrari.json: "V12 Twin-Turbo Custom" (2 occurrences)
|
|
│ └── lamborghini.json: "V10 Plus" (43 occurrences)
|
|
└── Empty engine arrays:
|
|
├── tesla.json: 24 models with empty engines
|
|
└── lucid.json: 3 models with empty engines
|
|
|
|
💡 Recommendations:
|
|
• Review unparseable engine formats
|
|
• Electric vehicle handling will create default "Electric Motor" entries
|
|
|
|
Overall Status: ✅ READY FOR PROCESSING
|
|
```
|
|
|
|
### Error Handling Output
|
|
```
|
|
$ python -m etl load-manual --make=invalid_make
|
|
|
|
❌ Error: Make 'invalid_make' not found
|
|
|
|
Available makes:
|
|
acura, alfa_romeo, aston_martin, audi, bentley, bmw,
|
|
buick, cadillac, chevrolet, chrysler, dodge, ferrari,
|
|
... (showing first 20)
|
|
|
|
💡 Tip: Use 'python -m etl validate-json' to see all available makes
|
|
```
|
|
|
|
## Integration with Existing Commands
|
|
|
|
### Command Compatibility
|
|
The new commands integrate seamlessly with existing ETL commands:
|
|
|
|
```bash
|
|
# Existing MSSQL pipeline (unchanged)
|
|
python -m etl build-catalog
|
|
|
|
# New manual JSON pipeline
|
|
python -m etl load-manual
|
|
|
|
# Test connections (works for both)
|
|
python -m etl test
|
|
|
|
# Scheduling (MSSQL only currently)
|
|
python -m etl schedule
|
|
```
|
|
|
|
### Configuration Integration
|
|
Uses existing config structure with new JSON-specific settings:
|
|
|
|
```python
|
|
# In config.py
|
|
JSON_SOURCES_DIR: str = "sources/makes"
|
|
MANUAL_LOAD_DEFAULT_MODE: str = "append"
|
|
MANUAL_LOAD_BATCH_SIZE: int = 1000
|
|
JSON_VALIDATION_STRICT: bool = False
|
|
```
|
|
|
|
## Help and Documentation
|
|
|
|
### Built-in Help
|
|
```bash
|
|
# Main command help
|
|
python -m etl load-manual --help
|
|
|
|
# All commands help
|
|
python -m etl --help
|
|
```
|
|
|
|
### Command Discovery
|
|
```bash
|
|
# List all available commands
|
|
python -m etl
|
|
|
|
# Shows:
|
|
# Commands:
|
|
# build-catalog Build vehicle catalog from MSSQL database
|
|
# load-manual Load vehicle data from JSON files
|
|
# validate-json Validate JSON files structure and data quality
|
|
# schedule Start ETL scheduler (default mode)
|
|
# test Test database connections
|
|
# update Run ETL update
|
|
```
|
|
|
|
## Future Enhancements
|
|
|
|
### Planned Command Options
|
|
- `--dry-run`: Show what would be processed without making changes
|
|
- `--since`: Process only files modified since timestamp
|
|
- `--parallel`: Enable parallel processing of makes
|
|
- `--rollback`: Rollback previous manual load operation
|
|
|
|
### Advanced Validation Options
|
|
- `--strict-parsing`: Fail on any engine parsing errors
|
|
- `--cross-validate`: Compare JSON data against MSSQL data where available
|
|
- `--performance-test`: Benchmark processing performance |