Step 1: Fetch Data from VehAPI
cd data/vehicle-etl python3 vehapi_fetch_snapshot.py --min-year 2015 --max-year 2025
Options:
| Flag | Default | Description |
|---|---|---|
| --min-year | 2015 | Start year |
| --max-year | 2022 | End year |
| --rate-per-min | 55 | API rate limit |
| --snapshot-dir | snapshots/ | Output directory |
| --no-response-cache | false | Disable resume caching |
Output: Creates snapshots//snapshot.sqlite
Step 2: Generate SQL Files
python3 etl_generate_sql.py --snapshot-path snapshots//snapshot.sqlite
Output: Creates output/01_engines.sql, output/02_transmissions.sql, output/03_vehicle_options.sql
Step 3: Import to PostgreSQL
./import_data.sh
Requires: mvp-postgres container running, SQL files in output/
Quick Test (single year)
python3 vehapi_fetch_snapshot.py --min-year 2020 --max-year 2020
Full ETL workflow
./reset_database.sh # Clear old data python3 vehapi_fetch_snapshot.py # Fetch from API python3 etl_generate_sql.py --snapshot-path snapshots/*.sqlite # Generate SQL ./import_data.sh # Import to Postgres docker compose exec mvp-redis redis-cli FLUSHALL # Flush Redis Cache for front end