Step 1: Fetch Data from VehAPI

  cd data/vehicle-etl
  python3 vehapi_fetch_snapshot.py --min-year 2015 --max-year 2025

  Options:
  | Flag                | Default           | Description            |
  |---------------------|-------------------|------------------------|
  | --min-year          | 2015              | Start year             |
  | --max-year          | 2022              | End year               |
  | --rate-per-min      | 55                | API rate limit         |
  | --snapshot-dir      | snapshots/<today> | Output directory       |
  | --no-response-cache | false             | Disable resume caching |

  Output: Creates snapshots/<date>/snapshot.sqlite

  ---
  Step 2: Generate SQL Files

  python3 etl_generate_sql.py --snapshot-path snapshots/<date>/snapshot.sqlite

  Output: Creates output/01_engines.sql, output/02_transmissions.sql, output/03_vehicle_options.sql

  ---
  Step 3: Import to PostgreSQL

  ./import_data.sh

  Requires: mvp-postgres container running, SQL files in output/

  ---
  Quick Test (single year)

  python3 vehapi_fetch_snapshot.py --min-year 2020 --max-year 2020

  # Full ETL workflow with cached results
  ./reset_database.sh                                                         # Clear old data
  python3 etl_generate_sql.py --snapshot-path snapshots/*.sqlite              # Generate SQL
  ./import_data.sh                                                            # Import to Postgres
  docker compose exec mvp-redis redis-cli FLUSHALL                            # Flush Redis Cache for front end