Vehicle ETL Process fixed. Admin settings fixed.

This commit is contained in:
Eric Gullickson
2025-12-15 20:51:52 -06:00
parent 1a9ead9d9d
commit b84d4c7fef
23 changed files with 4553 additions and 2450 deletions

View File

@@ -0,0 +1,41 @@
Step 1: Fetch Data from VehAPI
cd data/vehicle-etl
python3 vehapi_fetch_snapshot.py --min-year 2015 --max-year 2025
Options:
| Flag | Default | Description |
|---------------------|-------------------|------------------------|
| --min-year | 2015 | Start year |
| --max-year | 2022 | End year |
| --rate-per-min | 55 | API rate limit |
| --snapshot-dir | snapshots/<today> | Output directory |
| --no-response-cache | false | Disable resume caching |
Output: Creates snapshots/<date>/snapshot.sqlite
---
Step 2: Generate SQL Files
python3 etl_generate_sql.py --snapshot-path snapshots/<date>/snapshot.sqlite
Output: Creates output/01_engines.sql, output/02_transmissions.sql, output/03_vehicle_options.sql
---
Step 3: Import to PostgreSQL
./import_data.sh
Requires: mvp-postgres container running, SQL files in output/
---
Quick Test (single year)
python3 vehapi_fetch_snapshot.py --min-year 2020 --max-year 2020
# Full ETL workflow
./reset_database.sh # Clear old data
python3 vehapi_fetch_snapshot.py # Fetch from API
python3 etl_generate_sql.py # Generate SQL
./import_data.sh # Import to Postgres
docker compose exec mvp-redis redis-cli FLUSHALL # Flush Redis Cache for front end