Vehicle ETL Process fixed. Admin settings fixed.
This commit is contained in:
41
data/vehicle-etl/README.md
Normal file
41
data/vehicle-etl/README.md
Normal file
@@ -0,0 +1,41 @@
|
||||
Step 1: Fetch Data from VehAPI
|
||||
|
||||
cd data/vehicle-etl
|
||||
python3 vehapi_fetch_snapshot.py --min-year 2015 --max-year 2025
|
||||
|
||||
Options:
|
||||
| Flag | Default | Description |
|
||||
|---------------------|-------------------|------------------------|
|
||||
| --min-year | 2015 | Start year |
|
||||
| --max-year | 2022 | End year |
|
||||
| --rate-per-min | 55 | API rate limit |
|
||||
| --snapshot-dir | snapshots/<today> | Output directory |
|
||||
| --no-response-cache | false | Disable resume caching |
|
||||
|
||||
Output: Creates snapshots/<date>/snapshot.sqlite
|
||||
|
||||
---
|
||||
Step 2: Generate SQL Files
|
||||
|
||||
python3 etl_generate_sql.py --snapshot-path snapshots/<date>/snapshot.sqlite
|
||||
|
||||
Output: Creates output/01_engines.sql, output/02_transmissions.sql, output/03_vehicle_options.sql
|
||||
|
||||
---
|
||||
Step 3: Import to PostgreSQL
|
||||
|
||||
./import_data.sh
|
||||
|
||||
Requires: mvp-postgres container running, SQL files in output/
|
||||
|
||||
---
|
||||
Quick Test (single year)
|
||||
|
||||
python3 vehapi_fetch_snapshot.py --min-year 2020 --max-year 2020
|
||||
|
||||
# Full ETL workflow
|
||||
./reset_database.sh # Clear old data
|
||||
python3 vehapi_fetch_snapshot.py # Fetch from API
|
||||
python3 etl_generate_sql.py # Generate SQL
|
||||
./import_data.sh # Import to Postgres
|
||||
docker compose exec mvp-redis redis-cli FLUSHALL # Flush Redis Cache for front end
|
||||
Reference in New Issue
Block a user