42 lines
1.5 KiB
Markdown
42 lines
1.5 KiB
Markdown
Step 1: Fetch Data from VehAPI
|
|
|
|
cd data/vehicle-etl
|
|
python3 vehapi_fetch_snapshot.py --min-year 2015 --max-year 2025
|
|
|
|
Options:
|
|
| Flag | Default | Description |
|
|
|---------------------|-------------------|------------------------|
|
|
| --min-year | 2015 | Start year |
|
|
| --max-year | 2022 | End year |
|
|
| --rate-per-min | 55 | API rate limit |
|
|
| --snapshot-dir | snapshots/<today> | Output directory |
|
|
| --no-response-cache | false | Disable resume caching |
|
|
|
|
Output: Creates snapshots/<date>/snapshot.sqlite
|
|
|
|
---
|
|
Step 2: Generate SQL Files
|
|
|
|
python3 etl_generate_sql.py --snapshot-path snapshots/<date>/snapshot.sqlite
|
|
|
|
Output: Creates output/01_engines.sql, output/02_transmissions.sql, output/03_vehicle_options.sql
|
|
|
|
---
|
|
Step 3: Import to PostgreSQL
|
|
|
|
./import_data.sh
|
|
|
|
Requires: mvp-postgres container running, SQL files in output/
|
|
|
|
---
|
|
Quick Test (single year)
|
|
|
|
python3 vehapi_fetch_snapshot.py --min-year 2020 --max-year 2020
|
|
|
|
# Full ETL workflow
|
|
./reset_database.sh # Clear old data
|
|
python3 vehapi_fetch_snapshot.py # Fetch from API
|
|
python3 etl_generate_sql.py # Generate SQL
|
|
./import_data.sh # Import to Postgres
|
|
docker compose exec mvp-redis redis-cli FLUSHALL # Flush Redis Cache for front end
|