Vehicle Catalog Data Export
Export the current vehicle catalog database to SQL files for GitLab CI/CD deployment.
Export Workflow
Export from Running Database
cd data/vehicle-etl
python3 export_from_postgres.py
Output: Creates output/01_engines.sql, output/02_transmissions.sql, output/03_vehicle_options.sql
Requirements:
- mvp-postgres container running
- Python 3.7+
Commit and Deploy
git add output/*.sql
git commit -m "Update vehicle catalog data from PostgreSQL export"
git push
GitLab CI/CD will automatically import these SQL files during deployment.
When to Export
| Scenario | Action |
|---|---|
| Admin uploaded CSVs to database | Export and commit |
| Manual corrections in PostgreSQL | Export and commit |
| After adding new vehicle data | Export and commit |
| Preparing for deployment | Export and commit |
Local Testing
# Export current database state
python3 export_from_postgres.py
# Test import locally
./reset_database.sh
./import_data.sh
docker compose exec mvp-redis redis-cli FLUSHALL
# Verify data
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "
SELECT
(SELECT COUNT(*) FROM engines) as engines,
(SELECT COUNT(*) FROM transmissions) as transmissions,
(SELECT COUNT(*) FROM vehicle_options) as vehicle_options,
(SELECT MIN(year) FROM vehicle_options) as min_year,
(SELECT MAX(year) FROM vehicle_options) as max_year;
"
GitLab CI/CD Integration
The pipeline automatically imports SQL files from output/ directory during deployment (/.gitlab-ci.yml lines 89-98):
- data/vehicle-etl/output/01_engines.sql
- data/vehicle-etl/output/02_transmissions.sql
- data/vehicle-etl/output/03_vehicle_options.sql
Commit updated SQL files to trigger deployment with new data.
Legacy Scripts (Not Used)
The following scripts are legacy from the VehAPI integration and are no longer used:
- vehapi_fetch_snapshot.py (obsolete - VehAPI not used)
- etl_generate_sql.py (obsolete - database export used instead)
These scripts are preserved for historical reference but should not be executed.