Vehicle Catalog Data Export

Export the current vehicle catalog database to SQL files for GitLab CI/CD deployment.

Export Workflow

Export from Running Database

cd data/vehicle-etl
python3 export_from_postgres.py

Output: Creates output/01_engines.sql, output/02_transmissions.sql, output/03_vehicle_options.sql

Requirements:

mvp-postgres container running
Python 3.7+

Commit and Deploy

git add output/*.sql
git commit -m "Update vehicle catalog data from PostgreSQL export"
git push

GitLab CI/CD will automatically import these SQL files during deployment.

When to Export

Scenario	Action
Admin uploaded CSVs to database	Export and commit
Manual corrections in PostgreSQL	Export and commit
After adding new vehicle data	Export and commit
Preparing for deployment	Export and commit

Local Testing

# Export current database state
python3 export_from_postgres.py

# Test import locally
./reset_database.sh
./import_data.sh
docker compose exec mvp-redis redis-cli FLUSHALL

# Verify data
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "
  SELECT
    (SELECT COUNT(*) FROM engines) as engines,
    (SELECT COUNT(*) FROM transmissions) as transmissions,
    (SELECT COUNT(*) FROM vehicle_options) as vehicle_options,
    (SELECT MIN(year) FROM vehicle_options) as min_year,
    (SELECT MAX(year) FROM vehicle_options) as max_year;
"

GitLab CI/CD Integration

The pipeline automatically imports SQL files from output/ directory during deployment (/.gitlab-ci.yml lines 89-98):

data/vehicle-etl/output/01_engines.sql
data/vehicle-etl/output/02_transmissions.sql
data/vehicle-etl/output/03_vehicle_options.sql

Commit updated SQL files to trigger deployment with new data.

Legacy Scripts (Not Used)

The following scripts are legacy from the VehAPI integration and are no longer used:

vehapi_fetch_snapshot.py (obsolete - VehAPI not used)
etl_generate_sql.py (obsolete - database export used instead)

These scripts are preserved for historical reference but should not be executed.

2.1 KiB Raw Blame History