85 lines
2.1 KiB
Markdown
85 lines
2.1 KiB
Markdown
# Vehicle Catalog Data Export
|
|
|
|
Export the current vehicle catalog database to SQL files for GitLab CI/CD deployment.
|
|
|
|
## Export Workflow
|
|
|
|
### Export from Running Database
|
|
|
|
```bash
|
|
cd data/vehicle-etl
|
|
python3 export_from_postgres.py
|
|
```
|
|
|
|
**Output:** Creates output/01_engines.sql, output/02_transmissions.sql, output/03_vehicle_options.sql
|
|
|
|
**Requirements:**
|
|
- mvp-postgres container running
|
|
- Python 3.7+
|
|
|
|
### Commit and Deploy
|
|
|
|
```bash
|
|
git add output/*.sql
|
|
git commit -m "Update vehicle catalog data from PostgreSQL export"
|
|
git push
|
|
```
|
|
|
|
GitLab CI/CD will automatically import these SQL files during deployment.
|
|
|
|
---
|
|
|
|
## When to Export
|
|
|
|
| Scenario | Action |
|
|
|----------|--------|
|
|
| Admin uploaded CSVs to database | Export and commit |
|
|
| Manual corrections in PostgreSQL | Export and commit |
|
|
| After adding new vehicle data | Export and commit |
|
|
| Preparing for deployment | Export and commit |
|
|
|
|
---
|
|
|
|
## Local Testing
|
|
|
|
```bash
|
|
# Export current database state
|
|
python3 export_from_postgres.py
|
|
|
|
# Test import locally
|
|
./reset_database.sh
|
|
./import_data.sh
|
|
docker compose exec mvp-redis redis-cli FLUSHALL
|
|
|
|
# Verify data
|
|
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "
|
|
SELECT
|
|
(SELECT COUNT(*) FROM engines) as engines,
|
|
(SELECT COUNT(*) FROM transmissions) as transmissions,
|
|
(SELECT COUNT(*) FROM vehicle_options) as vehicle_options,
|
|
(SELECT MIN(year) FROM vehicle_options) as min_year,
|
|
(SELECT MAX(year) FROM vehicle_options) as max_year;
|
|
"
|
|
```
|
|
|
|
---
|
|
|
|
## GitLab CI/CD Integration
|
|
|
|
The pipeline automatically imports SQL files from `output/` directory during deployment (/.gitlab-ci.yml lines 89-98):
|
|
- data/vehicle-etl/output/01_engines.sql
|
|
- data/vehicle-etl/output/02_transmissions.sql
|
|
- data/vehicle-etl/output/03_vehicle_options.sql
|
|
|
|
Commit updated SQL files to trigger deployment with new data.
|
|
|
|
---
|
|
|
|
## Legacy Scripts (Not Used)
|
|
|
|
The following scripts are legacy from the VehAPI integration and are no longer used:
|
|
- vehapi_fetch_snapshot.py (obsolete - VehAPI not used)
|
|
- etl_generate_sql.py (obsolete - database export used instead)
|
|
|
|
These scripts are preserved for historical reference but should not be executed.
|