Database Tooling
This commit is contained in:
Eric Gullickson
419
docs/DATABASE-MIGRATION.md
Normal file
419
docs/DATABASE-MIGRATION.md
Normal file
@@ -0,0 +1,419 @@
|
||||
# Database Migration Guide
|
||||
|
||||
This guide explains how to export and import the MotoVaultPro database for deployment migration, backups, and disaster recovery.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Export Database
|
||||
|
||||
```bash
|
||||
# Full database export (compressed SQL)
|
||||
./scripts/export-database.sh
|
||||
|
||||
# Schema only (for creating new environments)
|
||||
./scripts/export-database.sh --schema-only
|
||||
|
||||
# Specific tables only
|
||||
./scripts/export-database.sh --include-table vehicles --include-table fuel_logs
|
||||
```
|
||||
|
||||
### Import Database
|
||||
|
||||
```bash
|
||||
# Import into new deployment
|
||||
./scripts/import-database.sh --create-db database-exports/motovaultpro_export_20250101_120000.sql.gz
|
||||
|
||||
# Import with existing database backup
|
||||
./scripts/import-database.sh database-exports/backup.sql.gz
|
||||
|
||||
# Replace existing database (DANGER!)
|
||||
./scripts/import-database.sh --drop-existing --force backup.sql.gz
|
||||
```
|
||||
|
||||
## Export Script
|
||||
|
||||
### Location
|
||||
`scripts/export-database.sh`
|
||||
|
||||
### Features
|
||||
- Multiple export formats (SQL, custom, directory)
|
||||
- Automatic compression
|
||||
- Schema-only or data-only exports
|
||||
- Table filtering (include/exclude)
|
||||
- Metadata generation
|
||||
- Import instructions generation
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
./scripts/export-database.sh [options]
|
||||
|
||||
Options:
|
||||
-h, --help Show help message
|
||||
-f, --format FORMAT Export format: sql, custom, directory (default: sql)
|
||||
-o, --output NAME Custom export filename (without extension)
|
||||
-n, --no-compress Don't compress the export
|
||||
--schema-only Export schema only (no data)
|
||||
--data-only Export data only (no schema)
|
||||
--exclude-table TABLE Exclude specific table(s)
|
||||
--include-table TABLE Include only specific table(s)
|
||||
-c, --container NAME Container name (default: mvp-postgres)
|
||||
```
|
||||
|
||||
### Export Formats
|
||||
|
||||
#### SQL Format (Default)
|
||||
- Plain text SQL dump
|
||||
- Human-readable
|
||||
- Automatically compressed with gzip
|
||||
- Best for version control and manual review
|
||||
|
||||
```bash
|
||||
./scripts/export-database.sh --format sql
|
||||
```
|
||||
|
||||
#### Custom Format
|
||||
- PostgreSQL custom format
|
||||
- Binary and compressed
|
||||
- Fastest for large databases
|
||||
- Supports parallel restore
|
||||
|
||||
```bash
|
||||
./scripts/export-database.sh --format custom
|
||||
```
|
||||
|
||||
#### Directory Format
|
||||
- Each table in separate file
|
||||
- Best for selective restore
|
||||
- Supports parallel processing
|
||||
|
||||
```bash
|
||||
./scripts/export-database.sh --format directory
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
#### Full Production Backup
|
||||
```bash
|
||||
./scripts/export-database.sh --output production_backup_$(date +%Y%m%d)
|
||||
```
|
||||
|
||||
#### Schema for New Environment
|
||||
```bash
|
||||
./scripts/export-database.sh --schema-only --output schema_template
|
||||
```
|
||||
|
||||
#### Specific Feature Data
|
||||
```bash
|
||||
./scripts/export-database.sh \
|
||||
--include-table vehicles \
|
||||
--include-table fuel_logs \
|
||||
--include-table maintenance_records \
|
||||
--output vehicle_data_backup
|
||||
```
|
||||
|
||||
#### Exclude Large Tables
|
||||
```bash
|
||||
./scripts/export-database.sh \
|
||||
--exclude-table audit_logs \
|
||||
--exclude-table system_events \
|
||||
--output core_data_backup
|
||||
```
|
||||
|
||||
### Output Files
|
||||
|
||||
Each export creates three files:
|
||||
|
||||
1. **Export File** (`.sql.gz`, `.dump`, or `.dir/`)
|
||||
- The actual database dump
|
||||
|
||||
2. **Metadata File** (`*_metadata.json`)
|
||||
```json
|
||||
{
|
||||
"export_timestamp": "2025-01-01T12:00:00Z",
|
||||
"database_name": "motovaultpro",
|
||||
"export_format": "sql",
|
||||
"compressed": true,
|
||||
"schema_included": true,
|
||||
"data_included": true,
|
||||
"postgresql_version": "PostgreSQL 15.x",
|
||||
"file_path": "/path/to/export.sql.gz",
|
||||
"file_size": "5.2M"
|
||||
}
|
||||
```
|
||||
|
||||
3. **Import Instructions** (`*_import_instructions.txt`)
|
||||
- Step-by-step import guide
|
||||
- Format-specific commands
|
||||
- Database preparation steps
|
||||
|
||||
## Import Script
|
||||
|
||||
### Location
|
||||
`scripts/import-database.sh`
|
||||
|
||||
### Features
|
||||
- Auto-detects export format
|
||||
- Automatic backup before import
|
||||
- Safety confirmations
|
||||
- Database creation
|
||||
- Import verification
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
./scripts/import-database.sh [options] <export-file>
|
||||
|
||||
Options:
|
||||
-h, --help Show help message
|
||||
-c, --container NAME Container name (default: mvp-postgres)
|
||||
-d, --database NAME Database name (default: motovaultpro)
|
||||
--create-db Create database if it doesn't exist
|
||||
--drop-existing Drop existing database before import (DANGER!)
|
||||
--no-backup Skip backup of existing database
|
||||
--force Skip confirmation prompts
|
||||
-f, --format FORMAT Import format (auto-detected if not specified)
|
||||
```
|
||||
|
||||
### Safety Features
|
||||
|
||||
1. **Automatic Backup**
|
||||
- Creates backup of existing database before destructive operations
|
||||
- Backup stored in `database-exports/`
|
||||
|
||||
2. **Confirmation Prompts**
|
||||
- Requires explicit "yes" for dangerous operations
|
||||
- Can be bypassed with `--force` for automation
|
||||
|
||||
3. **Validation**
|
||||
- Verifies export file exists
|
||||
- Checks PostgreSQL container is running
|
||||
- Confirms successful import
|
||||
|
||||
### Examples
|
||||
|
||||
#### Import into New Deployment
|
||||
```bash
|
||||
# Ensure database doesn't exist
|
||||
./scripts/import-database.sh --create-db backup.sql.gz
|
||||
```
|
||||
|
||||
#### Import with Backup
|
||||
```bash
|
||||
# Automatically backs up existing database
|
||||
./scripts/import-database.sh backup.sql.gz
|
||||
```
|
||||
|
||||
#### Replace Existing Database
|
||||
```bash
|
||||
# Interactive confirmation required
|
||||
./scripts/import-database.sh --drop-existing backup.sql.gz
|
||||
|
||||
# Automated (use with caution!)
|
||||
./scripts/import-database.sh --drop-existing --force backup.sql.gz
|
||||
```
|
||||
|
||||
#### Import Different Format
|
||||
```bash
|
||||
# Custom format
|
||||
./scripts/import-database.sh --format custom backup.dump
|
||||
|
||||
# Directory format
|
||||
./scripts/import-database.sh --format directory backup.dir/
|
||||
```
|
||||
|
||||
## Deployment Migration Workflow
|
||||
|
||||
### 1. Export from Source
|
||||
|
||||
```bash
|
||||
# On source server
|
||||
cd /path/to/motovaultpro
|
||||
./scripts/export-database.sh --output migration_$(date +%Y%m%d)
|
||||
```
|
||||
|
||||
### 2. Transfer to Target
|
||||
|
||||
```bash
|
||||
# Copy export files
|
||||
scp database-exports/migration_*.{sql.gz,json,txt} user@target:/path/to/motovaultpro/database-exports/
|
||||
```
|
||||
|
||||
### 3. Import on Target
|
||||
|
||||
```bash
|
||||
# On target server
|
||||
cd /path/to/motovaultpro
|
||||
|
||||
# First time setup
|
||||
./scripts/import-database.sh --create-db database-exports/migration_20250101.sql.gz
|
||||
|
||||
# Subsequent imports
|
||||
./scripts/import-database.sh --drop-existing database-exports/migration_20250101.sql.gz
|
||||
```
|
||||
|
||||
### 4. Verify Import
|
||||
|
||||
```bash
|
||||
# Check table count
|
||||
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "\dt"
|
||||
|
||||
# Check row counts
|
||||
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "
|
||||
SELECT schemaname, tablename, n_live_tup as rows
|
||||
FROM pg_stat_user_tables
|
||||
ORDER BY n_live_tup DESC;
|
||||
"
|
||||
```
|
||||
|
||||
## Automated Backups
|
||||
|
||||
### Daily Backup Cron Job
|
||||
|
||||
```bash
|
||||
# Add to crontab (crontab -e)
|
||||
0 2 * * * cd /path/to/motovaultpro && ./scripts/export-database.sh --output daily_backup_$(date +%Y%m%d) >> /var/log/motovaultpro-backup.log 2>&1
|
||||
```
|
||||
|
||||
### Weekly Full Backup
|
||||
|
||||
```bash
|
||||
# Weekly on Sunday at 3 AM
|
||||
0 3 * * 0 cd /path/to/motovaultpro && ./scripts/export-database.sh --format custom --output weekly_backup_$(date +%Y%m%d) >> /var/log/motovaultpro-backup.log 2>&1
|
||||
```
|
||||
|
||||
### Retention Script
|
||||
|
||||
```bash
|
||||
# Keep last 7 daily backups and 4 weekly backups
|
||||
find database-exports/ -name "daily_backup_*.sql.gz" -mtime +7 -delete
|
||||
find database-exports/ -name "weekly_backup_*.dump" -mtime +28 -delete
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Export Issues
|
||||
|
||||
#### Container Not Running
|
||||
```bash
|
||||
Error: PostgreSQL container 'mvp-postgres' is not running
|
||||
|
||||
Solution:
|
||||
docker compose up -d mvp-postgres
|
||||
```
|
||||
|
||||
#### Permission Denied
|
||||
```bash
|
||||
Error: Permission denied: /database-exports/
|
||||
|
||||
Solution:
|
||||
chmod +x scripts/export-database.sh
|
||||
mkdir -p database-exports
|
||||
chmod 755 database-exports
|
||||
```
|
||||
|
||||
### Import Issues
|
||||
|
||||
#### Database Already Exists
|
||||
```bash
|
||||
Error: Database 'motovaultpro' does not exist. Use --create-db to create it
|
||||
|
||||
Solution:
|
||||
./scripts/import-database.sh --create-db backup.sql.gz
|
||||
```
|
||||
|
||||
#### Import Failed Mid-Way
|
||||
```bash
|
||||
# Restore from automatic backup
|
||||
./scripts/import-database.sh --drop-existing database-exports/motovaultpro_backup_*.sql.gz
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Production Exports
|
||||
1. **Schedule during low-traffic periods**
|
||||
2. **Use custom format for large databases** (faster, compressed)
|
||||
3. **Store exports offsite** (S3, backup server)
|
||||
4. **Test restores regularly** (monthly verification)
|
||||
5. **Keep multiple generations** (daily, weekly, monthly)
|
||||
|
||||
### Development Imports
|
||||
1. **Use schema-only for clean environments**
|
||||
2. **Sanitize production data** before importing to dev
|
||||
3. **Keep test data exports** for consistent testing
|
||||
4. **Document data dependencies** between tables
|
||||
|
||||
### Security
|
||||
1. **Encrypt exports in transit** (SCP with SSH keys)
|
||||
2. **Encrypt exports at rest** (filesystem encryption)
|
||||
3. **Limit export access** (sudo/docker group only)
|
||||
4. **Audit export operations** (log all exports)
|
||||
5. **Rotate encryption keys** regularly
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Incremental Backups
|
||||
|
||||
For very large databases, use PostgreSQL WAL archiving:
|
||||
|
||||
```bash
|
||||
# Enable WAL archiving in postgresql.conf
|
||||
wal_level = replica
|
||||
archive_mode = on
|
||||
archive_command = 'cp %p /path/to/archive/%f'
|
||||
|
||||
# Base backup
|
||||
docker exec mvp-postgres pg_basebackup -D /backup/base -Ft -z -P
|
||||
|
||||
# Restore with WAL replay
|
||||
# See PostgreSQL documentation for WAL restore process
|
||||
```
|
||||
|
||||
### Selective Table Restore
|
||||
|
||||
```bash
|
||||
# Export specific table
|
||||
./scripts/export-database.sh --include-table vehicles --output vehicles_only
|
||||
|
||||
# Import into existing database (appends data)
|
||||
gunzip -c database-exports/vehicles_only.sql.gz | \
|
||||
docker exec -i mvp-postgres psql -U postgres -d motovaultpro
|
||||
```
|
||||
|
||||
### Data Migration Between Versions
|
||||
|
||||
```bash
|
||||
# Export from old version
|
||||
./scripts/export-database.sh --format custom --output migration_v1_to_v2
|
||||
|
||||
# Run migrations on target
|
||||
docker exec mvp-backend node dist/_system/migrations/run-all.js
|
||||
|
||||
# Import data
|
||||
./scripts/import-database.sh migration_v1_to_v2.dump
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Export Size Tracking
|
||||
|
||||
```bash
|
||||
# Track export sizes over time
|
||||
ls -lh database-exports/*.sql.gz | awk '{print $5, $9}' >> backup-sizes.log
|
||||
```
|
||||
|
||||
### Import Duration
|
||||
|
||||
The import script automatically logs duration. For detailed monitoring:
|
||||
|
||||
```bash
|
||||
time ./scripts/import-database.sh backup.sql.gz
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check the import instructions file generated with each export
|
||||
2. Review logs in `docker logs mvp-postgres`
|
||||
3. Consult PostgreSQL documentation for pg_dump/pg_restore
|
||||
4. Create an issue in the project repository
|
||||
@@ -9,6 +9,7 @@ Project documentation hub for the hybrid platform (platform microservices) and m
|
||||
- Vehicles API (authoritative): `docs/VEHICLES-API.md`
|
||||
- Database schema: `docs/DATABASE-SCHEMA.md`
|
||||
- Testing (containers only): `docs/TESTING.md`
|
||||
- Database Migration: `docs/DATABASE-MIGRATION.md`
|
||||
- Development commands: `Makefile`, `docker-compose.yml`
|
||||
- Application features (start at each README):
|
||||
- `backend/src/features/vehicles/README.md`
|
||||
|
||||
Reference in New Issue
Block a user