motovaultpro/docs/DATABASE-MIGRATION.md

Database Migration Guide

This guide explains how to export and import the MotoVaultPro database for deployment migration, backups, and disaster recovery.

Quick Start

Export Database

# Full database export (compressed SQL)
./scripts/export-database.sh

# Schema only (for creating new environments)
./scripts/export-database.sh --schema-only

# Specific tables only
./scripts/export-database.sh --include-table vehicles --include-table fuel_logs

Import Database

# Import into new deployment
./scripts/import-database.sh --create-db database-exports/motovaultpro_export_20250101_120000.sql.gz

# Import with existing database backup
./scripts/import-database.sh database-exports/backup.sql.gz

# Replace existing database (DANGER!)
./scripts/import-database.sh --drop-existing --force backup.sql.gz

Export Script

Location

scripts/export-database.sh

Features

• Multiple export formats (SQL, custom, directory)
• Automatic compression
• Schema-only or data-only exports
• Table filtering (include/exclude)
• Metadata generation
• Import instructions generation

Options

./scripts/export-database.sh [options]

Options:
  -h, --help              Show help message
  -f, --format FORMAT     Export format: sql, custom, directory (default: sql)
  -o, --output NAME       Custom export filename (without extension)
  -n, --no-compress       Don't compress the export
  --schema-only           Export schema only (no data)
  --data-only             Export data only (no schema)
  --exclude-table TABLE   Exclude specific table(s)
  --include-table TABLE   Include only specific table(s)
  -c, --container NAME    Container name (default: mvp-postgres)

Export Formats

SQL Format (Default)

• Plain text SQL dump
• Human-readable
• Automatically compressed with gzip
• Best for version control and manual review

./scripts/export-database.sh --format sql

Custom Format

• PostgreSQL custom format
• Binary and compressed
• Fastest for large databases
• Supports parallel restore

./scripts/export-database.sh --format custom

Directory Format

• Each table in separate file
• Best for selective restore
• Supports parallel processing

./scripts/export-database.sh --format directory

Examples

Full Production Backup

./scripts/export-database.sh --output production_backup_$(date +%Y%m%d)

Schema for New Environment

./scripts/export-database.sh --schema-only --output schema_template

Specific Feature Data

./scripts/export-database.sh \
  --include-table vehicles \
  --include-table fuel_logs \
  --include-table maintenance_records \
  --output vehicle_data_backup

Exclude Large Tables

./scripts/export-database.sh \
  --exclude-table audit_logs \
  --exclude-table system_events \
  --output core_data_backup

Output Files

Each export creates three files:

1. Export File (.sql.gz, .dump, or .dir/)

   • The actual database dump

2. Metadata File (*_metadata.json)

   {
     "export_timestamp": "2025-01-01T12:00:00Z",
     "database_name": "motovaultpro",
     "export_format": "sql",
     "compressed": true,
     "schema_included": true,
     "data_included": true,
     "postgresql_version": "PostgreSQL 15.x",
     "file_path": "/path/to/export.sql.gz",
     "file_size": "5.2M"
   }
   

3. Import Instructions (*_import_instructions.txt)

   • Step-by-step import guide
   • Format-specific commands
   • Database preparation steps

Import Script

Location

scripts/import-database.sh

Features

• Auto-detects export format
• Automatic backup before import
• Safety confirmations
• Database creation
• Import verification

Options

./scripts/import-database.sh [options] <export-file>

Options:
  -h, --help              Show help message
  -c, --container NAME    Container name (default: mvp-postgres)
  -d, --database NAME     Database name (default: motovaultpro)
  --create-db             Create database if it doesn't exist
  --drop-existing         Drop existing database before import (DANGER!)
  --no-backup             Skip backup of existing database
  --force                 Skip confirmation prompts
  -f, --format FORMAT     Import format (auto-detected if not specified)

Safety Features

1. Automatic Backup

   • Creates backup of existing database before destructive operations
   • Backup stored in database-exports/

2. Confirmation Prompts

   • Requires explicit "yes" for dangerous operations
   • Can be bypassed with --force for automation

3. Validation

   • Verifies export file exists
   • Checks PostgreSQL container is running
   • Confirms successful import

Examples

Import into New Deployment

# Ensure database doesn't exist
./scripts/import-database.sh --create-db backup.sql.gz

Import with Backup

# Automatically backs up existing database
./scripts/import-database.sh backup.sql.gz

Replace Existing Database

# Interactive confirmation required
./scripts/import-database.sh --drop-existing backup.sql.gz

# Automated (use with caution!)
./scripts/import-database.sh --drop-existing --force backup.sql.gz

Import Different Format

# Custom format
./scripts/import-database.sh --format custom backup.dump

# Directory format
./scripts/import-database.sh --format directory backup.dir/

Deployment Migration Workflow

1. Export from Source

# On source server
cd /path/to/motovaultpro
./scripts/export-database.sh --output migration_$(date +%Y%m%d)

2. Transfer to Target

# Copy export files
scp database-exports/migration_*.{sql.gz,json,txt} user@target:/path/to/motovaultpro/database-exports/

3. Import on Target

# On target server
cd /path/to/motovaultpro

# First time setup
./scripts/import-database.sh --create-db database-exports/migration_20250101.sql.gz

# Subsequent imports
./scripts/import-database.sh --drop-existing database-exports/migration_20250101.sql.gz

4. Verify Import

# Check table count
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "\dt"

# Check row counts
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "
  SELECT schemaname, tablename, n_live_tup as rows
  FROM pg_stat_user_tables
  ORDER BY n_live_tup DESC;
"

Automated Backups

Daily Backup Cron Job

# Add to crontab (crontab -e)
0 2 * * * cd /path/to/motovaultpro && ./scripts/export-database.sh --output daily_backup_$(date +%Y%m%d) >> /var/log/motovaultpro-backup.log 2>&1

Weekly Full Backup

# Weekly on Sunday at 3 AM
0 3 * * 0 cd /path/to/motovaultpro && ./scripts/export-database.sh --format custom --output weekly_backup_$(date +%Y%m%d) >> /var/log/motovaultpro-backup.log 2>&1

Retention Script

# Keep last 7 daily backups and 4 weekly backups
find database-exports/ -name "daily_backup_*.sql.gz" -mtime +7 -delete
find database-exports/ -name "weekly_backup_*.dump" -mtime +28 -delete

Troubleshooting

Export Issues

Container Not Running

Error: PostgreSQL container 'mvp-postgres' is not running

Solution:
docker compose up -d mvp-postgres

Permission Denied

Error: Permission denied: /database-exports/

Solution:
chmod +x scripts/export-database.sh
mkdir -p database-exports
chmod 755 database-exports

Import Issues

Database Already Exists

Error: Database 'motovaultpro' does not exist. Use --create-db to create it

Solution:
./scripts/import-database.sh --create-db backup.sql.gz

Import Failed Mid-Way

# Restore from automatic backup
./scripts/import-database.sh --drop-existing database-exports/motovaultpro_backup_*.sql.gz

Best Practices

Production Exports

1. Schedule during low-traffic periods
2. Use custom format for large databases (faster, compressed)
3. Store exports offsite (S3, backup server)
4. Test restores regularly (monthly verification)
5. Keep multiple generations (daily, weekly, monthly)

Development Imports

1. Use schema-only for clean environments
2. Sanitize production data before importing to dev
3. Keep test data exports for consistent testing
4. Document data dependencies between tables

Security

1. Encrypt exports in transit (SCP with SSH keys)
2. Encrypt exports at rest (filesystem encryption)
3. Limit export access (sudo/docker group only)
4. Audit export operations (log all exports)
5. Rotate encryption keys regularly

Advanced Usage

Incremental Backups

For very large databases, use PostgreSQL WAL archiving:

# Enable WAL archiving in postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'cp %p /path/to/archive/%f'

# Base backup
docker exec mvp-postgres pg_basebackup -D /backup/base -Ft -z -P

# Restore with WAL replay
# See PostgreSQL documentation for WAL restore process

Selective Table Restore

# Export specific table
./scripts/export-database.sh --include-table vehicles --output vehicles_only

# Import into existing database (appends data)
gunzip -c database-exports/vehicles_only.sql.gz | \
  docker exec -i mvp-postgres psql -U postgres -d motovaultpro

Data Migration Between Versions

# Export from old version
./scripts/export-database.sh --format custom --output migration_v1_to_v2

# Run migrations on target
docker exec mvp-backend node dist/_system/migrations/run-all.js

# Import data
./scripts/import-database.sh migration_v1_to_v2.dump

Monitoring

Export Size Tracking

# Track export sizes over time
ls -lh database-exports/*.sql.gz | awk '{print $5, $9}' >> backup-sizes.log

Import Duration

The import script automatically logs duration. For detailed monitoring:

time ./scripts/import-database.sh backup.sql.gz

Support

For issues or questions:

1. Check the import instructions file generated with each export
2. Review logs in docker logs mvp-postgres
3. Consult PostgreSQL documentation for pg_dump/pg_restore
4. Create an issue in the project repository