motovaultpro/docs/DATABASE-MIGRATION.md

# Database Migration Guide

This guide explains how to export and import the MotoVaultPro database for deployment migration, backups, and disaster recovery.

## Quick Start

### Export Database

```bash
# Full database export (compressed SQL)
./scripts/export-database.sh

# Schema only (for creating new environments)
./scripts/export-database.sh --schema-only

# Specific tables only
./scripts/export-database.sh --include-table vehicles --include-table fuel_logs
```

### Import Database

```bash
# Import into new deployment
./scripts/import-database.sh --create-db database-exports/motovaultpro_export_20250101_120000.sql.gz

# Import with existing database backup
./scripts/import-database.sh database-exports/backup.sql.gz

# Replace existing database (DANGER!)
./scripts/import-database.sh --drop-existing --force backup.sql.gz
```

## Export Script

### Location
`scripts/export-database.sh`

### Features
- Multiple export formats (SQL, custom, directory)
- Automatic compression
- Schema-only or data-only exports
- Table filtering (include/exclude)
- Metadata generation
- Import instructions generation

### Options

```bash
./scripts/export-database.sh [options]

Options:
  -h, --help              Show help message
  -f, --format FORMAT     Export format: sql, custom, directory (default: sql)
  -o, --output NAME       Custom export filename (without extension)
  -n, --no-compress       Don't compress the export
  --schema-only           Export schema only (no data)
  --data-only             Export data only (no schema)
  --exclude-table TABLE   Exclude specific table(s)
  --include-table TABLE   Include only specific table(s)
  -c, --container NAME    Container name (default: mvp-postgres)
```

### Export Formats

#### SQL Format (Default)
- Plain text SQL dump
- Human-readable
- Automatically compressed with gzip
- Best for version control and manual review

```bash
./scripts/export-database.sh --format sql
```

#### Custom Format
- PostgreSQL custom format
- Binary and compressed
- Fastest for large databases
- Supports parallel restore

```bash
./scripts/export-database.sh --format custom
```

#### Directory Format
- Each table in separate file
- Best for selective restore
- Supports parallel processing

```bash
./scripts/export-database.sh --format directory
```

### Examples

#### Full Production Backup
```bash
./scripts/export-database.sh --output production_backup_$(date +%Y%m%d)
```

#### Schema for New Environment
```bash
./scripts/export-database.sh --schema-only --output schema_template
```

#### Specific Feature Data
```bash
./scripts/export-database.sh \
  --include-table vehicles \
  --include-table fuel_logs \
  --include-table maintenance_records \
  --output vehicle_data_backup
```

#### Exclude Large Tables
```bash
./scripts/export-database.sh \
  --exclude-table audit_logs \
  --exclude-table system_events \
  --output core_data_backup
```

### Output Files

Each export creates three files:

1. **Export File** (`.sql.gz`, `.dump`, or `.dir/`)
   - The actual database dump

2. **Metadata File** (`*_metadata.json`)
   ```json
   {
     "export_timestamp": "2025-01-01T12:00:00Z",
     "database_name": "motovaultpro",
     "export_format": "sql",
     "compressed": true,
     "schema_included": true,
     "data_included": true,
     "postgresql_version": "PostgreSQL 15.x",
     "file_path": "/path/to/export.sql.gz",
     "file_size": "5.2M"
   }
   ```

3. **Import Instructions** (`*_import_instructions.txt`)
   - Step-by-step import guide
   - Format-specific commands
   - Database preparation steps

## Import Script

### Location
`scripts/import-database.sh`

### Features
- Auto-detects export format
- Automatic backup before import
- Safety confirmations
- Database creation
- Import verification

### Options

```bash
./scripts/import-database.sh [options] <export-file>

Options:
  -h, --help              Show help message
  -c, --container NAME    Container name (default: mvp-postgres)
  -d, --database NAME     Database name (default: motovaultpro)
  --create-db             Create database if it doesn't exist
  --drop-existing         Drop existing database before import (DANGER!)
  --no-backup             Skip backup of existing database
  --force                 Skip confirmation prompts
  -f, --format FORMAT     Import format (auto-detected if not specified)
```

### Safety Features

1. **Automatic Backup**
   - Creates backup of existing database before destructive operations
   - Backup stored in `database-exports/`

2. **Confirmation Prompts**
   - Requires explicit "yes" for dangerous operations
   - Can be bypassed with `--force` for automation

3. **Validation**
   - Verifies export file exists
   - Checks PostgreSQL container is running
   - Confirms successful import

### Examples

#### Import into New Deployment
```bash
# Ensure database doesn't exist
./scripts/import-database.sh --create-db backup.sql.gz
```

#### Import with Backup
```bash
# Automatically backs up existing database
./scripts/import-database.sh backup.sql.gz
```

#### Replace Existing Database
```bash
# Interactive confirmation required
./scripts/import-database.sh --drop-existing backup.sql.gz

# Automated (use with caution!)
./scripts/import-database.sh --drop-existing --force backup.sql.gz
```

#### Import Different Format
```bash
# Custom format
./scripts/import-database.sh --format custom backup.dump

# Directory format
./scripts/import-database.sh --format directory backup.dir/
```

## Deployment Migration Workflow

### 1. Export from Source

```bash
# On source server
cd /path/to/motovaultpro
./scripts/export-database.sh --output migration_$(date +%Y%m%d)
```

### 2. Transfer to Target

```bash
# Copy export files
scp database-exports/migration_*.{sql.gz,json,txt} user@target:/path/to/motovaultpro/database-exports/
```

### 3. Import on Target

```bash
# On target server
cd /path/to/motovaultpro

# First time setup
./scripts/import-database.sh --create-db database-exports/migration_20250101.sql.gz

# Subsequent imports
./scripts/import-database.sh --drop-existing database-exports/migration_20250101.sql.gz
```

### 4. Verify Import

```bash
# Check table count
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "\dt"

# Check row counts
docker exec mvp-postgres psql -U postgres -d motovaultpro -c "
  SELECT schemaname, tablename, n_live_tup as rows
  FROM pg_stat_user_tables
  ORDER BY n_live_tup DESC;
"
```

## Automated Backups

### Daily Backup Cron Job

```bash
# Add to crontab (crontab -e)
0 2 * * * cd /path/to/motovaultpro && ./scripts/export-database.sh --output daily_backup_$(date +%Y%m%d) >> /var/log/motovaultpro-backup.log 2>&1
```

### Weekly Full Backup

```bash
# Weekly on Sunday at 3 AM
0 3 * * 0 cd /path/to/motovaultpro && ./scripts/export-database.sh --format custom --output weekly_backup_$(date +%Y%m%d) >> /var/log/motovaultpro-backup.log 2>&1
```

### Retention Script

```bash
# Keep last 7 daily backups and 4 weekly backups
find database-exports/ -name "daily_backup_*.sql.gz" -mtime +7 -delete
find database-exports/ -name "weekly_backup_*.dump" -mtime +28 -delete
```

## Troubleshooting

### Export Issues

#### Container Not Running
```bash
Error: PostgreSQL container 'mvp-postgres' is not running

Solution:
docker compose up -d mvp-postgres
```

#### Permission Denied
```bash
Error: Permission denied: /database-exports/

Solution:
chmod +x scripts/export-database.sh
mkdir -p database-exports
chmod 755 database-exports
```

### Import Issues

#### Database Already Exists
```bash
Error: Database 'motovaultpro' does not exist. Use --create-db to create it

Solution:
./scripts/import-database.sh --create-db backup.sql.gz
```

#### Import Failed Mid-Way
```bash
# Restore from automatic backup
./scripts/import-database.sh --drop-existing database-exports/motovaultpro_backup_*.sql.gz
```

## Best Practices

### Production Exports
1. **Schedule during low-traffic periods**
2. **Use custom format for large databases** (faster, compressed)
3. **Store exports offsite** (S3, backup server)
4. **Test restores regularly** (monthly verification)
5. **Keep multiple generations** (daily, weekly, monthly)

### Development Imports
1. **Use schema-only for clean environments**
2. **Sanitize production data** before importing to dev
3. **Keep test data exports** for consistent testing
4. **Document data dependencies** between tables

### Security
1. **Encrypt exports in transit** (SCP with SSH keys)
2. **Encrypt exports at rest** (filesystem encryption)
3. **Limit export access** (sudo/docker group only)
4. **Audit export operations** (log all exports)
5. **Rotate encryption keys** regularly

## Advanced Usage

### Incremental Backups

For very large databases, use PostgreSQL WAL archiving:

```bash
# Enable WAL archiving in postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'cp %p /path/to/archive/%f'

# Base backup
docker exec mvp-postgres pg_basebackup -D /backup/base -Ft -z -P

# Restore with WAL replay
# See PostgreSQL documentation for WAL restore process
```

### Selective Table Restore

```bash
# Export specific table
./scripts/export-database.sh --include-table vehicles --output vehicles_only

# Import into existing database (appends data)
gunzip -c database-exports/vehicles_only.sql.gz | \
  docker exec -i mvp-postgres psql -U postgres -d motovaultpro
```

### Data Migration Between Versions

```bash
# Export from old version
./scripts/export-database.sh --format custom --output migration_v1_to_v2

# Run migrations on target
docker exec mvp-backend node dist/_system/migrations/run-all.js

# Import data
./scripts/import-database.sh migration_v1_to_v2.dump
```

## Monitoring

### Export Size Tracking

```bash
# Track export sizes over time
ls -lh database-exports/*.sql.gz | awk '{print $5, $9}' >> backup-sizes.log
```

### Import Duration

The import script automatically logs duration. For detailed monitoring:

```bash
time ./scripts/import-database.sh backup.sql.gz
```

## Support

For issues or questions:
1. Check the import instructions file generated with each export
2. Review logs in `docker logs mvp-postgres`
3. Consult PostgreSQL documentation for pg_dump/pg_restore
4. Create an issue in the project repository