motovaultpro/docs/CICD-DEPLOY.md

# MotoVaultPro GitLab CI/CD Deployment Guide

Complete guide for deploying MotoVaultPro using GitLab CI/CD with shell executor runners.

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [GitLab Runner Setup](#gitlab-runner-setup)
3. [CI/CD Variables Configuration](#cicd-variables-configuration)
4. [Secrets Architecture](#secrets-architecture)
5. [Pipeline Overview](#pipeline-overview)
6. [Deployment Process](#deployment-process)
7. [Rollback Procedure](#rollback-procedure)
8. [Troubleshooting](#troubleshooting)

---

## Prerequisites

### Server Requirements

- Linux server with Docker Engine installed
- Docker Compose v2 (plugin version)
- GitLab Runner installed and registered
- Git installed
- curl installed (for health checks)

### GitLab Requirements

- GitLab 18.6+ (tested with 18.6.2)
- Project with CI/CD enabled
- Protected `main` branch
- Maintainer access for CI/CD variable configuration

---

## GitLab Runner Setup

### 1. Verify Runner Registration

```bash
sudo gitlab-runner verify
```

Expected output should show your runner as active with shell executor.

### 2. Verify Docker Permissions

The `gitlab-runner` user must have Docker access:

```bash
# Add gitlab-runner to docker group (if not already done)
sudo usermod -aG docker gitlab-runner

# Verify access
sudo -u gitlab-runner docker info
sudo -u gitlab-runner docker compose version
```

### 3. Verify Deployment Directory

Ensure the deployment directory exists and is accessible:

```bash
# Create deployment directory
sudo mkdir -p /opt/motovaultpro
sudo chown gitlab-runner:gitlab-runner /opt/motovaultpro

# Clone repository (first time only)
sudo -u gitlab-runner git clone <repository-url> /opt/motovaultpro
```

---

## CI/CD Variables Configuration

Navigate to **Settings > CI/CD > Variables** in your GitLab project.

### Secrets (File Type Variables)

These variables use GitLab's **File** type, which writes the value to a temporary file and provides the path as the environment variable. This replicates the Kubernetes secrets pattern used by the application.

| Variable Name | Type | Protected | Masked | Description |
|--------------|------|-----------|--------|-------------|
| `POSTGRES_PASSWORD` | File | Yes | Yes | PostgreSQL database password |
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes | Auth0 client secret for backend |
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes | Google Maps API key |
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No | Google Maps Map ID |

### Configuration Variables

| Variable Name | Type | Protected | Masked | Value |
|--------------|------|-----------|--------|-------|
| `VITE_AUTH0_DOMAIN` | Variable | No | No | `motovaultpro.us.auth0.com` |
| `VITE_AUTH0_CLIENT_ID` | Variable | No | No | Your Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | Variable | No | No | `https://api.motovaultpro.com` |

Note: `DEPLOY_PATH` is automatically set in `.gitlab-ci.yml` using `GIT_CLONE_PATH` for a stable path.

### Setting Up a File Type Variable

1. Go to **Settings > CI/CD > Variables**
2. Click **Add variable**
3. Enter the variable key (e.g., `POSTGRES_PASSWORD`)
4. Enter the secret value in the **Value** field
5. Set **Type** to **File**
6. Enable **Protect variable** (recommended)
7. Enable **Mask variable** (for sensitive data)
8. Click **Add variable**

---

## Secrets Architecture

MotoVaultPro uses a Kubernetes-style secrets pattern where secrets are mounted as files at `/run/secrets/` inside containers.

### How It Works

1. **GitLab stores secrets** as File type CI/CD variables
2. **During pipeline execution**, GitLab writes each secret to a temporary file
3. **The `inject-secrets.sh` script** copies these files to `secrets/app/` directory
4. **Docker Compose** mounts these files to `/run/secrets/` in containers
5. **Application code** reads secrets from the filesystem (not environment variables)

### Secret Files

```
secrets/app/
  postgres-password.txt     -> /run/secrets/postgres-password
  auth0-client-secret.txt   -> /run/secrets/auth0-client-secret
  google-maps-api-key.txt   -> /run/secrets/google-maps-api-key
  google-maps-map-id.txt    -> /run/secrets/google-maps-map-id
```

### Security Benefits

- Secrets never appear as environment variables (not visible in `env` or `printenv`)
- File permissions (600) restrict access
- Masked variables prevent accidental log exposure
- Protected variables only available on protected branches

---

## Pipeline Overview

The CI/CD pipeline consists of four stages:

### Stage 1: Validate

Verifies deployment prerequisites:
- Docker is accessible
- Docker Compose is available
- Deployment directory exists

### Stage 2: Build

Builds Docker images:
- Pulls latest code from repository
- Builds all service images with `--no-cache`

### Stage 3: Deploy

Deploys the application:
1. Injects secrets from GitLab variables
2. Stops existing services gracefully
3. Pulls base images
4. Starts database services (PostgreSQL, Redis)
5. Runs database migrations
6. Starts all services

### Stage 4: Verify

Validates deployment health:
- Checks all containers are running
- Tests backend health endpoint
- Reports deployment status

### Pipeline Diagram

```
[Validate] -> [Build] -> [Deploy] -> [Verify]
     |            |          |           |
   Check       Build      Inject      Health
  prereqs     images     secrets      checks
                            |
                         Migrate
                            |
                         Start
                        services
```

---

## Deployment Process

### Automatic Deployment

Deployments are triggered automatically when:
- Code is pushed to the `main` branch
- A merge request is merged into `main`

### Manual Deployment

To trigger a manual deployment:

1. Go to **CI/CD > Pipelines**
2. Click **Run pipeline**
3. Select the `main` branch
4. Click **Run pipeline**

### Deployment Steps (What Happens)

1. **Secrets Injection**
   - `inject-secrets.sh` copies GitLab File variables to `secrets/app/`
   - Permissions are set to 600 for security

2. **Service Shutdown**
   - Existing containers are stopped gracefully (30s timeout)
   - Volumes are preserved

3. **Database Startup**
   - PostgreSQL and Redis start first
   - 15-second wait for database readiness

4. **Migrations**
   - Backend container runs database migrations
   - Ensures schema is up-to-date

5. **Full Service Startup**
   - All services start via `docker compose up -d`
   - Traefik routes traffic automatically

6. **Health Verification**
   - Container status checks
   - Backend health endpoint validation

---

## Rollback Procedure

### Automatic Rollback

If the verify stage fails, the pipeline will report failure but services remain running. Manual intervention is required.

### Manual Rollback

Use the rollback script:

```bash
# SSH to server
ssh user@server

# Run rollback to previous commit
cd /opt/motovaultpro
./scripts/rollback.sh HEAD~1

# Or rollback to specific tag/commit
./scripts/rollback.sh v1.0.0
```

### Rollback Script Details

The script performs:
1. Stops all current services
2. Checks out the specified version
3. Rebuilds images
4. Starts services

### Emergency Recovery

If rollback fails:

```bash
cd /opt/motovaultpro

# Stop everything
docker compose down

# Check git history
git log --oneline -10

# Checkout known working version
git checkout <commit-hash>

# Rebuild and start
docker compose build
docker compose up -d

# Verify
docker compose ps
```

---

## Troubleshooting

### Pipeline Fails at Validate Stage

**Symptom**: `DEPLOY_PATH not found`

**Solution**:
```bash
# Create directory on runner server
sudo mkdir -p /opt/motovaultpro
sudo chown gitlab-runner:gitlab-runner /opt/motovaultpro
```

### Pipeline Fails at Build Stage

**Symptom**: Docker build errors

**Solutions**:
1. Check Dockerfile syntax
2. Verify network connectivity for npm/package downloads
3. Check disk space: `df -h`
4. Clear Docker cache: `docker system prune -a`

### Pipeline Fails at Deploy Stage

**Symptom**: Secrets injection fails

**Solutions**:
1. Verify CI/CD variables are configured correctly
2. Check variable types are set to **File** for secrets
3. Ensure variables are not restricted to specific environments

**Symptom**: Migration fails

**Solutions**:
1. Check database connectivity
2. Verify PostgreSQL is healthy: `docker logs mvp-postgres`
3. Run migrations manually:
   ```bash
   docker compose exec mvp-backend npm run migrate
   ```

### Pipeline Fails at Verify Stage

**Symptom**: Container not running

**Solutions**:
1. Check container logs: `docker logs <container-name>`
2. Verify secrets are correctly mounted
3. Check for port conflicts

**Symptom**: Health check fails

**Solutions**:
1. Wait longer (service might be starting)
2. Check backend logs: `docker logs mvp-backend`
3. Verify database connection

### Services Start But Application Doesn't Work

**Check secrets are mounted**:
```bash
docker compose exec mvp-backend ls -la /run/secrets/
```

**Check configuration**:
```bash
docker compose exec mvp-backend cat /app/config/production.yml
```

**Check network connectivity**:
```bash
docker network ls
docker network inspect motovaultpro_backend
```

### Viewing Logs

```bash
# All services
docker compose logs -f

# Specific service
docker compose logs -f mvp-backend

# Last 100 lines
docker compose logs --tail 100 mvp-backend
```

---

## Maintenance

### Updating Secrets

1. Update the CI/CD variable in GitLab
2. Trigger a new pipeline (push or manual)
3. The new secrets will be injected during deployment

### Database Backups

Backups should be configured separately. Recommended approach:

```bash
# Manual backup
docker compose exec mvp-postgres pg_dump -U postgres motovaultpro > backup.sql

# Automated backup (add to cron)
0 2 * * * cd /opt/motovaultpro && docker compose exec -T mvp-postgres pg_dump -U postgres motovaultpro > /backups/mvp-$(date +\%Y\%m\%d).sql
```

### Monitoring

Consider adding:
- Prometheus metrics (Traefik already configured)
- Health check alerts
- Log aggregation

---

## Quick Reference

### Common Commands

```bash
# View pipeline status
# GitLab UI: CI/CD > Pipelines

# SSH to server
ssh user@your-server

# Navigate to project
cd /opt/motovaultpro

# View running containers
docker compose ps

# View logs
docker compose logs -f

# Restart a service
docker compose restart mvp-backend

# Run migrations manually
docker compose exec mvp-backend npm run migrate

# Access database
docker compose exec mvp-postgres psql -U postgres motovaultpro

# Health check
curl http://localhost:3001/health
```

### Important Paths

| Path | Description |
|------|-------------|
| `$CI_BUILDS_DIR/motovaultpro` | Application root (stable clone path) |
| `$CI_BUILDS_DIR/motovaultpro/secrets/app/` | Secrets directory |
| `$CI_BUILDS_DIR/motovaultpro/data/documents/` | Document storage |
| `$CI_BUILDS_DIR/motovaultpro/config/` | Configuration files |

Note: `CI_BUILDS_DIR` is typically `/opt/gitlab-runner/builds` for shell executors.

### Container Names

| Container | Purpose |
|-----------|---------|
| `mvp-traefik` | Reverse proxy, TLS termination |
| `mvp-frontend` | React SPA |
| `mvp-backend` | Node.js API |
| `mvp-postgres` | PostgreSQL database |
| `mvp-redis` | Redis cache |