motovaultpro/docs/CICD-DEPLOY.md

# MotoVaultPro GitLab CI/CD Deployment Guide

Complete guide for deploying MotoVaultPro using GitLab CI/CD with blue-green deployment and auto-rollback.

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Prerequisites](#prerequisites)
3. [Pipeline Stages](#pipeline-stages)
4. [Blue-Green Deployment](#blue-green-deployment)
5. [CI/CD Variables Configuration](#cicd-variables-configuration)
6. [Container Registry](#container-registry)
7. [Deployment Process](#deployment-process)
8. [Rollback Procedures](#rollback-procedures)
9. [Maintenance Migrations](#maintenance-migrations)
10. [Notifications](#notifications)
11. [Troubleshooting](#troubleshooting)

---

## Architecture Overview

MotoVaultPro uses a blue-green deployment strategy with automatic rollback:

```
+---------------------------------------------------+
|              GitLab (CI/CD + Registry)            |
+---------------------------------------------------+
         |                           |
         v                           v
+------------------+     +-----------------------+
|   Build VPS      |     |   Production Server   |
| (Docker Runner)  |     | (Shell Runner)        |
| Tags: build      |     | Tags: production      |
+------------------+     +-----------+-----------+
         |                           |
         | Push images               | Pull + Deploy
         v                           v
+---------------------------------------------------+
|              GitLab Container Registry            |
| registry.motovaultpro.com/motovaultpro/           |
+---------------------------------------------------+
                         |
         +---------------+---------------+
         |                               |
+--------v--------+             +--------v--------+
|   BLUE Stack    |             |  GREEN Stack    |
| mvp-frontend    |             | mvp-frontend    |
| mvp-backend     |             | mvp-backend     |
+-----------------+             +-----------------+
         |                               |
         +----------- Traefik -----------+
                  (weighted LB)
                         |
         +---------------+---------------+
         |                               |
+--------v--------+             +--------v--------+
|   PostgreSQL    |             |     Redis       |
|   (shared)      |             |    (shared)     |
+-----------------+             +-----------------+
```

### Key Features

- **Zero-downtime deployments**: Traffic switches in under 5 seconds
- **Instant rollback**: Previous version remains running
- **Automatic rollback**: On health check failure
- **Email notifications**: Via Resend API
- **Container registry**: Self-hosted on GitLab (no Docker Hub)

---

## Prerequisites

### Server Requirements

| Server | Purpose | Specs | Runner Tags |
|--------|---------|-------|-------------|
| Build VPS | Docker image builds | 2 CPU, 4GB RAM | `build` |
| Prod Server | Application hosting | 8GB+ RAM | `production` |

See [BUILD-SERVER-SETUP.md](BUILD-SERVER-SETUP.md) for build server setup.

### Software Requirements

- GitLab 18.6+
- Docker Engine 24.0+
- Docker Compose v2
- GitLab Runner (shell executor on both servers)
- `jq` for JSON processing

---

## Pipeline Stages

The CI/CD pipeline consists of 7 stages:

```
validate -> build -> deploy-prepare -> deploy-switch -> verify -> [rollback] -> notify
```

| Stage | Runner | Purpose |
|-------|--------|---------|
| `validate` | prod | Check prerequisites, determine target stack |
| `build` | build | Build and push images to GitLab registry |
| `deploy-prepare` | prod | Pull images, start inactive stack, health check |
| `deploy-switch` | prod | Switch Traefik traffic weights |
| `verify` | prod | Production health verification |
| `rollback` | prod | Auto-triggered on verify failure |
| `notify` | prod | Email success/failure notifications |

### Pipeline Flow

```
[Push to main]
      |
      v
[validate] - Checks Docker, paths, registry
      |
      v
[build] - Builds backend + frontend images
      |        Pushes to registry.motovaultpro.com
      v
[deploy-prepare] - Pulls new images
      |             Starts inactive stack (blue or green)
      |             Runs health checks
      v
[deploy-switch] - Updates Traefik weights
      |            Switches traffic instantly
      v
[verify] - External health check
      |     Container status verification
      |
      +--[SUCCESS]--> [notify-success] - Sends success email
      |
      +--[FAILURE]--> [rollback] - Switches back to previous stack
                           |
                           v
                    [notify-failure] - Sends failure email
```

---

## Blue-Green Deployment

### Stack Configuration

Both stacks share the same database layer:

| Component | Blue Stack | Green Stack | Shared |
|-----------|------------|-------------|--------|
| Frontend | `mvp-frontend-blue` | `mvp-frontend-green` | - |
| Backend | `mvp-backend-blue` | `mvp-backend-green` | - |
| PostgreSQL | - | - | `mvp-postgres` |
| Redis | - | - | `mvp-redis` |
| Traefik | - | - | `mvp-traefik` |

### Traffic Routing

Traefik uses weighted services for traffic distribution:

```yaml
# config/traefik/dynamic/blue-green.yml
services:
  mvp-frontend-weighted:
    weighted:
      services:
        - name: mvp-frontend-blue-svc
          weight: 100  # Active
        - name: mvp-frontend-green-svc
          weight: 0    # Standby
```

### Deployment State

State is tracked in `config/deployment/state.json`:

```json
{
  "active_stack": "blue",
  "inactive_stack": "green",
  "last_deployment": "2024-01-15T10:30:00Z",
  "last_deployment_commit": "abc123",
  "rollback_available": true
}
```

---

## CI/CD Variables Configuration

Navigate to **Settings > CI/CD > Variables** in your GitLab project.

### Required Variables

| Variable | Type | Protected | Purpose |
|----------|------|-----------|---------|
| `DEPLOY_NOTIFY_EMAIL` | Variable | Yes | Notification recipient |
| `VITE_AUTH0_DOMAIN` | Variable | No | Auth0 domain |
| `VITE_AUTH0_CLIENT_ID` | Variable | No | Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | Variable | No | Auth0 audience |

### Secret Files

These use GitLab's **File** type and are injected via `scripts/inject-secrets.sh`:

| Variable | Type | Protected | Masked |
|----------|------|-----------|--------|
| `POSTGRES_PASSWORD` | File | Yes | Yes |
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes |
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes |
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No |
| `CF_DNS_API_TOKEN` | File | Yes | Yes |
| `RESEND_API_KEY` | File | Yes | Yes |

### Registry Authentication

GitLab provides these automatically:
- `CI_REGISTRY_USER` - Registry username
- `CI_REGISTRY_PASSWORD` - Registry token
- `CI_REGISTRY` - Registry URL

---

## Container Registry

All images are hosted on the GitLab Container Registry to avoid Docker Hub rate limits.

### Registry URL

```
registry.motovaultpro.com
```

### Image Paths

| Image | Path |
|-------|------|
| Backend | `registry.motovaultpro.com/motovaultpro/backend:$TAG` |
| Frontend | `registry.motovaultpro.com/motovaultpro/frontend:$TAG` |
| Mirrors | `registry.motovaultpro.com/mirrors/` |

### Base Image Mirrors

Mirror upstream images to avoid rate limits:

```bash
# Run manually or via scheduled pipeline
./scripts/ci/mirror-base-images.sh
```

Mirrored images:
- `node:20-alpine`
- `nginx:alpine`
- `postgres:18-alpine`
- `redis:8.4-alpine`
- `traefik:v3.6`
- `docker:24.0`
- `docker:24.0-dind`

---

## Deployment Process

### Automatic Deployment

Deployments trigger automatically on push to `main`:

1. **Validate**: Check prerequisites, determine target stack
2. **Build**: Build images on dedicated build server
3. **Prepare**: Start inactive stack, run health checks
4. **Switch**: Update Traefik weights (instant)
5. **Verify**: External health check
6. **Notify**: Send email notification

### Manual Deployment

1. Go to **CI/CD > Pipelines**
2. Click **Run pipeline**
3. Select `main` branch
4. Click **Run pipeline**

### Deployment Timeline

| Phase | Duration |
|-------|----------|
| Validate | ~5s |
| Build | ~2 min |
| Deploy-prepare | ~30s |
| Deploy-switch | ~3s |
| Verify | ~30s |
| **Total** | ~3 min |

---

## Rollback Procedures

### Automatic Rollback

Triggers automatically when:
- Health check fails in `deploy-prepare`
- `verify` stage fails after switch
- Container becomes unhealthy within verification period

The pipeline runs `scripts/ci/auto-rollback.sh` which:
1. Verifies previous stack is healthy
2. Switches traffic back
3. Sends notification

### Manual Rollback

SSH to production server:

```bash
cd /opt/motovaultpro

# Check current state
cat config/deployment/state.json | jq .

# Switch to other stack
./scripts/ci/switch-traffic.sh blue   # or green
```

### Emergency Recovery

If both stacks are unhealthy:

```bash
# Stop everything
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml down

# Restart shared services
docker compose up -d mvp-postgres mvp-redis mvp-traefik

# Wait for database
sleep 15

# Start one stack
export BACKEND_IMAGE=registry.motovaultpro.com/motovaultpro/backend:latest
export FRONTEND_IMAGE=registry.motovaultpro.com/motovaultpro/frontend:latest
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
  mvp-frontend-blue mvp-backend-blue

# Switch traffic
./scripts/ci/switch-traffic.sh blue
```

---

## Maintenance Migrations

For breaking database changes requiring downtime:

### Via Pipeline (Recommended)

1. Go to **CI/CD > Pipelines**
2. Find the `maintenance-migration` job
3. Click **Play** to trigger manually

### Via Script

```bash
cd /opt/motovaultpro

# With backup
./scripts/ci/maintenance-migrate.sh backup

# Without backup
./scripts/ci/maintenance-migrate.sh
```

### What Happens

1. Sends maintenance notification
2. Enables maintenance mode (stops traffic)
3. Creates database backup (if requested)
4. Runs migrations
5. Restarts backends
6. Restores traffic
7. Sends completion notification

---

## Notifications

Email notifications via Resend API for:

| Event | Subject |
|-------|---------|
| `success` | Deployment Successful |
| `failure` | Deployment Failed |
| `rollback` | Auto-Rollback Executed |
| `rollback_failed` | CRITICAL: Rollback Failed |
| `maintenance_start` | Maintenance Mode Started |
| `maintenance_end` | Maintenance Complete |

Configure recipient in GitLab CI/CD variables:
```
DEPLOY_NOTIFY_EMAIL = admin@example.com
```

---

## Troubleshooting

### Pipeline Fails at Build Stage

**Check build server connectivity:**
```bash
# On build server
sudo gitlab-runner verify
docker login registry.motovaultpro.com
```

**Check disk space:**
```bash
df -h
docker system prune -af
```

### Pipeline Fails at Deploy-Prepare

**Container won't start:**
```bash
docker logs mvp-backend-blue --tail 100
docker logs mvp-frontend-blue --tail 100
```

**Health check timeout:**
```bash
# Increase timeout in .gitlab-ci.yml
HEALTH_CHECK_TIMEOUT: "90"
```

### Traffic Not Switching

**Check Traefik config:**
```bash
cat config/traefik/dynamic/blue-green.yml
docker exec mvp-traefik traefik healthcheck
```

**Check routing:**
```bash
curl -I https://motovaultpro.com/api/health
```

### Verify Stage Fails

**Check external connectivity:**
```bash
curl -sf https://motovaultpro.com/api/health
```

**Check container health:**
```bash
docker inspect --format='{{.State.Health.Status}}' mvp-backend-blue
```

---

## Quick Reference

### Important Paths

| Path | Description |
|------|-------------|
| `config/deployment/state.json` | Deployment state |
| `config/traefik/dynamic/blue-green.yml` | Traffic routing |
| `scripts/ci/` | Deployment scripts |

### Common Commands

```bash
# View current state
cat config/deployment/state.json | jq .

# Check container status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Health}}"

# View logs
docker logs mvp-backend-blue -f

# Manual traffic switch
./scripts/ci/switch-traffic.sh green

# Run health check
./scripts/ci/health-check.sh blue

# Send test notification
./scripts/ci/notify.sh success "Test message"
```

### Memory Budget (8GB Server)

| Component | RAM |
|-----------|-----|
| Blue frontend | 512MB |
| Blue backend | 1GB |
| Green frontend | 512MB |
| Green backend | 1GB |
| PostgreSQL | 2GB |
| Redis | 512MB |
| Traefik | 128MB |
| System | 1.3GB |
| **Total** | ~7GB |