motovaultpro/docs/CICD-DEPLOY.md

# MotoVaultPro CI/CD Deployment Guide

Complete guide for deploying MotoVaultPro using Gitea Actions with staging-first deployment and manual production approval.

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Prerequisites](#prerequisites)
3. [Workflow Structure](#workflow-structure)
4. [Deployment Process](#deployment-process)
5. [Secrets and Variables](#secrets-and-variables)
6. [Container Registry](#container-registry)
7. [Rollback Procedures](#rollback-procedures)
8. [Maintenance Migrations](#maintenance-migrations)
9. [Troubleshooting](#troubleshooting)

---

## Architecture Overview

MotoVaultPro uses a **staging-first deployment** strategy with manual approval:

```
+---------------------------------------------------+
|         Gitea (git.motovaultpro.com)              |
|  +--------------------+  +--------------------+   |
|  |   Gitea Actions    |  | Package Registry   |   |
|  +--------------------+  +--------------------+   |
+---------------------------------------------------+
         |                           |
         v                           v
+-------------------+     +--------------------+
| Build/Staging VPS |     | Production Server  |
| (mvp-build)       |     | (mvp-prod)         |
| act_runner        |     | act_runner         |
+--------+----------+     +----------+---------+
         |                           |
         v                           v
staging.motovaultpro.com    motovaultpro.com
   (Full Stack)             (Blue-Green)
```

### Key Features

- **Staging-first**: All changes verified on staging before production
- **Manual approval**: Production deploy requires manual trigger
- **Blue-green production**: Zero-downtime deployments
- **Auto-rollback**: Automatic rollback on health check failure
- **Email notifications**: Via Resend API

---

## Prerequisites

### Server Requirements

| Server | Purpose | Specs | Runner Label |
|--------|---------|-------|--------------|
| Build/Staging VPS | Build + Staging | 4 CPU, 8GB RAM | `mvp-build` |
| Prod Server | Production | 8GB+ RAM | `mvp-prod` |

See [BUILD-SERVER-SETUP.md](BUILD-SERVER-SETUP.md) for setup instructions.

### Software Requirements

- Gitea 1.21+ with Actions enabled
- Docker Engine 24.0+
- Docker Compose v2
- act_runner (Gitea Actions runner)
- `jq` for JSON processing

---

## Workflow Structure

### Two-Workflow Strategy

| Workflow | Trigger | Purpose |
|----------|---------|---------|
| `staging.yaml` | Push to main | Build, deploy to staging, verify |
| `production.yaml` | Manual (workflow_dispatch) | Deploy to production |

### Staging Workflow (Automatic)

```
[Push to main]
      |
      v
[build] -------- Build images, push to registry
      |
      v
[deploy-staging] - Deploy full stack to staging
      |
      v
[verify-staging] - Health checks
      |
      +--[FAIL]--> [notify-staging-failure]
      |
      v
[notify-staging-ready] - Email with production deploy link
```

### Production Workflow (Manual)

```
[Manual Trigger] - User clicks "Run workflow"
      |
      v
[validate] - Check prerequisites, determine stack
      |
      v
[deploy-prod] - Blue-green deployment
      |
      v
[verify-prod] - External health checks
      |
      +--[FAIL]--> [rollback] --> [notify-failure]
      |
      v
[notify-success]
```

---

## Deployment Process

### 1. Push to Main Branch

When you push to `main`:
1. Staging workflow triggers automatically
2. Images are built and pushed to Gitea Package Registry
3. Full stack deploys to staging.motovaultpro.com
4. Health checks verify staging works
5. Email notification sent with production deploy link

### 2. Review Staging

After receiving the "Staging Ready" email:
1. Visit https://staging.motovaultpro.com
2. Test functionality
3. Review logs if needed: `docker logs mvp-backend-staging`

### 3. Deploy to Production

When ready to deploy:
1. Go to `git.motovaultpro.com/egullickson/motovaultpro/actions`
2. Select "Deploy to Production" workflow
3. Click "Run workflow"
4. Optionally specify image tag (defaults to `latest`)
5. Click "Run workflow" to confirm

### 4. Monitor Production Deploy

The production workflow will:
1. Determine target stack (blue or green)
2. Pull and start the new stack
3. Run health checks
4. Switch traffic
5. Verify external health
6. Auto-rollback if verification fails
7. Send email notification

---

## Secrets and Variables

### Secrets Configuration

Navigate to: `git.motovaultpro.com/egullickson/motovaultpro/settings/actions/secrets`

| Secret | Description |
|--------|-------------|
| `REGISTRY_USER` | Gitea username (egullickson) |
| `REGISTRY_PASSWORD` | Gitea access token |
| `POSTGRES_PASSWORD` | Production PostgreSQL password |
| `STAGING_POSTGRES_PASSWORD` | Staging PostgreSQL password |
| `AUTH0_CLIENT_SECRET` | Auth0 secret |
| `AUTH0_MANAGEMENT_CLIENT_ID` | Auth0 Management API ID |
| `AUTH0_MANAGEMENT_CLIENT_SECRET` | Auth0 Management API secret |
| `GOOGLE_MAPS_API_KEY` | Google Maps key |
| `GOOGLE_MAPS_MAP_ID` | Google Maps Map ID |
| `CF_DNS_API_TOKEN` | Cloudflare DNS token |
| `RESEND_API_KEY` | Resend email API key |

### Variables Configuration

Navigate to: `git.motovaultpro.com/egullickson/motovaultpro/settings/actions/variables`

| Variable | Value |
|----------|-------|
| `DEPLOY_NOTIFY_EMAIL` | Notification recipient |
| `VITE_AUTH0_DOMAIN` | motovaultpro.us.auth0.com |
| `VITE_AUTH0_CLIENT_ID` | Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | https://api.motovaultpro.com |

---

## Container Registry

All images are hosted on Gitea Package Registry.

### Registry URL

```
git.motovaultpro.com
```

### Image Paths

| Image | Path |
|-------|------|
| Backend | `git.motovaultpro.com/egullickson/backend:$TAG` |
| Frontend | `git.motovaultpro.com/egullickson/frontend:$TAG` |
| Mirrors | `git.motovaultpro.com/egullickson/mirrors/` |

### Base Image Mirrors

Run the mirror workflow to avoid Docker Hub rate limits:

1. Go to Actions tab
2. Select "Mirror Base Images"
3. Click "Run workflow"

Mirrored images:
- `node:20-alpine`
- `nginx:alpine`
- `postgres:18-alpine`
- `redis:8.4-alpine`
- `traefik:v3.6`

---

## Rollback Procedures

### Automatic Rollback

Production workflow auto-rolls back when:
- Health check fails after traffic switch
- Container becomes unhealthy during verification

### Manual Rollback

SSH to production server:

```bash
cd /opt/motovaultpro

# Check current state
cat config/deployment/state.json | jq .

# Switch to other stack
./scripts/ci/switch-traffic.sh blue   # or green
```

### Emergency Recovery

If both stacks are unhealthy:

```bash
cd /opt/motovaultpro

# Stop everything
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml down

# Restart shared services
docker compose up -d mvp-postgres mvp-redis mvp-traefik

# Wait for database
sleep 15

# Start one stack
export BACKEND_IMAGE=git.motovaultpro.com/egullickson/backend:latest
export FRONTEND_IMAGE=git.motovaultpro.com/egullickson/frontend:latest
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
  mvp-frontend-blue mvp-backend-blue

# Switch traffic
./scripts/ci/switch-traffic.sh blue
```

---

## Maintenance Migrations

For breaking database changes requiring downtime:

### Via Gitea Actions

1. Go to Actions tab
2. Select "Maintenance Migration"
3. Click "Run workflow"
4. Choose whether to create backup
5. Click "Run workflow"

### Via Script

```bash
cd /opt/motovaultpro

# With backup
./scripts/ci/maintenance-migrate.sh backup

# Without backup
./scripts/ci/maintenance-migrate.sh
```

---

## Troubleshooting

### Staging Workflow Failures

**Build fails:**
```bash
# On build server
docker system df
docker system prune -af
```

**Deploy fails:**
```bash
docker logs mvp-backend-staging
docker logs mvp-frontend-staging
```

### Production Workflow Failures

**Health check timeout:**
```bash
# Check containers
docker ps
docker logs mvp-backend-blue  # or green
```

**Traffic not switching:**
```bash
# Check Traefik config
cat config/traefik/dynamic/blue-green.yml
docker exec mvp-traefik traefik healthcheck
```

### Runner Issues

**Runner offline:**
```bash
sudo systemctl status act_runner
sudo journalctl -u act_runner -f
```

**Permission denied:**
```bash
sudo usermod -aG docker act_runner
sudo systemctl restart act_runner
```

---

## Quick Reference

### Workflow Locations

| Workflow | File |
|----------|------|
| Staging | `.gitea/workflows/staging.yaml` |
| Production | `.gitea/workflows/production.yaml` |
| Maintenance | `.gitea/workflows/maintenance.yaml` |
| Mirror Images | `.gitea/workflows/mirror-images.yaml` |

### Important Paths

| Path | Description |
|------|-------------|
| `config/deployment/state.json` | Deployment state |
| `config/traefik/dynamic/blue-green.yml` | Traffic routing |
| `scripts/ci/` | Deployment scripts |

### Common Commands

```bash
# View deployment state
cat config/deployment/state.json | jq .

# Check containers
docker ps --format "table {{.Names}}\t{{.Status}}"

# View logs
docker logs mvp-backend-blue -f

# Manual traffic switch
./scripts/ci/switch-traffic.sh green

# Run health check
./scripts/ci/health-check.sh blue
```

### Email Notifications

| Event | Trigger |
|-------|---------|
| Staging Ready | Staging verified successfully |
| Success | Production deployed successfully |
| Failure | Deployment or verification failed |
| Rollback | Auto-rollback executed |
| Maintenance | Migration started/completed |