egullickson/motovaultpro
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9b0de6a5b81ca78285864696e0ef07238ec4d5d7
motovaultpro/docs/CICD-DEPLOY.md
Eric Gullickson 9b0de6a5b8 fix: I dunno, I'm making git server changes
2025-12-29 08:44:49 -06:00

12 KiB
Raw Blame History

MotoVaultPro GitLab CI/CD Deployment Guide

Complete guide for deploying MotoVaultPro using GitLab CI/CD with blue-green deployment and auto-rollback.

Table of Contents

  1. Architecture Overview
  2. Prerequisites
  3. Pipeline Stages
  4. Blue-Green Deployment
  5. CI/CD Variables Configuration
  6. Container Registry
  7. Deployment Process
  8. Rollback Procedures
  9. Maintenance Migrations
  10. Notifications
  11. Troubleshooting

Architecture Overview

MotoVaultPro uses a blue-green deployment strategy with automatic rollback:

+---------------------------------------------------+
|              GitLab (CI/CD + Registry)            |
+---------------------------------------------------+
         |                           |
         v                           v
+------------------+     +-----------------------+
|   Build VPS      |     |   Production Server   |
| (Docker Runner)  |     | (Shell Runner)        |
| Tags: build      |     | Tags: production      |
+------------------+     +-----------+-----------+
         |                           |
         | Push images               | Pull + Deploy
         v                           v
+---------------------------------------------------+
|              GitLab Container Registry            |
| registry.motovaultpro.com/motovaultpro/           |
+---------------------------------------------------+
                         |
         +---------------+---------------+
         |                               |
+--------v--------+             +--------v--------+
|   BLUE Stack    |             |  GREEN Stack    |
| mvp-frontend    |             | mvp-frontend    |
| mvp-backend     |             | mvp-backend     |
+-----------------+             +-----------------+
         |                               |
         +----------- Traefik -----------+
                  (weighted LB)
                         |
         +---------------+---------------+
         |                               |
+--------v--------+             +--------v--------+
|   PostgreSQL    |             |     Redis       |
|   (shared)      |             |    (shared)     |
+-----------------+             +-----------------+

Key Features

  • Zero-downtime deployments: Traffic switches in under 5 seconds
  • Instant rollback: Previous version remains running
  • Automatic rollback: On health check failure
  • Email notifications: Via Resend API
  • Container registry: Self-hosted on GitLab (no Docker Hub)

Prerequisites

Server Requirements

Server Purpose Specs Runner Tags
Build VPS Docker image builds 2 CPU, 4GB RAM build
Prod Server Application hosting 8GB+ RAM production

See BUILD-SERVER-SETUP.md for build server setup.

Software Requirements

  • GitLab 18.6+
  • Docker Engine 24.0+
  • Docker Compose v2
  • GitLab Runner (shell executor on both servers)
  • jq for JSON processing

Pipeline Stages

The CI/CD pipeline consists of 7 stages:

validate -> build -> deploy-prepare -> deploy-switch -> verify -> [rollback] -> notify
Stage Runner Purpose
validate prod Check prerequisites, determine target stack
build build Build and push images to GitLab registry
deploy-prepare prod Pull images, start inactive stack, health check
deploy-switch prod Switch Traefik traffic weights
verify prod Production health verification
rollback prod Auto-triggered on verify failure
notify prod Email success/failure notifications

Pipeline Flow

[Push to main]
      |
      v
[validate] - Checks Docker, paths, registry
      |
      v
[build] - Builds backend + frontend images
      |        Pushes to registry.motovaultpro.com
      v
[deploy-prepare] - Pulls new images
      |             Starts inactive stack (blue or green)
      |             Runs health checks
      v
[deploy-switch] - Updates Traefik weights
      |            Switches traffic instantly
      v
[verify] - External health check
      |     Container status verification
      |
      +--[SUCCESS]--> [notify-success] - Sends success email
      |
      +--[FAILURE]--> [rollback] - Switches back to previous stack
                           |
                           v
                    [notify-failure] - Sends failure email

Blue-Green Deployment

Stack Configuration

Both stacks share the same database layer:

Component Blue Stack Green Stack Shared
Frontend mvp-frontend-blue mvp-frontend-green -
Backend mvp-backend-blue mvp-backend-green -
PostgreSQL - - mvp-postgres
Redis - - mvp-redis
Traefik - - mvp-traefik

Traffic Routing

Traefik uses weighted services for traffic distribution:

# config/traefik/dynamic/blue-green.yml
services:
  mvp-frontend-weighted:
    weighted:
      services:
        - name: mvp-frontend-blue-svc
          weight: 100  # Active
        - name: mvp-frontend-green-svc
          weight: 0    # Standby

Deployment State

State is tracked in config/deployment/state.json:

{
  "active_stack": "blue",
  "inactive_stack": "green",
  "last_deployment": "2024-01-15T10:30:00Z",
  "last_deployment_commit": "abc123",
  "rollback_available": true
}

CI/CD Variables Configuration

Navigate to Settings > CI/CD > Variables in your GitLab project.

Required Variables

Variable Type Protected Purpose
DEPLOY_NOTIFY_EMAIL Variable Yes Notification recipient
VITE_AUTH0_DOMAIN Variable No Auth0 domain
VITE_AUTH0_CLIENT_ID Variable No Auth0 client ID
VITE_AUTH0_AUDIENCE Variable No Auth0 audience

Secret Files

These use GitLab's File type and are injected via scripts/inject-secrets.sh:

Variable Type Protected Masked
POSTGRES_PASSWORD File Yes Yes
AUTH0_CLIENT_SECRET File Yes Yes
GOOGLE_MAPS_API_KEY File Yes Yes
GOOGLE_MAPS_MAP_ID File Yes No
CF_DNS_API_TOKEN File Yes Yes
RESEND_API_KEY File Yes Yes

Registry Authentication

GitLab provides these automatically:

  • CI_REGISTRY_USER - Registry username
  • CI_REGISTRY_PASSWORD - Registry token
  • CI_REGISTRY - Registry URL

Container Registry

All images are hosted on the GitLab Container Registry to avoid Docker Hub rate limits.

Registry URL

registry.motovaultpro.com

Image Paths

Image Path
Backend registry.motovaultpro.com/motovaultpro/backend:$TAG
Frontend registry.motovaultpro.com/motovaultpro/frontend:$TAG
Mirrors registry.motovaultpro.com/mirrors/

Base Image Mirrors

Mirror upstream images to avoid rate limits:

# Run manually or via scheduled pipeline
./scripts/ci/mirror-base-images.sh

Mirrored images:

  • node:20-alpine
  • nginx:alpine
  • postgres:18-alpine
  • redis:8.4-alpine
  • traefik:v3.6
  • docker:24.0
  • docker:24.0-dind

Deployment Process

Automatic Deployment

Deployments trigger automatically on push to main:

  1. Validate: Check prerequisites, determine target stack
  2. Build: Build images on dedicated build server
  3. Prepare: Start inactive stack, run health checks
  4. Switch: Update Traefik weights (instant)
  5. Verify: External health check
  6. Notify: Send email notification

Manual Deployment

  1. Go to CI/CD > Pipelines
  2. Click Run pipeline
  3. Select main branch
  4. Click Run pipeline

Deployment Timeline

Phase Duration
Validate ~5s
Build ~2 min
Deploy-prepare ~30s
Deploy-switch ~3s
Verify ~30s
Total ~3 min

Rollback Procedures

Automatic Rollback

Triggers automatically when:

  • Health check fails in deploy-prepare
  • verify stage fails after switch
  • Container becomes unhealthy within verification period

The pipeline runs scripts/ci/auto-rollback.sh which:

  1. Verifies previous stack is healthy
  2. Switches traffic back
  3. Sends notification

Manual Rollback

SSH to production server:

cd /opt/motovaultpro

# Check current state
cat config/deployment/state.json | jq .

# Switch to other stack
./scripts/ci/switch-traffic.sh blue   # or green

Emergency Recovery

If both stacks are unhealthy:

# Stop everything
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml down

# Restart shared services
docker compose up -d mvp-postgres mvp-redis mvp-traefik

# Wait for database
sleep 15

# Start one stack
export BACKEND_IMAGE=registry.motovaultpro.com/motovaultpro/backend:latest
export FRONTEND_IMAGE=registry.motovaultpro.com/motovaultpro/frontend:latest
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
  mvp-frontend-blue mvp-backend-blue

# Switch traffic
./scripts/ci/switch-traffic.sh blue

Maintenance Migrations

For breaking database changes requiring downtime:

Via Pipeline (Recommended)

  1. Go to CI/CD > Pipelines
  2. Find the maintenance-migration job
  3. Click Play to trigger manually

Via Script

cd /opt/motovaultpro

# With backup
./scripts/ci/maintenance-migrate.sh backup

# Without backup
./scripts/ci/maintenance-migrate.sh

What Happens

  1. Sends maintenance notification
  2. Enables maintenance mode (stops traffic)
  3. Creates database backup (if requested)
  4. Runs migrations
  5. Restarts backends
  6. Restores traffic
  7. Sends completion notification

Notifications

Email notifications via Resend API for:

Event Subject
success Deployment Successful
failure Deployment Failed
rollback Auto-Rollback Executed
rollback_failed CRITICAL: Rollback Failed
maintenance_start Maintenance Mode Started
maintenance_end Maintenance Complete

Configure recipient in GitLab CI/CD variables:

DEPLOY_NOTIFY_EMAIL = admin@example.com

Troubleshooting

Pipeline Fails at Build Stage

Check build server connectivity:

# On build server
sudo gitlab-runner verify
docker login registry.motovaultpro.com

Check disk space:

df -h
docker system prune -af

Pipeline Fails at Deploy-Prepare

Container won't start:

docker logs mvp-backend-blue --tail 100
docker logs mvp-frontend-blue --tail 100

Health check timeout:

# Increase timeout in .gitlab-ci.yml
HEALTH_CHECK_TIMEOUT: "90"

Traffic Not Switching

Check Traefik config:

cat config/traefik/dynamic/blue-green.yml
docker exec mvp-traefik traefik healthcheck

Check routing:

curl -I https://motovaultpro.com/api/health

Verify Stage Fails

Check external connectivity:

curl -sf https://motovaultpro.com/api/health

Check container health:

docker inspect --format='{{.State.Health.Status}}' mvp-backend-blue

Quick Reference

Important Paths

Path Description
config/deployment/state.json Deployment state
config/traefik/dynamic/blue-green.yml Traffic routing
scripts/ci/ Deployment scripts

Common Commands

# View current state
cat config/deployment/state.json | jq .

# Check container status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Health}}"

# View logs
docker logs mvp-backend-blue -f

# Manual traffic switch
./scripts/ci/switch-traffic.sh green

# Run health check
./scripts/ci/health-check.sh blue

# Send test notification
./scripts/ci/notify.sh success "Test message"

Memory Budget (8GB Server)

Component RAM
Blue frontend 512MB
Blue backend 1GB
Green frontend 512MB
Green backend 1GB
PostgreSQL 2GB
Redis 512MB
Traefik 128MB
System 1.3GB
Total ~7GB
Reference in New Issue View Git Blame Copy Permalink