egullickson/motovaultpro
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: I dunno, I'm making git server changes

Browse Source
This commit is contained in:
Eric Gullickson
2025-12-29 08:44:49 -06:00
parent 57d2c43da7
commit 9b0de6a5b8
18 changed files with 2584 additions and 512 deletions
Download Patch File Download Diff File Expand all files Collapse all files

737
docs/CICD-DEPLOY.md
View File

@@ -1,17 +1,72 @@
# MotoVaultPro GitLab CI/CD Deployment Guide
Complete guide for deploying MotoVaultPro using GitLab CI/CD with shell executor runners.
Complete guide for deploying MotoVaultPro using GitLab CI/CD with blue-green deployment and auto-rollback.
## Table of Contents
1. [Prerequisites](#prerequisites)
2. [GitLab Runner Setup](#gitlab-runner-setup)
3. [CI/CD Variables Configuration](#cicd-variables-configuration)
4. [Secrets Architecture](#secrets-architecture)
5. [Pipeline Overview](#pipeline-overview)
6. [Deployment Process](#deployment-process)
7. [Rollback Procedure](#rollback-procedure)
8. [Troubleshooting](#troubleshooting)
1. [Architecture Overview](#architecture-overview)
2. [Prerequisites](#prerequisites)
3. [Pipeline Stages](#pipeline-stages)
4. [Blue-Green Deployment](#blue-green-deployment)
5. [CI/CD Variables Configuration](#cicd-variables-configuration)
6. [Container Registry](#container-registry)
7. [Deployment Process](#deployment-process)
8. [Rollback Procedures](#rollback-procedures)
9. [Maintenance Migrations](#maintenance-migrations)
10. [Notifications](#notifications)
11. [Troubleshooting](#troubleshooting)
---
## Architecture Overview
MotoVaultPro uses a blue-green deployment strategy with automatic rollback:
```
+---------------------------------------------------+
| GitLab (CI/CD + Registry) |
+---------------------------------------------------+
| |
v v
+------------------+ +-----------------------+
| Build VPS | | Production Server |
| (Docker Runner) | | (Shell Runner) |
| Tags: build | | Tags: production |
+------------------+ +-----------+-----------+
| |
| Push images | Pull + Deploy
v v
+---------------------------------------------------+
| GitLab Container Registry |
| registry.motovaultpro.com/motovaultpro/ |
+---------------------------------------------------+
|
+---------------+---------------+
| |
+--------v--------+ +--------v--------+
| BLUE Stack | | GREEN Stack |
| mvp-frontend | | mvp-frontend |
| mvp-backend | | mvp-backend |
+-----------------+ +-----------------+
| |
+----------- Traefik -----------+
(weighted LB)
|
+---------------+---------------+
| |
+--------v--------+ +--------v--------+
| PostgreSQL | | Redis |
| (shared) | | (shared) |
+-----------------+ +-----------------+
```
### Key Features
- **Zero-downtime deployments**: Traffic switches in under 5 seconds
- **Instant rollback**: Previous version remains running
- **Automatic rollback**: On health check failure
- **Email notifications**: Via Resend API
- **Container registry**: Self-hosted on GitLab (no Docker Hub)
---
@@ -19,55 +74,115 @@ Complete guide for deploying MotoVaultPro using GitLab CI/CD with shell executor
### Server Requirements
- Linux server with Docker Engine installed
- Docker Compose v2 (plugin version)
- GitLab Runner installed and registered
- Git installed
- curl installed (for health checks)
| Server | Purpose | Specs | Runner Tags |
|--------|---------|-------|-------------|
| Build VPS | Docker image builds | 2 CPU, 4GB RAM | `build` |
| Prod Server | Application hosting | 8GB+ RAM | `production` |
### GitLab Requirements
See [BUILD-SERVER-SETUP.md](BUILD-SERVER-SETUP.md) for build server setup.
- GitLab 18.6+ (tested with 18.6.2)
- Project with CI/CD enabled
- Protected `main` branch
- Maintainer access for CI/CD variable configuration
### Software Requirements
- GitLab 18.6+
- Docker Engine 24.0+
- Docker Compose v2
- GitLab Runner (shell executor on both servers)
- `jq` for JSON processing
---
## GitLab Runner Setup
## Pipeline Stages
### 1. Verify Runner Registration
The CI/CD pipeline consists of 7 stages:
```bash
sudo gitlab-runner verify
```
validate -> build -> deploy-prepare -> deploy-switch -> verify -> [rollback] -> notify
```
Expected output should show your runner as active with shell executor.
| Stage | Runner | Purpose |
|-------|--------|---------|
| `validate` | prod | Check prerequisites, determine target stack |
| `build` | build | Build and push images to GitLab registry |
| `deploy-prepare` | prod | Pull images, start inactive stack, health check |
| `deploy-switch` | prod | Switch Traefik traffic weights |
| `verify` | prod | Production health verification |
| `rollback` | prod | Auto-triggered on verify failure |
| `notify` | prod | Email success/failure notifications |
### 2. Verify Docker Permissions
### Pipeline Flow
The `gitlab-runner` user must have Docker access:
```bash
# Add gitlab-runner to docker group (if not already done)
sudo usermod -aG docker gitlab-runner
# Verify access
sudo -u gitlab-runner docker info
sudo -u gitlab-runner docker compose version
```
[Push to main]
|
v
[validate] - Checks Docker, paths, registry
|
v
[build] - Builds backend + frontend images
| Pushes to registry.motovaultpro.com
v
[deploy-prepare] - Pulls new images
| Starts inactive stack (blue or green)
| Runs health checks
v
[deploy-switch] - Updates Traefik weights
| Switches traffic instantly
v
[verify] - External health check
| Container status verification
|
+--[SUCCESS]--> [notify-success] - Sends success email
|
+--[FAILURE]--> [rollback] - Switches back to previous stack
|
v
[notify-failure] - Sends failure email
```
### 3. Verify Deployment Directory
---
Ensure the deployment directory exists and is accessible:
## Blue-Green Deployment
```bash
# Create deployment directory
sudo mkdir -p /opt/motovaultpro
sudo chown gitlab-runner:gitlab-runner /opt/motovaultpro
### Stack Configuration
# Clone repository (first time only)
sudo -u gitlab-runner git clone <repository-url> /opt/motovaultpro
Both stacks share the same database layer:
| Component | Blue Stack | Green Stack | Shared |
|-----------|------------|-------------|--------|
| Frontend | `mvp-frontend-blue` | `mvp-frontend-green` | - |
| Backend | `mvp-backend-blue` | `mvp-backend-green` | - |
| PostgreSQL | - | - | `mvp-postgres` |
| Redis | - | - | `mvp-redis` |
| Traefik | - | - | `mvp-traefik` |
### Traffic Routing
Traefik uses weighted services for traffic distribution:
```yaml
# config/traefik/dynamic/blue-green.yml
services:
mvp-frontend-weighted:
weighted:
services:
- name: mvp-frontend-blue-svc
weight: 100 # Active
- name: mvp-frontend-green-svc
weight: 0 # Standby
```
### Deployment State
State is tracked in `config/deployment/state.json`:
```json
{
"active_stack": "blue",
"inactive_stack": "green",
"last_deployment": "2024-01-15T10:30:00Z",
"last_deployment_commit": "abc123",
"rollback_available": true
}
```
---
@@ -76,410 +191,316 @@ sudo -u gitlab-runner git clone <repository-url> /opt/motovaultpro
Navigate to **Settings > CI/CD > Variables** in your GitLab project.
### Secrets (File Type Variables)
### Required Variables
These variables use GitLab's **File** type, which writes the value to a temporary file and provides the path as the environment variable. This replicates the Kubernetes secrets pattern used by the application.
| Variable Name | Type | Protected | Masked | Description |
|--------------|------|-----------|--------|-------------|
| `POSTGRES_PASSWORD` | File | Yes | Yes | PostgreSQL database password |
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes | Auth0 client secret for backend |
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes | Google Maps API key |
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No | Google Maps Map ID |
| `CF_DNS_API_TOKEN` | File | Yes | Yes | Cloudflare API token for Let's Encrypt DNS challenge |
| `RESEND_API_KEY` | File | Yes | Yes | Resend API key for email notifications |
### Configuration Variables
| Variable Name | Type | Protected | Masked | Value |
|--------------|------|-----------|--------|-------|
| `VITE_AUTH0_DOMAIN` | Variable | No | No | `motovaultpro.us.auth0.com` |
| `VITE_AUTH0_CLIENT_ID` | Variable | No | No | Your Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | Variable | No | No | `https://api.motovaultpro.com` |
Note: `DEPLOY_PATH` is automatically set in `.gitlab-ci.yml` using `GIT_CLONE_PATH` for a stable path.
### Creating Cloudflare API Token
The `CF_DNS_API_TOKEN` is required for automatic SSL certificate generation via Let's Encrypt DNS-01 challenge.
1. Go to [Cloudflare Dashboard](https://dash.cloudflare.com/profile/api-tokens)
2. Click **Create Token**
3. Use template: **Edit zone DNS**
4. Configure permissions:
- **Permissions**: Zone > DNS > Edit
- **Zone Resources**: Include > Specific zone > `motovaultpro.com`
5. Click **Continue to summary** then **Create Token**
6. Copy the token value immediately (it won't be shown again)
7. Add as `CF_DNS_API_TOKEN` File variable in GitLab
### Setting Up a File Type Variable
1. Go to **Settings > CI/CD > Variables**
2. Click **Add variable**
3. Enter the variable key (e.g., `POSTGRES_PASSWORD`)
4. Enter the secret value in the **Value** field
5. Set **Type** to **File**
6. Enable **Protect variable** (recommended)
7. Enable **Mask variable** (for sensitive data)
8. Click **Add variable**
---
## Secrets Architecture
MotoVaultPro uses a Kubernetes-style secrets pattern where secrets are mounted as files at `/run/secrets/` inside containers.
### How It Works
1. **GitLab stores secrets** as File type CI/CD variables
2. **During pipeline execution**, GitLab writes each secret to a temporary file
3. **The `inject-secrets.sh` script** copies these files to `secrets/app/` directory
4. **Docker Compose** mounts these files to `/run/secrets/` in containers
5. **Application code** reads secrets from the filesystem (not environment variables)
| Variable | Type | Protected | Purpose |
|----------|------|-----------|---------|
| `DEPLOY_NOTIFY_EMAIL` | Variable | Yes | Notification recipient |
| `VITE_AUTH0_DOMAIN` | Variable | No | Auth0 domain |
| `VITE_AUTH0_CLIENT_ID` | Variable | No | Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | Variable | No | Auth0 audience |
### Secret Files
```
secrets/app/
postgres-password.txt -> /run/secrets/postgres-password
auth0-client-secret.txt -> /run/secrets/auth0-client-secret
google-maps-api-key.txt -> /run/secrets/google-maps-api-key
google-maps-map-id.txt -> /run/secrets/google-maps-map-id
cloudflare-dns-token.txt -> /run/secrets/cloudflare-dns-token
resend-api-key.txt -> /run/secrets/resend-api-key
```
These use GitLab's **File** type and are injected via `scripts/inject-secrets.sh`:
### Security Benefits
| Variable | Type | Protected | Masked |
|----------|------|-----------|--------|
| `POSTGRES_PASSWORD` | File | Yes | Yes |
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes |
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes |
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No |
| `CF_DNS_API_TOKEN` | File | Yes | Yes |
| `RESEND_API_KEY` | File | Yes | Yes |
- Secrets never appear as environment variables (not visible in `env` or `printenv`)
- File permissions (600) restrict access
- Masked variables prevent accidental log exposure
- Protected variables only available on protected branches
### Registry Authentication
GitLab provides these automatically:
- `CI_REGISTRY_USER` - Registry username
- `CI_REGISTRY_PASSWORD` - Registry token
- `CI_REGISTRY` - Registry URL
---
## Pipeline Overview
## Container Registry
The CI/CD pipeline consists of four stages:
All images are hosted on the GitLab Container Registry to avoid Docker Hub rate limits.
### Stage 1: Validate
Verifies deployment prerequisites:
- Docker is accessible
- Docker Compose is available
- Deployment directory exists
### Stage 2: Build
Builds Docker images:
- Pulls latest code from repository
- Builds all service images with `--no-cache`
### Stage 3: Deploy
Deploys the application:
1. Injects secrets from GitLab variables
2. Stops existing services gracefully
3. Pulls base images
4. Starts database services (PostgreSQL, Redis)
5. Runs database migrations
6. Starts all services
### Stage 4: Verify
Validates deployment health:
- Checks all containers are running
- Tests backend health endpoint
- Reports deployment status
### Pipeline Diagram
### Registry URL
```
[Validate] -> [Build] -> [Deploy] -> [Verify]
| | | |
Check Build Inject Health
prereqs images secrets checks
|
Migrate
|
Start
services
registry.motovaultpro.com
```
### Image Paths
| Image | Path |
|-------|------|
| Backend | `registry.motovaultpro.com/motovaultpro/backend:$TAG` |
| Frontend | `registry.motovaultpro.com/motovaultpro/frontend:$TAG` |
| Mirrors | `registry.motovaultpro.com/mirrors/` |
### Base Image Mirrors
Mirror upstream images to avoid rate limits:
```bash
# Run manually or via scheduled pipeline
./scripts/ci/mirror-base-images.sh
```
Mirrored images:
- `node:20-alpine`
- `nginx:alpine`
- `postgres:18-alpine`
- `redis:8.4-alpine`
- `traefik:v3.6`
- `docker:24.0`
- `docker:24.0-dind`
---
## Deployment Process
### Automatic Deployment
Deployments are triggered automatically when:
- Code is pushed to the `main` branch
- A merge request is merged into `main`
Deployments trigger automatically on push to `main`:
1. **Validate**: Check prerequisites, determine target stack
2. **Build**: Build images on dedicated build server
3. **Prepare**: Start inactive stack, run health checks
4. **Switch**: Update Traefik weights (instant)
5. **Verify**: External health check
6. **Notify**: Send email notification
### Manual Deployment
To trigger a manual deployment:
1. Go to **CI/CD > Pipelines**
2. Click **Run pipeline**
3. Select the `main` branch
3. Select `main` branch
4. Click **Run pipeline**
### Deployment Steps (What Happens)
### Deployment Timeline
1. **Secrets Injection**
- `inject-secrets.sh` copies GitLab File variables to `secrets/app/`
- Permissions are set to 600 for security
2. **Service Shutdown**
- Existing containers are stopped gracefully (30s timeout)
- Volumes are preserved
3. **Database Startup**
- PostgreSQL and Redis start first
- 15-second wait for database readiness
4. **Migrations**
- Backend container runs database migrations
- Ensures schema is up-to-date
5. **Full Service Startup**
- All services start via `docker compose up -d`
- Traefik routes traffic automatically
6. **Health Verification**
- Container status checks
- Backend health endpoint validation
| Phase | Duration |
|-------|----------|
| Validate | ~5s |
| Build | ~2 min |
| Deploy-prepare | ~30s |
| Deploy-switch | ~3s |
| Verify | ~30s |
| **Total** | ~3 min |
---
## Rollback Procedure
## Rollback Procedures
### Automatic Rollback
If the verify stage fails, the pipeline will report failure but services remain running. Manual intervention is required.
Triggers automatically when:
- Health check fails in `deploy-prepare`
- `verify` stage fails after switch
- Container becomes unhealthy within verification period
The pipeline runs `scripts/ci/auto-rollback.sh` which:
1. Verifies previous stack is healthy
2. Switches traffic back
3. Sends notification
### Manual Rollback
Use the rollback script:
SSH to production server:
```bash
# SSH to server
ssh user@server
# Run rollback to previous commit
cd /opt/motovaultpro
./scripts/rollback.sh HEAD~1
# Or rollback to specific tag/commit
./scripts/rollback.sh v1.0.0
# Check current state
cat config/deployment/state.json | jq .
# Switch to other stack
./scripts/ci/switch-traffic.sh blue # or green
```
### Rollback Script Details
The script performs:
1. Stops all current services
2. Checks out the specified version
3. Rebuilds images
4. Starts services
### Emergency Recovery
If rollback fails:
If both stacks are unhealthy:
```bash
# Stop everything
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml down
# Restart shared services
docker compose up -d mvp-postgres mvp-redis mvp-traefik
# Wait for database
sleep 15
# Start one stack
export BACKEND_IMAGE=registry.motovaultpro.com/motovaultpro/backend:latest
export FRONTEND_IMAGE=registry.motovaultpro.com/motovaultpro/frontend:latest
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
mvp-frontend-blue mvp-backend-blue
# Switch traffic
./scripts/ci/switch-traffic.sh blue
```
---
## Maintenance Migrations
For breaking database changes requiring downtime:
### Via Pipeline (Recommended)
1. Go to **CI/CD > Pipelines**
2. Find the `maintenance-migration` job
3. Click **Play** to trigger manually
### Via Script
```bash
cd /opt/motovaultpro
# Stop everything
docker compose down
# With backup
./scripts/ci/maintenance-migrate.sh backup
# Check git history
git log --oneline -10
# Without backup
./scripts/ci/maintenance-migrate.sh
```
# Checkout known working version
git checkout <commit-hash>
### What Happens
# Rebuild and start
docker compose build
docker compose up -d
1. Sends maintenance notification
2. Enables maintenance mode (stops traffic)
3. Creates database backup (if requested)
4. Runs migrations
5. Restarts backends
6. Restores traffic
7. Sends completion notification
# Verify
docker compose ps
---
## Notifications
Email notifications via Resend API for:
| Event | Subject |
|-------|---------|
| `success` | Deployment Successful |
| `failure` | Deployment Failed |
| `rollback` | Auto-Rollback Executed |
| `rollback_failed` | CRITICAL: Rollback Failed |
| `maintenance_start` | Maintenance Mode Started |
| `maintenance_end` | Maintenance Complete |
Configure recipient in GitLab CI/CD variables:
```
DEPLOY_NOTIFY_EMAIL = admin@example.com
```
---
## Troubleshooting
### Pipeline Fails at Validate Stage
**Symptom**: `DEPLOY_PATH not found`
**Solution**:
```bash
# Create directory on runner server
sudo mkdir -p /opt/motovaultpro
sudo chown gitlab-runner:gitlab-runner /opt/motovaultpro
```
### Pipeline Fails at Build Stage
**Symptom**: Docker build errors
**Solutions**:
1. Check Dockerfile syntax
2. Verify network connectivity for npm/package downloads
3. Check disk space: `df -h`
4. Clear Docker cache: `docker system prune -a`
### Pipeline Fails at Deploy Stage
**Symptom**: Secrets injection fails
**Solutions**:
1. Verify CI/CD variables are configured correctly
2. Check variable types are set to **File** for secrets
3. Ensure variables are not restricted to specific environments
**Symptom**: Migration fails
**Solutions**:
1. Check database connectivity
2. Verify PostgreSQL is healthy: `docker logs mvp-postgres`
3. Run migrations manually:
```bash
docker compose exec mvp-backend npm run migrate
```
### Pipeline Fails at Verify Stage
**Symptom**: Container not running
**Solutions**:
1. Check container logs: `docker logs <container-name>`
2. Verify secrets are correctly mounted
3. Check for port conflicts
**Symptom**: Health check fails
**Solutions**:
1. Wait longer (service might be starting)
2. Check backend logs: `docker logs mvp-backend`
3. Verify database connection
### Services Start But Application Doesn't Work
**Check secrets are mounted**:
**Check build server connectivity:**
```bash
docker compose exec mvp-backend ls -la /run/secrets/
# On build server
sudo gitlab-runner verify
docker login registry.motovaultpro.com
```
**Check configuration**:
**Check disk space:**
```bash
docker compose exec mvp-backend cat /app/config/production.yml
df -h
docker system prune -af
```
**Check network connectivity**:
### Pipeline Fails at Deploy-Prepare
**Container won't start:**
```bash
docker network ls
docker network inspect motovaultpro_backend
docker logs mvp-backend-blue --tail 100
docker logs mvp-frontend-blue --tail 100
```
### Viewing Logs
**Health check timeout:**
```bash
# All services
docker compose logs -f
# Specific service
docker compose logs -f mvp-backend
# Last 100 lines
docker compose logs --tail 100 mvp-backend
# Increase timeout in .gitlab-ci.yml
HEALTH_CHECK_TIMEOUT: "90"
```
---
## Maintenance
### Updating Secrets
1. Update the CI/CD variable in GitLab
2. Trigger a new pipeline (push or manual)
3. The new secrets will be injected during deployment
### Database Backups
Backups should be configured separately. Recommended approach:
### Traffic Not Switching
**Check Traefik config:**
```bash
# Manual backup
docker compose exec mvp-postgres pg_dump -U postgres motovaultpro > backup.sql
# Automated backup (add to cron)
0 2 * * * cd /opt/motovaultpro && docker compose exec -T mvp-postgres pg_dump -U postgres motovaultpro > /backups/mvp-$(date +\%Y\%m\%d).sql
cat config/traefik/dynamic/blue-green.yml
docker exec mvp-traefik traefik healthcheck
```
### Monitoring
**Check routing:**
```bash
curl -I https://motovaultpro.com/api/health
```
Consider adding:
- Prometheus metrics (Traefik already configured)
- Health check alerts
- Log aggregation
### Verify Stage Fails
**Check external connectivity:**
```bash
curl -sf https://motovaultpro.com/api/health
```
**Check container health:**
```bash
docker inspect --format='{{.State.Health.Status}}' mvp-backend-blue
```
---
## Quick Reference
### Common Commands
```bash
# View pipeline status
# GitLab UI: CI/CD > Pipelines
# SSH to server
ssh user@your-server
# Navigate to project
cd /opt/motovaultpro
# View running containers
docker compose ps
# View logs
docker compose logs -f
# Restart a service
docker compose restart mvp-backend
# Run migrations manually
docker compose exec mvp-backend npm run migrate
# Access database
docker compose exec mvp-postgres psql -U postgres motovaultpro
# Health check
curl http://localhost:3001/health
```
### Important Paths
| Path | Description |
|------|-------------|
| `$CI_BUILDS_DIR/motovaultpro` | Application root (stable clone path) |
| `$CI_BUILDS_DIR/motovaultpro/secrets/app/` | Secrets directory |
| `$CI_BUILDS_DIR/motovaultpro/data/documents/` | Document storage |
| `$CI_BUILDS_DIR/motovaultpro/config/` | Configuration files |
| `config/deployment/state.json` | Deployment state |
| `config/traefik/dynamic/blue-green.yml` | Traffic routing |
| `scripts/ci/` | Deployment scripts |
Note: `CI_BUILDS_DIR` is typically `/opt/gitlab-runner/builds` for shell executors.
### Common Commands
### Container Names
```bash
# View current state
cat config/deployment/state.json | jq .
| Container | Purpose |
|-----------|---------|
| `mvp-traefik` | Reverse proxy, TLS termination |
| `mvp-frontend` | React SPA |
| `mvp-backend` | Node.js API |
| `mvp-postgres` | PostgreSQL database |
| `mvp-redis` | Redis cache |
# Check container status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Health}}"
# View logs
docker logs mvp-backend-blue -f
# Manual traffic switch
./scripts/ci/switch-traffic.sh green
# Run health check
./scripts/ci/health-check.sh blue
# Send test notification
./scripts/ci/notify.sh success "Test message"
```
### Memory Budget (8GB Server)
| Component | RAM |
|-----------|-----|
| Blue frontend | 512MB |
| Blue backend | 1GB |
| Green frontend | 512MB |
| Green backend | 1GB |
| PostgreSQL | 2GB |
| Redis | 512MB |
| Traefik | 128MB |
| System | 1.3GB |
| **Total** | ~7GB |