egullickson/motovaultpro
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

CI/CD Gitea v1.0
Some checks failed
Deploy to Staging / Build Images (push) Failing after 7s
Details
Deploy to Staging / Deploy to Staging (push) Has been skipped
Details
Deploy to Staging / Verify Staging (push) Has been skipped
Details
Deploy to Staging / Notify Staging Ready (push) Has been skipped
Details
Deploy to Staging / Notify Staging Failure (push) Failing after 6s
Details

Browse Source
This commit is contained in:
Eric Gullickson
2025-12-29 18:51:41 -06:00
parent 9b0de6a5b8
commit 83d79da3aa
15 changed files with 1101 additions and 929 deletions
Download Patch File Download Diff File Expand all files Collapse all files

466
docs/CICD-DEPLOY.md
View File

@@ -1,72 +1,52 @@
# MotoVaultPro GitLab CI/CD Deployment Guide
# MotoVaultPro CI/CD Deployment Guide
Complete guide for deploying MotoVaultPro using GitLab CI/CD with blue-green deployment and auto-rollback.
Complete guide for deploying MotoVaultPro using Gitea Actions with staging-first deployment and manual production approval.
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Prerequisites](#prerequisites)
3. [Pipeline Stages](#pipeline-stages)
4. [Blue-Green Deployment](#blue-green-deployment)
5. [CI/CD Variables Configuration](#cicd-variables-configuration)
3. [Workflow Structure](#workflow-structure)
4. [Deployment Process](#deployment-process)
5. [Secrets and Variables](#secrets-and-variables)
6. [Container Registry](#container-registry)
7. [Deployment Process](#deployment-process)
8. [Rollback Procedures](#rollback-procedures)
9. [Maintenance Migrations](#maintenance-migrations)
10. [Notifications](#notifications)
11. [Troubleshooting](#troubleshooting)
7. [Rollback Procedures](#rollback-procedures)
8. [Maintenance Migrations](#maintenance-migrations)
9. [Troubleshooting](#troubleshooting)
---
## Architecture Overview
MotoVaultPro uses a blue-green deployment strategy with automatic rollback:
MotoVaultPro uses a **staging-first deployment** strategy with manual approval:
```
+---------------------------------------------------+
| GitLab (CI/CD + Registry) |
| Gitea (git.motovaultpro.com) |
| +--------------------+ +--------------------+ |
| | Gitea Actions | | Package Registry | |
| +--------------------+ +--------------------+ |
+---------------------------------------------------+
| |
v v
+------------------+ +-----------------------+
| Build VPS | | Production Server |
| (Docker Runner) | | (Shell Runner) |
| Tags: build | | Tags: production |
+------------------+ +-----------+-----------+
+-------------------+ +--------------------+
| Build/Staging VPS | | Production Server |
| (mvp-build) | | (mvp-prod) |
| act_runner | | act_runner |
+--------+----------+ +----------+---------+
| |
| Push images | Pull + Deploy
v v
+---------------------------------------------------+
| GitLab Container Registry |
| registry.motovaultpro.com/motovaultpro/ |
+---------------------------------------------------+
|
+---------------+---------------+
| |
+--------v--------+ +--------v--------+
| BLUE Stack | | GREEN Stack |
| mvp-frontend | | mvp-frontend |
| mvp-backend | | mvp-backend |
+-----------------+ +-----------------+
| |
+----------- Traefik -----------+
(weighted LB)
|
+---------------+---------------+
| |
+--------v--------+ +--------v--------+
| PostgreSQL | | Redis |
| (shared) | | (shared) |
+-----------------+ +-----------------+
staging.motovaultpro.com motovaultpro.com
(Full Stack) (Blue-Green)
```
### Key Features
- **Zero-downtime deployments**: Traffic switches in under 5 seconds
- **Instant rollback**: Previous version remains running
- **Automatic rollback**: On health check failure
- **Staging-first**: All changes verified on staging before production
- **Manual approval**: Production deploy requires manual trigger
- **Blue-green production**: Zero-downtime deployments
- **Auto-rollback**: Automatic rollback on health check failure
- **Email notifications**: Via Resend API
- **Container registry**: Self-hosted on GitLab (no Docker Hub)
---
@@ -74,180 +54,172 @@ MotoVaultPro uses a blue-green deployment strategy with automatic rollback:
### Server Requirements
| Server | Purpose | Specs | Runner Tags |
|--------|---------|-------|-------------|
| Build VPS | Docker image builds | 2 CPU, 4GB RAM | `build` |
| Prod Server | Application hosting | 8GB+ RAM | `production` |
| Server | Purpose | Specs | Runner Label |
|--------|---------|-------|--------------|
| Build/Staging VPS | Build + Staging | 4 CPU, 8GB RAM | `mvp-build` |
| Prod Server | Production | 8GB+ RAM | `mvp-prod` |
See [BUILD-SERVER-SETUP.md](BUILD-SERVER-SETUP.md) for build server setup.
See [BUILD-SERVER-SETUP.md](BUILD-SERVER-SETUP.md) for setup instructions.
### Software Requirements
- GitLab 18.6+
- Gitea 1.21+ with Actions enabled
- Docker Engine 24.0+
- Docker Compose v2
- GitLab Runner (shell executor on both servers)
- act_runner (Gitea Actions runner)
- `jq` for JSON processing
---
## Pipeline Stages
## Workflow Structure
The CI/CD pipeline consists of 7 stages:
### Two-Workflow Strategy
```
validate -> build -> deploy-prepare -> deploy-switch -> verify -> [rollback] -> notify
```
| Workflow | Trigger | Purpose |
|----------|---------|---------|
| `staging.yaml` | Push to main | Build, deploy to staging, verify |
| `production.yaml` | Manual (workflow_dispatch) | Deploy to production |
| Stage | Runner | Purpose |
|-------|--------|---------|
| `validate` | prod | Check prerequisites, determine target stack |
| `build` | build | Build and push images to GitLab registry |
| `deploy-prepare` | prod | Pull images, start inactive stack, health check |
| `deploy-switch` | prod | Switch Traefik traffic weights |
| `verify` | prod | Production health verification |
| `rollback` | prod | Auto-triggered on verify failure |
| `notify` | prod | Email success/failure notifications |
### Pipeline Flow
### Staging Workflow (Automatic)
```
[Push to main]
|
v
[validate] - Checks Docker, paths, registry
[build] -------- Build images, push to registry
|
v
[build] - Builds backend + frontend images
| Pushes to registry.motovaultpro.com
v
[deploy-prepare] - Pulls new images
| Starts inactive stack (blue or green)
| Runs health checks
v
[deploy-switch] - Updates Traefik weights
| Switches traffic instantly
v
[verify] - External health check
| Container status verification
[deploy-staging] - Deploy full stack to staging
|
+--[SUCCESS]--> [notify-success] - Sends success email
v
[verify-staging] - Health checks
|
+--[FAILURE]--> [rollback] - Switches back to previous stack
|
v
[notify-failure] - Sends failure email
+--[FAIL]--> [notify-staging-failure]
|
v
[notify-staging-ready] - Email with production deploy link
```
### Production Workflow (Manual)
```
[Manual Trigger] - User clicks "Run workflow"
|
v
[validate] - Check prerequisites, determine stack
|
v
[deploy-prod] - Blue-green deployment
|
v
[verify-prod] - External health checks
|
+--[FAIL]--> [rollback] --> [notify-failure]
|
v
[notify-success]
```
---
## Blue-Green Deployment
## Deployment Process
### Stack Configuration
### 1. Push to Main Branch
Both stacks share the same database layer:
When you push to `main`:
1. Staging workflow triggers automatically
2. Images are built and pushed to Gitea Package Registry
3. Full stack deploys to staging.motovaultpro.com
4. Health checks verify staging works
5. Email notification sent with production deploy link
| Component | Blue Stack | Green Stack | Shared |
|-----------|------------|-------------|--------|
| Frontend | `mvp-frontend-blue` | `mvp-frontend-green` | - |
| Backend | `mvp-backend-blue` | `mvp-backend-green` | - |
| PostgreSQL | - | - | `mvp-postgres` |
| Redis | - | - | `mvp-redis` |
| Traefik | - | - | `mvp-traefik` |
### 2. Review Staging
### Traffic Routing
After receiving the "Staging Ready" email:
1. Visit https://staging.motovaultpro.com
2. Test functionality
3. Review logs if needed: `docker logs mvp-backend-staging`
Traefik uses weighted services for traffic distribution:
### 3. Deploy to Production
```yaml
# config/traefik/dynamic/blue-green.yml
services:
mvp-frontend-weighted:
weighted:
services:
- name: mvp-frontend-blue-svc
weight: 100 # Active
- name: mvp-frontend-green-svc
weight: 0 # Standby
```
When ready to deploy:
1. Go to `git.motovaultpro.com/egullickson/motovaultpro/actions`
2. Select "Deploy to Production" workflow
3. Click "Run workflow"
4. Optionally specify image tag (defaults to `latest`)
5. Click "Run workflow" to confirm
### Deployment State
### 4. Monitor Production Deploy
State is tracked in `config/deployment/state.json`:
```json
{
"active_stack": "blue",
"inactive_stack": "green",
"last_deployment": "2024-01-15T10:30:00Z",
"last_deployment_commit": "abc123",
"rollback_available": true
}
```
The production workflow will:
1. Determine target stack (blue or green)
2. Pull and start the new stack
3. Run health checks
4. Switch traffic
5. Verify external health
6. Auto-rollback if verification fails
7. Send email notification
---
## CI/CD Variables Configuration
## Secrets and Variables
Navigate to **Settings > CI/CD > Variables** in your GitLab project.
### Secrets Configuration
### Required Variables
Navigate to: `git.motovaultpro.com/egullickson/motovaultpro/settings/actions/secrets`
| Variable | Type | Protected | Purpose |
|----------|------|-----------|---------|
| `DEPLOY_NOTIFY_EMAIL` | Variable | Yes | Notification recipient |
| `VITE_AUTH0_DOMAIN` | Variable | No | Auth0 domain |
| `VITE_AUTH0_CLIENT_ID` | Variable | No | Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | Variable | No | Auth0 audience |
| Secret | Description |
|--------|-------------|
| `REGISTRY_USER` | Gitea username (egullickson) |
| `REGISTRY_PASSWORD` | Gitea access token |
| `POSTGRES_PASSWORD` | Production PostgreSQL password |
| `STAGING_POSTGRES_PASSWORD` | Staging PostgreSQL password |
| `AUTH0_CLIENT_SECRET` | Auth0 secret |
| `AUTH0_MANAGEMENT_CLIENT_ID` | Auth0 Management API ID |
| `AUTH0_MANAGEMENT_CLIENT_SECRET` | Auth0 Management API secret |
| `GOOGLE_MAPS_API_KEY` | Google Maps key |
| `GOOGLE_MAPS_MAP_ID` | Google Maps Map ID |
| `CF_DNS_API_TOKEN` | Cloudflare DNS token |
| `RESEND_API_KEY` | Resend email API key |
### Secret Files
### Variables Configuration
These use GitLab's **File** type and are injected via `scripts/inject-secrets.sh`:
Navigate to: `git.motovaultpro.com/egullickson/motovaultpro/settings/actions/variables`
| Variable | Type | Protected | Masked |
|----------|------|-----------|--------|
| `POSTGRES_PASSWORD` | File | Yes | Yes |
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes |
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes |
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No |
| `CF_DNS_API_TOKEN` | File | Yes | Yes |
| `RESEND_API_KEY` | File | Yes | Yes |
### Registry Authentication
GitLab provides these automatically:
- `CI_REGISTRY_USER` - Registry username
- `CI_REGISTRY_PASSWORD` - Registry token
- `CI_REGISTRY` - Registry URL
| Variable | Value |
|----------|-------|
| `DEPLOY_NOTIFY_EMAIL` | Notification recipient |
| `VITE_AUTH0_DOMAIN` | motovaultpro.us.auth0.com |
| `VITE_AUTH0_CLIENT_ID` | Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | https://api.motovaultpro.com |
---
## Container Registry
All images are hosted on the GitLab Container Registry to avoid Docker Hub rate limits.
All images are hosted on Gitea Package Registry.
### Registry URL
```
registry.motovaultpro.com
git.motovaultpro.com
```
### Image Paths
| Image | Path |
|-------|------|
| Backend | `registry.motovaultpro.com/motovaultpro/backend:$TAG` |
| Frontend | `registry.motovaultpro.com/motovaultpro/frontend:$TAG` |
| Mirrors | `registry.motovaultpro.com/mirrors/` |
| Backend | `git.motovaultpro.com/egullickson/backend:$TAG` |
| Frontend | `git.motovaultpro.com/egullickson/frontend:$TAG` |
| Mirrors | `git.motovaultpro.com/egullickson/mirrors/` |
### Base Image Mirrors
Mirror upstream images to avoid rate limits:
Run the mirror workflow to avoid Docker Hub rate limits:
```bash
# Run manually or via scheduled pipeline
./scripts/ci/mirror-base-images.sh
```
1. Go to Actions tab
2. Select "Mirror Base Images"
3. Click "Run workflow"
Mirrored images:
- `node:20-alpine`
@@ -255,41 +227,6 @@ Mirrored images:
- `postgres:18-alpine`
- `redis:8.4-alpine`
- `traefik:v3.6`
- `docker:24.0`
- `docker:24.0-dind`
---
## Deployment Process
### Automatic Deployment
Deployments trigger automatically on push to `main`:
1. **Validate**: Check prerequisites, determine target stack
2. **Build**: Build images on dedicated build server
3. **Prepare**: Start inactive stack, run health checks
4. **Switch**: Update Traefik weights (instant)
5. **Verify**: External health check
6. **Notify**: Send email notification
### Manual Deployment
1. Go to **CI/CD > Pipelines**
2. Click **Run pipeline**
3. Select `main` branch
4. Click **Run pipeline**
### Deployment Timeline
| Phase | Duration |
|-------|----------|
| Validate | ~5s |
| Build | ~2 min |
| Deploy-prepare | ~30s |
| Deploy-switch | ~3s |
| Verify | ~30s |
| **Total** | ~3 min |
---
@@ -297,15 +234,9 @@ Deployments trigger automatically on push to `main`:
### Automatic Rollback
Triggers automatically when:
- Health check fails in `deploy-prepare`
- `verify` stage fails after switch
- Container becomes unhealthy within verification period
The pipeline runs `scripts/ci/auto-rollback.sh` which:
1. Verifies previous stack is healthy
2. Switches traffic back
3. Sends notification
Production workflow auto-rolls back when:
- Health check fails after traffic switch
- Container becomes unhealthy during verification
### Manual Rollback
@@ -326,6 +257,8 @@ cat config/deployment/state.json | jq .
If both stacks are unhealthy:
```bash
cd /opt/motovaultpro
# Stop everything
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml down
@@ -336,8 +269,8 @@ docker compose up -d mvp-postgres mvp-redis mvp-traefik
sleep 15
# Start one stack
export BACKEND_IMAGE=registry.motovaultpro.com/motovaultpro/backend:latest
export FRONTEND_IMAGE=registry.motovaultpro.com/motovaultpro/frontend:latest
export BACKEND_IMAGE=git.motovaultpro.com/egullickson/backend:latest
export FRONTEND_IMAGE=git.motovaultpro.com/egullickson/frontend:latest
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
mvp-frontend-blue mvp-backend-blue
@@ -351,11 +284,13 @@ docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
For breaking database changes requiring downtime:
### Via Pipeline (Recommended)
### Via Gitea Actions
1. Go to **CI/CD > Pipelines**
2. Find the `maintenance-migration` job
3. Click **Play** to trigger manually
1. Go to Actions tab
2. Select "Maintenance Migration"
3. Click "Run workflow"
4. Choose whether to create backup
5. Click "Run workflow"
### Via Script
@@ -369,98 +304,68 @@ cd /opt/motovaultpro
./scripts/ci/maintenance-migrate.sh
```
### What Happens
1. Sends maintenance notification
2. Enables maintenance mode (stops traffic)
3. Creates database backup (if requested)
4. Runs migrations
5. Restarts backends
6. Restores traffic
7. Sends completion notification
---
## Notifications
Email notifications via Resend API for:
| Event | Subject |
|-------|---------|
| `success` | Deployment Successful |
| `failure` | Deployment Failed |
| `rollback` | Auto-Rollback Executed |
| `rollback_failed` | CRITICAL: Rollback Failed |
| `maintenance_start` | Maintenance Mode Started |
| `maintenance_end` | Maintenance Complete |
Configure recipient in GitLab CI/CD variables:
```
DEPLOY_NOTIFY_EMAIL = admin@example.com
```
---
## Troubleshooting
### Pipeline Fails at Build Stage
### Staging Workflow Failures
**Check build server connectivity:**
**Build fails:**
```bash
# On build server
sudo gitlab-runner verify
docker login registry.motovaultpro.com
```
**Check disk space:**
```bash
df -h
docker system df
docker system prune -af
```
### Pipeline Fails at Deploy-Prepare
**Container won't start:**
**Deploy fails:**
```bash
docker logs mvp-backend-blue --tail 100
docker logs mvp-frontend-blue --tail 100
docker logs mvp-backend-staging
docker logs mvp-frontend-staging
```
### Production Workflow Failures
**Health check timeout:**
```bash
# Increase timeout in .gitlab-ci.yml
HEALTH_CHECK_TIMEOUT: "90"
# Check containers
docker ps
docker logs mvp-backend-blue # or green
```
### Traffic Not Switching
**Check Traefik config:**
**Traffic not switching:**
```bash
# Check Traefik config
cat config/traefik/dynamic/blue-green.yml
docker exec mvp-traefik traefik healthcheck
```
**Check routing:**
### Runner Issues
**Runner offline:**
```bash
curl -I https://motovaultpro.com/api/health
sudo systemctl status act_runner
sudo journalctl -u act_runner -f
```
### Verify Stage Fails
**Check external connectivity:**
**Permission denied:**
```bash
curl -sf https://motovaultpro.com/api/health
```
**Check container health:**
```bash
docker inspect --format='{{.State.Health.Status}}' mvp-backend-blue
sudo usermod -aG docker act_runner
sudo systemctl restart act_runner
```
---
## Quick Reference
### Workflow Locations
| Workflow | File |
|----------|------|
| Staging | `.gitea/workflows/staging.yaml` |
| Production | `.gitea/workflows/production.yaml` |
| Maintenance | `.gitea/workflows/maintenance.yaml` |
| Mirror Images | `.gitea/workflows/mirror-images.yaml` |
### Important Paths
| Path | Description |
@@ -472,11 +377,11 @@ docker inspect --format='{{.State.Health.Status}}' mvp-backend-blue
### Common Commands
```bash
# View current state
# View deployment state
cat config/deployment/state.json | jq .
# Check container status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Health}}"
# Check containers
docker ps --format "table {{.Names}}\t{{.Status}}"
# View logs
docker logs mvp-backend-blue -f
@@ -486,21 +391,14 @@ docker logs mvp-backend-blue -f
# Run health check
./scripts/ci/health-check.sh blue
# Send test notification
./scripts/ci/notify.sh success "Test message"
```
### Memory Budget (8GB Server)
### Email Notifications
| Component | RAM |
|-----------|-----|
| Blue frontend | 512MB |
| Blue backend | 1GB |
| Green frontend | 512MB |
| Green backend | 1GB |
| PostgreSQL | 2GB |
| Redis | 512MB |
| Traefik | 128MB |
| System | 1.3GB |
| **Total** | ~7GB |
| Event | Trigger |
|-------|---------|
| Staging Ready | Staging verified successfully |
| Success | Production deployed successfully |
| Failure | Deployment or verification failed |
| Rollback | Auto-rollback executed |
| Maintenance | Migration started/completed |