egullickson/motovaultpro
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

CI/CD Gitea v1.0
Some checks failed
Deploy to Staging / Build Images (push) Failing after 7s
Details
Deploy to Staging / Deploy to Staging (push) Has been skipped
Details
Deploy to Staging / Verify Staging (push) Has been skipped
Details
Deploy to Staging / Notify Staging Ready (push) Has been skipped
Details
Deploy to Staging / Notify Staging Failure (push) Failing after 6s
Details

Browse Source
This commit is contained in:
Eric Gullickson
2025-12-29 18:51:41 -06:00
parent 9b0de6a5b8
commit 83d79da3aa
15 changed files with 1101 additions and 929 deletions
Download Patch File Download Diff File Expand all files Collapse all files

335
docs/BUILD-SERVER-SETUP.md
View File

@@ -1,22 +1,28 @@
# Build Server Setup Guide
# Build/Staging Server Setup Guide
Complete guide for setting up a dedicated build VPS for MotoVaultPro CI/CD pipeline.
Complete guide for setting up the build and staging server for MotoVaultPro CI/CD with Gitea Actions.
## Overview
The build server isolates resource-intensive Docker builds from the production server, ensuring deployments don't impact application performance.
The build server serves dual purposes:
1. **Build Server**: Builds Docker images and pushes to Gitea Package Registry
2. **Staging Server**: Runs full application stack at staging.motovaultpro.com
```
+-------------------+ +--------------------+
| GitLab Server | | Production Server |
| (CI/CD + Registry)| | (Shell Runner) |
+--------+----------+ +----------+---------+
| |
v v
+--------+----------+ +----------+---------+
| Build VPS | | Blue-Green Stacks |
| (Docker Runner) |---->| + Shared Data |
+-------------------+ +--------------------+
| Gitea Server | | Production Server |
| git.motovaultpro | | (mvp-prod runner) |
| + Package Registry| +----------+---------+
+--------+----------+ |
| v
v motovaultpro.com
+--------+----------+
| Build/Staging VPS |
| (mvp-build runner)|
+-------------------+
|
v
staging.motovaultpro.com
```
## Server Requirements
@@ -25,16 +31,16 @@ The build server isolates resource-intensive Docker builds from the production s
| Resource | Requirement |
|----------|-------------|
| CPU | 2 cores |
| RAM | 4GB |
| Storage | 50GB SSD |
| CPU | 4 cores |
| RAM | 8GB |
| Storage | 100GB SSD |
| Network | 100Mbps+ |
| OS | Ubuntu 22.04 LTS / Debian 12 |
### Network Requirements
- Outbound HTTPS to GitLab instance
- Outbound HTTPS to Docker registries (for fallback)
- Port 80/443 open (for staging.motovaultpro.com)
- Outbound HTTPS to git.motovaultpro.com
- SSH access for administration
---
@@ -45,7 +51,7 @@ The build server isolates resource-intensive Docker builds from the production s
```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl git ca-certificates gnupg
sudo apt install -y curl git ca-certificates gnupg jq
```
### 2. Install Docker Engine
@@ -56,7 +62,7 @@ sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# Add the repository to Apt sources
# Add the repository
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
@@ -71,95 +77,162 @@ docker --version
docker compose version
```
### 3. Install GitLab Runner
### 3. Install act_runner
```bash
# Add GitLab Runner repository
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
# Install GitLab Runner
sudo apt install gitlab-runner
# Download act_runner binary
curl -L https://gitea.com/gitea/act_runner/releases/download/v0.2.12/act_runner-0.2.12-linux-amd64 -o /tmp/act_runner
sudo mv /tmp/act_runner /usr/local/bin/act_runner
sudo chmod +x /usr/local/bin/act_runner
# Verify installation
gitlab-runner --version
act_runner --version
```
### 4. Register Runner with Shell Executor
### 4. Create act_runner User
```bash
sudo gitlab-runner register \
--non-interactive \
--url "https://git.motovaultpro.com" \
--registration-token "YOUR_REGISTRATION_TOKEN" \
--executor "shell" \
--description "Build Server - Shell Executor" \
--tag-list "build" \
--run-untagged="false" \
--locked="true"
# Create user for running act_runner
sudo useradd -r -m -s /bin/bash act_runner
sudo usermod -aG docker act_runner
# Create config directory
sudo mkdir -p /etc/act_runner
sudo chown act_runner:act_runner /etc/act_runner
```
**Notes:**
- Replace `YOUR_REGISTRATION_TOKEN` with the token from GitLab Admin > CI/CD > Runners
- Shell executor runs jobs directly on the host with access to Docker
- Tag `build` is used in `.gitlab-ci.yml` to route build jobs to this server
### 5. Register Runner with Gitea
### 5. Add gitlab-runner to Docker Group
The gitlab-runner user needs access to Docker:
Get a registration token from: `git.motovaultpro.com/egullickson/motovaultpro/settings/actions/runners`
```bash
sudo usermod -aG docker gitlab-runner
# Generate config
sudo -u act_runner act_runner generate-config > /etc/act_runner/config.yaml
# Verify access
sudo -u gitlab-runner docker info
sudo -u gitlab-runner docker compose version
# Register runner with staging/build label
sudo -u act_runner act_runner register --no-interactive \
--instance https://git.motovaultpro.com \
--token <REGISTRATION_TOKEN> \
--name "Build/Staging Server" \
--labels "mvp-build:host"
```
### 6. Configure Docker Registry Authentication
Create credentials file for GitLab Container Registry:
### 6. Create Systemd Service
```bash
# Login to GitLab Container Registry (creates ~/.docker/config.json)
docker login registry.motovaultpro.com -u <deploy-token-username> -p <deploy-token>
cat << 'EOF' | sudo tee /etc/systemd/system/act_runner.service
[Unit]
Description=Gitea Actions Runner
After=docker.service network.target
[Service]
ExecStart=/usr/local/bin/act_runner daemon --config /etc/act_runner/config.yaml
WorkingDirectory=/home/act_runner
User=act_runner
Group=act_runner
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable act_runner --now
sudo systemctl status act_runner
```
**Creating Deploy Token:**
1. Go to GitLab Project > Settings > Repository > Deploy Tokens
2. Create token with `read_registry` and `write_registry` scopes
3. Use the token username/password for Docker login
---
## Staging Environment Setup
### 1. Clone Repository
```bash
sudo mkdir -p /opt/motovaultpro
sudo chown act_runner:act_runner /opt/motovaultpro
sudo -u act_runner git clone https://git.motovaultpro.com/egullickson/motovaultpro.git /opt/motovaultpro
```
### 2. Create Staging Secrets Directory
```bash
sudo mkdir -p /opt/motovaultpro/secrets/staging
sudo chown -R act_runner:act_runner /opt/motovaultpro/secrets
sudo chmod 700 /opt/motovaultpro/secrets/staging
```
### 3. Configure DNS
Add DNS A record:
```
staging.motovaultpro.com -> <build-server-ip>
```
### 4. Configure Cloudflare (if using)
Ensure `staging.motovaultpro.com` is proxied through Cloudflare or has a valid SSL certificate configured.
### 5. Initialize Data Directories
```bash
cd /opt/motovaultpro
sudo mkdir -p data/backups data/documents
sudo chown -R 1001:1001 data/backups data/documents
```
---
## Docker Registry Authentication
### Login to Gitea Package Registry
```bash
# Login as act_runner user
sudo -u act_runner docker login git.motovaultpro.com -u egullickson
# Enter your Gitea access token when prompted
```
### Create Access Token
1. Go to `git.motovaultpro.com/user/settings/applications`
2. Create new token with scopes:
- `read:packages`
- `write:packages`
3. Save token securely
---
## Verification
### Test Runner Registration
### Check Runner Status
```bash
sudo gitlab-runner verify
sudo systemctl status act_runner
```
Expected output:
```
Verifying runner... is alive runner=XXXXXX
```
### Check Runner Registration
Go to `git.motovaultpro.com/egullickson/motovaultpro/settings/actions/runners` and verify the runner appears as "Online".
### Test Docker Access
```bash
sudo gitlab-runner exec docker --docker-privileged test-job
sudo -u act_runner docker info
sudo -u act_runner docker compose version
```
### Test Registry Push
```bash
# Build and push a test image
docker build -t registry.motovaultpro.com/motovaultpro/test:latest -f- . <<EOF
sudo -u act_runner docker build -t git.motovaultpro.com/egullickson/test:latest -f- . <<EOF
FROM alpine:latest
RUN echo "test"
EOF
docker push registry.motovaultpro.com/motovaultpro/test:latest
sudo -u act_runner docker push git.motovaultpro.com/egullickson/test:latest
```
---
@@ -168,8 +241,6 @@ docker push registry.motovaultpro.com/motovaultpro/test:latest
### Disk Cleanup
Docker builds accumulate disk space. Set up automated cleanup:
```bash
# Create cleanup script
sudo tee /usr/local/bin/docker-cleanup.sh > /dev/null <<'EOF'
@@ -185,102 +256,64 @@ sudo chmod +x /usr/local/bin/docker-cleanup.sh
echo "0 3 * * * /usr/local/bin/docker-cleanup.sh >> /var/log/docker-cleanup.log 2>&1" | sudo crontab -
```
### Log Rotation
Configure log rotation for GitLab Runner:
```bash
sudo tee /etc/logrotate.d/gitlab-runner > /dev/null <<EOF
/var/log/gitlab-runner/*.log {
daily
rotate 7
compress
missingok
notifempty
}
EOF
```
### Update Runner
```bash
# Update GitLab Runner
sudo apt update
sudo apt upgrade gitlab-runner
# Download new version
curl -L https://gitea.com/gitea/act_runner/releases/download/v0.2.12/act_runner-0.2.12-linux-amd64 -o /tmp/act_runner
sudo mv /tmp/act_runner /usr/local/bin/act_runner
sudo chmod +x /usr/local/bin/act_runner
# Restart runner
sudo gitlab-runner restart
# Restart service
sudo systemctl restart act_runner
```
---
## Security Considerations
### Firewall Configuration
```bash
# Allow only necessary outbound traffic
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable
```
### Runner Security
- **Locked runner**: Only accepts jobs from the specific project
- **Protected tags**: Only runs on protected branches (main)
- **Docker socket**: Mounted read-only where possible
### Secrets Management
The build server does NOT store application secrets. All secrets are:
- Stored in GitLab CI/CD Variables
- Injected at runtime on the production server
- Never cached in Docker layers
---
## Troubleshooting
### Runner Not Picking Up Jobs
```bash
# Check runner status
sudo gitlab-runner status
# Check service status
sudo systemctl status act_runner
# View runner logs
sudo journalctl -u gitlab-runner -f
# View logs
sudo journalctl -u act_runner -f
# Re-register runner if needed
sudo gitlab-runner unregister --all-runners
sudo gitlab-runner register
# Check registration
sudo -u act_runner act_runner list
```
### Docker Build Failures
### Docker Permission Issues
```bash
# Check Docker daemon
sudo systemctl status docker
# Ensure act_runner is in docker group
sudo usermod -aG docker act_runner
# Check available disk space
df -h
# Clear Docker cache
docker system prune -af
# Restart service
sudo systemctl restart act_runner
```
### Registry Push Failures
### Registry Authentication Failures
```bash
# Verify registry login
docker login registry.motovaultpro.com
# Re-login to registry
sudo -u act_runner docker logout git.motovaultpro.com
sudo -u act_runner docker login git.motovaultpro.com -u egullickson
```
# Check network connectivity
curl -v https://registry.motovaultpro.com/v2/
### Staging Not Accessible
# Verify image exists
docker images | grep motovaultpro
```bash
# Check containers
docker ps
# Check Traefik logs
docker logs mvp-traefik-staging
# Check SSL certificate
curl -vI https://staging.motovaultpro.com
```
---
@@ -291,25 +324,27 @@ docker images | grep motovaultpro
| Path | Description |
|------|-------------|
| `/etc/gitlab-runner/config.toml` | Runner configuration |
| `/var/log/gitlab-runner/` | Runner logs |
| `~/.docker/config.json` | Docker registry credentials |
| `/var/lib/docker/` | Docker data |
| `/opt/motovaultpro` | Application root |
| `/opt/motovaultpro/secrets/staging` | Staging secrets |
| `/etc/act_runner/config.yaml` | Runner configuration |
| `/home/act_runner/.docker/config.json` | Registry credentials |
### Common Commands
```bash
# Runner management
sudo gitlab-runner status
sudo gitlab-runner restart
sudo gitlab-runner verify
sudo systemctl status act_runner
sudo systemctl restart act_runner
sudo journalctl -u act_runner -f
# Docker management
docker system df # Check disk usage
docker system prune -af # Clean all unused resources
docker images # List images
docker ps -a # List containers
docker system df
docker system prune -af
docker ps
docker logs -f mvp-backend-staging
# View build logs
sudo journalctl -u gitlab-runner --since "1 hour ago"
# Staging stack
cd /opt/motovaultpro
docker compose -f docker-compose.yml -f docker-compose.staging.yml ps
docker compose -f docker-compose.yml -f docker-compose.staging.yml logs -f
```

466
docs/CICD-DEPLOY.md
View File

@@ -1,72 +1,52 @@
# MotoVaultPro GitLab CI/CD Deployment Guide
# MotoVaultPro CI/CD Deployment Guide
Complete guide for deploying MotoVaultPro using GitLab CI/CD with blue-green deployment and auto-rollback.
Complete guide for deploying MotoVaultPro using Gitea Actions with staging-first deployment and manual production approval.
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Prerequisites](#prerequisites)
3. [Pipeline Stages](#pipeline-stages)
4. [Blue-Green Deployment](#blue-green-deployment)
5. [CI/CD Variables Configuration](#cicd-variables-configuration)
3. [Workflow Structure](#workflow-structure)
4. [Deployment Process](#deployment-process)
5. [Secrets and Variables](#secrets-and-variables)
6. [Container Registry](#container-registry)
7. [Deployment Process](#deployment-process)
8. [Rollback Procedures](#rollback-procedures)
9. [Maintenance Migrations](#maintenance-migrations)
10. [Notifications](#notifications)
11. [Troubleshooting](#troubleshooting)
7. [Rollback Procedures](#rollback-procedures)
8. [Maintenance Migrations](#maintenance-migrations)
9. [Troubleshooting](#troubleshooting)
---
## Architecture Overview
MotoVaultPro uses a blue-green deployment strategy with automatic rollback:
MotoVaultPro uses a **staging-first deployment** strategy with manual approval:
```
+---------------------------------------------------+
| GitLab (CI/CD + Registry) |
| Gitea (git.motovaultpro.com) |
| +--------------------+ +--------------------+ |
| | Gitea Actions | | Package Registry | |
| +--------------------+ +--------------------+ |
+---------------------------------------------------+
| |
v v
+------------------+ +-----------------------+
| Build VPS | | Production Server |
| (Docker Runner) | | (Shell Runner) |
| Tags: build | | Tags: production |
+------------------+ +-----------+-----------+
+-------------------+ +--------------------+
| Build/Staging VPS | | Production Server |
| (mvp-build) | | (mvp-prod) |
| act_runner | | act_runner |
+--------+----------+ +----------+---------+
| |
| Push images | Pull + Deploy
v v
+---------------------------------------------------+
| GitLab Container Registry |
| registry.motovaultpro.com/motovaultpro/ |
+---------------------------------------------------+
|
+---------------+---------------+
| |
+--------v--------+ +--------v--------+
| BLUE Stack | | GREEN Stack |
| mvp-frontend | | mvp-frontend |
| mvp-backend | | mvp-backend |
+-----------------+ +-----------------+
| |
+----------- Traefik -----------+
(weighted LB)
|
+---------------+---------------+
| |
+--------v--------+ +--------v--------+
| PostgreSQL | | Redis |
| (shared) | | (shared) |
+-----------------+ +-----------------+
staging.motovaultpro.com motovaultpro.com
(Full Stack) (Blue-Green)
```
### Key Features
- **Zero-downtime deployments**: Traffic switches in under 5 seconds
- **Instant rollback**: Previous version remains running
- **Automatic rollback**: On health check failure
- **Staging-first**: All changes verified on staging before production
- **Manual approval**: Production deploy requires manual trigger
- **Blue-green production**: Zero-downtime deployments
- **Auto-rollback**: Automatic rollback on health check failure
- **Email notifications**: Via Resend API
- **Container registry**: Self-hosted on GitLab (no Docker Hub)
---
@@ -74,180 +54,172 @@ MotoVaultPro uses a blue-green deployment strategy with automatic rollback:
### Server Requirements
| Server | Purpose | Specs | Runner Tags |
|--------|---------|-------|-------------|
| Build VPS | Docker image builds | 2 CPU, 4GB RAM | `build` |
| Prod Server | Application hosting | 8GB+ RAM | `production` |
| Server | Purpose | Specs | Runner Label |
|--------|---------|-------|--------------|
| Build/Staging VPS | Build + Staging | 4 CPU, 8GB RAM | `mvp-build` |
| Prod Server | Production | 8GB+ RAM | `mvp-prod` |
See [BUILD-SERVER-SETUP.md](BUILD-SERVER-SETUP.md) for build server setup.
See [BUILD-SERVER-SETUP.md](BUILD-SERVER-SETUP.md) for setup instructions.
### Software Requirements
- GitLab 18.6+
- Gitea 1.21+ with Actions enabled
- Docker Engine 24.0+
- Docker Compose v2
- GitLab Runner (shell executor on both servers)
- act_runner (Gitea Actions runner)
- `jq` for JSON processing
---
## Pipeline Stages
## Workflow Structure
The CI/CD pipeline consists of 7 stages:
### Two-Workflow Strategy
```
validate -> build -> deploy-prepare -> deploy-switch -> verify -> [rollback] -> notify
```
| Workflow | Trigger | Purpose |
|----------|---------|---------|
| `staging.yaml` | Push to main | Build, deploy to staging, verify |
| `production.yaml` | Manual (workflow_dispatch) | Deploy to production |
| Stage | Runner | Purpose |
|-------|--------|---------|
| `validate` | prod | Check prerequisites, determine target stack |
| `build` | build | Build and push images to GitLab registry |
| `deploy-prepare` | prod | Pull images, start inactive stack, health check |
| `deploy-switch` | prod | Switch Traefik traffic weights |
| `verify` | prod | Production health verification |
| `rollback` | prod | Auto-triggered on verify failure |
| `notify` | prod | Email success/failure notifications |
### Pipeline Flow
### Staging Workflow (Automatic)
```
[Push to main]
|
v
[validate] - Checks Docker, paths, registry
[build] -------- Build images, push to registry
|
v
[build] - Builds backend + frontend images
| Pushes to registry.motovaultpro.com
v
[deploy-prepare] - Pulls new images
| Starts inactive stack (blue or green)
| Runs health checks
v
[deploy-switch] - Updates Traefik weights
| Switches traffic instantly
v
[verify] - External health check
| Container status verification
[deploy-staging] - Deploy full stack to staging
|
+--[SUCCESS]--> [notify-success] - Sends success email
v
[verify-staging] - Health checks
|
+--[FAILURE]--> [rollback] - Switches back to previous stack
|
v
[notify-failure] - Sends failure email
+--[FAIL]--> [notify-staging-failure]
|
v
[notify-staging-ready] - Email with production deploy link
```
### Production Workflow (Manual)
```
[Manual Trigger] - User clicks "Run workflow"
|
v
[validate] - Check prerequisites, determine stack
|
v
[deploy-prod] - Blue-green deployment
|
v
[verify-prod] - External health checks
|
+--[FAIL]--> [rollback] --> [notify-failure]
|
v
[notify-success]
```
---
## Blue-Green Deployment
## Deployment Process
### Stack Configuration
### 1. Push to Main Branch
Both stacks share the same database layer:
When you push to `main`:
1. Staging workflow triggers automatically
2. Images are built and pushed to Gitea Package Registry
3. Full stack deploys to staging.motovaultpro.com
4. Health checks verify staging works
5. Email notification sent with production deploy link
| Component | Blue Stack | Green Stack | Shared |
|-----------|------------|-------------|--------|
| Frontend | `mvp-frontend-blue` | `mvp-frontend-green` | - |
| Backend | `mvp-backend-blue` | `mvp-backend-green` | - |
| PostgreSQL | - | - | `mvp-postgres` |
| Redis | - | - | `mvp-redis` |
| Traefik | - | - | `mvp-traefik` |
### 2. Review Staging
### Traffic Routing
After receiving the "Staging Ready" email:
1. Visit https://staging.motovaultpro.com
2. Test functionality
3. Review logs if needed: `docker logs mvp-backend-staging`
Traefik uses weighted services for traffic distribution:
### 3. Deploy to Production
```yaml
# config/traefik/dynamic/blue-green.yml
services:
mvp-frontend-weighted:
weighted:
services:
- name: mvp-frontend-blue-svc
weight: 100 # Active
- name: mvp-frontend-green-svc
weight: 0 # Standby
```
When ready to deploy:
1. Go to `git.motovaultpro.com/egullickson/motovaultpro/actions`
2. Select "Deploy to Production" workflow
3. Click "Run workflow"
4. Optionally specify image tag (defaults to `latest`)
5. Click "Run workflow" to confirm
### Deployment State
### 4. Monitor Production Deploy
State is tracked in `config/deployment/state.json`:
```json
{
"active_stack": "blue",
"inactive_stack": "green",
"last_deployment": "2024-01-15T10:30:00Z",
"last_deployment_commit": "abc123",
"rollback_available": true
}
```
The production workflow will:
1. Determine target stack (blue or green)
2. Pull and start the new stack
3. Run health checks
4. Switch traffic
5. Verify external health
6. Auto-rollback if verification fails
7. Send email notification
---
## CI/CD Variables Configuration
## Secrets and Variables
Navigate to **Settings > CI/CD > Variables** in your GitLab project.
### Secrets Configuration
### Required Variables
Navigate to: `git.motovaultpro.com/egullickson/motovaultpro/settings/actions/secrets`
| Variable | Type | Protected | Purpose |
|----------|------|-----------|---------|
| `DEPLOY_NOTIFY_EMAIL` | Variable | Yes | Notification recipient |
| `VITE_AUTH0_DOMAIN` | Variable | No | Auth0 domain |
| `VITE_AUTH0_CLIENT_ID` | Variable | No | Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | Variable | No | Auth0 audience |
| Secret | Description |
|--------|-------------|
| `REGISTRY_USER` | Gitea username (egullickson) |
| `REGISTRY_PASSWORD` | Gitea access token |
| `POSTGRES_PASSWORD` | Production PostgreSQL password |
| `STAGING_POSTGRES_PASSWORD` | Staging PostgreSQL password |
| `AUTH0_CLIENT_SECRET` | Auth0 secret |
| `AUTH0_MANAGEMENT_CLIENT_ID` | Auth0 Management API ID |
| `AUTH0_MANAGEMENT_CLIENT_SECRET` | Auth0 Management API secret |
| `GOOGLE_MAPS_API_KEY` | Google Maps key |
| `GOOGLE_MAPS_MAP_ID` | Google Maps Map ID |
| `CF_DNS_API_TOKEN` | Cloudflare DNS token |
| `RESEND_API_KEY` | Resend email API key |
### Secret Files
### Variables Configuration
These use GitLab's **File** type and are injected via `scripts/inject-secrets.sh`:
Navigate to: `git.motovaultpro.com/egullickson/motovaultpro/settings/actions/variables`
| Variable | Type | Protected | Masked |
|----------|------|-----------|--------|
| `POSTGRES_PASSWORD` | File | Yes | Yes |
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes |
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes |
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No |
| `CF_DNS_API_TOKEN` | File | Yes | Yes |
| `RESEND_API_KEY` | File | Yes | Yes |
### Registry Authentication
GitLab provides these automatically:
- `CI_REGISTRY_USER` - Registry username
- `CI_REGISTRY_PASSWORD` - Registry token
- `CI_REGISTRY` - Registry URL
| Variable | Value |
|----------|-------|
| `DEPLOY_NOTIFY_EMAIL` | Notification recipient |
| `VITE_AUTH0_DOMAIN` | motovaultpro.us.auth0.com |
| `VITE_AUTH0_CLIENT_ID` | Auth0 client ID |
| `VITE_AUTH0_AUDIENCE` | https://api.motovaultpro.com |
---
## Container Registry
All images are hosted on the GitLab Container Registry to avoid Docker Hub rate limits.
All images are hosted on Gitea Package Registry.
### Registry URL
```
registry.motovaultpro.com
git.motovaultpro.com
```
### Image Paths
| Image | Path |
|-------|------|
| Backend | `registry.motovaultpro.com/motovaultpro/backend:$TAG` |
| Frontend | `registry.motovaultpro.com/motovaultpro/frontend:$TAG` |
| Mirrors | `registry.motovaultpro.com/mirrors/` |
| Backend | `git.motovaultpro.com/egullickson/backend:$TAG` |
| Frontend | `git.motovaultpro.com/egullickson/frontend:$TAG` |
| Mirrors | `git.motovaultpro.com/egullickson/mirrors/` |
### Base Image Mirrors
Mirror upstream images to avoid rate limits:
Run the mirror workflow to avoid Docker Hub rate limits:
```bash
# Run manually or via scheduled pipeline
./scripts/ci/mirror-base-images.sh
```
1. Go to Actions tab
2. Select "Mirror Base Images"
3. Click "Run workflow"
Mirrored images:
- `node:20-alpine`
@@ -255,41 +227,6 @@ Mirrored images:
- `postgres:18-alpine`
- `redis:8.4-alpine`
- `traefik:v3.6`
- `docker:24.0`
- `docker:24.0-dind`
---
## Deployment Process
### Automatic Deployment
Deployments trigger automatically on push to `main`:
1. **Validate**: Check prerequisites, determine target stack
2. **Build**: Build images on dedicated build server
3. **Prepare**: Start inactive stack, run health checks
4. **Switch**: Update Traefik weights (instant)
5. **Verify**: External health check
6. **Notify**: Send email notification
### Manual Deployment
1. Go to **CI/CD > Pipelines**
2. Click **Run pipeline**
3. Select `main` branch
4. Click **Run pipeline**
### Deployment Timeline
| Phase | Duration |
|-------|----------|
| Validate | ~5s |
| Build | ~2 min |
| Deploy-prepare | ~30s |
| Deploy-switch | ~3s |
| Verify | ~30s |
| **Total** | ~3 min |
---
@@ -297,15 +234,9 @@ Deployments trigger automatically on push to `main`:
### Automatic Rollback
Triggers automatically when:
- Health check fails in `deploy-prepare`
- `verify` stage fails after switch
- Container becomes unhealthy within verification period
The pipeline runs `scripts/ci/auto-rollback.sh` which:
1. Verifies previous stack is healthy
2. Switches traffic back
3. Sends notification
Production workflow auto-rolls back when:
- Health check fails after traffic switch
- Container becomes unhealthy during verification
### Manual Rollback
@@ -326,6 +257,8 @@ cat config/deployment/state.json | jq .
If both stacks are unhealthy:
```bash
cd /opt/motovaultpro
# Stop everything
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml down
@@ -336,8 +269,8 @@ docker compose up -d mvp-postgres mvp-redis mvp-traefik
sleep 15
# Start one stack
export BACKEND_IMAGE=registry.motovaultpro.com/motovaultpro/backend:latest
export FRONTEND_IMAGE=registry.motovaultpro.com/motovaultpro/frontend:latest
export BACKEND_IMAGE=git.motovaultpro.com/egullickson/backend:latest
export FRONTEND_IMAGE=git.motovaultpro.com/egullickson/frontend:latest
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
mvp-frontend-blue mvp-backend-blue
@@ -351,11 +284,13 @@ docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
For breaking database changes requiring downtime:
### Via Pipeline (Recommended)
### Via Gitea Actions
1. Go to **CI/CD > Pipelines**
2. Find the `maintenance-migration` job
3. Click **Play** to trigger manually
1. Go to Actions tab
2. Select "Maintenance Migration"
3. Click "Run workflow"
4. Choose whether to create backup
5. Click "Run workflow"
### Via Script
@@ -369,98 +304,68 @@ cd /opt/motovaultpro
./scripts/ci/maintenance-migrate.sh
```
### What Happens
1. Sends maintenance notification
2. Enables maintenance mode (stops traffic)
3. Creates database backup (if requested)
4. Runs migrations
5. Restarts backends
6. Restores traffic
7. Sends completion notification
---
## Notifications
Email notifications via Resend API for:
| Event | Subject |
|-------|---------|
| `success` | Deployment Successful |
| `failure` | Deployment Failed |
| `rollback` | Auto-Rollback Executed |
| `rollback_failed` | CRITICAL: Rollback Failed |
| `maintenance_start` | Maintenance Mode Started |
| `maintenance_end` | Maintenance Complete |
Configure recipient in GitLab CI/CD variables:
```
DEPLOY_NOTIFY_EMAIL = admin@example.com
```
---
## Troubleshooting
### Pipeline Fails at Build Stage
### Staging Workflow Failures
**Check build server connectivity:**
**Build fails:**
```bash
# On build server
sudo gitlab-runner verify
docker login registry.motovaultpro.com
```
**Check disk space:**
```bash
df -h
docker system df
docker system prune -af
```
### Pipeline Fails at Deploy-Prepare
**Container won't start:**
**Deploy fails:**
```bash
docker logs mvp-backend-blue --tail 100
docker logs mvp-frontend-blue --tail 100
docker logs mvp-backend-staging
docker logs mvp-frontend-staging
```
### Production Workflow Failures
**Health check timeout:**
```bash
# Increase timeout in .gitlab-ci.yml
HEALTH_CHECK_TIMEOUT: "90"
# Check containers
docker ps
docker logs mvp-backend-blue # or green
```
### Traffic Not Switching
**Check Traefik config:**
**Traffic not switching:**
```bash
# Check Traefik config
cat config/traefik/dynamic/blue-green.yml
docker exec mvp-traefik traefik healthcheck
```
**Check routing:**
### Runner Issues
**Runner offline:**
```bash
curl -I https://motovaultpro.com/api/health
sudo systemctl status act_runner
sudo journalctl -u act_runner -f
```
### Verify Stage Fails
**Check external connectivity:**
**Permission denied:**
```bash
curl -sf https://motovaultpro.com/api/health
```
**Check container health:**
```bash
docker inspect --format='{{.State.Health.Status}}' mvp-backend-blue
sudo usermod -aG docker act_runner
sudo systemctl restart act_runner
```
---
## Quick Reference
### Workflow Locations
| Workflow | File |
|----------|------|
| Staging | `.gitea/workflows/staging.yaml` |
| Production | `.gitea/workflows/production.yaml` |
| Maintenance | `.gitea/workflows/maintenance.yaml` |
| Mirror Images | `.gitea/workflows/mirror-images.yaml` |
### Important Paths
| Path | Description |
@@ -472,11 +377,11 @@ docker inspect --format='{{.State.Health.Status}}' mvp-backend-blue
### Common Commands
```bash
# View current state
# View deployment state
cat config/deployment/state.json | jq .
# Check container status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Health}}"
# Check containers
docker ps --format "table {{.Names}}\t{{.Status}}"
# View logs
docker logs mvp-backend-blue -f
@@ -486,21 +391,14 @@ docker logs mvp-backend-blue -f
# Run health check
./scripts/ci/health-check.sh blue
# Send test notification
./scripts/ci/notify.sh success "Test message"
```
### Memory Budget (8GB Server)
### Email Notifications
| Component | RAM |
|-----------|-----|
| Blue frontend | 512MB |
| Blue backend | 1GB |
| Green frontend | 512MB |
| Green backend | 1GB |
| PostgreSQL | 2GB |
| Redis | 512MB |
| Traefik | 128MB |
| System | 1.3GB |
| **Total** | ~7GB |
| Event | Trigger |
|-------|---------|
| Staging Ready | Staging verified successfully |
| Success | Production deployed successfully |
| Failure | Deployment or verification failed |
| Rollback | Auto-rollback executed |
| Maintenance | Migration started/completed |