fix: I dunno, I'm making git server changes
This commit is contained in:
Eric Gullickson
315
docs/BUILD-SERVER-SETUP.md
Normal file
315
docs/BUILD-SERVER-SETUP.md
Normal file
@@ -0,0 +1,315 @@
|
||||
# Build Server Setup Guide
|
||||
|
||||
Complete guide for setting up a dedicated build VPS for MotoVaultPro CI/CD pipeline.
|
||||
|
||||
## Overview
|
||||
|
||||
The build server isolates resource-intensive Docker builds from the production server, ensuring deployments don't impact application performance.
|
||||
|
||||
```
|
||||
+-------------------+ +--------------------+
|
||||
| GitLab Server | | Production Server |
|
||||
| (CI/CD + Registry)| | (Shell Runner) |
|
||||
+--------+----------+ +----------+---------+
|
||||
| |
|
||||
v v
|
||||
+--------+----------+ +----------+---------+
|
||||
| Build VPS | | Blue-Green Stacks |
|
||||
| (Docker Runner) |---->| + Shared Data |
|
||||
+-------------------+ +--------------------+
|
||||
```
|
||||
|
||||
## Server Requirements
|
||||
|
||||
### Minimum Specifications
|
||||
|
||||
| Resource | Requirement |
|
||||
|----------|-------------|
|
||||
| CPU | 2 cores |
|
||||
| RAM | 4GB |
|
||||
| Storage | 50GB SSD |
|
||||
| Network | 100Mbps+ |
|
||||
| OS | Ubuntu 22.04 LTS / Debian 12 |
|
||||
|
||||
### Network Requirements
|
||||
|
||||
- Outbound HTTPS to GitLab instance
|
||||
- Outbound HTTPS to Docker registries (for fallback)
|
||||
- SSH access for administration
|
||||
|
||||
---
|
||||
|
||||
## Installation Steps
|
||||
|
||||
### 1. Update System
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
sudo apt install -y curl git ca-certificates gnupg
|
||||
```
|
||||
|
||||
### 2. Install Docker Engine
|
||||
|
||||
```bash
|
||||
# Add Docker's official GPG key
|
||||
sudo install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
|
||||
sudo chmod a+r /etc/apt/keyrings/docker.gpg
|
||||
|
||||
# Add the repository to Apt sources
|
||||
echo \
|
||||
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
|
||||
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
|
||||
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
|
||||
|
||||
# Install Docker
|
||||
sudo apt update
|
||||
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
|
||||
# Verify installation
|
||||
docker --version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
### 3. Install GitLab Runner
|
||||
|
||||
```bash
|
||||
# Add GitLab Runner repository
|
||||
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
|
||||
|
||||
# Install GitLab Runner
|
||||
sudo apt install gitlab-runner
|
||||
|
||||
# Verify installation
|
||||
gitlab-runner --version
|
||||
```
|
||||
|
||||
### 4. Register Runner with Shell Executor
|
||||
|
||||
```bash
|
||||
sudo gitlab-runner register \
|
||||
--non-interactive \
|
||||
--url "https://git.motovaultpro.com" \
|
||||
--registration-token "YOUR_REGISTRATION_TOKEN" \
|
||||
--executor "shell" \
|
||||
--description "Build Server - Shell Executor" \
|
||||
--tag-list "build" \
|
||||
--run-untagged="false" \
|
||||
--locked="true"
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
- Replace `YOUR_REGISTRATION_TOKEN` with the token from GitLab Admin > CI/CD > Runners
|
||||
- Shell executor runs jobs directly on the host with access to Docker
|
||||
- Tag `build` is used in `.gitlab-ci.yml` to route build jobs to this server
|
||||
|
||||
### 5. Add gitlab-runner to Docker Group
|
||||
|
||||
The gitlab-runner user needs access to Docker:
|
||||
|
||||
```bash
|
||||
sudo usermod -aG docker gitlab-runner
|
||||
|
||||
# Verify access
|
||||
sudo -u gitlab-runner docker info
|
||||
sudo -u gitlab-runner docker compose version
|
||||
```
|
||||
|
||||
### 6. Configure Docker Registry Authentication
|
||||
|
||||
Create credentials file for GitLab Container Registry:
|
||||
|
||||
```bash
|
||||
# Login to GitLab Container Registry (creates ~/.docker/config.json)
|
||||
docker login registry.motovaultpro.com -u <deploy-token-username> -p <deploy-token>
|
||||
```
|
||||
|
||||
**Creating Deploy Token:**
|
||||
1. Go to GitLab Project > Settings > Repository > Deploy Tokens
|
||||
2. Create token with `read_registry` and `write_registry` scopes
|
||||
3. Use the token username/password for Docker login
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
### Test Runner Registration
|
||||
|
||||
```bash
|
||||
sudo gitlab-runner verify
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Verifying runner... is alive runner=XXXXXX
|
||||
```
|
||||
|
||||
### Test Docker Access
|
||||
|
||||
```bash
|
||||
sudo gitlab-runner exec docker --docker-privileged test-job
|
||||
```
|
||||
|
||||
### Test Registry Push
|
||||
|
||||
```bash
|
||||
# Build and push a test image
|
||||
docker build -t registry.motovaultpro.com/motovaultpro/test:latest -f- . <<EOF
|
||||
FROM alpine:latest
|
||||
RUN echo "test"
|
||||
EOF
|
||||
|
||||
docker push registry.motovaultpro.com/motovaultpro/test:latest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Disk Cleanup
|
||||
|
||||
Docker builds accumulate disk space. Set up automated cleanup:
|
||||
|
||||
```bash
|
||||
# Create cleanup script
|
||||
sudo tee /usr/local/bin/docker-cleanup.sh > /dev/null <<'EOF'
|
||||
#!/bin/bash
|
||||
# Remove unused Docker resources older than 7 days
|
||||
docker system prune -af --filter "until=168h"
|
||||
docker volume prune -f
|
||||
EOF
|
||||
|
||||
sudo chmod +x /usr/local/bin/docker-cleanup.sh
|
||||
|
||||
# Add to crontab (run daily at 3 AM)
|
||||
echo "0 3 * * * /usr/local/bin/docker-cleanup.sh >> /var/log/docker-cleanup.log 2>&1" | sudo crontab -
|
||||
```
|
||||
|
||||
### Log Rotation
|
||||
|
||||
Configure log rotation for GitLab Runner:
|
||||
|
||||
```bash
|
||||
sudo tee /etc/logrotate.d/gitlab-runner > /dev/null <<EOF
|
||||
/var/log/gitlab-runner/*.log {
|
||||
daily
|
||||
rotate 7
|
||||
compress
|
||||
missingok
|
||||
notifempty
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
### Update Runner
|
||||
|
||||
```bash
|
||||
# Update GitLab Runner
|
||||
sudo apt update
|
||||
sudo apt upgrade gitlab-runner
|
||||
|
||||
# Restart runner
|
||||
sudo gitlab-runner restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Firewall Configuration
|
||||
|
||||
```bash
|
||||
# Allow only necessary outbound traffic
|
||||
sudo ufw default deny incoming
|
||||
sudo ufw default allow outgoing
|
||||
sudo ufw allow ssh
|
||||
sudo ufw enable
|
||||
```
|
||||
|
||||
### Runner Security
|
||||
|
||||
- **Locked runner**: Only accepts jobs from the specific project
|
||||
- **Protected tags**: Only runs on protected branches (main)
|
||||
- **Docker socket**: Mounted read-only where possible
|
||||
|
||||
### Secrets Management
|
||||
|
||||
The build server does NOT store application secrets. All secrets are:
|
||||
- Stored in GitLab CI/CD Variables
|
||||
- Injected at runtime on the production server
|
||||
- Never cached in Docker layers
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Runner Not Picking Up Jobs
|
||||
|
||||
```bash
|
||||
# Check runner status
|
||||
sudo gitlab-runner status
|
||||
|
||||
# View runner logs
|
||||
sudo journalctl -u gitlab-runner -f
|
||||
|
||||
# Re-register runner if needed
|
||||
sudo gitlab-runner unregister --all-runners
|
||||
sudo gitlab-runner register
|
||||
```
|
||||
|
||||
### Docker Build Failures
|
||||
|
||||
```bash
|
||||
# Check Docker daemon
|
||||
sudo systemctl status docker
|
||||
|
||||
# Check available disk space
|
||||
df -h
|
||||
|
||||
# Clear Docker cache
|
||||
docker system prune -af
|
||||
```
|
||||
|
||||
### Registry Push Failures
|
||||
|
||||
```bash
|
||||
# Verify registry login
|
||||
docker login registry.motovaultpro.com
|
||||
|
||||
# Check network connectivity
|
||||
curl -v https://registry.motovaultpro.com/v2/
|
||||
|
||||
# Verify image exists
|
||||
docker images | grep motovaultpro
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Important Paths
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `/etc/gitlab-runner/config.toml` | Runner configuration |
|
||||
| `/var/log/gitlab-runner/` | Runner logs |
|
||||
| `~/.docker/config.json` | Docker registry credentials |
|
||||
| `/var/lib/docker/` | Docker data |
|
||||
|
||||
### Common Commands
|
||||
|
||||
```bash
|
||||
# Runner management
|
||||
sudo gitlab-runner status
|
||||
sudo gitlab-runner restart
|
||||
sudo gitlab-runner verify
|
||||
|
||||
# Docker management
|
||||
docker system df # Check disk usage
|
||||
docker system prune -af # Clean all unused resources
|
||||
docker images # List images
|
||||
docker ps -a # List containers
|
||||
|
||||
# View build logs
|
||||
sudo journalctl -u gitlab-runner --since "1 hour ago"
|
||||
```
|
||||
@@ -1,17 +1,72 @@
|
||||
# MotoVaultPro GitLab CI/CD Deployment Guide
|
||||
|
||||
Complete guide for deploying MotoVaultPro using GitLab CI/CD with shell executor runners.
|
||||
Complete guide for deploying MotoVaultPro using GitLab CI/CD with blue-green deployment and auto-rollback.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Prerequisites](#prerequisites)
|
||||
2. [GitLab Runner Setup](#gitlab-runner-setup)
|
||||
3. [CI/CD Variables Configuration](#cicd-variables-configuration)
|
||||
4. [Secrets Architecture](#secrets-architecture)
|
||||
5. [Pipeline Overview](#pipeline-overview)
|
||||
6. [Deployment Process](#deployment-process)
|
||||
7. [Rollback Procedure](#rollback-procedure)
|
||||
8. [Troubleshooting](#troubleshooting)
|
||||
1. [Architecture Overview](#architecture-overview)
|
||||
2. [Prerequisites](#prerequisites)
|
||||
3. [Pipeline Stages](#pipeline-stages)
|
||||
4. [Blue-Green Deployment](#blue-green-deployment)
|
||||
5. [CI/CD Variables Configuration](#cicd-variables-configuration)
|
||||
6. [Container Registry](#container-registry)
|
||||
7. [Deployment Process](#deployment-process)
|
||||
8. [Rollback Procedures](#rollback-procedures)
|
||||
9. [Maintenance Migrations](#maintenance-migrations)
|
||||
10. [Notifications](#notifications)
|
||||
11. [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
MotoVaultPro uses a blue-green deployment strategy with automatic rollback:
|
||||
|
||||
```
|
||||
+---------------------------------------------------+
|
||||
| GitLab (CI/CD + Registry) |
|
||||
+---------------------------------------------------+
|
||||
| |
|
||||
v v
|
||||
+------------------+ +-----------------------+
|
||||
| Build VPS | | Production Server |
|
||||
| (Docker Runner) | | (Shell Runner) |
|
||||
| Tags: build | | Tags: production |
|
||||
+------------------+ +-----------+-----------+
|
||||
| |
|
||||
| Push images | Pull + Deploy
|
||||
v v
|
||||
+---------------------------------------------------+
|
||||
| GitLab Container Registry |
|
||||
| registry.motovaultpro.com/motovaultpro/ |
|
||||
+---------------------------------------------------+
|
||||
|
|
||||
+---------------+---------------+
|
||||
| |
|
||||
+--------v--------+ +--------v--------+
|
||||
| BLUE Stack | | GREEN Stack |
|
||||
| mvp-frontend | | mvp-frontend |
|
||||
| mvp-backend | | mvp-backend |
|
||||
+-----------------+ +-----------------+
|
||||
| |
|
||||
+----------- Traefik -----------+
|
||||
(weighted LB)
|
||||
|
|
||||
+---------------+---------------+
|
||||
| |
|
||||
+--------v--------+ +--------v--------+
|
||||
| PostgreSQL | | Redis |
|
||||
| (shared) | | (shared) |
|
||||
+-----------------+ +-----------------+
|
||||
```
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Zero-downtime deployments**: Traffic switches in under 5 seconds
|
||||
- **Instant rollback**: Previous version remains running
|
||||
- **Automatic rollback**: On health check failure
|
||||
- **Email notifications**: Via Resend API
|
||||
- **Container registry**: Self-hosted on GitLab (no Docker Hub)
|
||||
|
||||
---
|
||||
|
||||
@@ -19,55 +74,115 @@ Complete guide for deploying MotoVaultPro using GitLab CI/CD with shell executor
|
||||
|
||||
### Server Requirements
|
||||
|
||||
- Linux server with Docker Engine installed
|
||||
- Docker Compose v2 (plugin version)
|
||||
- GitLab Runner installed and registered
|
||||
- Git installed
|
||||
- curl installed (for health checks)
|
||||
| Server | Purpose | Specs | Runner Tags |
|
||||
|--------|---------|-------|-------------|
|
||||
| Build VPS | Docker image builds | 2 CPU, 4GB RAM | `build` |
|
||||
| Prod Server | Application hosting | 8GB+ RAM | `production` |
|
||||
|
||||
### GitLab Requirements
|
||||
See [BUILD-SERVER-SETUP.md](BUILD-SERVER-SETUP.md) for build server setup.
|
||||
|
||||
- GitLab 18.6+ (tested with 18.6.2)
|
||||
- Project with CI/CD enabled
|
||||
- Protected `main` branch
|
||||
- Maintainer access for CI/CD variable configuration
|
||||
### Software Requirements
|
||||
|
||||
- GitLab 18.6+
|
||||
- Docker Engine 24.0+
|
||||
- Docker Compose v2
|
||||
- GitLab Runner (shell executor on both servers)
|
||||
- `jq` for JSON processing
|
||||
|
||||
---
|
||||
|
||||
## GitLab Runner Setup
|
||||
## Pipeline Stages
|
||||
|
||||
### 1. Verify Runner Registration
|
||||
The CI/CD pipeline consists of 7 stages:
|
||||
|
||||
```bash
|
||||
sudo gitlab-runner verify
|
||||
```
|
||||
validate -> build -> deploy-prepare -> deploy-switch -> verify -> [rollback] -> notify
|
||||
```
|
||||
|
||||
Expected output should show your runner as active with shell executor.
|
||||
| Stage | Runner | Purpose |
|
||||
|-------|--------|---------|
|
||||
| `validate` | prod | Check prerequisites, determine target stack |
|
||||
| `build` | build | Build and push images to GitLab registry |
|
||||
| `deploy-prepare` | prod | Pull images, start inactive stack, health check |
|
||||
| `deploy-switch` | prod | Switch Traefik traffic weights |
|
||||
| `verify` | prod | Production health verification |
|
||||
| `rollback` | prod | Auto-triggered on verify failure |
|
||||
| `notify` | prod | Email success/failure notifications |
|
||||
|
||||
### 2. Verify Docker Permissions
|
||||
### Pipeline Flow
|
||||
|
||||
The `gitlab-runner` user must have Docker access:
|
||||
|
||||
```bash
|
||||
# Add gitlab-runner to docker group (if not already done)
|
||||
sudo usermod -aG docker gitlab-runner
|
||||
|
||||
# Verify access
|
||||
sudo -u gitlab-runner docker info
|
||||
sudo -u gitlab-runner docker compose version
|
||||
```
|
||||
[Push to main]
|
||||
|
|
||||
v
|
||||
[validate] - Checks Docker, paths, registry
|
||||
|
|
||||
v
|
||||
[build] - Builds backend + frontend images
|
||||
| Pushes to registry.motovaultpro.com
|
||||
v
|
||||
[deploy-prepare] - Pulls new images
|
||||
| Starts inactive stack (blue or green)
|
||||
| Runs health checks
|
||||
v
|
||||
[deploy-switch] - Updates Traefik weights
|
||||
| Switches traffic instantly
|
||||
v
|
||||
[verify] - External health check
|
||||
| Container status verification
|
||||
|
|
||||
+--[SUCCESS]--> [notify-success] - Sends success email
|
||||
|
|
||||
+--[FAILURE]--> [rollback] - Switches back to previous stack
|
||||
|
|
||||
v
|
||||
[notify-failure] - Sends failure email
|
||||
```
|
||||
|
||||
### 3. Verify Deployment Directory
|
||||
---
|
||||
|
||||
Ensure the deployment directory exists and is accessible:
|
||||
## Blue-Green Deployment
|
||||
|
||||
```bash
|
||||
# Create deployment directory
|
||||
sudo mkdir -p /opt/motovaultpro
|
||||
sudo chown gitlab-runner:gitlab-runner /opt/motovaultpro
|
||||
### Stack Configuration
|
||||
|
||||
# Clone repository (first time only)
|
||||
sudo -u gitlab-runner git clone <repository-url> /opt/motovaultpro
|
||||
Both stacks share the same database layer:
|
||||
|
||||
| Component | Blue Stack | Green Stack | Shared |
|
||||
|-----------|------------|-------------|--------|
|
||||
| Frontend | `mvp-frontend-blue` | `mvp-frontend-green` | - |
|
||||
| Backend | `mvp-backend-blue` | `mvp-backend-green` | - |
|
||||
| PostgreSQL | - | - | `mvp-postgres` |
|
||||
| Redis | - | - | `mvp-redis` |
|
||||
| Traefik | - | - | `mvp-traefik` |
|
||||
|
||||
### Traffic Routing
|
||||
|
||||
Traefik uses weighted services for traffic distribution:
|
||||
|
||||
```yaml
|
||||
# config/traefik/dynamic/blue-green.yml
|
||||
services:
|
||||
mvp-frontend-weighted:
|
||||
weighted:
|
||||
services:
|
||||
- name: mvp-frontend-blue-svc
|
||||
weight: 100 # Active
|
||||
- name: mvp-frontend-green-svc
|
||||
weight: 0 # Standby
|
||||
```
|
||||
|
||||
### Deployment State
|
||||
|
||||
State is tracked in `config/deployment/state.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"active_stack": "blue",
|
||||
"inactive_stack": "green",
|
||||
"last_deployment": "2024-01-15T10:30:00Z",
|
||||
"last_deployment_commit": "abc123",
|
||||
"rollback_available": true
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
@@ -76,410 +191,316 @@ sudo -u gitlab-runner git clone <repository-url> /opt/motovaultpro
|
||||
|
||||
Navigate to **Settings > CI/CD > Variables** in your GitLab project.
|
||||
|
||||
### Secrets (File Type Variables)
|
||||
### Required Variables
|
||||
|
||||
These variables use GitLab's **File** type, which writes the value to a temporary file and provides the path as the environment variable. This replicates the Kubernetes secrets pattern used by the application.
|
||||
|
||||
| Variable Name | Type | Protected | Masked | Description |
|
||||
|--------------|------|-----------|--------|-------------|
|
||||
| `POSTGRES_PASSWORD` | File | Yes | Yes | PostgreSQL database password |
|
||||
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes | Auth0 client secret for backend |
|
||||
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes | Google Maps API key |
|
||||
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No | Google Maps Map ID |
|
||||
| `CF_DNS_API_TOKEN` | File | Yes | Yes | Cloudflare API token for Let's Encrypt DNS challenge |
|
||||
| `RESEND_API_KEY` | File | Yes | Yes | Resend API key for email notifications |
|
||||
|
||||
### Configuration Variables
|
||||
|
||||
| Variable Name | Type | Protected | Masked | Value |
|
||||
|--------------|------|-----------|--------|-------|
|
||||
| `VITE_AUTH0_DOMAIN` | Variable | No | No | `motovaultpro.us.auth0.com` |
|
||||
| `VITE_AUTH0_CLIENT_ID` | Variable | No | No | Your Auth0 client ID |
|
||||
| `VITE_AUTH0_AUDIENCE` | Variable | No | No | `https://api.motovaultpro.com` |
|
||||
|
||||
Note: `DEPLOY_PATH` is automatically set in `.gitlab-ci.yml` using `GIT_CLONE_PATH` for a stable path.
|
||||
|
||||
### Creating Cloudflare API Token
|
||||
|
||||
The `CF_DNS_API_TOKEN` is required for automatic SSL certificate generation via Let's Encrypt DNS-01 challenge.
|
||||
|
||||
1. Go to [Cloudflare Dashboard](https://dash.cloudflare.com/profile/api-tokens)
|
||||
2. Click **Create Token**
|
||||
3. Use template: **Edit zone DNS**
|
||||
4. Configure permissions:
|
||||
- **Permissions**: Zone > DNS > Edit
|
||||
- **Zone Resources**: Include > Specific zone > `motovaultpro.com`
|
||||
5. Click **Continue to summary** then **Create Token**
|
||||
6. Copy the token value immediately (it won't be shown again)
|
||||
7. Add as `CF_DNS_API_TOKEN` File variable in GitLab
|
||||
|
||||
### Setting Up a File Type Variable
|
||||
|
||||
1. Go to **Settings > CI/CD > Variables**
|
||||
2. Click **Add variable**
|
||||
3. Enter the variable key (e.g., `POSTGRES_PASSWORD`)
|
||||
4. Enter the secret value in the **Value** field
|
||||
5. Set **Type** to **File**
|
||||
6. Enable **Protect variable** (recommended)
|
||||
7. Enable **Mask variable** (for sensitive data)
|
||||
8. Click **Add variable**
|
||||
|
||||
---
|
||||
|
||||
## Secrets Architecture
|
||||
|
||||
MotoVaultPro uses a Kubernetes-style secrets pattern where secrets are mounted as files at `/run/secrets/` inside containers.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **GitLab stores secrets** as File type CI/CD variables
|
||||
2. **During pipeline execution**, GitLab writes each secret to a temporary file
|
||||
3. **The `inject-secrets.sh` script** copies these files to `secrets/app/` directory
|
||||
4. **Docker Compose** mounts these files to `/run/secrets/` in containers
|
||||
5. **Application code** reads secrets from the filesystem (not environment variables)
|
||||
| Variable | Type | Protected | Purpose |
|
||||
|----------|------|-----------|---------|
|
||||
| `DEPLOY_NOTIFY_EMAIL` | Variable | Yes | Notification recipient |
|
||||
| `VITE_AUTH0_DOMAIN` | Variable | No | Auth0 domain |
|
||||
| `VITE_AUTH0_CLIENT_ID` | Variable | No | Auth0 client ID |
|
||||
| `VITE_AUTH0_AUDIENCE` | Variable | No | Auth0 audience |
|
||||
|
||||
### Secret Files
|
||||
|
||||
```
|
||||
secrets/app/
|
||||
postgres-password.txt -> /run/secrets/postgres-password
|
||||
auth0-client-secret.txt -> /run/secrets/auth0-client-secret
|
||||
google-maps-api-key.txt -> /run/secrets/google-maps-api-key
|
||||
google-maps-map-id.txt -> /run/secrets/google-maps-map-id
|
||||
cloudflare-dns-token.txt -> /run/secrets/cloudflare-dns-token
|
||||
resend-api-key.txt -> /run/secrets/resend-api-key
|
||||
```
|
||||
These use GitLab's **File** type and are injected via `scripts/inject-secrets.sh`:
|
||||
|
||||
### Security Benefits
|
||||
| Variable | Type | Protected | Masked |
|
||||
|----------|------|-----------|--------|
|
||||
| `POSTGRES_PASSWORD` | File | Yes | Yes |
|
||||
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes |
|
||||
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes |
|
||||
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No |
|
||||
| `CF_DNS_API_TOKEN` | File | Yes | Yes |
|
||||
| `RESEND_API_KEY` | File | Yes | Yes |
|
||||
|
||||
- Secrets never appear as environment variables (not visible in `env` or `printenv`)
|
||||
- File permissions (600) restrict access
|
||||
- Masked variables prevent accidental log exposure
|
||||
- Protected variables only available on protected branches
|
||||
### Registry Authentication
|
||||
|
||||
GitLab provides these automatically:
|
||||
- `CI_REGISTRY_USER` - Registry username
|
||||
- `CI_REGISTRY_PASSWORD` - Registry token
|
||||
- `CI_REGISTRY` - Registry URL
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Overview
|
||||
## Container Registry
|
||||
|
||||
The CI/CD pipeline consists of four stages:
|
||||
All images are hosted on the GitLab Container Registry to avoid Docker Hub rate limits.
|
||||
|
||||
### Stage 1: Validate
|
||||
|
||||
Verifies deployment prerequisites:
|
||||
- Docker is accessible
|
||||
- Docker Compose is available
|
||||
- Deployment directory exists
|
||||
|
||||
### Stage 2: Build
|
||||
|
||||
Builds Docker images:
|
||||
- Pulls latest code from repository
|
||||
- Builds all service images with `--no-cache`
|
||||
|
||||
### Stage 3: Deploy
|
||||
|
||||
Deploys the application:
|
||||
1. Injects secrets from GitLab variables
|
||||
2. Stops existing services gracefully
|
||||
3. Pulls base images
|
||||
4. Starts database services (PostgreSQL, Redis)
|
||||
5. Runs database migrations
|
||||
6. Starts all services
|
||||
|
||||
### Stage 4: Verify
|
||||
|
||||
Validates deployment health:
|
||||
- Checks all containers are running
|
||||
- Tests backend health endpoint
|
||||
- Reports deployment status
|
||||
|
||||
### Pipeline Diagram
|
||||
### Registry URL
|
||||
|
||||
```
|
||||
[Validate] -> [Build] -> [Deploy] -> [Verify]
|
||||
| | | |
|
||||
Check Build Inject Health
|
||||
prereqs images secrets checks
|
||||
|
|
||||
Migrate
|
||||
|
|
||||
Start
|
||||
services
|
||||
registry.motovaultpro.com
|
||||
```
|
||||
|
||||
### Image Paths
|
||||
|
||||
| Image | Path |
|
||||
|-------|------|
|
||||
| Backend | `registry.motovaultpro.com/motovaultpro/backend:$TAG` |
|
||||
| Frontend | `registry.motovaultpro.com/motovaultpro/frontend:$TAG` |
|
||||
| Mirrors | `registry.motovaultpro.com/mirrors/` |
|
||||
|
||||
### Base Image Mirrors
|
||||
|
||||
Mirror upstream images to avoid rate limits:
|
||||
|
||||
```bash
|
||||
# Run manually or via scheduled pipeline
|
||||
./scripts/ci/mirror-base-images.sh
|
||||
```
|
||||
|
||||
Mirrored images:
|
||||
- `node:20-alpine`
|
||||
- `nginx:alpine`
|
||||
- `postgres:18-alpine`
|
||||
- `redis:8.4-alpine`
|
||||
- `traefik:v3.6`
|
||||
- `docker:24.0`
|
||||
- `docker:24.0-dind`
|
||||
|
||||
---
|
||||
|
||||
## Deployment Process
|
||||
|
||||
### Automatic Deployment
|
||||
|
||||
Deployments are triggered automatically when:
|
||||
- Code is pushed to the `main` branch
|
||||
- A merge request is merged into `main`
|
||||
Deployments trigger automatically on push to `main`:
|
||||
|
||||
1. **Validate**: Check prerequisites, determine target stack
|
||||
2. **Build**: Build images on dedicated build server
|
||||
3. **Prepare**: Start inactive stack, run health checks
|
||||
4. **Switch**: Update Traefik weights (instant)
|
||||
5. **Verify**: External health check
|
||||
6. **Notify**: Send email notification
|
||||
|
||||
### Manual Deployment
|
||||
|
||||
To trigger a manual deployment:
|
||||
|
||||
1. Go to **CI/CD > Pipelines**
|
||||
2. Click **Run pipeline**
|
||||
3. Select the `main` branch
|
||||
3. Select `main` branch
|
||||
4. Click **Run pipeline**
|
||||
|
||||
### Deployment Steps (What Happens)
|
||||
### Deployment Timeline
|
||||
|
||||
1. **Secrets Injection**
|
||||
- `inject-secrets.sh` copies GitLab File variables to `secrets/app/`
|
||||
- Permissions are set to 600 for security
|
||||
|
||||
2. **Service Shutdown**
|
||||
- Existing containers are stopped gracefully (30s timeout)
|
||||
- Volumes are preserved
|
||||
|
||||
3. **Database Startup**
|
||||
- PostgreSQL and Redis start first
|
||||
- 15-second wait for database readiness
|
||||
|
||||
4. **Migrations**
|
||||
- Backend container runs database migrations
|
||||
- Ensures schema is up-to-date
|
||||
|
||||
5. **Full Service Startup**
|
||||
- All services start via `docker compose up -d`
|
||||
- Traefik routes traffic automatically
|
||||
|
||||
6. **Health Verification**
|
||||
- Container status checks
|
||||
- Backend health endpoint validation
|
||||
| Phase | Duration |
|
||||
|-------|----------|
|
||||
| Validate | ~5s |
|
||||
| Build | ~2 min |
|
||||
| Deploy-prepare | ~30s |
|
||||
| Deploy-switch | ~3s |
|
||||
| Verify | ~30s |
|
||||
| **Total** | ~3 min |
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedure
|
||||
## Rollback Procedures
|
||||
|
||||
### Automatic Rollback
|
||||
|
||||
If the verify stage fails, the pipeline will report failure but services remain running. Manual intervention is required.
|
||||
Triggers automatically when:
|
||||
- Health check fails in `deploy-prepare`
|
||||
- `verify` stage fails after switch
|
||||
- Container becomes unhealthy within verification period
|
||||
|
||||
The pipeline runs `scripts/ci/auto-rollback.sh` which:
|
||||
1. Verifies previous stack is healthy
|
||||
2. Switches traffic back
|
||||
3. Sends notification
|
||||
|
||||
### Manual Rollback
|
||||
|
||||
Use the rollback script:
|
||||
SSH to production server:
|
||||
|
||||
```bash
|
||||
# SSH to server
|
||||
ssh user@server
|
||||
|
||||
# Run rollback to previous commit
|
||||
cd /opt/motovaultpro
|
||||
./scripts/rollback.sh HEAD~1
|
||||
|
||||
# Or rollback to specific tag/commit
|
||||
./scripts/rollback.sh v1.0.0
|
||||
# Check current state
|
||||
cat config/deployment/state.json | jq .
|
||||
|
||||
# Switch to other stack
|
||||
./scripts/ci/switch-traffic.sh blue # or green
|
||||
```
|
||||
|
||||
### Rollback Script Details
|
||||
|
||||
The script performs:
|
||||
1. Stops all current services
|
||||
2. Checks out the specified version
|
||||
3. Rebuilds images
|
||||
4. Starts services
|
||||
|
||||
### Emergency Recovery
|
||||
|
||||
If rollback fails:
|
||||
If both stacks are unhealthy:
|
||||
|
||||
```bash
|
||||
# Stop everything
|
||||
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml down
|
||||
|
||||
# Restart shared services
|
||||
docker compose up -d mvp-postgres mvp-redis mvp-traefik
|
||||
|
||||
# Wait for database
|
||||
sleep 15
|
||||
|
||||
# Start one stack
|
||||
export BACKEND_IMAGE=registry.motovaultpro.com/motovaultpro/backend:latest
|
||||
export FRONTEND_IMAGE=registry.motovaultpro.com/motovaultpro/frontend:latest
|
||||
docker compose -f docker-compose.yml -f docker-compose.blue-green.yml up -d \
|
||||
mvp-frontend-blue mvp-backend-blue
|
||||
|
||||
# Switch traffic
|
||||
./scripts/ci/switch-traffic.sh blue
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance Migrations
|
||||
|
||||
For breaking database changes requiring downtime:
|
||||
|
||||
### Via Pipeline (Recommended)
|
||||
|
||||
1. Go to **CI/CD > Pipelines**
|
||||
2. Find the `maintenance-migration` job
|
||||
3. Click **Play** to trigger manually
|
||||
|
||||
### Via Script
|
||||
|
||||
```bash
|
||||
cd /opt/motovaultpro
|
||||
|
||||
# Stop everything
|
||||
docker compose down
|
||||
# With backup
|
||||
./scripts/ci/maintenance-migrate.sh backup
|
||||
|
||||
# Check git history
|
||||
git log --oneline -10
|
||||
# Without backup
|
||||
./scripts/ci/maintenance-migrate.sh
|
||||
```
|
||||
|
||||
# Checkout known working version
|
||||
git checkout <commit-hash>
|
||||
### What Happens
|
||||
|
||||
# Rebuild and start
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
1. Sends maintenance notification
|
||||
2. Enables maintenance mode (stops traffic)
|
||||
3. Creates database backup (if requested)
|
||||
4. Runs migrations
|
||||
5. Restarts backends
|
||||
6. Restores traffic
|
||||
7. Sends completion notification
|
||||
|
||||
# Verify
|
||||
docker compose ps
|
||||
---
|
||||
|
||||
## Notifications
|
||||
|
||||
Email notifications via Resend API for:
|
||||
|
||||
| Event | Subject |
|
||||
|-------|---------|
|
||||
| `success` | Deployment Successful |
|
||||
| `failure` | Deployment Failed |
|
||||
| `rollback` | Auto-Rollback Executed |
|
||||
| `rollback_failed` | CRITICAL: Rollback Failed |
|
||||
| `maintenance_start` | Maintenance Mode Started |
|
||||
| `maintenance_end` | Maintenance Complete |
|
||||
|
||||
Configure recipient in GitLab CI/CD variables:
|
||||
```
|
||||
DEPLOY_NOTIFY_EMAIL = admin@example.com
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Pipeline Fails at Validate Stage
|
||||
|
||||
**Symptom**: `DEPLOY_PATH not found`
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Create directory on runner server
|
||||
sudo mkdir -p /opt/motovaultpro
|
||||
sudo chown gitlab-runner:gitlab-runner /opt/motovaultpro
|
||||
```
|
||||
|
||||
### Pipeline Fails at Build Stage
|
||||
|
||||
**Symptom**: Docker build errors
|
||||
|
||||
**Solutions**:
|
||||
1. Check Dockerfile syntax
|
||||
2. Verify network connectivity for npm/package downloads
|
||||
3. Check disk space: `df -h`
|
||||
4. Clear Docker cache: `docker system prune -a`
|
||||
|
||||
### Pipeline Fails at Deploy Stage
|
||||
|
||||
**Symptom**: Secrets injection fails
|
||||
|
||||
**Solutions**:
|
||||
1. Verify CI/CD variables are configured correctly
|
||||
2. Check variable types are set to **File** for secrets
|
||||
3. Ensure variables are not restricted to specific environments
|
||||
|
||||
**Symptom**: Migration fails
|
||||
|
||||
**Solutions**:
|
||||
1. Check database connectivity
|
||||
2. Verify PostgreSQL is healthy: `docker logs mvp-postgres`
|
||||
3. Run migrations manually:
|
||||
```bash
|
||||
docker compose exec mvp-backend npm run migrate
|
||||
```
|
||||
|
||||
### Pipeline Fails at Verify Stage
|
||||
|
||||
**Symptom**: Container not running
|
||||
|
||||
**Solutions**:
|
||||
1. Check container logs: `docker logs <container-name>`
|
||||
2. Verify secrets are correctly mounted
|
||||
3. Check for port conflicts
|
||||
|
||||
**Symptom**: Health check fails
|
||||
|
||||
**Solutions**:
|
||||
1. Wait longer (service might be starting)
|
||||
2. Check backend logs: `docker logs mvp-backend`
|
||||
3. Verify database connection
|
||||
|
||||
### Services Start But Application Doesn't Work
|
||||
|
||||
**Check secrets are mounted**:
|
||||
**Check build server connectivity:**
|
||||
```bash
|
||||
docker compose exec mvp-backend ls -la /run/secrets/
|
||||
# On build server
|
||||
sudo gitlab-runner verify
|
||||
docker login registry.motovaultpro.com
|
||||
```
|
||||
|
||||
**Check configuration**:
|
||||
**Check disk space:**
|
||||
```bash
|
||||
docker compose exec mvp-backend cat /app/config/production.yml
|
||||
df -h
|
||||
docker system prune -af
|
||||
```
|
||||
|
||||
**Check network connectivity**:
|
||||
### Pipeline Fails at Deploy-Prepare
|
||||
|
||||
**Container won't start:**
|
||||
```bash
|
||||
docker network ls
|
||||
docker network inspect motovaultpro_backend
|
||||
docker logs mvp-backend-blue --tail 100
|
||||
docker logs mvp-frontend-blue --tail 100
|
||||
```
|
||||
|
||||
### Viewing Logs
|
||||
|
||||
**Health check timeout:**
|
||||
```bash
|
||||
# All services
|
||||
docker compose logs -f
|
||||
|
||||
# Specific service
|
||||
docker compose logs -f mvp-backend
|
||||
|
||||
# Last 100 lines
|
||||
docker compose logs --tail 100 mvp-backend
|
||||
# Increase timeout in .gitlab-ci.yml
|
||||
HEALTH_CHECK_TIMEOUT: "90"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Updating Secrets
|
||||
|
||||
1. Update the CI/CD variable in GitLab
|
||||
2. Trigger a new pipeline (push or manual)
|
||||
3. The new secrets will be injected during deployment
|
||||
|
||||
### Database Backups
|
||||
|
||||
Backups should be configured separately. Recommended approach:
|
||||
### Traffic Not Switching
|
||||
|
||||
**Check Traefik config:**
|
||||
```bash
|
||||
# Manual backup
|
||||
docker compose exec mvp-postgres pg_dump -U postgres motovaultpro > backup.sql
|
||||
|
||||
# Automated backup (add to cron)
|
||||
0 2 * * * cd /opt/motovaultpro && docker compose exec -T mvp-postgres pg_dump -U postgres motovaultpro > /backups/mvp-$(date +\%Y\%m\%d).sql
|
||||
cat config/traefik/dynamic/blue-green.yml
|
||||
docker exec mvp-traefik traefik healthcheck
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
**Check routing:**
|
||||
```bash
|
||||
curl -I https://motovaultpro.com/api/health
|
||||
```
|
||||
|
||||
Consider adding:
|
||||
- Prometheus metrics (Traefik already configured)
|
||||
- Health check alerts
|
||||
- Log aggregation
|
||||
### Verify Stage Fails
|
||||
|
||||
**Check external connectivity:**
|
||||
```bash
|
||||
curl -sf https://motovaultpro.com/api/health
|
||||
```
|
||||
|
||||
**Check container health:**
|
||||
```bash
|
||||
docker inspect --format='{{.State.Health.Status}}' mvp-backend-blue
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Common Commands
|
||||
|
||||
```bash
|
||||
# View pipeline status
|
||||
# GitLab UI: CI/CD > Pipelines
|
||||
|
||||
# SSH to server
|
||||
ssh user@your-server
|
||||
|
||||
# Navigate to project
|
||||
cd /opt/motovaultpro
|
||||
|
||||
# View running containers
|
||||
docker compose ps
|
||||
|
||||
# View logs
|
||||
docker compose logs -f
|
||||
|
||||
# Restart a service
|
||||
docker compose restart mvp-backend
|
||||
|
||||
# Run migrations manually
|
||||
docker compose exec mvp-backend npm run migrate
|
||||
|
||||
# Access database
|
||||
docker compose exec mvp-postgres psql -U postgres motovaultpro
|
||||
|
||||
# Health check
|
||||
curl http://localhost:3001/health
|
||||
```
|
||||
|
||||
### Important Paths
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `$CI_BUILDS_DIR/motovaultpro` | Application root (stable clone path) |
|
||||
| `$CI_BUILDS_DIR/motovaultpro/secrets/app/` | Secrets directory |
|
||||
| `$CI_BUILDS_DIR/motovaultpro/data/documents/` | Document storage |
|
||||
| `$CI_BUILDS_DIR/motovaultpro/config/` | Configuration files |
|
||||
| `config/deployment/state.json` | Deployment state |
|
||||
| `config/traefik/dynamic/blue-green.yml` | Traffic routing |
|
||||
| `scripts/ci/` | Deployment scripts |
|
||||
|
||||
Note: `CI_BUILDS_DIR` is typically `/opt/gitlab-runner/builds` for shell executors.
|
||||
### Common Commands
|
||||
|
||||
### Container Names
|
||||
```bash
|
||||
# View current state
|
||||
cat config/deployment/state.json | jq .
|
||||
|
||||
| Container | Purpose |
|
||||
|-----------|---------|
|
||||
| `mvp-traefik` | Reverse proxy, TLS termination |
|
||||
| `mvp-frontend` | React SPA |
|
||||
| `mvp-backend` | Node.js API |
|
||||
| `mvp-postgres` | PostgreSQL database |
|
||||
| `mvp-redis` | Redis cache |
|
||||
# Check container status
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Health}}"
|
||||
|
||||
# View logs
|
||||
docker logs mvp-backend-blue -f
|
||||
|
||||
# Manual traffic switch
|
||||
./scripts/ci/switch-traffic.sh green
|
||||
|
||||
# Run health check
|
||||
./scripts/ci/health-check.sh blue
|
||||
|
||||
# Send test notification
|
||||
./scripts/ci/notify.sh success "Test message"
|
||||
```
|
||||
|
||||
### Memory Budget (8GB Server)
|
||||
|
||||
| Component | RAM |
|
||||
|-----------|-----|
|
||||
| Blue frontend | 512MB |
|
||||
| Blue backend | 1GB |
|
||||
| Green frontend | 512MB |
|
||||
| Green backend | 1GB |
|
||||
| PostgreSQL | 2GB |
|
||||
| Redis | 512MB |
|
||||
| Traefik | 128MB |
|
||||
| System | 1.3GB |
|
||||
| **Total** | ~7GB |
|
||||
|
||||
Reference in New Issue
Block a user