CI/CD Improvements
This commit is contained in:
Eric Gullickson
464
docs/CICD-DEPLOY.md
Normal file
464
docs/CICD-DEPLOY.md
Normal file
@@ -0,0 +1,464 @@
|
||||
# MotoVaultPro GitLab CI/CD Deployment Guide
|
||||
|
||||
Complete guide for deploying MotoVaultPro using GitLab CI/CD with shell executor runners.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Prerequisites](#prerequisites)
|
||||
2. [GitLab Runner Setup](#gitlab-runner-setup)
|
||||
3. [CI/CD Variables Configuration](#cicd-variables-configuration)
|
||||
4. [Secrets Architecture](#secrets-architecture)
|
||||
5. [Pipeline Overview](#pipeline-overview)
|
||||
6. [Deployment Process](#deployment-process)
|
||||
7. [Rollback Procedure](#rollback-procedure)
|
||||
8. [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Server Requirements
|
||||
|
||||
- Linux server with Docker Engine installed
|
||||
- Docker Compose v2 (plugin version)
|
||||
- GitLab Runner installed and registered
|
||||
- Git installed
|
||||
- curl installed (for health checks)
|
||||
|
||||
### GitLab Requirements
|
||||
|
||||
- GitLab 18.6+ (tested with 18.6.2)
|
||||
- Project with CI/CD enabled
|
||||
- Protected `main` branch
|
||||
- Maintainer access for CI/CD variable configuration
|
||||
|
||||
---
|
||||
|
||||
## GitLab Runner Setup
|
||||
|
||||
### 1. Verify Runner Registration
|
||||
|
||||
```bash
|
||||
sudo gitlab-runner verify
|
||||
```
|
||||
|
||||
Expected output should show your runner as active with shell executor.
|
||||
|
||||
### 2. Verify Docker Permissions
|
||||
|
||||
The `gitlab-runner` user must have Docker access:
|
||||
|
||||
```bash
|
||||
# Add gitlab-runner to docker group (if not already done)
|
||||
sudo usermod -aG docker gitlab-runner
|
||||
|
||||
# Verify access
|
||||
sudo -u gitlab-runner docker info
|
||||
sudo -u gitlab-runner docker compose version
|
||||
```
|
||||
|
||||
### 3. Verify Deployment Directory
|
||||
|
||||
Ensure the deployment directory exists and is accessible:
|
||||
|
||||
```bash
|
||||
# Create deployment directory
|
||||
sudo mkdir -p /opt/motovaultpro
|
||||
sudo chown gitlab-runner:gitlab-runner /opt/motovaultpro
|
||||
|
||||
# Clone repository (first time only)
|
||||
sudo -u gitlab-runner git clone <repository-url> /opt/motovaultpro
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Variables Configuration
|
||||
|
||||
Navigate to **Settings > CI/CD > Variables** in your GitLab project.
|
||||
|
||||
### Secrets (File Type Variables)
|
||||
|
||||
These variables use GitLab's **File** type, which writes the value to a temporary file and provides the path as the environment variable. This replicates the Kubernetes secrets pattern used by the application.
|
||||
|
||||
| Variable Name | Type | Protected | Masked | Description |
|
||||
|--------------|------|-----------|--------|-------------|
|
||||
| `POSTGRES_PASSWORD` | File | Yes | Yes | PostgreSQL database password |
|
||||
| `AUTH0_CLIENT_SECRET` | File | Yes | Yes | Auth0 client secret for backend |
|
||||
| `GOOGLE_MAPS_API_KEY` | File | Yes | Yes | Google Maps API key |
|
||||
| `GOOGLE_MAPS_MAP_ID` | File | Yes | No | Google Maps Map ID |
|
||||
|
||||
### Configuration Variables
|
||||
|
||||
| Variable Name | Type | Protected | Masked | Value |
|
||||
|--------------|------|-----------|--------|-------|
|
||||
| `DEPLOY_PATH` | Variable | No | No | `/opt/motovaultpro` |
|
||||
| `VITE_AUTH0_DOMAIN` | Variable | No | No | `motovaultpro.us.auth0.com` |
|
||||
| `VITE_AUTH0_CLIENT_ID` | Variable | No | No | Your Auth0 client ID |
|
||||
| `VITE_AUTH0_AUDIENCE` | Variable | No | No | `https://api.motovaultpro.com` |
|
||||
|
||||
### Setting Up a File Type Variable
|
||||
|
||||
1. Go to **Settings > CI/CD > Variables**
|
||||
2. Click **Add variable**
|
||||
3. Enter the variable key (e.g., `POSTGRES_PASSWORD`)
|
||||
4. Enter the secret value in the **Value** field
|
||||
5. Set **Type** to **File**
|
||||
6. Enable **Protect variable** (recommended)
|
||||
7. Enable **Mask variable** (for sensitive data)
|
||||
8. Click **Add variable**
|
||||
|
||||
---
|
||||
|
||||
## Secrets Architecture
|
||||
|
||||
MotoVaultPro uses a Kubernetes-style secrets pattern where secrets are mounted as files at `/run/secrets/` inside containers.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **GitLab stores secrets** as File type CI/CD variables
|
||||
2. **During pipeline execution**, GitLab writes each secret to a temporary file
|
||||
3. **The `inject-secrets.sh` script** copies these files to `secrets/app/` directory
|
||||
4. **Docker Compose** mounts these files to `/run/secrets/` in containers
|
||||
5. **Application code** reads secrets from the filesystem (not environment variables)
|
||||
|
||||
### Secret Files
|
||||
|
||||
```
|
||||
secrets/app/
|
||||
postgres-password.txt -> /run/secrets/postgres-password
|
||||
auth0-client-secret.txt -> /run/secrets/auth0-client-secret
|
||||
google-maps-api-key.txt -> /run/secrets/google-maps-api-key
|
||||
google-maps-map-id.txt -> /run/secrets/google-maps-map-id
|
||||
```
|
||||
|
||||
### Security Benefits
|
||||
|
||||
- Secrets never appear as environment variables (not visible in `env` or `printenv`)
|
||||
- File permissions (600) restrict access
|
||||
- Masked variables prevent accidental log exposure
|
||||
- Protected variables only available on protected branches
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Overview
|
||||
|
||||
The CI/CD pipeline consists of four stages:
|
||||
|
||||
### Stage 1: Validate
|
||||
|
||||
Verifies deployment prerequisites:
|
||||
- Docker is accessible
|
||||
- Docker Compose is available
|
||||
- Deployment directory exists
|
||||
|
||||
### Stage 2: Build
|
||||
|
||||
Builds Docker images:
|
||||
- Pulls latest code from repository
|
||||
- Builds all service images with `--no-cache`
|
||||
|
||||
### Stage 3: Deploy
|
||||
|
||||
Deploys the application:
|
||||
1. Injects secrets from GitLab variables
|
||||
2. Stops existing services gracefully
|
||||
3. Pulls base images
|
||||
4. Starts database services (PostgreSQL, Redis)
|
||||
5. Runs database migrations
|
||||
6. Starts all services
|
||||
|
||||
### Stage 4: Verify
|
||||
|
||||
Validates deployment health:
|
||||
- Checks all containers are running
|
||||
- Tests backend health endpoint
|
||||
- Reports deployment status
|
||||
|
||||
### Pipeline Diagram
|
||||
|
||||
```
|
||||
[Validate] -> [Build] -> [Deploy] -> [Verify]
|
||||
| | | |
|
||||
Check Build Inject Health
|
||||
prereqs images secrets checks
|
||||
|
|
||||
Migrate
|
||||
|
|
||||
Start
|
||||
services
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deployment Process
|
||||
|
||||
### Automatic Deployment
|
||||
|
||||
Deployments are triggered automatically when:
|
||||
- Code is pushed to the `main` branch
|
||||
- A merge request is merged into `main`
|
||||
|
||||
### Manual Deployment
|
||||
|
||||
To trigger a manual deployment:
|
||||
|
||||
1. Go to **CI/CD > Pipelines**
|
||||
2. Click **Run pipeline**
|
||||
3. Select the `main` branch
|
||||
4. Click **Run pipeline**
|
||||
|
||||
### Deployment Steps (What Happens)
|
||||
|
||||
1. **Secrets Injection**
|
||||
- `inject-secrets.sh` copies GitLab File variables to `secrets/app/`
|
||||
- Permissions are set to 600 for security
|
||||
|
||||
2. **Service Shutdown**
|
||||
- Existing containers are stopped gracefully (30s timeout)
|
||||
- Volumes are preserved
|
||||
|
||||
3. **Database Startup**
|
||||
- PostgreSQL and Redis start first
|
||||
- 15-second wait for database readiness
|
||||
|
||||
4. **Migrations**
|
||||
- Backend container runs database migrations
|
||||
- Ensures schema is up-to-date
|
||||
|
||||
5. **Full Service Startup**
|
||||
- All services start via `docker compose up -d`
|
||||
- Traefik routes traffic automatically
|
||||
|
||||
6. **Health Verification**
|
||||
- Container status checks
|
||||
- Backend health endpoint validation
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
### Automatic Rollback
|
||||
|
||||
If the verify stage fails, the pipeline will report failure but services remain running. Manual intervention is required.
|
||||
|
||||
### Manual Rollback
|
||||
|
||||
Use the rollback script:
|
||||
|
||||
```bash
|
||||
# SSH to server
|
||||
ssh user@server
|
||||
|
||||
# Run rollback to previous commit
|
||||
cd /opt/motovaultpro
|
||||
./scripts/rollback.sh HEAD~1
|
||||
|
||||
# Or rollback to specific tag/commit
|
||||
./scripts/rollback.sh v1.0.0
|
||||
```
|
||||
|
||||
### Rollback Script Details
|
||||
|
||||
The script performs:
|
||||
1. Stops all current services
|
||||
2. Checks out the specified version
|
||||
3. Rebuilds images
|
||||
4. Starts services
|
||||
|
||||
### Emergency Recovery
|
||||
|
||||
If rollback fails:
|
||||
|
||||
```bash
|
||||
cd /opt/motovaultpro
|
||||
|
||||
# Stop everything
|
||||
docker compose down
|
||||
|
||||
# Check git history
|
||||
git log --oneline -10
|
||||
|
||||
# Checkout known working version
|
||||
git checkout <commit-hash>
|
||||
|
||||
# Rebuild and start
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
|
||||
# Verify
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Pipeline Fails at Validate Stage
|
||||
|
||||
**Symptom**: `DEPLOY_PATH not found`
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Create directory on runner server
|
||||
sudo mkdir -p /opt/motovaultpro
|
||||
sudo chown gitlab-runner:gitlab-runner /opt/motovaultpro
|
||||
```
|
||||
|
||||
### Pipeline Fails at Build Stage
|
||||
|
||||
**Symptom**: Docker build errors
|
||||
|
||||
**Solutions**:
|
||||
1. Check Dockerfile syntax
|
||||
2. Verify network connectivity for npm/package downloads
|
||||
3. Check disk space: `df -h`
|
||||
4. Clear Docker cache: `docker system prune -a`
|
||||
|
||||
### Pipeline Fails at Deploy Stage
|
||||
|
||||
**Symptom**: Secrets injection fails
|
||||
|
||||
**Solutions**:
|
||||
1. Verify CI/CD variables are configured correctly
|
||||
2. Check variable types are set to **File** for secrets
|
||||
3. Ensure variables are not restricted to specific environments
|
||||
|
||||
**Symptom**: Migration fails
|
||||
|
||||
**Solutions**:
|
||||
1. Check database connectivity
|
||||
2. Verify PostgreSQL is healthy: `docker logs mvp-postgres`
|
||||
3. Run migrations manually:
|
||||
```bash
|
||||
docker compose exec mvp-backend npm run migrate
|
||||
```
|
||||
|
||||
### Pipeline Fails at Verify Stage
|
||||
|
||||
**Symptom**: Container not running
|
||||
|
||||
**Solutions**:
|
||||
1. Check container logs: `docker logs <container-name>`
|
||||
2. Verify secrets are correctly mounted
|
||||
3. Check for port conflicts
|
||||
|
||||
**Symptom**: Health check fails
|
||||
|
||||
**Solutions**:
|
||||
1. Wait longer (service might be starting)
|
||||
2. Check backend logs: `docker logs mvp-backend`
|
||||
3. Verify database connection
|
||||
|
||||
### Services Start But Application Doesn't Work
|
||||
|
||||
**Check secrets are mounted**:
|
||||
```bash
|
||||
docker compose exec mvp-backend ls -la /run/secrets/
|
||||
```
|
||||
|
||||
**Check configuration**:
|
||||
```bash
|
||||
docker compose exec mvp-backend cat /app/config/production.yml
|
||||
```
|
||||
|
||||
**Check network connectivity**:
|
||||
```bash
|
||||
docker network ls
|
||||
docker network inspect motovaultpro_backend
|
||||
```
|
||||
|
||||
### Viewing Logs
|
||||
|
||||
```bash
|
||||
# All services
|
||||
docker compose logs -f
|
||||
|
||||
# Specific service
|
||||
docker compose logs -f mvp-backend
|
||||
|
||||
# Last 100 lines
|
||||
docker compose logs --tail 100 mvp-backend
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Updating Secrets
|
||||
|
||||
1. Update the CI/CD variable in GitLab
|
||||
2. Trigger a new pipeline (push or manual)
|
||||
3. The new secrets will be injected during deployment
|
||||
|
||||
### Database Backups
|
||||
|
||||
Backups should be configured separately. Recommended approach:
|
||||
|
||||
```bash
|
||||
# Manual backup
|
||||
docker compose exec mvp-postgres pg_dump -U postgres motovaultpro > backup.sql
|
||||
|
||||
# Automated backup (add to cron)
|
||||
0 2 * * * cd /opt/motovaultpro && docker compose exec -T mvp-postgres pg_dump -U postgres motovaultpro > /backups/mvp-$(date +\%Y\%m\%d).sql
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
Consider adding:
|
||||
- Prometheus metrics (Traefik already configured)
|
||||
- Health check alerts
|
||||
- Log aggregation
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Common Commands
|
||||
|
||||
```bash
|
||||
# View pipeline status
|
||||
# GitLab UI: CI/CD > Pipelines
|
||||
|
||||
# SSH to server
|
||||
ssh user@your-server
|
||||
|
||||
# Navigate to project
|
||||
cd /opt/motovaultpro
|
||||
|
||||
# View running containers
|
||||
docker compose ps
|
||||
|
||||
# View logs
|
||||
docker compose logs -f
|
||||
|
||||
# Restart a service
|
||||
docker compose restart mvp-backend
|
||||
|
||||
# Run migrations manually
|
||||
docker compose exec mvp-backend npm run migrate
|
||||
|
||||
# Access database
|
||||
docker compose exec mvp-postgres psql -U postgres motovaultpro
|
||||
|
||||
# Health check
|
||||
curl http://localhost:3001/health
|
||||
```
|
||||
|
||||
### Important Paths
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `/opt/motovaultpro` | Application root |
|
||||
| `/opt/motovaultpro/secrets/app/` | Secrets directory |
|
||||
| `/opt/motovaultpro/data/documents/` | Document storage |
|
||||
| `/opt/motovaultpro/config/` | Configuration files |
|
||||
|
||||
### Container Names
|
||||
|
||||
| Container | Purpose |
|
||||
|-----------|---------|
|
||||
| `mvp-traefik` | Reverse proxy, TLS termination |
|
||||
| `mvp-frontend` | React SPA |
|
||||
| `mvp-backend` | Node.js API |
|
||||
| `mvp-postgres` | PostgreSQL database |
|
||||
| `mvp-redis` | Redis cache |
|
||||
Reference in New Issue
Block a user