fix: I dunno, I'm making git server changes
This commit is contained in:
Eric Gullickson
315
docs/BUILD-SERVER-SETUP.md
Normal file
315
docs/BUILD-SERVER-SETUP.md
Normal file
@@ -0,0 +1,315 @@
|
||||
# Build Server Setup Guide
|
||||
|
||||
Complete guide for setting up a dedicated build VPS for MotoVaultPro CI/CD pipeline.
|
||||
|
||||
## Overview
|
||||
|
||||
The build server isolates resource-intensive Docker builds from the production server, ensuring deployments don't impact application performance.
|
||||
|
||||
```
|
||||
+-------------------+ +--------------------+
|
||||
| GitLab Server | | Production Server |
|
||||
| (CI/CD + Registry)| | (Shell Runner) |
|
||||
+--------+----------+ +----------+---------+
|
||||
| |
|
||||
v v
|
||||
+--------+----------+ +----------+---------+
|
||||
| Build VPS | | Blue-Green Stacks |
|
||||
| (Docker Runner) |---->| + Shared Data |
|
||||
+-------------------+ +--------------------+
|
||||
```
|
||||
|
||||
## Server Requirements
|
||||
|
||||
### Minimum Specifications
|
||||
|
||||
| Resource | Requirement |
|
||||
|----------|-------------|
|
||||
| CPU | 2 cores |
|
||||
| RAM | 4GB |
|
||||
| Storage | 50GB SSD |
|
||||
| Network | 100Mbps+ |
|
||||
| OS | Ubuntu 22.04 LTS / Debian 12 |
|
||||
|
||||
### Network Requirements
|
||||
|
||||
- Outbound HTTPS to GitLab instance
|
||||
- Outbound HTTPS to Docker registries (for fallback)
|
||||
- SSH access for administration
|
||||
|
||||
---
|
||||
|
||||
## Installation Steps
|
||||
|
||||
### 1. Update System
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
sudo apt install -y curl git ca-certificates gnupg
|
||||
```
|
||||
|
||||
### 2. Install Docker Engine
|
||||
|
||||
```bash
|
||||
# Add Docker's official GPG key
|
||||
sudo install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
|
||||
sudo chmod a+r /etc/apt/keyrings/docker.gpg
|
||||
|
||||
# Add the repository to Apt sources
|
||||
echo \
|
||||
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
|
||||
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
|
||||
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
|
||||
|
||||
# Install Docker
|
||||
sudo apt update
|
||||
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
|
||||
# Verify installation
|
||||
docker --version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
### 3. Install GitLab Runner
|
||||
|
||||
```bash
|
||||
# Add GitLab Runner repository
|
||||
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
|
||||
|
||||
# Install GitLab Runner
|
||||
sudo apt install gitlab-runner
|
||||
|
||||
# Verify installation
|
||||
gitlab-runner --version
|
||||
```
|
||||
|
||||
### 4. Register Runner with Shell Executor
|
||||
|
||||
```bash
|
||||
sudo gitlab-runner register \
|
||||
--non-interactive \
|
||||
--url "https://git.motovaultpro.com" \
|
||||
--registration-token "YOUR_REGISTRATION_TOKEN" \
|
||||
--executor "shell" \
|
||||
--description "Build Server - Shell Executor" \
|
||||
--tag-list "build" \
|
||||
--run-untagged="false" \
|
||||
--locked="true"
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
- Replace `YOUR_REGISTRATION_TOKEN` with the token from GitLab Admin > CI/CD > Runners
|
||||
- Shell executor runs jobs directly on the host with access to Docker
|
||||
- Tag `build` is used in `.gitlab-ci.yml` to route build jobs to this server
|
||||
|
||||
### 5. Add gitlab-runner to Docker Group
|
||||
|
||||
The gitlab-runner user needs access to Docker:
|
||||
|
||||
```bash
|
||||
sudo usermod -aG docker gitlab-runner
|
||||
|
||||
# Verify access
|
||||
sudo -u gitlab-runner docker info
|
||||
sudo -u gitlab-runner docker compose version
|
||||
```
|
||||
|
||||
### 6. Configure Docker Registry Authentication
|
||||
|
||||
Create credentials file for GitLab Container Registry:
|
||||
|
||||
```bash
|
||||
# Login to GitLab Container Registry (creates ~/.docker/config.json)
|
||||
docker login registry.motovaultpro.com -u <deploy-token-username> -p <deploy-token>
|
||||
```
|
||||
|
||||
**Creating Deploy Token:**
|
||||
1. Go to GitLab Project > Settings > Repository > Deploy Tokens
|
||||
2. Create token with `read_registry` and `write_registry` scopes
|
||||
3. Use the token username/password for Docker login
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
### Test Runner Registration
|
||||
|
||||
```bash
|
||||
sudo gitlab-runner verify
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Verifying runner... is alive runner=XXXXXX
|
||||
```
|
||||
|
||||
### Test Docker Access
|
||||
|
||||
```bash
|
||||
sudo gitlab-runner exec docker --docker-privileged test-job
|
||||
```
|
||||
|
||||
### Test Registry Push
|
||||
|
||||
```bash
|
||||
# Build and push a test image
|
||||
docker build -t registry.motovaultpro.com/motovaultpro/test:latest -f- . <<EOF
|
||||
FROM alpine:latest
|
||||
RUN echo "test"
|
||||
EOF
|
||||
|
||||
docker push registry.motovaultpro.com/motovaultpro/test:latest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Disk Cleanup
|
||||
|
||||
Docker builds accumulate disk space. Set up automated cleanup:
|
||||
|
||||
```bash
|
||||
# Create cleanup script
|
||||
sudo tee /usr/local/bin/docker-cleanup.sh > /dev/null <<'EOF'
|
||||
#!/bin/bash
|
||||
# Remove unused Docker resources older than 7 days
|
||||
docker system prune -af --filter "until=168h"
|
||||
docker volume prune -f
|
||||
EOF
|
||||
|
||||
sudo chmod +x /usr/local/bin/docker-cleanup.sh
|
||||
|
||||
# Add to crontab (run daily at 3 AM)
|
||||
echo "0 3 * * * /usr/local/bin/docker-cleanup.sh >> /var/log/docker-cleanup.log 2>&1" | sudo crontab -
|
||||
```
|
||||
|
||||
### Log Rotation
|
||||
|
||||
Configure log rotation for GitLab Runner:
|
||||
|
||||
```bash
|
||||
sudo tee /etc/logrotate.d/gitlab-runner > /dev/null <<EOF
|
||||
/var/log/gitlab-runner/*.log {
|
||||
daily
|
||||
rotate 7
|
||||
compress
|
||||
missingok
|
||||
notifempty
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
### Update Runner
|
||||
|
||||
```bash
|
||||
# Update GitLab Runner
|
||||
sudo apt update
|
||||
sudo apt upgrade gitlab-runner
|
||||
|
||||
# Restart runner
|
||||
sudo gitlab-runner restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Firewall Configuration
|
||||
|
||||
```bash
|
||||
# Allow only necessary outbound traffic
|
||||
sudo ufw default deny incoming
|
||||
sudo ufw default allow outgoing
|
||||
sudo ufw allow ssh
|
||||
sudo ufw enable
|
||||
```
|
||||
|
||||
### Runner Security
|
||||
|
||||
- **Locked runner**: Only accepts jobs from the specific project
|
||||
- **Protected tags**: Only runs on protected branches (main)
|
||||
- **Docker socket**: Mounted read-only where possible
|
||||
|
||||
### Secrets Management
|
||||
|
||||
The build server does NOT store application secrets. All secrets are:
|
||||
- Stored in GitLab CI/CD Variables
|
||||
- Injected at runtime on the production server
|
||||
- Never cached in Docker layers
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Runner Not Picking Up Jobs
|
||||
|
||||
```bash
|
||||
# Check runner status
|
||||
sudo gitlab-runner status
|
||||
|
||||
# View runner logs
|
||||
sudo journalctl -u gitlab-runner -f
|
||||
|
||||
# Re-register runner if needed
|
||||
sudo gitlab-runner unregister --all-runners
|
||||
sudo gitlab-runner register
|
||||
```
|
||||
|
||||
### Docker Build Failures
|
||||
|
||||
```bash
|
||||
# Check Docker daemon
|
||||
sudo systemctl status docker
|
||||
|
||||
# Check available disk space
|
||||
df -h
|
||||
|
||||
# Clear Docker cache
|
||||
docker system prune -af
|
||||
```
|
||||
|
||||
### Registry Push Failures
|
||||
|
||||
```bash
|
||||
# Verify registry login
|
||||
docker login registry.motovaultpro.com
|
||||
|
||||
# Check network connectivity
|
||||
curl -v https://registry.motovaultpro.com/v2/
|
||||
|
||||
# Verify image exists
|
||||
docker images | grep motovaultpro
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Important Paths
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `/etc/gitlab-runner/config.toml` | Runner configuration |
|
||||
| `/var/log/gitlab-runner/` | Runner logs |
|
||||
| `~/.docker/config.json` | Docker registry credentials |
|
||||
| `/var/lib/docker/` | Docker data |
|
||||
|
||||
### Common Commands
|
||||
|
||||
```bash
|
||||
# Runner management
|
||||
sudo gitlab-runner status
|
||||
sudo gitlab-runner restart
|
||||
sudo gitlab-runner verify
|
||||
|
||||
# Docker management
|
||||
docker system df # Check disk usage
|
||||
docker system prune -af # Clean all unused resources
|
||||
docker images # List images
|
||||
docker ps -a # List containers
|
||||
|
||||
# View build logs
|
||||
sudo journalctl -u gitlab-runner --since "1 hour ago"
|
||||
```
|
||||
Reference in New Issue
Block a user