Redesign
This commit is contained in:
Eric Gullickson
545
docs/redesign/ROLLBACK-STRATEGY.md
Normal file
545
docs/redesign/ROLLBACK-STRATEGY.md
Normal file
@@ -0,0 +1,545 @@
|
||||
# Rollback Strategy - Recovery Procedures
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides comprehensive rollback procedures for each phase of the simplification. Each phase can be rolled back independently, and full system rollback is available.
|
||||
|
||||
## Pre-Execution Backup
|
||||
|
||||
### Before Starting ANY Phase
|
||||
|
||||
```bash
|
||||
# 1. Create backup branch
|
||||
git checkout -b backup-$(date +%Y%m%d-%H%M%S)
|
||||
git push origin backup-$(date +%Y%m%d-%H%M%S)
|
||||
|
||||
# 2. Tag current state
|
||||
git tag -a pre-simplification-$(date +%Y%m%d) \
|
||||
-m "State before architecture simplification"
|
||||
git push origin --tags
|
||||
|
||||
# 3. Export docker volumes
|
||||
docker run --rm -v admin_postgres_data:/data -v $(pwd):/backup \
|
||||
alpine tar czf /backup/postgres-backup-$(date +%Y%m%d).tar.gz /data
|
||||
|
||||
docker run --rm -v admin_redis_data:/data -v $(pwd):/backup \
|
||||
alpine tar czf /backup/redis-backup-$(date +%Y%m%d).tar.gz /data
|
||||
|
||||
# 4. Export MinIO data (if documents exist)
|
||||
docker run --rm -v admin_minio_data:/data -v $(pwd):/backup \
|
||||
alpine tar czf /backup/minio-backup-$(date +%Y%m%d).tar.gz /data
|
||||
|
||||
# 5. Document current state
|
||||
docker compose ps > container-state-$(date +%Y%m%d).txt
|
||||
docker network ls > network-state-$(date +%Y%m%d).txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Per-Phase Rollback Procedures
|
||||
|
||||
### Phase 1: Docker Compose Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- docker-compose.yml validation fails
|
||||
- Containers fail to start
|
||||
- Network errors
|
||||
- Volume mount issues
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Stop current containers
|
||||
docker compose down
|
||||
|
||||
# 2. Restore docker-compose.yml
|
||||
git checkout HEAD~1 -- docker-compose.yml
|
||||
|
||||
# 3. Restart with original config
|
||||
docker compose up -d
|
||||
|
||||
# 4. Verify original state
|
||||
docker compose ps # Should show 14 containers
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] 14 containers running
|
||||
- [ ] All containers healthy
|
||||
- [ ] No errors in logs
|
||||
|
||||
**Risk:** Low - file-based rollback, no data loss
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Remove Tenant Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Build errors after tenant code removal
|
||||
- Application won't start
|
||||
- Tests failing
|
||||
- Missing functionality
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Restore deleted files
|
||||
git checkout HEAD~1 -- backend/src/core/middleware/tenant.ts
|
||||
git checkout HEAD~1 -- backend/src/core/config/tenant.ts
|
||||
git checkout HEAD~1 -- backend/src/features/tenant-management/
|
||||
|
||||
# 2. Restore modified files
|
||||
git checkout HEAD~1 -- backend/src/app.ts
|
||||
git checkout HEAD~1 -- backend/src/core/plugins/auth.plugin.ts
|
||||
|
||||
# 3. Rebuild backend
|
||||
cd backend
|
||||
npm install
|
||||
npm run build
|
||||
|
||||
# 4. Restart backend container
|
||||
docker compose restart mvp-backend # or admin-backend if Phase 1 not done
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] Backend builds successfully
|
||||
- [ ] Backend starts without errors
|
||||
- [ ] Tests pass
|
||||
- [ ] Tenant functionality restored
|
||||
|
||||
**Risk:** Low-Medium - code rollback, no data impact
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Filesystem Storage Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Document upload/download fails
|
||||
- File system errors
|
||||
- Permission issues
|
||||
- Data access errors
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Stop backend
|
||||
docker compose stop mvp-backend
|
||||
|
||||
# 2. Restore storage adapter
|
||||
git checkout HEAD~1 -- backend/src/core/storage/
|
||||
|
||||
# 3. Restore documents feature
|
||||
git checkout HEAD~1 -- backend/src/features/documents/
|
||||
|
||||
# 4. Re-add MinIO to docker-compose.yml
|
||||
git checkout HEAD~1 -- docker-compose.yml
|
||||
# (Only MinIO service, keep other Phase 1 changes if applicable)
|
||||
|
||||
# 5. Restore MinIO data if backed up
|
||||
docker volume create admin_minio_data
|
||||
docker run --rm -v admin_minio_data:/data -v $(pwd):/backup \
|
||||
alpine tar xzf /backup/minio-backup-YYYYMMDD.tar.gz -C /
|
||||
|
||||
# 6. Rebuild and restart
|
||||
docker compose up -d admin-minio
|
||||
docker compose restart mvp-backend
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] MinIO container running
|
||||
- [ ] Document upload works
|
||||
- [ ] Document download works
|
||||
- [ ] Existing documents accessible
|
||||
|
||||
**Risk:** Medium - requires MinIO restore, potential data migration
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Config Cleanup Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Service connection failures
|
||||
- Authentication errors
|
||||
- Missing configuration
|
||||
- Environment variable errors
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Restore config files
|
||||
git checkout HEAD~1 -- config/app/production.yml
|
||||
git checkout HEAD~1 -- .env
|
||||
git checkout HEAD~1 -- .env.development
|
||||
|
||||
# 2. Restore secrets
|
||||
git checkout HEAD~1 -- secrets/app/
|
||||
git checkout HEAD~1 -- secrets/platform/
|
||||
|
||||
# 3. Restart affected services
|
||||
docker compose restart mvp-backend mvp-platform
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] Backend connects to database
|
||||
- [ ] Backend connects to Redis
|
||||
- [ ] Platform service accessible
|
||||
- [ ] Auth0 integration works
|
||||
|
||||
**Risk:** Low - configuration rollback, no data loss
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Network Simplification Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Service discovery failures
|
||||
- Network isolation broken
|
||||
- Container communication errors
|
||||
- Traefik routing issues
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Stop all services
|
||||
docker compose down
|
||||
|
||||
# 2. Remove simplified networks
|
||||
docker network rm motovaultpro_frontend motovaultpro_backend motovaultpro_database
|
||||
|
||||
# 3. Restore network configuration
|
||||
git checkout HEAD~1 -- docker-compose.yml
|
||||
# (Restore only networks section if possible)
|
||||
|
||||
# 4. Recreate networks and restart
|
||||
docker compose up -d
|
||||
|
||||
# 5. Verify routing
|
||||
curl http://localhost:8080/api/http/routers # Traefik dashboard
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] All 5 networks exist
|
||||
- [ ] Services can communicate
|
||||
- [ ] Traefik routes correctly
|
||||
- [ ] No network errors
|
||||
|
||||
**Risk:** Medium - requires container restart, brief downtime
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Backend Updates Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Service reference errors
|
||||
- API connection failures
|
||||
- Database connection issues
|
||||
- Build failures
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Restore backend code
|
||||
git checkout HEAD~1 -- backend/src/core/config/config-loader.ts
|
||||
git checkout HEAD~1 -- backend/src/features/vehicles/external/
|
||||
|
||||
# 2. Rebuild backend
|
||||
cd backend
|
||||
npm run build
|
||||
|
||||
# 3. Restart backend
|
||||
docker compose restart mvp-backend
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] Backend starts successfully
|
||||
- [ ] Connects to database
|
||||
- [ ] Platform client works
|
||||
- [ ] Tests pass
|
||||
|
||||
**Risk:** Low - code rollback, no data impact
|
||||
|
||||
---
|
||||
|
||||
### Phase 7: Database Updates Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Database connection failures
|
||||
- Schema errors
|
||||
- Migration failures
|
||||
- Data access issues
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Restore database configuration
|
||||
git checkout HEAD~1 -- backend/src/_system/migrations/
|
||||
git checkout HEAD~1 -- docker-compose.yml
|
||||
# (Only database section)
|
||||
|
||||
# 2. Restore database volume if corrupted
|
||||
docker compose down mvp-postgres
|
||||
docker volume rm mvp_postgres_data
|
||||
docker volume create admin_postgres_data
|
||||
docker run --rm -v admin_postgres_data:/data -v $(pwd):/backup \
|
||||
alpine tar xzf /backup/postgres-backup-YYYYMMDD.tar.gz -C /
|
||||
|
||||
# 3. Restart database
|
||||
docker compose up -d mvp-postgres
|
||||
|
||||
# 4. Re-run migrations if needed
|
||||
docker compose exec mvp-backend node dist/_system/migrations/run-all.js
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] Database accessible
|
||||
- [ ] All tables exist
|
||||
- [ ] Data intact
|
||||
- [ ] Migrations current
|
||||
|
||||
**Risk:** High - potential data loss if volume restore needed
|
||||
|
||||
---
|
||||
|
||||
### Phase 8: Platform Service Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Platform API failures
|
||||
- Database connection errors
|
||||
- Service crashes
|
||||
- API endpoint errors
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Stop simplified platform service
|
||||
docker compose down mvp-platform
|
||||
|
||||
# 2. Restore platform service files
|
||||
git checkout HEAD~1 -- mvp-platform-services/vehicles/
|
||||
|
||||
# 3. Restore full platform architecture in docker-compose.yml
|
||||
git checkout HEAD~1 -- docker-compose.yml
|
||||
# (Only platform services section)
|
||||
|
||||
# 4. Restore platform database
|
||||
docker volume create mvp_platform_vehicles_db_data
|
||||
docker run --rm -v mvp_platform_vehicles_db_data:/data -v $(pwd):/backup \
|
||||
alpine tar xzf /backup/platform-db-backup-YYYYMMDD.tar.gz -C /
|
||||
|
||||
# 5. Restart all platform services
|
||||
docker compose up -d mvp-platform-vehicles-api mvp-platform-vehicles-db \
|
||||
mvp-platform-vehicles-redis mvp-platform-vehicles-etl
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] Platform service accessible
|
||||
- [ ] API endpoints work
|
||||
- [ ] VIN decode works
|
||||
- [ ] Hierarchical data loads
|
||||
|
||||
**Risk:** Medium-High - requires multi-container restore
|
||||
|
||||
---
|
||||
|
||||
### Phase 9: Documentation Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Incorrect documentation
|
||||
- Missing instructions
|
||||
- Broken links
|
||||
- Confusion among team
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Restore all documentation
|
||||
git checkout HEAD~1 -- README.md CLAUDE.md AI-INDEX.md
|
||||
git checkout HEAD~1 -- docs/
|
||||
git checkout HEAD~1 -- .ai/context.json
|
||||
git checkout HEAD~1 -- Makefile
|
||||
git checkout HEAD~1 -- backend/src/features/*/README.md
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] Documentation accurate
|
||||
- [ ] Examples work
|
||||
- [ ] Makefile commands work
|
||||
|
||||
**Risk:** None - documentation only, no functional impact
|
||||
|
||||
---
|
||||
|
||||
### Phase 10: Frontend Rollback
|
||||
|
||||
**Rollback Trigger:**
|
||||
- Build errors
|
||||
- Runtime errors
|
||||
- UI broken
|
||||
- API calls failing
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Restore frontend code
|
||||
git checkout HEAD~1 -- frontend/src/
|
||||
|
||||
# 2. Rebuild frontend
|
||||
cd frontend
|
||||
npm install
|
||||
npm run build
|
||||
|
||||
# 3. Restart frontend container
|
||||
docker compose restart mvp-frontend
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] Frontend builds successfully
|
||||
- [ ] UI loads without errors
|
||||
- [ ] Auth works
|
||||
- [ ] API calls work
|
||||
|
||||
**Risk:** Low - frontend rollback, no data impact
|
||||
|
||||
---
|
||||
|
||||
### Phase 11: Testing Rollback
|
||||
|
||||
**Note:** Testing phase doesn't modify code, only validates. If tests fail, rollback appropriate phases based on failure analysis.
|
||||
|
||||
---
|
||||
|
||||
## Full System Rollback
|
||||
|
||||
### Complete Rollback to Pre-Simplification State
|
||||
|
||||
**When to Use:**
|
||||
- Multiple phases failing
|
||||
- Unrecoverable errors
|
||||
- Production blocker
|
||||
- Need to abort entire simplification
|
||||
|
||||
**Rollback Steps:**
|
||||
```bash
|
||||
# 1. Stop all services
|
||||
docker compose down
|
||||
|
||||
# 2. Restore entire codebase
|
||||
git checkout pre-simplification
|
||||
|
||||
# 3. Restore volumes
|
||||
docker volume rm mvp_postgres_data mvp_redis_data
|
||||
docker volume create admin_postgres_data admin_redis_data admin_minio_data
|
||||
|
||||
docker run --rm -v admin_postgres_data:/data -v $(pwd):/backup \
|
||||
alpine tar xzf /backup/postgres-backup-YYYYMMDD.tar.gz -C /
|
||||
|
||||
docker run --rm -v admin_redis_data:/data -v $(pwd):/backup \
|
||||
alpine tar xzf /backup/redis-backup-YYYYMMDD.tar.gz -C /
|
||||
|
||||
docker run --rm -v admin_minio_data:/data -v $(pwd):/backup \
|
||||
alpine tar xzf /backup/minio-backup-YYYYMMDD.tar.gz -C /
|
||||
|
||||
# 4. Restart all services
|
||||
docker compose up -d
|
||||
|
||||
# 5. Verify original state
|
||||
docker compose ps # Should show 14 containers
|
||||
make test
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- [ ] All 14 containers running
|
||||
- [ ] All tests passing
|
||||
- [ ] Application functional
|
||||
- [ ] Data intact
|
||||
|
||||
**Duration:** 15-30 minutes
|
||||
**Risk:** Low if backups are good
|
||||
|
||||
---
|
||||
|
||||
## Partial Rollback Scenarios
|
||||
|
||||
### Scenario 1: Keep Infrastructure Changes, Rollback Backend
|
||||
```bash
|
||||
# Keep Phase 1 (Docker), rollback Phases 2-11
|
||||
git checkout pre-simplification -- backend/ frontend/
|
||||
docker compose restart mvp-backend mvp-frontend
|
||||
```
|
||||
|
||||
### Scenario 2: Keep Config Cleanup, Rollback Code
|
||||
```bash
|
||||
# Keep Phase 4, rollback Phases 1-3, 5-11
|
||||
git checkout pre-simplification -- docker-compose.yml backend/src/ frontend/src/
|
||||
```
|
||||
|
||||
### Scenario 3: Rollback Only Storage
|
||||
```bash
|
||||
# Rollback Phase 3 only
|
||||
git checkout HEAD~1 -- backend/src/core/storage/ backend/src/features/documents/
|
||||
docker compose up -d admin-minio
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback Decision Matrix
|
||||
|
||||
| Failure Type | Rollback Scope | Risk | Duration |
|
||||
|--------------|---------------|------|----------|
|
||||
| Container start fails | Phase 1 | Low | 5 min |
|
||||
| Build errors | Specific phase | Low | 10 min |
|
||||
| Test failures | Investigate, partial | Medium | 15-30 min |
|
||||
| Data corruption | Full + restore | High | 30-60 min |
|
||||
| Network issues | Phase 5 | Medium | 10 min |
|
||||
| Platform API down | Phase 8 | Medium | 15 min |
|
||||
| Critical production bug | Full system | Medium | 30 min |
|
||||
|
||||
---
|
||||
|
||||
## Post-Rollback Actions
|
||||
|
||||
After any rollback:
|
||||
|
||||
1. **Document the Issue:**
|
||||
```bash
|
||||
# Create incident report
|
||||
echo "Rollback performed: $(date)" >> docs/redesign/ROLLBACK-LOG.md
|
||||
echo "Reason: [description]" >> docs/redesign/ROLLBACK-LOG.md
|
||||
echo "Phases rolled back: [list]" >> docs/redesign/ROLLBACK-LOG.md
|
||||
```
|
||||
|
||||
2. **Analyze Root Cause:**
|
||||
- Review logs
|
||||
- Identify failure point
|
||||
- Document lessons learned
|
||||
|
||||
3. **Plan Fix:**
|
||||
- Address root cause
|
||||
- Update phase documentation
|
||||
- Add validation checks
|
||||
|
||||
4. **Retry (if appropriate):**
|
||||
- Apply fix
|
||||
- Re-execute phase
|
||||
- Validate thoroughly
|
||||
|
||||
---
|
||||
|
||||
## Emergency Contacts
|
||||
|
||||
If rollback fails or assistance needed:
|
||||
- Technical Lead: [contact]
|
||||
- DevOps Lead: [contact]
|
||||
- Database Admin: [contact]
|
||||
- Emergency Hotline: [contact]
|
||||
|
||||
---
|
||||
|
||||
## Rollback Testing
|
||||
|
||||
Before starting simplification, test rollback procedures:
|
||||
|
||||
```bash
|
||||
# Dry run rollback
|
||||
git checkout -b rollback-test
|
||||
# Make test change
|
||||
echo "test" > test.txt
|
||||
git add test.txt
|
||||
git commit -m "test"
|
||||
|
||||
# Rollback test
|
||||
git checkout HEAD~1 -- test.txt
|
||||
# Verify rollback works
|
||||
|
||||
git checkout main
|
||||
git branch -D rollback-test
|
||||
```
|
||||
Reference in New Issue
Block a user