motovaultpro/docs/K8S-REFACTOR.md

# Kubernetes Modernization Plan for MotoVaultPro

## Executive Summary

This document outlines a comprehensive plan to modernize MotoVaultPro from a traditional self-hosted application to a cloud-native, highly available system running on Kubernetes. The modernization focuses on transforming the current monolithic ASP.NET Core application into a resilient, scalable platform capable of handling enterprise-level workloads while maintaining the existing feature set and user experience.

### Key Objectives
- **High Availability**: Eliminate single points of failure through distributed architecture
- **Scalability**: Enable horizontal scaling to handle increased user loads
- **Resilience**: Implement fault tolerance and automatic recovery mechanisms
- **Cloud-Native**: Adopt Kubernetes-native patterns and best practices
- **Operational Excellence**: Improve monitoring, logging, and maintenance capabilities

### Strategic Benefits
- **Reduced Downtime**: Multi-replica deployments with automatic failover
- **Improved Performance**: Distributed caching and optimized data access patterns
- **Enhanced Security**: Pod-level isolation and secret management
- **Cost Optimization**: Efficient resource utilization through auto-scaling
- **Future-Ready**: Foundation for microservices and advanced cloud features

## Current Architecture Analysis

### Existing System Overview
MotoVaultPro is currently deployed as a monolithic ASP.NET Core 8.0 application with the following characteristics:

#### Application Architecture
- **Monolithic Design**: Single deployable unit containing all functionality
- **MVC Pattern**: Traditional Model-View-Controller architecture
- **Dual Database Support**: LiteDB (embedded) and PostgreSQL (external)
- **File Storage**: Local filesystem for document attachments
- **Session Management**: In-memory or cookie-based sessions
- **Configuration**: File-based configuration with environment variables

#### Current Deployment Model
- **Single Instance**: Typically deployed as a single container or VM
- **Stateful**: Relies on local storage for files and embedded database
- **Limited Scalability**: Cannot horizontally scale due to state dependencies
- **Single Point of Failure**: No redundancy or automatic recovery

#### Identified Limitations for Kubernetes
1. **State Dependencies**: LiteDB and local file storage prevent stateless operation
2. **Configuration Management**: File-based configuration not suitable for container orchestration
3. **Health Monitoring**: Lacks Kubernetes-compatible health check endpoints
4. **Logging**: Basic logging not optimized for centralized log aggregation
5. **Resource Management**: No resource constraints or auto-scaling capabilities
6. **Secret Management**: Sensitive configuration stored in plain text files

## Target Architecture

### Cloud-Native Design Principles
The modernized architecture will embrace the following cloud-native principles:

#### Stateless Application Design
- **External State Storage**: All state moved to external, highly available services
- **Horizontal Scalability**: Multiple application replicas with load balancing
- **Configuration as Code**: All configuration externalized to ConfigMaps and Secrets
- **Ephemeral Containers**: Pods can be created, destroyed, and recreated without data loss

#### Distributed Data Architecture
- **PostgreSQL Cluster**: Primary/replica configuration with automatic failover
- **MinIO High Availability**: Distributed object storage for file attachments
- **Redis Cluster**: Distributed caching and session storage
- **Backup Strategy**: Automated backups with point-in-time recovery

#### Observability and Operations
- **Structured Logging**: JSON logging with correlation IDs for distributed tracing
- **Metrics Collection**: Prometheus-compatible metrics for monitoring
- **Health Checks**: Kubernetes-native readiness and liveness probes
- **Distributed Tracing**: OpenTelemetry integration for request flow analysis

### High-Level Architecture Diagram
```
┌─────────────────────────────────────────────────────────────────┐
│                        Kubernetes Cluster                       │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐  │
│  │   MotoVault     │  │   MotoVault     │  │   MotoVault     │  │
│  │   Pod (1)       │  │   Pod (2)       │  │   Pod (3)       │  │
│  │                 │  │                 │  │                 │  │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘  │
│           │                     │                     │          │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │                   Load Balancer Service                     │ │
│  └─────────────────────────────────────────────────────────────┘ │
│           │                     │                     │          │
├───────────┼─────────────────────┼─────────────────────┼──────────┤
│  ┌────────▼──────┐    ┌─────────▼──────┐    ┌─────────▼──────┐   │
│  │ PostgreSQL    │    │ Redis Cluster  │    │ MinIO Cluster  │   │
│  │ Primary       │    │ (3 nodes)      │    │ (4+ nodes)     │   │
│  │ + 2 Replicas  │    │                │    │ Erasure Coded  │   │
│  └───────────────┘    └────────────────┘    └────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
```

## Detailed Implementation Phases

### Phase 1: Core Kubernetes Readiness (Weeks 1-4)

This phase focuses on making the application compatible with Kubernetes deployment patterns while maintaining existing functionality.

#### 1.1 Configuration Externalization

**Objective**: Move all configuration from files to Kubernetes-native configuration management.

**Current State**: 
- Configuration stored in `appsettings.json` and environment variables
- Database connection strings in configuration files
- Feature flags and application settings mixed with deployment configuration

**Target State**:
- All configuration externalized to ConfigMaps and Secrets
- Environment-specific configuration separated from application code
- Sensitive data (passwords, API keys) managed through Kubernetes Secrets

**Implementation Tasks**:
1. **Create ConfigMap templates** for non-sensitive configuration
   ```yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: motovault-config
   data:
     APP_NAME: "MotoVaultPro"
     LOG_LEVEL: "Information"
     ENABLE_FEATURES: "OpenIDConnect,EmailNotifications"
     CACHE_EXPIRY_MINUTES: "30"
   ```

2. **Create Secret templates** for sensitive configuration
   ```yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: motovault-secrets
   type: Opaque
   data:
     POSTGRES_CONNECTION: <base64-encoded-connection-string>
     MINIO_ACCESS_KEY: <base64-encoded-access-key>
     MINIO_SECRET_KEY: <base64-encoded-secret-key>
     JWT_SECRET: <base64-encoded-jwt-secret>
   ```

3. **Modify application startup** to read from environment variables
4. **Remove file-based configuration** dependencies
5. **Implement configuration validation** at startup

#### 1.2 Database Architecture Modernization

**Objective**: Eliminate LiteDB dependency and optimize PostgreSQL usage for Kubernetes.

**Current State**:
- Dual database support with LiteDB as default
- Single PostgreSQL connection for external database mode
- No connection pooling optimization for multiple instances

**Target State**:
- PostgreSQL-only configuration with high availability
- Optimized connection pooling for horizontal scaling
- Database migration strategy for existing LiteDB installations

**Implementation Tasks**:
1. **Remove LiteDB implementation** and dependencies
2. **Implement PostgreSQL HA configuration**:
   ```csharp
   services.AddDbContext<MotoVaultContext>(options =>
   {
       options.UseNpgsql(connectionString, npgsqlOptions =>
       {
           npgsqlOptions.EnableRetryOnFailure(
               maxRetryCount: 3,
               maxRetryDelay: TimeSpan.FromSeconds(5),
               errorCodesToAdd: null);
       });
   });
   ```
3. **Add connection pooling configuration**:
   ```csharp
   // Configure connection pooling for multiple instances
   services.Configure<NpgsqlConnectionStringBuilder>(options =>
   {
       options.MaxPoolSize = 100;
       options.MinPoolSize = 10;
       options.ConnectionLifetime = 300; // 5 minutes
   });
   ```
4. **Create data migration tools** for LiteDB to PostgreSQL conversion
5. **Implement database health checks** for Kubernetes probes

#### 1.3 Health Check Implementation

**Objective**: Add Kubernetes-compatible health check endpoints for proper orchestration.

**Current State**:
- No dedicated health check endpoints
- Application startup/shutdown not optimized for Kubernetes

**Target State**:
- Comprehensive health checks for all dependencies
- Proper readiness and liveness probe endpoints
- Graceful shutdown handling for pod termination

**Implementation Tasks**:
1. **Add health check middleware**:
   ```csharp
   // Program.cs
   builder.Services.AddHealthChecks()
       .AddNpgSql(connectionString, name: "database")
       .AddRedis(redisConnectionString, name: "cache")
       .AddCheck<MinIOHealthCheck>("minio");
   
   app.MapHealthChecks("/health/ready", new HealthCheckOptions
   {
       Predicate = check => check.Tags.Contains("ready"),
       ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
   });
   
   app.MapHealthChecks("/health/live", new HealthCheckOptions
   {
       Predicate = _ => false // Only check if the app is responsive
   });
   ```

2. **Implement custom health checks**:
   ```csharp
   public class MinIOHealthCheck : IHealthCheck
   {
       private readonly IMinioClient _minioClient;
       
       public async Task<HealthCheckResult> CheckHealthAsync(
           HealthCheckContext context, 
           CancellationToken cancellationToken = default)
       {
           try
           {
               await _minioClient.ListBucketsAsync(cancellationToken);
               return HealthCheckResult.Healthy("MinIO is accessible");
           }
           catch (Exception ex)
           {
               return HealthCheckResult.Unhealthy("MinIO is not accessible", ex);
           }
       }
   }
   ```

3. **Add graceful shutdown handling**:
   ```csharp
   builder.Services.Configure<HostOptions>(options =>
   {
       options.ShutdownTimeout = TimeSpan.FromSeconds(30);
   });
   ```

#### 1.4 Logging Enhancement

**Objective**: Implement structured logging suitable for centralized log aggregation.

**Current State**:
- Basic logging with simple string messages
- No correlation IDs for distributed tracing
- Log levels not optimized for production monitoring

**Target State**:
- JSON-structured logging with correlation IDs
- Centralized log aggregation compatibility
- Performance and error metrics embedded in logs

**Implementation Tasks**:
1. **Configure structured logging**:
   ```csharp
   builder.Services.AddLogging(loggingBuilder =>
   {
       loggingBuilder.ClearProviders();
       loggingBuilder.AddJsonConsole(options =>
       {
           options.IncludeScopes = true;
           options.TimestampFormat = "yyyy-MM-ddTHH:mm:ss.fffZ";
           options.JsonWriterOptions = new JsonWriterOptions
           {
               Indented = false
           };
       });
   });
   ```

2. **Add correlation ID middleware**:
   ```csharp
   public class CorrelationIdMiddleware
   {
       public async Task InvokeAsync(HttpContext context, RequestDelegate next)
       {
           var correlationId = context.Request.Headers["X-Correlation-ID"]
               .FirstOrDefault() ?? Guid.NewGuid().ToString();
           
           using var scope = _logger.BeginScope(new Dictionary<string, object>
           {
               ["CorrelationId"] = correlationId,
               ["UserId"] = context.User?.Identity?.Name
           });
           
           context.Response.Headers.Add("X-Correlation-ID", correlationId);
           await next(context);
       }
   }
   ```

3. **Implement performance logging** for critical operations

### Phase 2: High Availability Infrastructure (Weeks 5-8)

This phase focuses on implementing the supporting infrastructure required for high availability.

#### 2.1 MinIO High Availability Setup

**Objective**: Deploy a highly available MinIO cluster for file storage with automatic failover.

**Architecture Overview**:
MinIO will be deployed as a distributed cluster with erasure coding for data protection and automatic healing capabilities.

**MinIO Cluster Configuration**:
```yaml
# MinIO Tenant Configuration
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: motovault-minio
  namespace: motovault
spec:
  image: minio/minio:RELEASE.2024-01-16T16-07-38Z
  creationDate: 2024-01-20T10:00:00Z
  pools:
  - servers: 4
    name: pool-0
    volumesPerServer: 4
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: fast-ssd
  mountPath: /export
  subPath: /data
  requestAutoCert: false
  certConfig:
    commonName: ""
    organizationName: []
    dnsNames: []
  console:
    image: minio/console:v0.22.5
    replicas: 2
    consoleSecret:
      name: motovault-minio-console-secret
  configuration:
    name: motovault-minio-config
  pools:
  - servers: 4
    volumesPerServer: 4
    volumeClaimTemplate:
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 100Gi
```

**Implementation Tasks**:
1. **Deploy MinIO Operator**:
   ```bash
   kubectl apply -k "github.com/minio/operator/resources"
   ```

2. **Create MinIO cluster configuration** with erasure coding for data protection
3. **Configure backup policies** for disaster recovery
4. **Set up monitoring** with Prometheus metrics
5. **Create service endpoints** for application connectivity

**MinIO High Availability Features**:
- **Erasure Coding**: Data is split across multiple drives with parity for automatic healing
- **Distributed Architecture**: No single point of failure
- **Automatic Healing**: Corrupted data is automatically detected and repaired
- **Load Balancing**: Built-in load balancing across cluster nodes
- **Bucket Policies**: Fine-grained access control for different data types

#### 2.2 File Storage Abstraction Implementation

**Objective**: Create an abstraction layer that allows seamless switching between local filesystem and MinIO object storage.

**Current State**:
- Direct filesystem operations throughout the application
- File paths hardcoded in various controllers and services
- No abstraction for different storage backends

**Target State**:
- Unified file storage interface
- Pluggable storage implementations
- Transparent migration between storage types

**Implementation Tasks**:
1. **Define storage abstraction interface**:
   ```csharp
   public interface IFileStorageService
   {
       Task<string> UploadFileAsync(Stream fileStream, string fileName, string contentType, CancellationToken cancellationToken = default);
       Task<Stream> DownloadFileAsync(string fileId, CancellationToken cancellationToken = default);
       Task<bool> DeleteFileAsync(string fileId, CancellationToken cancellationToken = default);
       Task<FileMetadata> GetFileMetadataAsync(string fileId, CancellationToken cancellationToken = default);
       Task<IEnumerable<FileMetadata>> ListFilesAsync(string prefix = null, CancellationToken cancellationToken = default);
       Task<string> GeneratePresignedUrlAsync(string fileId, TimeSpan expiration, CancellationToken cancellationToken = default);
   }
   
   public class FileMetadata
   {
       public string Id { get; set; }
       public string FileName { get; set; }
       public string ContentType { get; set; }
       public long Size { get; set; }
       public DateTime CreatedDate { get; set; }
       public DateTime ModifiedDate { get; set; }
       public Dictionary<string, string> Tags { get; set; }
   }
   ```

2. **Implement MinIO storage service**:
   ```csharp
   public class MinIOFileStorageService : IFileStorageService
   {
       private readonly IMinioClient _minioClient;
       private readonly ILogger<MinIOFileStorageService> _logger;
       private readonly string _bucketName;
       
       public MinIOFileStorageService(IMinioClient minioClient, IConfiguration configuration, ILogger<MinIOFileStorageService> logger)
       {
           _minioClient = minioClient;
           _logger = logger;
           _bucketName = configuration["MinIO:BucketName"] ?? "motovault-files";
       }
       
       public async Task<string> UploadFileAsync(Stream fileStream, string fileName, string contentType, CancellationToken cancellationToken = default)
       {
           var fileId = $"{Guid.NewGuid()}/{fileName}";
           
           try
           {
               await _minioClient.PutObjectAsync(new PutObjectArgs()
                   .WithBucket(_bucketName)
                   .WithObject(fileId)
                   .WithStreamData(fileStream)
                   .WithObjectSize(fileStream.Length)
                   .WithContentType(contentType)
                   .WithHeaders(new Dictionary<string, string>
                   {
                       ["X-Amz-Meta-Original-Name"] = fileName,
                       ["X-Amz-Meta-Upload-Date"] = DateTime.UtcNow.ToString("O")
                   }), cancellationToken);
               
               _logger.LogInformation("File uploaded successfully: {FileId}", fileId);
               return fileId;
           }
           catch (Exception ex)
           {
               _logger.LogError(ex, "Failed to upload file: {FileName}", fileName);
               throw;
           }
       }
       
       // Additional method implementations...
   }
   ```

3. **Create fallback storage service** for graceful degradation:
   ```csharp
   public class FallbackFileStorageService : IFileStorageService
   {
       private readonly IFileStorageService _primaryService;
       private readonly IFileStorageService _fallbackService;
       private readonly ILogger<FallbackFileStorageService> _logger;
       
       // Implementation with automatic fallback logic
   }
   ```

4. **Update all file operations** to use the abstraction layer
5. **Implement file migration utility** for existing local files

#### 2.3 PostgreSQL High Availability Configuration

**Objective**: Set up a PostgreSQL cluster with automatic failover and read replicas.

**Architecture Overview**:
PostgreSQL will be deployed using an operator (like CloudNativePG or Postgres Operator) to provide automated failover, backup, and scaling capabilities.

**PostgreSQL Cluster Configuration**:
```yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: motovault-postgres
  namespace: motovault
spec:
  instances: 3
  primaryUpdateStrategy: unsupervised
  
  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "256MB"
      effective_cache_size: "1GB"
      maintenance_work_mem: "64MB"
      checkpoint_completion_target: "0.9"
      wal_buffers: "16MB"
      default_statistics_target: "100"
      random_page_cost: "1.1"
      effective_io_concurrency: "200"
      
  resources:
    requests:
      memory: "2Gi"
      cpu: "1000m"
    limits:
      memory: "4Gi"
      cpu: "2000m"
      
  storage:
    size: "100Gi"
    storageClass: "fast-ssd"
    
  monitoring:
    enabled: true
    
  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
      destinationPath: "s3://motovault-backups/postgres"
      s3Credentials:
        accessKeyId:
          name: postgres-backup-credentials
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: postgres-backup-credentials
          key: SECRET_ACCESS_KEY
      wal:
        retention: "5d"
      data:
        retention: "30d"
        jobs: 1
```

**Implementation Tasks**:
1. **Deploy PostgreSQL operator** (CloudNativePG recommended)
2. **Configure cluster with primary/replica setup**
3. **Set up automated backups** to MinIO or external storage
4. **Implement connection pooling** with PgBouncer
5. **Configure monitoring** and alerting for database health

#### 2.4 Redis Cluster for Session Management

**Objective**: Implement distributed session storage and caching using Redis cluster.

**Current State**:
- In-memory session storage tied to individual application instances
- No distributed caching for expensive operations
- Configuration and translation data loaded on each application start

**Target State**:
- Redis cluster for distributed session storage
- Centralized caching for frequently accessed data
- High availability with automatic failover

**Redis Cluster Configuration**:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-config
  namespace: motovault
data:
  redis.conf: |
    cluster-enabled yes
    cluster-require-full-coverage no
    cluster-node-timeout 15000
    cluster-config-file /data/nodes.conf
    cluster-migration-barrier 1
    appendonly yes
    appendfsync everysec
    save 900 1
    save 300 10
    save 60 10000

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
  namespace: motovault
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      containers:
      - name: redis
        image: redis:7-alpine
        command:
        - redis-server
        - /etc/redis/redis.conf
        ports:
        - containerPort: 6379
        - containerPort: 16379
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis
        - name: redis-data
          mountPath: /data
      volumes:
      - name: redis-config
        configMap:
          name: redis-cluster-config
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
```

**Implementation Tasks**:
1. **Deploy Redis cluster** with 6 nodes (3 masters, 3 replicas)
2. **Configure session storage**:
   ```csharp
   services.AddStackExchangeRedisCache(options =>
   {
       options.Configuration = configuration.GetConnectionString("Redis");
       options.InstanceName = "MotoVault";
   });
   
   services.AddSession(options =>
   {
       options.IdleTimeout = TimeSpan.FromMinutes(30);
       options.Cookie.HttpOnly = true;
       options.Cookie.IsEssential = true;
       options.Cookie.SecurePolicy = CookieSecurePolicy.Always;
   });
   ```

3. **Implement distributed caching**:
   ```csharp
   public class CachedTranslationService : ITranslationService
   {
       private readonly IDistributedCache _cache;
       private readonly ITranslationService _translationService;
       private readonly ILogger<CachedTranslationService> _logger;
       
       public async Task<string> GetTranslationAsync(string key, string language)
       {
           var cacheKey = $"translation:{language}:{key}";
           var cached = await _cache.GetStringAsync(cacheKey);
           
           if (cached != null)
           {
               return cached;
           }
           
           var translation = await _translationService.GetTranslationAsync(key, language);
           
           await _cache.SetStringAsync(cacheKey, translation, new DistributedCacheEntryOptions
           {
               SlidingExpiration = TimeSpan.FromHours(1)
           });
           
           return translation;
       }
   }
   ```

4. **Add cache monitoring** and performance metrics

### Phase 3: Production Deployment (Weeks 9-12)

This phase focuses on deploying the modernized application with proper production configurations and operational procedures.

#### 3.1 Kubernetes Deployment Configuration

**Objective**: Create production-ready Kubernetes manifests with proper resource management and high availability.

**Application Deployment Configuration**:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: motovault-app
  namespace: motovault
  labels:
    app: motovault
    version: v1.0.0
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: motovault
  template:
    metadata:
      labels:
        app: motovault
        version: v1.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "8080"
    spec:
      serviceAccountName: motovault-service-account
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - motovault
              topologyKey: kubernetes.io/hostname
          - weight: 50
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - motovault
              topologyKey: topology.kubernetes.io/zone
      containers:
      - name: motovault
        image: motovault:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: "Production"
        - name: ASPNETCORE_URLS
          value: "http://+:8080"
        envFrom:
        - configMapRef:
            name: motovault-config
        - secretRef:
            name: motovault-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
        volumeMounts:
        - name: tmp-volume
          mountPath: /tmp
        - name: app-logs
          mountPath: /app/logs
      volumes:
      - name: tmp-volume
        emptyDir: {}
      - name: app-logs
        emptyDir: {}
      terminationGracePeriodSeconds: 30

---
apiVersion: v1
kind: Service
metadata:
  name: motovault-service
  namespace: motovault
  labels:
    app: motovault
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: motovault

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: motovault-pdb
  namespace: motovault
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: motovault
```

**Horizontal Pod Autoscaler Configuration**:
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: motovault-hpa
  namespace: motovault
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: motovault-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60
```

#### 3.2 Ingress and TLS Configuration

**Objective**: Configure secure external access with proper TLS termination and routing.

**Ingress Configuration**:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: motovault-ingress
  namespace: motovault
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/rate-limit-window: "1m"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - motovault.example.com
    secretName: motovault-tls
  rules:
  - host: motovault.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: motovault-service
            port:
              number: 80
```

#### 3.3 Monitoring and Observability Setup

**Objective**: Implement comprehensive monitoring, logging, and alerting for production operations.

**Prometheus ServiceMonitor Configuration**:
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: motovault-metrics
  namespace: motovault
  labels:
    app: motovault
spec:
  selector:
    matchLabels:
      app: motovault
  endpoints:
  - port: http
    path: /metrics
    interval: 30s
    scrapeTimeout: 10s
```

**Application Metrics Implementation**:
```csharp
public class MetricsService
{
    private readonly Counter _httpRequestsTotal;
    private readonly Histogram _httpRequestDuration;
    private readonly Gauge _activeConnections;
    private readonly Counter _databaseOperationsTotal;
    private readonly Histogram _databaseOperationDuration;
    
    public MetricsService()
    {
        _httpRequestsTotal = Metrics.CreateCounter(
            "motovault_http_requests_total",
            "Total number of HTTP requests",
            new[] { "method", "endpoint", "status_code" });
            
        _httpRequestDuration = Metrics.CreateHistogram(
            "motovault_http_request_duration_seconds",
            "Duration of HTTP requests in seconds",
            new[] { "method", "endpoint" });
            
        _activeConnections = Metrics.CreateGauge(
            "motovault_active_connections",
            "Number of active database connections");
            
        _databaseOperationsTotal = Metrics.CreateCounter(
            "motovault_database_operations_total",
            "Total number of database operations",
            new[] { "operation", "table", "status" });
            
        _databaseOperationDuration = Metrics.CreateHistogram(
            "motovault_database_operation_duration_seconds",
            "Duration of database operations in seconds",
            new[] { "operation", "table" });
    }
    
    public void RecordHttpRequest(string method, string endpoint, int statusCode, double duration)
    {
        _httpRequestsTotal.WithLabels(method, endpoint, statusCode.ToString()).Inc();
        _httpRequestDuration.WithLabels(method, endpoint).Observe(duration);
    }
    
    public void RecordDatabaseOperation(string operation, string table, bool success, double duration)
    {
        var status = success ? "success" : "error";
        _databaseOperationsTotal.WithLabels(operation, table, status).Inc();
        _databaseOperationDuration.WithLabels(operation, table).Observe(duration);
    }
}
```

**Custom Grafana Dashboard Configuration**:
```json
{
  "dashboard": {
    "title": "MotoVaultPro Application Dashboard",
    "panels": [
      {
        "title": "HTTP Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(motovault_http_requests_total[5m])",
            "legendFormat": "{{method}} {{endpoint}}"
          }
        ]
      },
      {
        "title": "Response Time Percentiles",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.50, rate(motovault_http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "50th percentile"
          },
          {
            "expr": "histogram_quantile(0.95, rate(motovault_http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "95th percentile"
          }
        ]
      },
      {
        "title": "Database Connection Pool",
        "type": "singlestat",
        "targets": [
          {
            "expr": "motovault_active_connections",
            "legendFormat": "Active Connections"
          }
        ]
      },
      {
        "title": "Error Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(motovault_http_requests_total{status_code=~\"5..\"}[5m])",
            "legendFormat": "5xx errors"
          }
        ]
      }
    ]
  }
}
```

#### 3.4 Backup and Disaster Recovery

**Objective**: Implement comprehensive backup strategies and disaster recovery procedures.

**Velero Backup Configuration**:
```yaml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: motovault-daily-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  template:
    includedNamespaces:
    - motovault
    includedResources:
    - "*"
    storageLocation: default
    ttl: 720h0m0s  # 30 days
    snapshotVolumes: true

---
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: motovault-weekly-backup
  namespace: velero
spec:
  schedule: "0 3 * * 0"  # Weekly on Sunday at 3 AM
  template:
    includedNamespaces:
    - motovault
    includedResources:
    - "*"
    storageLocation: default
    ttl: 2160h0m0s  # 90 days
    snapshotVolumes: true
```

**Database Backup Strategy**:
```bash
#!/bin/bash
# Automated database backup script

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="motovault_backup_${BACKUP_DATE}.sql"
S3_BUCKET="motovault-backups"

# Create database backup
kubectl exec -n motovault motovault-postgres-1 -- \
  pg_dump -U postgres motovault > "${BACKUP_FILE}"

# Compress backup
gzip "${BACKUP_FILE}"

# Upload to S3/MinIO
aws s3 cp "${BACKUP_FILE}.gz" "s3://${S3_BUCKET}/database/"

# Clean up local file
rm "${BACKUP_FILE}.gz"

# Retain only last 30 days of backups
aws s3api list-objects-v2 \
  --bucket "${S3_BUCKET}" \
  --prefix "database/" \
  --query 'Contents[?LastModified<=`'$(date -d "30 days ago" --iso-8601)'`].[Key]' \
  --output text | \
  xargs -I {} aws s3 rm "s3://${S3_BUCKET}/{}"
```

### Phase 4: Advanced Features and Optimization (Weeks 13-16)

This phase focuses on advanced cloud-native features and performance optimization.

#### 4.1 Advanced Caching Strategies

**Objective**: Implement multi-layer caching for optimal performance and reduced database load.

**Cache Architecture**:
```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Browser       │    │   CDN/Proxy     │    │   Application   │
│   Cache         │◄──►│   Cache         │◄──►│   Memory Cache  │
│   (Static)      │    │   (Static +     │    │   (L1)          │
│                 │    │    Dynamic)     │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
                                               ┌─────────────────┐
                                               │   Redis Cache   │
                                               │   (L2)          │
                                               │   Distributed   │
                                               └─────────────────┘
                                                        │
                                               ┌─────────────────┐
                                               │   Database      │
                                               │   (Source)      │
                                               │                 │
                                               └─────────────────┘
```

**Implementation Details**:
```csharp
public class MultiLevelCacheService
{
    private readonly IMemoryCache _memoryCache;
    private readonly IDistributedCache _distributedCache;
    private readonly ILogger<MultiLevelCacheService> _logger;
    
    public async Task<T> GetAsync<T>(string key, Func<Task<T>> factory, TimeSpan? expiration = null)
    {
        // L1 Cache - Memory
        if (_memoryCache.TryGetValue(key, out T cachedValue))
        {
            _logger.LogDebug("Cache hit (L1): {Key}", key);
            return cachedValue;
        }
        
        // L2 Cache - Redis
        var distributedValue = await _distributedCache.GetStringAsync(key);
        if (distributedValue != null)
        {
            var deserializedValue = JsonSerializer.Deserialize<T>(distributedValue);
            _memoryCache.Set(key, deserializedValue, TimeSpan.FromMinutes(5)); // Short-lived L1 cache
            _logger.LogDebug("Cache hit (L2): {Key}", key);
            return deserializedValue;
        }
        
        // Cache miss - fetch from source
        _logger.LogDebug("Cache miss: {Key}", key);
        var value = await factory();
        
        // Store in both cache levels
        var serializedValue = JsonSerializer.Serialize(value);
        await _distributedCache.SetStringAsync(key, serializedValue, new DistributedCacheEntryOptions
        {
            SlidingExpiration = expiration ?? TimeSpan.FromHours(1)
        });
        
        _memoryCache.Set(key, value, TimeSpan.FromMinutes(5));
        
        return value;
    }
}
```

#### 4.2 Performance Optimization

**Objective**: Optimize application performance for high-load scenarios.

**Database Query Optimization**:
```csharp
public class OptimizedVehicleService
{
    private readonly IDbContextFactory<MotoVaultContext> _dbContextFactory;
    private readonly IMemoryCache _cache;
    
    public async Task<VehicleDashboardData> GetDashboardDataAsync(int userId, int vehicleId)
    {
        var cacheKey = $"dashboard:{userId}:{vehicleId}";
        
        if (_cache.TryGetValue(cacheKey, out VehicleDashboardData cached))
        {
            return cached;
        }
        
        using var context = _dbContextFactory.CreateDbContext();
        
        // Optimized single query with projections
        var dashboardData = await context.Vehicles
            .Where(v => v.Id == vehicleId && v.UserId == userId)
            .Select(v => new VehicleDashboardData
            {
                Vehicle = v,
                RecentServices = v.ServiceRecords
                    .OrderByDescending(s => s.Date)
                    .Take(5)
                    .ToList(),
                UpcomingReminders = v.ReminderRecords
                    .Where(r => r.IsActive && r.DueDate > DateTime.Now)
                    .OrderBy(r => r.DueDate)
                    .Take(5)
                    .ToList(),
                FuelEfficiency = v.GasRecords
                    .Where(g => g.Date >= DateTime.Now.AddMonths(-3))
                    .Average(g => g.Efficiency),
                TotalMileage = v.OdometerRecords
                    .OrderByDescending(o => o.Date)
                    .FirstOrDefault().Mileage ?? 0
            })
            .AsNoTracking()
            .FirstOrDefaultAsync();
        
        _cache.Set(cacheKey, dashboardData, TimeSpan.FromMinutes(15));
        return dashboardData;
    }
}
```

**Connection Pool Optimization**:
```csharp
services.AddDbContextFactory<MotoVaultContext>(options =>
{
    options.UseNpgsql(connectionString, npgsqlOptions =>
    {
        npgsqlOptions.EnableRetryOnFailure(
            maxRetryCount: 3,
            maxRetryDelay: TimeSpan.FromSeconds(5),
            errorCodesToAdd: null);
        npgsqlOptions.CommandTimeout(30);
    });
    
    // Optimize for read-heavy workloads
    options.EnableSensitiveDataLogging(false);
    options.EnableServiceProviderCaching();
    options.EnableDetailedErrors(false);
}, ServiceLifetime.Singleton);

// Configure connection pooling
services.Configure<NpgsqlConnectionStringBuilder>(builder =>
{
    builder.MaxPoolSize = 100;
    builder.MinPoolSize = 10;
    builder.ConnectionLifetime = 300;
    builder.ConnectionPruningInterval = 10;
    builder.ConnectionIdleLifetime = 300;
});
```

#### 4.3 Security Enhancements

**Objective**: Implement advanced security features for production deployment.

**Network Security Policies**:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: motovault-network-policy
  namespace: motovault
spec:
  podSelector:
    matchLabels:
      app: motovault
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: nginx-ingress
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: motovault
    ports:
    - protocol: TCP
      port: 5432  # PostgreSQL
    - protocol: TCP
      port: 6379  # Redis
    - protocol: TCP
      port: 9000  # MinIO
  - to: []  # Allow external HTTPS for OIDC
    ports:
    - protocol: TCP
      port: 443
    - protocol: TCP
      port: 80
```

**Pod Security Standards**:
```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: motovault
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
```

**Secret Management with External Secrets Operator**:
```yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: motovault
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "motovault-role"

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: motovault-secrets
  namespace: motovault
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: motovault-secrets
    creationPolicy: Owner
  data:
  - secretKey: POSTGRES_CONNECTION
    remoteRef:
      key: motovault/database
      property: connection_string
  - secretKey: JWT_SECRET
    remoteRef:
      key: motovault/auth
      property: jwt_secret
```

## Migration Strategy

### Pre-Migration Assessment

**Current State Analysis**:
1. **Data Inventory**: Catalog all existing data, configurations, and file attachments
2. **Dependency Mapping**: Identify all external dependencies and integrations
3. **Performance Baseline**: Establish current performance metrics for comparison
4. **User Impact Assessment**: Analyze potential downtime and user experience changes

**Migration Prerequisites**:
1. **Kubernetes Cluster Ready**: Properly configured cluster with required operators
2. **Infrastructure Deployed**: PostgreSQL, MinIO, and Redis clusters operational
3. **Backup Strategy**: Complete backup of current system and data
4. **Rollback Plan**: Detailed procedure for reverting to current system if needed

### Migration Execution Plan

#### Phase 1: Parallel Environment Setup (Week 1)
1. **Deploy target infrastructure** in parallel to existing system
2. **Configure monitoring and logging** for new environment
3. **Run initial data migration tests** with sample data
4. **Validate all health checks** and monitoring alerts

#### Phase 2: Data Migration (Week 2)
1. **Initial data sync**: Migrate historical data during low-usage periods
2. **File migration**: Transfer all attachments to MinIO with validation
3. **Configuration migration**: Convert all settings to ConfigMaps/Secrets
4. **User data validation**: Verify data integrity and completeness

#### Phase 3: Application Cutover (Week 3)
1. **Final data sync**: Synchronize any changes made during migration
2. **DNS cutover**: Redirect traffic to new Kubernetes deployment
3. **Monitor closely**: Watch for any issues or performance problems
4. **User acceptance testing**: Validate all functionality works correctly

#### Phase 4: Optimization and Cleanup (Week 4)
1. **Performance tuning**: Optimize based on real-world usage patterns
2. **Clean up old infrastructure**: Decommission legacy deployment
3. **Update documentation**: Finalize operational procedures
4. **Training**: Train operations team on new procedures

### Data Migration Tools

**LiteDB to PostgreSQL Migration Utility**:
```csharp
public class DataMigrationService
{
    private readonly ILiteDatabase _liteDb;
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<DataMigrationService> _logger;
    
    public async Task<MigrationResult> MigrateAllDataAsync()
    {
        var result = new MigrationResult();
        
        try
        {
            using var scope = _serviceProvider.CreateScope();
            var context = scope.ServiceProvider.GetRequiredService<MotoVaultContext>();
            
            // Migrate users first (dependencies)
            result.UsersProcessed = await MigrateUsersAsync(context);
            
            // Migrate vehicles
            result.VehiclesProcessed = await MigrateVehiclesAsync(context);
            
            // Migrate all record types
            result.ServiceRecordsProcessed = await MigrateServiceRecordsAsync(context);
            result.GasRecordsProcessed = await MigrateGasRecordsAsync(context);
            result.FilesProcessed = await MigrateFilesAsync();
            
            await context.SaveChangesAsync();
            result.Success = true;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Migration failed");
            result.Success = false;
            result.ErrorMessage = ex.Message;
        }
        
        return result;
    }
    
    private async Task<int> MigrateFilesAsync()
    {
        var fileStorage = _serviceProvider.GetRequiredService<IFileStorageService>();
        var filesProcessed = 0;
        
        var localFilesPath = "data/files";
        if (Directory.Exists(localFilesPath))
        {
            var files = Directory.GetFiles(localFilesPath, "*", SearchOption.AllDirectories);
            
            foreach (var filePath in files)
            {
                using var fileStream = File.OpenRead(filePath);
                var fileName = Path.GetFileName(filePath);
                var contentType = GetContentType(fileName);
                
                await fileStorage.UploadFileAsync(fileStream, fileName, contentType);
                filesProcessed++;
                
                _logger.LogInformation("Migrated file: {FileName}", fileName);
            }
        }
        
        return filesProcessed;
    }
}
```

### Rollback Procedures

**Emergency Rollback Plan**:
1. **Immediate Actions** (0-15 minutes):
   - Redirect DNS back to original system
   - Activate incident response team
   - Begin root cause analysis

2. **Data Consistency** (15-30 minutes):
   - Verify data integrity in original system
   - Sync any changes made during brief cutover period
   - Validate all services are operational

3. **Communication** (30-60 minutes):
   - Notify stakeholders of rollback
   - Provide status updates to users
   - Document lessons learned

4. **Post-Rollback Analysis** (1-24 hours):
   - Complete root cause analysis
   - Update migration plan based on findings
   - Plan next migration attempt

## Risk Assessment and Mitigation

### Technical Risks

#### High Impact Risks

**1. Data Loss or Corruption**
- **Probability**: Low
- **Impact**: Critical
- **Mitigation**: 
  - Multiple backup strategies with point-in-time recovery
  - Comprehensive data validation during migration
  - Parallel running systems during cutover
  - Automated data integrity checks

**2. Extended Downtime During Migration**
- **Probability**: Medium
- **Impact**: High
- **Mitigation**:
  - Phased migration approach with minimal downtime windows
  - Blue-green deployment strategy
  - Comprehensive rollback procedures
  - 24/7 monitoring during cutover

**3. Performance Degradation**
- **Probability**: Medium
- **Impact**: Medium
- **Mitigation**:
  - Extensive load testing before migration
  - Performance monitoring and alerting
  - Auto-scaling capabilities
  - Database query optimization

#### Medium Impact Risks

**4. Integration Failures**
- **Probability**: Medium
- **Impact**: Medium
- **Mitigation**:
  - Thorough integration testing
  - Circuit breaker patterns for external dependencies
  - Graceful degradation for non-critical features
  - Health check monitoring

**5. Security Vulnerabilities**
- **Probability**: Low
- **Impact**: High
- **Mitigation**:
  - Security scanning of all container images
  - Network policies and Pod Security Standards
  - Secret management best practices
  - Regular security audits

### Operational Risks

**6. Team Knowledge Gaps**
- **Probability**: Medium
- **Impact**: Medium
- **Mitigation**:
  - Comprehensive training program
  - Detailed operational documentation
  - On-call procedures and runbooks
  - Knowledge transfer sessions

**7. Infrastructure Capacity Issues**
- **Probability**: Low
- **Impact**: Medium
- **Mitigation**:
  - Capacity planning and resource monitoring
  - Auto-scaling policies
  - Resource quotas and limits
  - Infrastructure as Code for rapid scaling

### Business Risks

**8. User Adoption Challenges**
- **Probability**: Low
- **Impact**: Medium
- **Mitigation**:
  - Transparent communication about changes
  - User training and documentation
  - Phased rollout to minimize impact
  - User feedback collection and response

## Testing Strategy

### Test Environment Architecture

**Multi-Environment Strategy**:
```
Development → Staging → Pre-Production → Production
     ↓           ↓            ↓             ↓
   Unit Tests  Integration  Load Testing  Monitoring
   API Tests   UI Tests     Security      Alerting
   DB Tests    E2E Tests    Performance   Backup Tests
```

### Comprehensive Testing Plan

#### Unit Testing
- **Coverage Target**: 80% code coverage minimum
- **Focus Areas**: Business logic, data access layer, API endpoints
- **Test Framework**: xUnit with Moq for dependency injection testing
- **Automated Execution**: Run on every commit and pull request

#### Integration Testing
- **Database Integration**: Test all repository implementations
- **External Service Integration**: MinIO, Redis, PostgreSQL connectivity
- **API Integration**: Full request/response cycle testing
- **Authentication Testing**: All authentication flows and authorization rules

#### Load Testing
- **Tools**: k6 or Artillery for load generation
- **Scenarios**: 
  - Normal load: 100 concurrent users
  - Peak load: 500 concurrent users
  - Stress test: 1000+ concurrent users
- **Metrics**: Response time, throughput, error rate, resource utilization

#### Security Testing
- **Container Security**: Scan images for vulnerabilities
- **Network Security**: Validate network policies and isolation
- **Authentication**: Test all authentication and authorization scenarios
- **Data Protection**: Verify encryption at rest and in transit

#### Disaster Recovery Testing
- **Database Failover**: Test automatic failover scenarios
- **Application Recovery**: Pod failure and recovery testing
- **Backup Restoration**: Full system restoration from backups
- **Network Partitioning**: Test behavior during network issues

### Performance Testing Scenarios

**Load Testing Script Example**:
```javascript
import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 20 }, // Ramp up
    { duration: '5m', target: 20 }, // Stay at 20 users
    { duration: '2m', target: 50 }, // Ramp up to 50
    { duration: '5m', target: 50 }, // Stay at 50
    { duration: '2m', target: 100 }, // Ramp up to 100
    { duration: '5m', target: 100 }, // Stay at 100
    { duration: '2m', target: 0 }, // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], // 95% of requests under 500ms
    http_req_failed: ['rate<0.1'], // Error rate under 10%
  },
};

export default function() {
  // Login
  let loginResponse = http.post('https://motovault.example.com/api/auth/login', {
    username: 'testuser',
    password: 'testpass'
  });
  
  check(loginResponse, {
    'login successful': (r) => r.status === 200,
  });
  
  let authToken = loginResponse.json('token');
  
  // Dashboard load
  let dashboardResponse = http.get('https://motovault.example.com/api/dashboard', {
    headers: { Authorization: `Bearer ${authToken}` },
  });
  
  check(dashboardResponse, {
    'dashboard loaded': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
  
  sleep(1);
}
```

## Operational Procedures

### Monitoring and Alerting

#### Application Metrics
```yaml
# Prometheus AlertManager Rules
groups:
- name: motovault.rules
  rules:
  - alert: HighErrorRate
    expr: rate(motovault_http_requests_total{status_code=~"5.."}[5m]) > 0.1
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High error rate detected"
      description: "Error rate is {{ $value }}% for the last 5 minutes"
      
  - alert: HighResponseTime
    expr: histogram_quantile(0.95, rate(motovault_http_request_duration_seconds_bucket[5m])) > 2
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High response time detected"
      description: "95th percentile response time is {{ $value }}s"
      
  - alert: DatabaseConnectionPoolExhaustion
    expr: motovault_active_connections > 80
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Database connection pool nearly exhausted"
      description: "Active connections: {{ $value }}/100"
      
  - alert: PodCrashLooping
    expr: rate(kube_pod_container_status_restarts_total{namespace="motovault"}[15m]) > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Pod is crash looping"
      description: "Pod {{ $labels.pod }} is restarting frequently"
```

#### Infrastructure Monitoring
- **Node Resources**: CPU, memory, disk usage across all nodes
- **Network Performance**: Latency, throughput, packet loss
- **Storage Performance**: IOPS, latency for persistent volumes
- **Kubernetes Health**: API server, etcd, scheduler performance

### Backup and Recovery Procedures

#### Automated Backup Schedule
```bash
# Daily backup script
#!/bin/bash
set -e

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_NAMESPACE="motovault"

# Database backup
echo "Starting database backup at $(date)"
kubectl exec -n $BACKUP_NAMESPACE motovault-postgres-1 -- \
  pg_dump -U postgres motovault | \
  gzip > "database_backup_${TIMESTAMP}.sql.gz"

# MinIO backup (metadata and small files)
echo "Starting MinIO backup at $(date)"
mc mirror motovault-minio/motovault-files backup/minio_${TIMESTAMP}/

# Kubernetes resources backup
echo "Starting Kubernetes backup at $(date)"
velero backup create "motovault-${TIMESTAMP}" \
  --include-namespaces motovault \
  --wait

# Upload to remote storage
echo "Uploading backups to remote storage"
aws s3 cp "database_backup_${TIMESTAMP}.sql.gz" s3://motovault-backups/daily/
aws s3 sync "backup/minio_${TIMESTAMP}/" s3://motovault-backups/minio/${TIMESTAMP}/

# Cleanup local files older than 7 days
find backup/ -name "*.gz" -mtime +7 -delete
find backup/minio_* -mtime +7 -exec rm -rf {} \;

echo "Backup completed successfully at $(date)"
```

#### Recovery Procedures
```bash
# Full system recovery script
#!/bin/bash
set -e

BACKUP_DATE=$1
if [ -z "$BACKUP_DATE" ]; then
  echo "Usage: $0 <backup_date>"
  echo "Example: $0 20240120_020000"
  exit 1
fi

# Stop application
echo "Scaling down application..."
kubectl scale deployment motovault-app --replicas=0 -n motovault

# Restore database
echo "Restoring database from backup..."
aws s3 cp "s3://motovault-backups/daily/database_backup_${BACKUP_DATE}.sql.gz" .
gunzip "database_backup_${BACKUP_DATE}.sql.gz"
kubectl exec -i motovault-postgres-1 -n motovault -- \
  psql -U postgres -d motovault < "database_backup_${BACKUP_DATE}.sql"

# Restore MinIO data
echo "Restoring MinIO data..."
aws s3 sync "s3://motovault-backups/minio/${BACKUP_DATE}/" /tmp/minio_restore/
mc mirror /tmp/minio_restore/ motovault-minio/motovault-files/

# Restart application
echo "Scaling up application..."
kubectl scale deployment motovault-app --replicas=3 -n motovault

# Verify health
echo "Waiting for application to be ready..."
kubectl wait --for=condition=ready pod -l app=motovault -n motovault --timeout=300s

echo "Recovery completed successfully"
```

### Maintenance Procedures

#### Rolling Updates
```yaml
# Zero-downtime deployment strategy
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: motovault-rollout
  namespace: motovault
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 1m}
      - setWeight: 40
      - pause: {duration: 2m}
      - setWeight: 60
      - pause: {duration: 2m}
      - setWeight: 80
      - pause: {duration: 2m}
      analysis:
        templates:
        - templateName: success-rate
        args:
        - name: service-name
          value: motovault-service
      canaryService: motovault-canary-service
      stableService: motovault-stable-service
  selector:
    matchLabels:
      app: motovault
  template:
    metadata:
      labels:
        app: motovault
    spec:
      containers:
      - name: motovault
        image: motovault:latest
        # ... container spec
```

#### Scaling Procedures
- **Horizontal Scaling**: Use HPA for automatic scaling based on metrics
- **Vertical Scaling**: Monitor resource usage and adjust requests/limits
- **Database Scaling**: Add read replicas for read-heavy workloads
- **Storage Scaling**: Monitor MinIO usage and add nodes as needed

## Implementation Timeline

### Detailed 16-Week Schedule

#### Weeks 1-4: Foundation Phase
**Week 1: Environment Setup**
- Day 1-2: Kubernetes cluster setup and configuration
- Day 3-4: Deploy PostgreSQL operator and cluster
- Day 5-7: Deploy MinIO operator and configure HA cluster

**Week 2: Redis and Monitoring**
- Day 1-3: Deploy Redis cluster with sentinel configuration
- Day 4-5: Set up Prometheus and Grafana
- Day 6-7: Configure initial monitoring dashboards

**Week 3: Application Changes**
- Day 1-2: Remove LiteDB dependencies
- Day 3-4: Implement configuration externalization
- Day 5-7: Add health check endpoints

**Week 4: File Storage Abstraction**
- Day 1-3: Implement IFileStorageService interface
- Day 4-5: Create MinIO implementation
- Day 6-7: Add fallback mechanisms

#### Weeks 5-8: Core Implementation
**Week 5: Database Integration**
- Day 1-3: Optimize PostgreSQL connections
- Day 4-5: Implement connection pooling
- Day 6-7: Add database health checks

**Week 6: Session and Caching**
- Day 1-2: Implement Redis session storage
- Day 3-4: Add distributed caching layer
- Day 5-7: Implement multi-level caching

**Week 7: Observability**
- Day 1-3: Add structured logging
- Day 4-5: Implement Prometheus metrics
- Day 6-7: Add distributed tracing

**Week 8: Security Implementation**
- Day 1-2: Configure Pod Security Standards
- Day 3-4: Implement network policies
- Day 5-7: Set up secret management

#### Weeks 9-12: Production Deployment
**Week 9: Kubernetes Manifests**
- Day 1-3: Create production Kubernetes manifests
- Day 4-5: Configure HPA and resource limits
- Day 6-7: Set up ingress and TLS

**Week 10: Backup and Recovery**
- Day 1-3: Implement backup strategies
- Day 4-5: Create recovery procedures
- Day 6-7: Test disaster recovery scenarios

**Week 11: Load Testing**
- Day 1-3: Create load testing scenarios
- Day 4-5: Execute performance tests
- Day 6-7: Optimize based on results

**Week 12: Migration Preparation**
- Day 1-3: Create data migration tools
- Day 4-5: Test migration procedures
- Day 6-7: Prepare rollback plans

#### Weeks 13-16: Advanced Features
**Week 13: Performance Optimization**
- Day 1-3: Implement advanced caching strategies
- Day 4-5: Optimize database queries
- Day 6-7: Fine-tune resource allocation

**Week 14: Advanced Security**
- Day 1-3: Implement external secret management
- Day 4-5: Add security scanning to CI/CD
- Day 6-7: Configure advanced network policies

**Week 15: Production Migration**
- Day 1-2: Execute data migration
- Day 3-4: Perform application cutover
- Day 5-7: Monitor and optimize

**Week 16: Optimization and Documentation**
- Day 1-3: Performance tuning based on production usage
- Day 4-5: Update operational documentation
- Day 6-7: Conduct team training

### Success Criteria

#### Technical Success Metrics
- **Availability**: 99.9% uptime (no more than 8.76 hours downtime per year)
- **Performance**: 95th percentile response time under 500ms
- **Scalability**: Ability to handle 10x current user load
- **Recovery**: RTO < 1 hour, RPO < 15 minutes

#### Operational Success Metrics
- **Deployment Frequency**: Enable weekly deployments with zero downtime
- **Mean Time to Recovery**: < 30 minutes for critical issues
- **Change Failure Rate**: < 5% of deployments require rollback
- **Monitoring Coverage**: 100% of critical services monitored

#### Business Success Metrics
- **User Satisfaction**: No degradation in user experience
- **Cost Efficiency**: Infrastructure costs within 20% of current spending
- **Maintenance Overhead**: Reduced operational maintenance time by 50%
- **Future Readiness**: Foundation for future enhancements and scaling

---

**Document Version**: 1.0  
**Last Updated**: January 2025  
**Author**: MotoVaultPro Modernization Team  
**Status**: Draft for Review

---

This comprehensive plan provides a detailed roadmap for modernizing MotoVaultPro to run efficiently on Kubernetes with high availability, scalability, and operational excellence. The phased approach ensures minimal risk while delivering maximum benefits for future growth and reliability.