motovaultpro/K8S-PHASE-1-DETAILED.md

Phase 1: Core Kubernetes Readiness - Detailed Implementation Plan

Executive Summary

This document provides a comprehensive, step-by-step implementation plan for Phase 1 that ensures minimal risk through incremental changes, thorough testing, and debugging at each step. Each change is isolated, tested, and verified before proceeding to the next step.

Improved Implementation Strategy

Key Principles

1. One Change at a Time: Each step focuses on a single, well-defined change
2. Non-Destructive First: Start with safest changes that don't affect data or core functionality
3. Comprehensive Testing: Automated and manual validation at each step
4. Rollback Ready: Every change includes a rollback procedure
5. Debugging First: Extensive debugging and diagnostic capabilities before making changes
6. Continuous Validation: Performance and functionality validation throughout

Risk Mitigation Improvements

• Comprehensive Logging: Extensive structured logging for troubleshooting
• Feature Flags: Enable/disable new functionality without code changes
• Automated Testing: Comprehensive test suite validation at each step
• Performance Monitoring: Baseline and continuous performance validation
• User Experience: Functional testing ensures no user-facing regressions

Step-by-Step Implementation Plan

Step 1: Structured Logging Implementation

Duration: 2-3 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Implement structured JSON logging to improve observability during subsequent changes.

Implementation

// 1.1: Add logging configuration (Program.cs)
builder.Services.AddLogging(loggingBuilder =>
{
    loggingBuilder.ClearProviders();
    
    if (builder.Environment.IsDevelopment())
    {
        loggingBuilder.AddConsole();
    }
    else
    {
        loggingBuilder.AddJsonConsole(options =>
        {
            options.IncludeScopes = true;
            options.TimestampFormat = "yyyy-MM-ddTHH:mm:ss.fffZ";
            options.JsonWriterOptions = new JsonWriterOptions { Indented = false };
        });
    }
});

// 1.2: Add correlation ID service
public class CorrelationIdService
{
    public string CorrelationId { get; } = Guid.NewGuid().ToString();
}

// 1.3: Add correlation ID middleware
public class CorrelationIdMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<CorrelationIdMiddleware> _logger;

    public async Task InvokeAsync(HttpContext context)
    {
        var correlationId = context.Request.Headers["X-Correlation-ID"]
            .FirstOrDefault() ?? Guid.NewGuid().ToString();

        context.Items["CorrelationId"] = correlationId;
        context.Response.Headers.Add("X-Correlation-ID", correlationId);

        using var scope = _logger.BeginScope(new Dictionary<string, object>
        {
            ["CorrelationId"] = correlationId,
            ["RequestPath"] = context.Request.Path,
            ["RequestMethod"] = context.Request.Method
        });

        await _next(context);
    }
}

Testing Plan

Automated Tests:

[Test]
public async Task StructuredLogging_ProducesValidJson()
{
    // Arrange
    var logOutput = new StringWriter();
    var logger = CreateTestLogger(logOutput);
    
    // Act
    logger.LogInformation("Test message", new { TestProperty = "TestValue" });
    
    // Assert
    var logEntry = JsonSerializer.Deserialize<LogEntry>(logOutput.ToString());
    Assert.IsNotNull(logEntry.Timestamp);
    Assert.AreEqual("Information", logEntry.Level);
    Assert.Contains("Test message", logEntry.Message);
}

[Test]
public async Task CorrelationId_PreservedAcrossRequests()
{
    // Test that correlation ID flows through request pipeline
    var client = _factory.CreateClient();
    var correlationId = Guid.NewGuid().ToString();
    
    client.DefaultRequestHeaders.Add("X-Correlation-ID", correlationId);
    var response = await client.GetAsync("/health");
    
    Assert.AreEqual(correlationId, response.Headers.GetValues("X-Correlation-ID").First());
}

Manual Validation:

1. Start application and verify JSON log format in console
2. Make HTTP requests and verify correlation IDs in logs
3. Check log aggregation works with external tools
4. Verify existing functionality unchanged

Success Criteria:

• All logs output in structured JSON format
• Correlation IDs generated and preserved
• No existing functionality affected
• Performance impact < 5ms per request

Rollback Procedure:

# Revert logging configuration
git checkout HEAD~1 -- Program.cs
# Remove middleware registration
# Restart application

---

Step 2: Health Check Infrastructure

Duration: 2-3 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Implement comprehensive health check endpoints for Kubernetes readiness and liveness probes.

Implementation

// 2.1: Add health check services
public class DatabaseHealthCheck : IHealthCheck
{
    private readonly IConfiguration _configuration;
    private readonly ILogger<DatabaseHealthCheck> _logger;

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, 
        CancellationToken cancellationToken = default)
    {
        try
        {
            var connectionString = _configuration.GetConnectionString("DefaultConnection");
            
            if (connectionString?.Contains("LiteDB") == true)
            {
                return await CheckLiteDBHealthAsync(connectionString);
            }
            else if (!string.IsNullOrEmpty(connectionString))
            {
                return await CheckPostgreSQLHealthAsync(connectionString, cancellationToken);
            }
            
            return HealthCheckResult.Unhealthy("No database configuration found");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Database health check failed");
            return HealthCheckResult.Unhealthy("Database health check failed", ex);
        }
    }

    private async Task<HealthCheckResult> CheckPostgreSQLHealthAsync(
        string connectionString, 
        CancellationToken cancellationToken)
    {
        using var connection = new NpgsqlConnection(connectionString);
        await connection.OpenAsync(cancellationToken);
        
        using var command = new NpgsqlCommand("SELECT 1", connection);
        var result = await command.ExecuteScalarAsync(cancellationToken);
        
        return HealthCheckResult.Healthy($"PostgreSQL connection successful. Result: {result}");
    }

    private async Task<HealthCheckResult> CheckLiteDBHealthAsync(string connectionString)
    {
        try
        {
            using var db = new LiteDatabase(connectionString);
            var collections = db.GetCollectionNames().ToList();
            return HealthCheckResult.Healthy($"LiteDB connection successful. Collections: {collections.Count}");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("LiteDB connection failed", ex);
        }
    }
}

// 2.2: Add application health check
public class ApplicationHealthCheck : IHealthCheck
{
    private readonly IServiceProvider _serviceProvider;

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, 
        CancellationToken cancellationToken = default)
    {
        try
        {
            // Verify essential services are available
            var vehicleLogic = _serviceProvider.GetService<VehicleLogic>();
            var userLogic = _serviceProvider.GetService<UserLogic>();
            
            if (vehicleLogic == null || userLogic == null)
            {
                return HealthCheckResult.Unhealthy("Essential services not available");
            }

            return HealthCheckResult.Healthy("Application services available");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy("Application health check failed", ex);
        }
    }
}

// 2.3: Configure health checks in Program.cs
builder.Services.AddHealthChecks()
    .AddCheck<DatabaseHealthCheck>("database", tags: new[] { "ready", "db" })
    .AddCheck<ApplicationHealthCheck>("application", tags: new[] { "ready", "app" });

// 2.4: Add health check endpoints
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = _ => false, // Only checks if the app is running
    ResponseWriter = async (context, report) =>
    {
        context.Response.ContentType = "application/json";
        var response = new
        {
            status = "Healthy",
            timestamp = DateTime.UtcNow,
            uptime = DateTime.UtcNow - Process.GetCurrentProcess().StartTime
        };
        await context.Response.WriteAsync(JsonSerializer.Serialize(response));
    }
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready"),
    ResponseWriter = async (context, report) =>
    {
        context.Response.ContentType = "application/json";
        var response = new
        {
            status = report.Status.ToString(),
            timestamp = DateTime.UtcNow,
            checks = report.Entries.Select(x => new
            {
                name = x.Key,
                status = x.Value.Status.ToString(),
                description = x.Value.Description,
                duration = x.Value.Duration.TotalMilliseconds
            })
        };
        await context.Response.WriteAsync(JsonSerializer.Serialize(response));
    }
});

Testing Plan

Automated Tests:

[Test]
public async Task HealthCheck_Live_ReturnsHealthy()
{
    var client = _factory.CreateClient();
    var response = await client.GetAsync("/health/live");
    
    Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
    
    var content = await response.Content.ReadAsStringAsync();
    var healthResponse = JsonSerializer.Deserialize<HealthResponse>(content);
    
    Assert.AreEqual("Healthy", healthResponse.Status);
    Assert.IsTrue(healthResponse.Uptime > TimeSpan.Zero);
}

[Test]
public async Task HealthCheck_Ready_ValidatesAllServices()
{
    var client = _factory.CreateClient();
    var response = await client.GetAsync("/health/ready");
    
    Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
    
    var content = await response.Content.ReadAsStringAsync();
    var healthResponse = JsonSerializer.Deserialize<HealthResponse>(content);
    
    Assert.Contains(healthResponse.Checks, c => c.Name == "database");
    Assert.Contains(healthResponse.Checks, c => c.Name == "application");
}

[Test]
public async Task HealthCheck_DatabaseFailure_ReturnsUnhealthy()
{
    // Test with invalid connection string
    var factory = CreateTestFactory(invalidConnectionString: true);
    var client = factory.CreateClient();
    
    var response = await client.GetAsync("/health/ready");
    Assert.AreEqual(HttpStatusCode.ServiceUnavailable, response.StatusCode);
}

Manual Validation:

1. Verify /health/live returns 200 OK with JSON response
2. Verify /health/ready returns detailed health information
3. Test with database disconnected - should return 503
4. Verify health checks work with both LiteDB and PostgreSQL
5. Test health check performance (< 100ms response time)

Success Criteria:

• Health endpoints return appropriate HTTP status codes
• JSON responses contain required health information
• Database connectivity properly validated
• Health checks complete within 100ms
• Unhealthy conditions properly detected

---

Step 3: Configuration Framework Enhancement

Duration: 2-3 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Enhance configuration framework to support both file-based and environment variable configuration.

Implementation

// 3.1: Create configuration validation service
public class ConfigurationValidationService
{
    private readonly IConfiguration _configuration;
    private readonly ILogger<ConfigurationValidationService> _logger;

    public ValidationResult ValidateConfiguration()
    {
        var result = new ValidationResult();
        
        // Database configuration
        ValidateDatabaseConfiguration(result);
        
        // Application configuration
        ValidateApplicationConfiguration(result);
        
        // External service configuration
        ValidateExternalServiceConfiguration(result);
        
        return result;
    }

    private void ValidateDatabaseConfiguration(ValidationResult result)
    {
        var postgresConnection = _configuration.GetConnectionString("DefaultConnection");
        var liteDbPath = _configuration["LiteDB:DatabasePath"];
        
        if (string.IsNullOrEmpty(postgresConnection) && string.IsNullOrEmpty(liteDbPath))
        {
            result.AddError("Database", "No database configuration found");
        }
        
        if (!string.IsNullOrEmpty(postgresConnection))
        {
            try
            {
                var builder = new NpgsqlConnectionStringBuilder(postgresConnection);
                if (string.IsNullOrEmpty(builder.Database))
                {
                    result.AddWarning("Database", "PostgreSQL database name not specified");
                }
            }
            catch (Exception ex)
            {
                result.AddError("Database", $"Invalid PostgreSQL connection string: {ex.Message}");
            }
        }
    }

    private void ValidateApplicationConfiguration(ValidationResult result)
    {
        var appName = _configuration["App:Name"];
        if (string.IsNullOrEmpty(appName))
        {
            result.AddWarning("Application", "Application name not configured");
        }

        var logLevel = _configuration["Logging:LogLevel:Default"];
        if (!IsValidLogLevel(logLevel))
        {
            result.AddWarning("Logging", $"Invalid log level: {logLevel}");
        }
    }

    private void ValidateExternalServiceConfiguration(ValidationResult result)
    {
        // Validate email configuration if enabled
        var emailEnabled = _configuration.GetValue<bool>("Email:Enabled");
        if (emailEnabled)
        {
            var smtpServer = _configuration["Email:SmtpServer"];
            if (string.IsNullOrEmpty(smtpServer))
            {
                result.AddError("Email", "SMTP server required when email is enabled");
            }
        }

        // Validate OIDC configuration if enabled
        var oidcEnabled = _configuration.GetValue<bool>("Authentication:OpenIDConnect:Enabled");
        if (oidcEnabled)
        {
            var authority = _configuration["Authentication:OpenIDConnect:Authority"];
            var clientId = _configuration["Authentication:OpenIDConnect:ClientId"];
            
            if (string.IsNullOrEmpty(authority) || string.IsNullOrEmpty(clientId))
            {
                result.AddError("OIDC", "Authority and ClientId required for OpenID Connect");
            }
        }
    }
}

// 3.2: Create startup configuration validation
public class StartupConfigurationValidator : IHostedService
{
    private readonly ConfigurationValidationService _validator;
    private readonly ILogger<StartupConfigurationValidator> _logger;
    private readonly IHostApplicationLifetime _lifetime;

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Validating application configuration...");
        
        var result = _validator.ValidateConfiguration();
        
        foreach (var warning in result.Warnings)
        {
            _logger.LogWarning("Configuration warning: {Category}: {Message}", 
                warning.Category, warning.Message);
        }

        if (result.HasErrors)
        {
            foreach (var error in result.Errors)
            {
                _logger.LogError("Configuration error: {Category}: {Message}", 
                    error.Category, error.Message);
            }
            
            _logger.LogCritical("Application startup failed due to configuration errors");
            _lifetime.StopApplication();
            return;
        }

        _logger.LogInformation("Configuration validation completed successfully");
    }

    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

// 3.3: Enhanced configuration builder
public static class ConfigurationBuilderExtensions
{
    public static IConfigurationBuilder AddMotoVaultConfiguration(
        this IConfigurationBuilder builder, 
        IWebHostEnvironment environment)
    {
        // Base configuration files
        builder.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
        builder.AddJsonFile($"appsettings.{environment.EnvironmentName}.json", 
            optional: true, reloadOnChange: true);

        // Environment variables with prefix
        builder.AddEnvironmentVariables("MOTOVAULT_");
        
        // Standard environment variables for compatibility
        builder.AddEnvironmentVariables();

        return builder;
    }
}

// 3.4: Register services
builder.Services.AddSingleton<ConfigurationValidationService>();
builder.Services.AddHostedService<StartupConfigurationValidator>();

Testing Plan

Automated Tests:

[Test]
public void ConfigurationValidation_ValidConfiguration_ReturnsSuccess()
{
    var configuration = CreateTestConfiguration(new Dictionary<string, string>
    {
        ["ConnectionStrings:DefaultConnection"] = "Host=localhost;Database=test;Username=test;Password=test",
        ["App:Name"] = "MotoVaultPro",
        ["Logging:LogLevel:Default"] = "Information"
    });

    var validator = new ConfigurationValidationService(configuration, Mock.Of<ILogger<ConfigurationValidationService>>());
    var result = validator.ValidateConfiguration();

    Assert.IsFalse(result.HasErrors);
    Assert.AreEqual(0, result.Errors.Count);
}

[Test]
public void ConfigurationValidation_MissingDatabase_ReturnsError()
{
    var configuration = CreateTestConfiguration(new Dictionary<string, string>
    {
        ["App:Name"] = "MotoVaultPro"
    });

    var validator = new ConfigurationValidationService(configuration, Mock.Of<ILogger<ConfigurationValidationService>>());
    var result = validator.ValidateConfiguration();

    Assert.IsTrue(result.HasErrors);
    Assert.Contains(result.Errors, e => e.Category == "Database");
}

[Test]
public async Task StartupValidator_InvalidConfiguration_StopsApplication()
{
    var mockLifetime = new Mock<IHostApplicationLifetime>();
    var validator = CreateStartupValidator(invalidConfig: true, mockLifetime.Object);

    await validator.StartAsync(CancellationToken.None);

    mockLifetime.Verify(x => x.StopApplication(), Times.Once);
}

Manual Validation:

1. Start application with valid configuration - should start normally
2. Start with missing database configuration - should fail with clear error
3. Start with invalid PostgreSQL connection string - should fail
4. Test environment variable override of JSON configuration
5. Verify configuration warnings are logged but don't stop startup

Success Criteria:

• Configuration validation runs at startup
• Invalid configuration prevents application startup
• Clear error messages for configuration issues
• Environment variables properly override JSON settings
• Existing functionality unchanged

---

Step 4: Configuration Externalization

Duration: 3-4 days
Risk Level: Medium
Rollback Complexity: Moderate

Objective

Externalize configuration to support Kubernetes ConfigMaps and Secrets while maintaining compatibility.

Implementation

// 4.1: Create Kubernetes configuration extensions
public static class KubernetesConfigurationExtensions
{
    public static IConfigurationBuilder AddKubernetesConfiguration(
        this IConfigurationBuilder builder)
    {
        // Check if running in Kubernetes
        var kubernetesServiceHost = Environment.GetEnvironmentVariable("KUBERNETES_SERVICE_HOST");
        if (!string.IsNullOrEmpty(kubernetesServiceHost))
        {
            builder.AddKubernetesSecrets();
            builder.AddKubernetesConfigMaps();
        }

        return builder;
    }

    private static IConfigurationBuilder AddKubernetesSecrets(this IConfigurationBuilder builder)
    {
        var secretsPath = "/var/secrets";
        if (Directory.Exists(secretsPath))
        {
            foreach (var secretFile in Directory.GetFiles(secretsPath))
            {
                var key = Path.GetFileName(secretFile);
                var value = File.ReadAllText(secretFile);
                builder.AddInMemoryCollection(new[] { new KeyValuePair<string, string>(key, value) });
            }
        }
        return builder;
    }

    private static IConfigurationBuilder AddKubernetesConfigMaps(this IConfigurationBuilder builder)
    {
        var configPath = "/var/config";
        if (Directory.Exists(configPath))
        {
            foreach (var configFile in Directory.GetFiles(configPath))
            {
                var key = Path.GetFileName(configFile);
                var value = File.ReadAllText(configFile);
                builder.AddInMemoryCollection(new[] { new KeyValuePair<string, string>(key, value) });
            }
        }
        return builder;
    }
}

// 4.2: Create configuration mapping service
public class ConfigurationMappingService
{
    private readonly IConfiguration _configuration;

    public DatabaseConfiguration GetDatabaseConfiguration()
    {
        return new DatabaseConfiguration
        {
            PostgreSQLConnectionString = GetConnectionString("POSTGRES_CONNECTION", "ConnectionStrings:DefaultConnection"),
            LiteDBPath = GetConfigValue("LITEDB_PATH", "LiteDB:DatabasePath"),
            CommandTimeout = GetConfigValue<int>("DB_COMMAND_TIMEOUT", "Database:CommandTimeout", 30),
            MaxPoolSize = GetConfigValue<int>("DB_MAX_POOL_SIZE", "Database:MaxPoolSize", 100),
            MinPoolSize = GetConfigValue<int>("DB_MIN_POOL_SIZE", "Database:MinPoolSize", 10)
        };
    }

    public ApplicationConfiguration GetApplicationConfiguration()
    {
        return new ApplicationConfiguration
        {
            Name = GetConfigValue("APP_NAME", "App:Name", "MotoVaultPro"),
            Environment = GetConfigValue("ASPNETCORE_ENVIRONMENT", "App:Environment", "Production"),
            LogLevel = GetConfigValue("LOG_LEVEL", "Logging:LogLevel:Default", "Information"),
            EnableFeatures = GetConfigValue("ENABLE_FEATURES", "App:EnableFeatures", "").Split(','),
            CacheExpiryMinutes = GetConfigValue<int>("CACHE_EXPIRY_MINUTES", "App:CacheExpiryMinutes", 30)
        };
    }

    public EmailConfiguration GetEmailConfiguration()
    {
        return new EmailConfiguration
        {
            Enabled = GetConfigValue<bool>("EMAIL_ENABLED", "Email:Enabled", false),
            SmtpServer = GetConfigValue("EMAIL_SMTP_SERVER", "Email:SmtpServer"),
            SmtpPort = GetConfigValue<int>("EMAIL_SMTP_PORT", "Email:SmtpPort", 587),
            Username = GetConfigValue("EMAIL_USERNAME", "Email:Username"),
            Password = GetConfigValue("EMAIL_PASSWORD", "Email:Password"),
            FromAddress = GetConfigValue("EMAIL_FROM_ADDRESS", "Email:FromAddress"),
            EnableSsl = GetConfigValue<bool>("EMAIL_ENABLE_SSL", "Email:EnableSsl", true)
        };
    }

    private string GetConnectionString(string envKey, string configKey)
    {
        return _configuration[envKey] ?? _configuration.GetConnectionString(configKey);
    }

    private string GetConfigValue(string envKey, string configKey, string defaultValue = null)
    {
        return _configuration[envKey] ?? _configuration[configKey] ?? defaultValue;
    }

    private T GetConfigValue<T>(string envKey, string configKey, T defaultValue = default)
    {
        var value = _configuration[envKey] ?? _configuration[configKey];
        if (string.IsNullOrEmpty(value))
            return defaultValue;

        return (T)Convert.ChangeType(value, typeof(T));
    }
}

// 4.3: Create configuration models
public class DatabaseConfiguration
{
    public string PostgreSQLConnectionString { get; set; }
    public string LiteDBPath { get; set; }
    public int CommandTimeout { get; set; }
    public int MaxPoolSize { get; set; }
    public int MinPoolSize { get; set; }
}

public class ApplicationConfiguration
{
    public string Name { get; set; }
    public string Environment { get; set; }
    public string LogLevel { get; set; }
    public string[] EnableFeatures { get; set; }
    public int CacheExpiryMinutes { get; set; }
}

public class EmailConfiguration
{
    public bool Enabled { get; set; }
    public string SmtpServer { get; set; }
    public int SmtpPort { get; set; }
    public string Username { get; set; }
    public string Password { get; set; }
    public string FromAddress { get; set; }
    public bool EnableSsl { get; set; }
}

// 4.4: Update Program.cs configuration
var builder = WebApplication.CreateBuilder(args);

// Enhanced configuration setup
builder.Configuration
    .AddMotoVaultConfiguration(builder.Environment)
    .AddKubernetesConfiguration();

// Register configuration services
builder.Services.AddSingleton<ConfigurationMappingService>();
builder.Services.Configure<DatabaseConfiguration>(config => 
    config = builder.Services.GetRequiredService<ConfigurationMappingService>().GetDatabaseConfiguration());

Testing Plan

Automated Tests:

[Test]
public void ConfigurationMapping_EnvironmentVariableOverride_TakesPrecedence()
{
    Environment.SetEnvironmentVariable("APP_NAME", "TestApp");
    var configuration = CreateTestConfiguration(new Dictionary<string, string>
    {
        ["App:Name"] = "ConfigApp"
    });

    var mapper = new ConfigurationMappingService(configuration);
    var appConfig = mapper.GetApplicationConfiguration();

    Assert.AreEqual("TestApp", appConfig.Name);
    
    Environment.SetEnvironmentVariable("APP_NAME", null); // Cleanup
}

[Test]
public void KubernetesConfiguration_SecretsPath_LoadsSecrets()
{
    // Create temporary secrets directory
    var secretsPath = Path.Combine(Path.GetTempPath(), "secrets");
    Directory.CreateDirectory(secretsPath);
    File.WriteAllText(Path.Combine(secretsPath, "POSTGRES_CONNECTION"), "test-connection-string");

    try
    {
        Environment.SetEnvironmentVariable("KUBERNETES_SERVICE_HOST", "localhost");
        var builder = new ConfigurationBuilder();
        builder.AddKubernetesConfiguration();
        var configuration = builder.Build();

        Assert.AreEqual("test-connection-string", configuration["POSTGRES_CONNECTION"]);
    }
    finally
    {
        Directory.Delete(secretsPath, true);
        Environment.SetEnvironmentVariable("KUBERNETES_SERVICE_HOST", null);
    }
}

[Test]
public async Task Application_StartupWithExternalizedConfig_Succeeds()
{
    var factory = new WebApplicationFactory<Program>()
        .WithWebHostBuilder(builder =>
        {
            builder.UseEnvironment("Testing");
            builder.ConfigureAppConfiguration((context, config) =>
            {
                config.AddInMemoryCollection(new[]
                {
                    new KeyValuePair<string, string>("POSTGRES_CONNECTION", "Host=localhost;Database=test;Username=test;Password=test"),
                    new KeyValuePair<string, string>("APP_NAME", "TestApp")
                });
            });
        });

    var client = factory.CreateClient();
    var response = await client.GetAsync("/health/ready");
    
    Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
}

Kubernetes Manifests for Testing:

# test-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: motovault-config-test
data:
  APP_NAME: "MotoVaultPro"
  LOG_LEVEL: "Information"
  CACHE_EXPIRY_MINUTES: "30"
  ENABLE_FEATURES: "OpenIDConnect,EmailNotifications"

---
# test-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: motovault-secrets-test
type: Opaque
data:
  POSTGRES_CONNECTION: <base64-encoded-connection-string>
  EMAIL_PASSWORD: <base64-encoded-password>
  JWT_SECRET: <base64-encoded-jwt-secret>

---
# test-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: motovault-config-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: motovault-test
  template:
    spec:
      containers:
      - name: motovault
        image: motovault:test
        envFrom:
        - configMapRef:
            name: motovault-config-test
        - secretRef:
            name: motovault-secrets-test
        volumeMounts:
        - name: config-volume
          mountPath: /var/config
        - name: secrets-volume
          mountPath: /var/secrets
      volumes:
      - name: config-volume
        configMap:
          name: motovault-config-test
      - name: secrets-volume
        secret:
          secretName: motovault-secrets-test

Manual Validation:

1. Test with environment variables only - application should start
2. Test with JSON configuration only - application should start
3. Test with Kubernetes ConfigMap/Secret simulation - application should start
4. Verify environment variables override JSON configuration
5. Test configuration validation with externalized config
6. Deploy to test Kubernetes environment and verify functionality

Success Criteria:

• Application starts with environment variables only
• Kubernetes ConfigMap/Secret integration works
• Environment variables override JSON configuration
• Configuration validation works with externalized config
• All existing functionality preserved
• No hardcoded configuration remains in code

---

Step 5: PostgreSQL Connection Optimization

Duration: 2-3 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Optimize PostgreSQL connections for high availability and performance without affecting LiteDB functionality.

Implementation

// 5.1: Enhanced PostgreSQL configuration
public class PostgreSQLConnectionService
{
    private readonly DatabaseConfiguration _config;
    private readonly ILogger<PostgreSQLConnectionService> _logger;
    private readonly IHostEnvironment _environment;

    public NpgsqlConnectionStringBuilder CreateOptimizedConnectionString()
    {
        var builder = new NpgsqlConnectionStringBuilder(_config.PostgreSQLConnectionString);
        
        // Connection pooling optimization
        builder.MaxPoolSize = _config.MaxPoolSize;
        builder.MinPoolSize = _config.MinPoolSize;
        builder.ConnectionLifetime = 300; // 5 minutes
        builder.ConnectionIdleLifetime = 300; // 5 minutes
        builder.ConnectionPruningInterval = 10; // 10 seconds
        
        // Performance optimization
        builder.CommandTimeout = _config.CommandTimeout;
        builder.NoResetOnClose = true;
        builder.Enlist = false; // Disable distributed transactions for performance
        
        // Reliability settings
        builder.KeepAlive = 30; // 30 seconds
        builder.TcpKeepAliveTime = 30;
        builder.TcpKeepAliveInterval = 5;
        
        // Application name for monitoring
        builder.ApplicationName = $"{_config.ApplicationName}-{_environment.EnvironmentName}";
        
        _logger.LogInformation("PostgreSQL connection configured: Pool({MinPoolSize}-{MaxPoolSize}), Timeout({CommandTimeout}s)", 
            builder.MinPoolSize, builder.MaxPoolSize, builder.CommandTimeout);
            
        return builder;
    }
    
    public async Task<bool> TestConnectionAsync(CancellationToken cancellationToken = default)
    {
        try
        {
            var connectionString = CreateOptimizedConnectionString().ConnectionString;
            using var connection = new NpgsqlConnection(connectionString);
            
            await connection.OpenAsync(cancellationToken);
            
            using var command = new NpgsqlCommand("SELECT version()", connection);
            var version = await command.ExecuteScalarAsync(cancellationToken);
            
            _logger.LogInformation("PostgreSQL connection test successful. Version: {Version}", version);
            return true;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "PostgreSQL connection test failed");
            return false;
        }
    }
}

// 5.2: Enhanced database context configuration
public static class DatabaseServiceExtensions
{
    public static IServiceCollection AddOptimizedDatabase(
        this IServiceCollection services, 
        DatabaseConfiguration config)
    {
        if (!string.IsNullOrEmpty(config.PostgreSQLConnectionString))
        {
            services.AddOptimizedPostgreSQL(config);
        }
        else if (!string.IsNullOrEmpty(config.LiteDBPath))
        {
            services.AddLiteDB(config);
        }
        else
        {
            throw new InvalidOperationException("No database configuration provided");
        }

        return services;
    }

    private static IServiceCollection AddOptimizedPostgreSQL(
        this IServiceCollection services, 
        DatabaseConfiguration config)
    {
        services.AddSingleton<PostgreSQLConnectionService>();
        
        services.AddDbContextFactory<MotoVaultContext>((serviceProvider, options) =>
        {
            var connectionService = serviceProvider.GetRequiredService<PostgreSQLConnectionService>();
            var connectionString = connectionService.CreateOptimizedConnectionString().ConnectionString;
            
            options.UseNpgsql(connectionString, npgsqlOptions =>
            {
                npgsqlOptions.EnableRetryOnFailure(
                    maxRetryCount: 3,
                    maxRetryDelay: TimeSpan.FromSeconds(5),
                    errorCodesToAdd: null);
                    
                npgsqlOptions.CommandTimeout(config.CommandTimeout);
                npgsqlOptions.MigrationsAssembly(typeof(MotoVaultContext).Assembly.FullName);
            });
            
            // Performance optimizations
            options.EnableSensitiveDataLogging(false);
            options.EnableServiceProviderCaching();
            options.EnableDetailedErrors(false);
            
        }, ServiceLifetime.Singleton);

        // Register data access implementations
        services.AddScoped<IVehicleDataAccess, PostgreSQL.VehicleDataAccess>();
        services.AddScoped<IServiceRecordDataAccess, PostgreSQL.ServiceRecordDataAccess>();
        services.AddScoped<IGasRecordDataAccess, PostgreSQL.GasRecordDataAccess>();
        services.AddScoped<IUserRecordDataAccess, PostgreSQL.UserRecordDataAccess>();

        return services;
    }

    private static IServiceCollection AddLiteDB(
        this IServiceCollection services, 
        DatabaseConfiguration config)
    {
        // Keep existing LiteDB configuration unchanged
        services.AddSingleton<ILiteDatabase>(provider =>
        {
            var connectionString = $"Filename={config.LiteDBPath};Connection=shared";
            return new LiteDatabase(connectionString);
        });

        // Register LiteDB data access implementations
        services.AddScoped<IVehicleDataAccess, LiteDB.VehicleDataAccess>();
        services.AddScoped<IServiceRecordDataAccess, LiteDB.ServiceRecordDataAccess>();
        services.AddScoped<IGasRecordDataAccess, LiteDB.GasRecordDataAccess>();
        services.AddScoped<IUserRecordDataAccess, LiteDB.UserRecordDataAccess>();

        return services;
    }
}

// 5.3: Connection monitoring service
public class DatabaseConnectionMonitoringService : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<DatabaseConnectionMonitoringService> _logger;
    private readonly Counter _connectionAttempts;
    private readonly Counter _connectionFailures;
    private readonly Gauge _activeConnections;

    public DatabaseConnectionMonitoringService(IServiceProvider serviceProvider, ILogger<DatabaseConnectionMonitoringService> logger)
    {
        _serviceProvider = serviceProvider;
        _logger = logger;
        
        _connectionAttempts = Metrics.CreateCounter(
            "motovault_db_connection_attempts_total",
            "Total database connection attempts");
            
        _connectionFailures = Metrics.CreateCounter(
            "motovault_db_connection_failures_total", 
            "Total database connection failures");
            
        _activeConnections = Metrics.CreateGauge(
            "motovault_db_active_connections",
            "Number of active database connections");
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await MonitorConnections();
            await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
        }
    }

    private async Task MonitorConnections()
    {
        try
        {
            using var scope = _serviceProvider.CreateScope();
            var connectionService = scope.ServiceProvider.GetService<PostgreSQLConnectionService>();
            
            if (connectionService != null)
            {
                _connectionAttempts.Inc();
                
                var isHealthy = await connectionService.TestConnectionAsync();
                if (!isHealthy)
                {
                    _connectionFailures.Inc();
                    _logger.LogWarning("Database connection health check failed");
                }
                
                // Monitor connection pool if available
                await MonitorConnectionPool();
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error monitoring database connections");
            _connectionFailures.Inc();
        }
    }

    private async Task MonitorConnectionPool()
    {
        // This would require access to Npgsql connection pool metrics
        // For now, we'll implement a basic check
        try
        {
            using var scope = _serviceProvider.CreateScope();
            var contextFactory = scope.ServiceProvider.GetService<IDbContextFactory<MotoVaultContext>>();
            
            if (contextFactory != null)
            {
                using var context = contextFactory.CreateDbContext();
                var connectionState = context.Database.GetDbConnection().State;
                
                _logger.LogDebug("Database connection state: {State}", connectionState);
            }
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex, "Failed to monitor connection pool");
        }
    }
}

// 5.4: Register services in Program.cs
var databaseConfig = builder.Services.GetRequiredService<ConfigurationMappingService>()
    .GetDatabaseConfiguration();

builder.Services.AddOptimizedDatabase(databaseConfig);
builder.Services.AddHostedService<DatabaseConnectionMonitoringService>();

Testing Plan

Automated Tests:

[Test]
public void PostgreSQLConnectionService_CreatesOptimizedConnectionString()
{
    var config = new DatabaseConfiguration
    {
        PostgreSQLConnectionString = "Host=localhost;Database=test;Username=test;Password=test",
        MaxPoolSize = 50,
        MinPoolSize = 5,
        CommandTimeout = 30
    };

    var service = new PostgreSQLConnectionService(config, Mock.Of<ILogger<PostgreSQLConnectionService>>(), Mock.Of<IHostEnvironment>());
    var builder = service.CreateOptimizedConnectionString();

    Assert.AreEqual(50, builder.MaxPoolSize);
    Assert.AreEqual(5, builder.MinPoolSize);
    Assert.AreEqual(30, builder.CommandTimeout);
    Assert.AreEqual(300, builder.ConnectionLifetime);
}

[Test]
public async Task PostgreSQLConnectionService_TestConnection_ValidConnection_ReturnsTrue()
{
    var config = CreateValidDatabaseConfiguration();
    var service = new PostgreSQLConnectionService(config, Mock.Of<ILogger<PostgreSQLConnectionService>>(), Mock.Of<IHostEnvironment>());

    var result = await service.TestConnectionAsync();

    Assert.IsTrue(result);
}

[Test]
public async Task DatabaseServiceExtensions_PostgreSQLConfiguration_RegistersCorrectServices()
{
    var services = new ServiceCollection();
    var config = new DatabaseConfiguration
    {
        PostgreSQLConnectionString = "Host=localhost;Database=test;Username=test;Password=test"
    };

    services.AddOptimizedDatabase(config);

    var serviceProvider = services.BuildServiceProvider();
    
    Assert.IsNotNull(serviceProvider.GetService<IDbContextFactory<MotoVaultContext>>());
    Assert.IsNotNull(serviceProvider.GetService<PostgreSQLConnectionService>());
    Assert.IsInstanceOf<PostgreSQL.VehicleDataAccess>(serviceProvider.GetService<IVehicleDataAccess>());
}

[Test]
public async Task DatabaseServiceExtensions_LiteDBConfiguration_RegistersCorrectServices()
{
    var services = new ServiceCollection();
    var config = new DatabaseConfiguration
    {
        LiteDBPath = ":memory:"
    };

    services.AddOptimizedDatabase(config);

    var serviceProvider = services.BuildServiceProvider();
    
    Assert.IsNotNull(serviceProvider.GetService<ILiteDatabase>());
    Assert.IsInstanceOf<LiteDB.VehicleDataAccess>(serviceProvider.GetService<IVehicleDataAccess>());
}

Performance Tests:

[Test]
public async Task PostgreSQLConnection_ConcurrentConnections_HandlesLoad()
{
    var config = CreateValidDatabaseConfiguration();
    var service = new PostgreSQLConnectionService(config, Mock.Of<ILogger<PostgreSQLConnectionService>>(), Mock.Of<IHostEnvironment>());
    
    var tasks = Enumerable.Range(0, 20).Select(async i =>
    {
        var stopwatch = Stopwatch.StartNew();
        var result = await service.TestConnectionAsync();
        stopwatch.Stop();
        
        return new { Success = result, Duration = stopwatch.ElapsedMilliseconds };
    });

    var results = await Task.WhenAll(tasks);

    Assert.IsTrue(results.All(r => r.Success));
    Assert.IsTrue(results.All(r => r.Duration < 1000)); // All connections under 1 second
}

[Test]
public async Task DatabaseContext_ConcurrentQueries_OptimalPerformance()
{
    using var factory = CreateDbContextFactory();
    
    var tasks = Enumerable.Range(0, 10).Select(async i =>
    {
        using var context = factory.CreateDbContext();
        var stopwatch = Stopwatch.StartNew();
        
        var count = await context.Vehicles.CountAsync();
        
        stopwatch.Stop();
        return stopwatch.ElapsedMilliseconds;
    });

    var durations = await Task.WhenAll(tasks);
    
    Assert.IsTrue(durations.All(d => d < 500)); // All queries under 500ms
    Assert.IsTrue(durations.Average() < 200);   // Average under 200ms
}

Manual Validation:

1. Test PostgreSQL connection with optimized settings
2. Verify connection pooling behavior under load
3. Test connection recovery after database restart
4. Verify LiteDB functionality remains unchanged
5. Monitor connection metrics during testing
6. Test with both PostgreSQL and LiteDB configurations

Success Criteria:

• PostgreSQL connections use optimized settings
• Connection pooling configured correctly
• Connection monitoring provides metrics
• LiteDB functionality unchanged
• Performance improvement measurable
• Connection recovery works after database restart

---

Step 6: Database Provider Selection and Debugging Infrastructure

Duration: 2-3 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Implement a clean database provider selection mechanism with comprehensive debugging and diagnostic capabilities.

Implementation

// 6.1: Database provider selector
public enum DatabaseProvider
{
    LiteDB,
    PostgreSQL
}

public class DatabaseProviderService
{
    private readonly DatabaseConfiguration _config;
    private readonly ILogger<DatabaseProviderService> _logger;

    public DatabaseProvider GetActiveProvider()
    {
        var hasPostgreSQL = !string.IsNullOrEmpty(_config.PostgreSQLConnectionString);
        var hasLiteDB = !string.IsNullOrEmpty(_config.LiteDBPath);

        if (hasPostgreSQL)
        {
            _logger.LogInformation("PostgreSQL database mode enabled. Connection: {ConnectionInfo}", 
                GetConnectionInfo(_config.PostgreSQLConnectionString));
            return DatabaseProvider.PostgreSQL;
        }
        
        if (hasLiteDB)
        {
            _logger.LogInformation("LiteDB database mode enabled. Path: {LiteDBPath}", _config.LiteDBPath);
            return DatabaseProvider.LiteDB;
        }

        throw new InvalidOperationException("No database provider configured");
    }

    private string GetConnectionInfo(string connectionString)
    {
        try
        {
            var builder = new NpgsqlConnectionStringBuilder(connectionString);
            return $"Host={builder.Host}, Database={builder.Database}, Port={builder.Port}";
        }
        catch
        {
            return "Invalid connection string";
        }
    }
}

// 6.2: Database diagnostics service
public class DatabaseDiagnosticsService
{
    private readonly ILogger<DatabaseDiagnosticsService> _logger;
    private readonly DatabaseConfiguration _config;
    private readonly DatabaseProviderService _providerService;

    public async Task<DatabaseDiagnosticResult> PerformDiagnosticsAsync()
    {
        var result = new DatabaseDiagnosticResult();
        var provider = _providerService.GetActiveProvider();

        _logger.LogInformation("Starting database diagnostics for provider: {Provider}", provider);

        switch (provider)
        {
            case DatabaseProvider.PostgreSQL:
                await DiagnosePostgreSQLAsync(result);
                break;
            case DatabaseProvider.LiteDB:
                await DiagnoseLiteDBAsync(result);
                break;
        }

        _logger.LogInformation("Database diagnostics completed. Status: {Status}, Issues: {IssueCount}", 
            result.OverallStatus, result.Issues.Count);

        return result;
    }

    private async Task DiagnosePostgreSQLAsync(DatabaseDiagnosticResult result)
    {
        result.Provider = "PostgreSQL";
        
        // Test connection string parsing
        try
        {
            var builder = new NpgsqlConnectionStringBuilder(_config.PostgreSQLConnectionString);
            result.ConnectionDetails = new Dictionary<string, object>
            {
                ["Host"] = builder.Host,
                ["Port"] = builder.Port,
                ["Database"] = builder.Database,
                ["Username"] = builder.Username,
                ["MaxPoolSize"] = builder.MaxPoolSize,
                ["MinPoolSize"] = builder.MinPoolSize,
                ["CommandTimeout"] = builder.CommandTimeout
            };
            _logger.LogDebug("PostgreSQL connection string parsed successfully");
        }
        catch (Exception ex)
        {
            result.Issues.Add($"Invalid PostgreSQL connection string: {ex.Message}");
            _logger.LogError(ex, "Failed to parse PostgreSQL connection string");
            result.OverallStatus = "Failed";
            return;
        }

        // Test connectivity
        try
        {
            using var connection = new NpgsqlConnection(_config.PostgreSQLConnectionString);
            var stopwatch = Stopwatch.StartNew();
            await connection.OpenAsync();
            stopwatch.Stop();

            result.ConnectionTime = stopwatch.ElapsedMilliseconds;
            _logger.LogDebug("PostgreSQL connection established in {ElapsedMs}ms", stopwatch.ElapsedMilliseconds);

            // Test basic query
            using var command = new NpgsqlCommand("SELECT version(), current_database(), current_user", connection);
            using var reader = await command.ExecuteReaderAsync();
            
            if (await reader.ReadAsync())
            {
                result.ServerInfo = new Dictionary<string, object>
                {
                    ["Version"] = reader.GetString(0),
                    ["Database"] = reader.GetString(1),
                    ["User"] = reader.GetString(2)
                };
            }

            result.OverallStatus = "Healthy";
            _logger.LogInformation("PostgreSQL diagnostics successful. Version: {Version}", 
                result.ServerInfo?["Version"]);
        }
        catch (Exception ex)
        {
            result.Issues.Add($"PostgreSQL connection failed: {ex.Message}");
            result.OverallStatus = "Failed";
            _logger.LogError(ex, "PostgreSQL connection failed during diagnostics");
        }
    }

    private async Task DiagnoseLiteDBAsync(DatabaseDiagnosticResult result)
    {
        result.Provider = "LiteDB";
        
        try
        {
            var dbPath = _config.LiteDBPath;
            var directory = Path.GetDirectoryName(dbPath);
            
            result.ConnectionDetails = new Dictionary<string, object>
            {
                ["DatabasePath"] = dbPath,
                ["Directory"] = directory,
                ["DirectoryExists"] = Directory.Exists(directory),
                ["FileExists"] = File.Exists(dbPath)
            };

            // Test directory access
            if (!Directory.Exists(directory))
            {
                _logger.LogWarning("LiteDB directory does not exist: {Directory}", directory);
                Directory.CreateDirectory(directory);
                _logger.LogInformation("Created LiteDB directory: {Directory}", directory);
            }

            // Test LiteDB access
            var stopwatch = Stopwatch.StartNew();
            using var db = new LiteDatabase($"Filename={dbPath};Connection=shared");
            var collections = db.GetCollectionNames().ToList();
            stopwatch.Stop();

            result.ConnectionTime = stopwatch.ElapsedMilliseconds;
            result.ServerInfo = new Dictionary<string, object>
            {
                ["Collections"] = collections,
                ["CollectionCount"] = collections.Count,
                ["FileSize"] = File.Exists(dbPath) ? new FileInfo(dbPath).Length : 0
            };

            result.OverallStatus = "Healthy";
            _logger.LogInformation("LiteDB diagnostics successful. Collections: {CollectionCount}, Size: {FileSize} bytes", 
                collections.Count, result.ServerInfo["FileSize"]);
        }
        catch (Exception ex)
        {
            result.Issues.Add($"LiteDB access failed: {ex.Message}");
            result.OverallStatus = "Failed";
            _logger.LogError(ex, "LiteDB access failed during diagnostics");
        }
    }
}

// 6.3: Database diagnostic result model
public class DatabaseDiagnosticResult
{
    public string Provider { get; set; }
    public string OverallStatus { get; set; } = "Unknown";
    public long ConnectionTime { get; set; }
    public Dictionary<string, object> ConnectionDetails { get; set; } = new();
    public Dictionary<string, object> ServerInfo { get; set; } = new();
    public List<string> Issues { get; set; } = new();
    public List<string> Recommendations { get; set; } = new();
}

// 6.4: Database startup diagnostics service
public class DatabaseStartupDiagnosticsService : IHostedService
{
    private readonly DatabaseDiagnosticsService _diagnostics;
    private readonly ILogger<DatabaseStartupDiagnosticsService> _logger;

    public DatabaseStartupDiagnosticsService(
        DatabaseDiagnosticsService diagnostics,
        ILogger<DatabaseStartupDiagnosticsService> logger)
    {
        _diagnostics = diagnostics;
        _logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        try
        {
            _logger.LogInformation("Running database diagnostics at startup");
            var result = await _diagnostics.PerformDiagnosticsAsync();
            
            _logger.LogInformation("Database diagnostics completed. Provider: {Provider}, Status: {Status}, ConnectionTime: {ConnectionTime}ms",
                result.Provider, result.OverallStatus, result.ConnectionTime);

            if (result.Issues.Any())
            {
                foreach (var issue in result.Issues)
                {
                    _logger.LogWarning("Database diagnostic issue: {Issue}", issue);
                }
            }

            foreach (var detail in result.ConnectionDetails)
            {
                _logger.LogDebug("Database connection detail - {Key}: {Value}", detail.Key, detail.Value);
            }

            foreach (var info in result.ServerInfo)
            {
                _logger.LogInformation("Database server info - {Key}: {Value}", info.Key, info.Value);
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Database startup diagnostics failed");
        }
    }

    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

// 6.5: Enhanced database service registration
public static class DatabaseServiceExtensions
{
    public static IServiceCollection AddDatabaseWithDiagnostics(
        this IServiceCollection services, 
        DatabaseConfiguration config)
    {
        services.AddSingleton<DatabaseProviderService>();
        services.AddSingleton<DatabaseDiagnosticsService>();
        
        var providerService = new DatabaseProviderService(config, null);
        var provider = providerService.GetActiveProvider();

        switch (provider)
        {
            case DatabaseProvider.PostgreSQL:
                services.AddOptimizedPostgreSQL(config);
                break;
                
            case DatabaseProvider.LiteDB:
                services.AddLiteDB(config);
                break;
        }

        return services;
    }
}

Testing Plan

Automated Tests:

[Test]
public void DatabaseProviderService_PostgreSQLConfigured_ReturnsPostgreSQL()
{
    var config = new DatabaseConfiguration
    {
        PostgreSQLConnectionString = "Host=localhost;Database=test;Username=test;Password=test"
    };

    var service = new DatabaseProviderService(config, Mock.Of<ILogger<DatabaseProviderService>>());
    var provider = service.GetActiveProvider();

    Assert.AreEqual(DatabaseProvider.PostgreSQL, provider);
}

[Test]
public void DatabaseProviderService_LiteDBConfigured_ReturnsLiteDB()
{
    var config = new DatabaseConfiguration
    {
        LiteDBPath = "/tmp/test.db"
    };

    var service = new DatabaseProviderService(config, Mock.Of<ILogger<DatabaseProviderService>>());
    var provider = service.GetActiveProvider();

    Assert.AreEqual(DatabaseProvider.LiteDB, provider);
}

[Test]
public void DatabaseProviderService_NoConfiguration_ThrowsException()
{
    var config = new DatabaseConfiguration();

    var service = new DatabaseProviderService(config, Mock.Of<ILogger<DatabaseProviderService>>());
    
    Assert.Throws<InvalidOperationException>(() => service.GetActiveProvider());
}

[Test]
public async Task DatabaseDiagnosticsService_PostgreSQL_ValidConnection_ReturnsHealthy()
{
    var config = new DatabaseConfiguration
    {
        PostgreSQLConnectionString = GetTestPostgreSQLConnectionString()
    };

    var providerService = new DatabaseProviderService(config, Mock.Of<ILogger<DatabaseProviderService>>());
    var diagnostics = new DatabaseDiagnosticsService(
        Mock.Of<ILogger<DatabaseDiagnosticsService>>(), 
        config, 
        providerService);

    var result = await diagnostics.PerformDiagnosticsAsync();

    Assert.AreEqual("Healthy", result.OverallStatus);
    Assert.AreEqual("PostgreSQL", result.Provider);
    Assert.IsTrue(result.ConnectionTime > 0);
    Assert.IsTrue(result.ServerInfo.ContainsKey("Version"));
}

[Test]
public async Task DatabaseDiagnosticsService_LiteDB_ValidPath_ReturnsHealthy()
{
    var tempPath = Path.GetTempFileName();
    var config = new DatabaseConfiguration
    {
        LiteDBPath = tempPath
    };

    try
    {
        var providerService = new DatabaseProviderService(config, Mock.Of<ILogger<DatabaseProviderService>>());
        var diagnostics = new DatabaseDiagnosticsService(
            Mock.Of<ILogger<DatabaseDiagnosticsService>>(), 
            config, 
            providerService);

        var result = await diagnostics.PerformDiagnosticsAsync();

        Assert.AreEqual("Healthy", result.OverallStatus);
        Assert.AreEqual("LiteDB", result.Provider);
        Assert.IsTrue(result.ConnectionTime >= 0);
    }
    finally
    {
        File.Delete(tempPath);
    }
}

[Test]
public async Task DatabaseStartupDiagnosticsService_RunsAtStartup_LogsResults()
{
    var mockLogger = new Mock<ILogger<DatabaseStartupDiagnosticsService>>();
    var mockDiagnostics = new Mock<DatabaseDiagnosticsService>();
    
    var diagnosticResult = new DatabaseDiagnosticResult
    {
        Provider = "PostgreSQL",
        OverallStatus = "Healthy",
        ConnectionTime = 50
    };
    
    mockDiagnostics.Setup(x => x.PerformDiagnosticsAsync())
        .ReturnsAsync(diagnosticResult);
    
    var service = new DatabaseStartupDiagnosticsService(mockDiagnostics.Object, mockLogger.Object);
    await service.StartAsync(CancellationToken.None);
    
    // Verify that diagnostic information was logged
    mockLogger.Verify(
        x => x.Log(
            LogLevel.Information,
            It.IsAny<EventId>(),
            It.Is<It.IsAnyType>((v, t) => v.ToString().Contains("Database diagnostics completed")),
            It.IsAny<Exception>(),
            It.IsAny<Func<It.IsAnyType, Exception, string>>()),
        Times.Once);
}

Manual Validation:

1. Test database provider selection with PostgreSQL configuration
2. Test database provider selection with LiteDB configuration
3. Review startup logs for diagnostic information
4. Test with invalid PostgreSQL connection string and verify error logging
5. Test with invalid LiteDB path and verify error logging
6. Verify logging output provides comprehensive debugging information

Success Criteria:

• Database provider correctly selected based on configuration
• Comprehensive diagnostic information logged at startup
• Error conditions properly detected and logged
• Logging provides sufficient detail for debugging
• Connection details and server info logged appropriately
• No impact on existing functionality

---

Step 7: Database Migration Preparation and Tooling

Duration: 3-4 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Create comprehensive database migration tools and validation utilities for future PostgreSQL transition.

Implementation

// 7.1: Database migration service
public class DatabaseMigrationService
{
    private readonly ILogger<DatabaseMigrationService> _logger;
    private readonly DatabaseConfiguration _config;

    public async Task<MigrationPreparationResult> PrepareMigrationAsync()
    {
        var result = new MigrationPreparationResult();
        
        _logger.LogInformation("Starting migration preparation analysis");

        // Analyze current data structure
        await AnalyzeDataStructureAsync(result);
        
        // Validate migration prerequisites  
        await ValidateMigrationPrerequisitesAsync(result);
        
        // Generate migration plan
        await GenerateMigrationPlanAsync(result);

        _logger.LogInformation("Migration preparation completed. Status: {Status}", result.Status);
        return result;
    }

    private async Task AnalyzeDataStructureAsync(MigrationPreparationResult result)
    {
        try
        {
            if (!string.IsNullOrEmpty(_config.LiteDBPath) && File.Exists(_config.LiteDBPath))
            {
                using var db = new LiteDatabase(_config.LiteDBPath);
                var collections = db.GetCollectionNames().ToList();
                
                result.DataAnalysis = new Dictionary<string, object>();
                
                foreach (var collectionName in collections)
                {
                    var collection = db.GetCollection(collectionName);
                    var count = collection.Count();
                    
                    result.DataAnalysis[collectionName] = new
                    {
                        RecordCount = count,
                        EstimatedSize = count * 1024 // Rough estimate
                    };
                    
                    _logger.LogDebug("Collection {Collection}: {Count} records", collectionName, count);
                }

                result.TotalRecords = result.DataAnalysis.Values
                    .Cast<dynamic>()
                    .Sum(x => (int)x.RecordCount);
                    
                _logger.LogInformation("Data analysis completed. Total records: {TotalRecords}", result.TotalRecords);
            }
            else
            {
                result.DataAnalysis = new Dictionary<string, object>();
                result.TotalRecords = 0;
                _logger.LogInformation("No LiteDB database found for analysis");
            }
        }
        catch (Exception ex)
        {
            result.Issues.Add($"Data analysis failed: {ex.Message}");
            _logger.LogError(ex, "Failed to analyze data structure");
        }
    }

    private async Task ValidateMigrationPrerequisitesAsync(MigrationPreparationResult result)
    {
        _logger.LogDebug("Validating migration prerequisites");

        // Check PostgreSQL availability if configured
        if (!string.IsNullOrEmpty(_config.PostgreSQLConnectionString))
        {
            try
            {
                using var connection = new NpgsqlConnection(_config.PostgreSQLConnectionString);
                await connection.OpenAsync();
                
                // Check if database is empty or has expected schema
                using var command = new NpgsqlCommand(
                    "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'public'", 
                    connection);
                var tableCount = (long)await command.ExecuteScalarAsync();
                
                result.Prerequisites["PostgreSQLConnectivity"] = true;
                result.Prerequisites["PostgreSQLTableCount"] = tableCount;
                
                if (tableCount > 0)
                {
                    result.Recommendations.Add("PostgreSQL database contains existing tables. Consider backup before migration.");
                }
                
                _logger.LogDebug("PostgreSQL validation successful. Table count: {TableCount}", tableCount);
            }
            catch (Exception ex)
            {
                result.Prerequisites["PostgreSQLConnectivity"] = false;
                result.Issues.Add($"PostgreSQL validation failed: {ex.Message}");
                _logger.LogWarning(ex, "PostgreSQL validation failed");
            }
        }

        // Check disk space
        try
        {
            var currentPath = Environment.CurrentDirectory;
            var drive = new DriveInfo(Path.GetPathRoot(currentPath));
            var freeSpaceGB = drive.AvailableFreeSpace / (1024 * 1024 * 1024);
            
            result.Prerequisites["DiskSpaceGB"] = freeSpaceGB;
            
            if (freeSpaceGB < 1)
            {
                result.Issues.Add("Insufficient disk space for migration (< 1GB available)");
            }
            else if (freeSpaceGB < 5)
            {
                result.Recommendations.Add("Limited disk space available. Monitor during migration.");
            }
            
            _logger.LogDebug("Disk space check: {FreeSpaceGB}GB available", freeSpaceGB);
        }
        catch (Exception ex)
        {
            result.Issues.Add($"Disk space check failed: {ex.Message}");
            _logger.LogWarning(ex, "Failed to check disk space");
        }
    }

    private async Task GenerateMigrationPlanAsync(MigrationPreparationResult result)
    {
        _logger.LogDebug("Generating migration plan");

        var plan = new List<string>();
        
        if (result.TotalRecords > 0)
        {
            plan.Add("1. Create PostgreSQL database schema");
            plan.Add("2. Export data from LiteDB");
            plan.Add("3. Transform data for PostgreSQL compatibility");
            plan.Add("4. Import data to PostgreSQL");
            plan.Add("5. Validate data integrity");
            plan.Add("6. Update configuration to use PostgreSQL");
            plan.Add("7. Test application functionality");
            plan.Add("8. Archive LiteDB data");

            // Estimate migration time based on record count
            var estimatedMinutes = Math.Max(5, result.TotalRecords / 1000); // Rough estimate
            result.EstimatedMigrationTime = TimeSpan.FromMinutes(estimatedMinutes);
            
            plan.Add($"Estimated migration time: {result.EstimatedMigrationTime.TotalMinutes:F0} minutes");
        }
        else
        {
            plan.Add("1. Create PostgreSQL database schema");
            plan.Add("2. Update configuration to use PostgreSQL");
            plan.Add("3. Test application functionality");
            result.EstimatedMigrationTime = TimeSpan.FromMinutes(5);
        }

        result.MigrationPlan = plan;
        _logger.LogInformation("Migration plan generated with {StepCount} steps", plan.Count);
    }
}

// 7.2: Migration result models
public class MigrationPreparationResult
{
    public string Status { get; set; } = "Success";
    public Dictionary<string, object> DataAnalysis { get; set; } = new();
    public Dictionary<string, object> Prerequisites { get; set; } = new();
    public List<string> Issues { get; set; } = new();
    public List<string> Recommendations { get; set; } = new();
    public List<string> MigrationPlan { get; set; } = new();
    public int TotalRecords { get; set; }
    public TimeSpan EstimatedMigrationTime { get; set; }
}

// 7.3: Data validation service
public class DataValidationService
{
    private readonly ILogger<DataValidationService> _logger;

    public async Task<DataValidationResult> ValidateDataIntegrityAsync(DatabaseProvider provider)
    {
        var result = new DataValidationResult { Provider = provider.ToString() };
        
        _logger.LogInformation("Starting data integrity validation for {Provider}", provider);

        switch (provider)
        {
            case DatabaseProvider.LiteDB:
                await ValidateLiteDBIntegrityAsync(result);
                break;
            case DatabaseProvider.PostgreSQL:
                await ValidatePostgreSQLIntegrityAsync(result);
                break;
        }

        _logger.LogInformation("Data validation completed for {Provider}. Status: {Status}, Issues: {IssueCount}", 
            provider, result.Status, result.Issues.Count);

        return result;
    }

    private async Task ValidateLiteDBIntegrityAsync(DataValidationResult result)
    {
        try
        {
            // Implement LiteDB-specific validation logic
            result.ValidationChecks["LiteDBAccessible"] = true;
            result.ValidationChecks["CollectionsAccessible"] = true;
            result.Status = "Healthy";
        }
        catch (Exception ex)
        {
            result.Issues.Add($"LiteDB validation failed: {ex.Message}");
            result.Status = "Failed";
            _logger.LogError(ex, "LiteDB validation failed");
        }
    }

    private async Task ValidatePostgreSQLIntegrityAsync(DataValidationResult result)
    {
        try
        {
            // Implement PostgreSQL-specific validation logic
            result.ValidationChecks["PostgreSQLAccessible"] = true;
            result.ValidationChecks["TablesAccessible"] = true;
            result.Status = "Healthy";
        }
        catch (Exception ex)
        {
            result.Issues.Add($"PostgreSQL validation failed: {ex.Message}");
            result.Status = "Failed";
            _logger.LogError(ex, "PostgreSQL validation failed");
        }
    }
}

public class DataValidationResult
{
    public string Provider { get; set; }
    public string Status { get; set; } = "Unknown";
    public Dictionary<string, bool> ValidationChecks { get; set; } = new();
    public List<string> Issues { get; set; } = new();
    public Dictionary<string, object> Statistics { get; set; } = new();
}

// 7.4: Migration analysis startup service
public class MigrationAnalysisService : IHostedService
{
    private readonly DatabaseMigrationService _migrationService;
    private readonly DataValidationService _validationService;
    private readonly DatabaseProviderService _providerService;
    private readonly FeatureFlagService _featureFlags;
    private readonly ILogger<MigrationAnalysisService> _logger;

    public MigrationAnalysisService(
        DatabaseMigrationService migrationService,
        DataValidationService validationService,
        DatabaseProviderService providerService,
        FeatureFlagService featureFlags,
        ILogger<MigrationAnalysisService> logger)
    {
        _migrationService = migrationService;
        _validationService = validationService;
        _providerService = providerService;
        _featureFlags = featureFlags;
        _logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        if (!_featureFlags.IsEnabled("MigrationTools", true))
        {
            _logger.LogDebug("Migration tools feature is disabled, skipping migration analysis");
            return;
        }

        try
        {
            var currentProvider = _providerService.GetActiveProvider();
            _logger.LogInformation("Running migration analysis for current provider: {Provider}", currentProvider);

            var migrationResult = await _migrationService.PrepareMigrationAsync();
            _logger.LogInformation("Migration analysis completed. Status: {Status}, Total Records: {TotalRecords}, Estimated Time: {EstimatedTime}",
                migrationResult.Status, migrationResult.TotalRecords, migrationResult.EstimatedMigrationTime);

            if (migrationResult.Issues.Any())
            {
                foreach (var issue in migrationResult.Issues)
                {
                    _logger.LogWarning("Migration analysis issue: {Issue}", issue);
                }
            }

            if (migrationResult.Recommendations.Any())
            {
                foreach (var recommendation in migrationResult.Recommendations)
                {
                    _logger.LogInformation("Migration recommendation: {Recommendation}", recommendation);
                }
            }

            foreach (var step in migrationResult.MigrationPlan)
            {
                _logger.LogDebug("Migration plan step: {Step}", step);
            }

            var validationResult = await _validationService.ValidateDataIntegrityAsync(currentProvider);
            _logger.LogInformation("Data validation completed for {Provider}. Status: {Status}",
                validationResult.Provider, validationResult.Status);

            if (validationResult.Issues.Any())
            {
                foreach (var issue in validationResult.Issues)
                {
                    _logger.LogWarning("Data validation issue: {Issue}", issue);
                }
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Migration analysis failed during startup");
        }
    }

    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

Testing Plan

Automated Tests:

[Test]
public async Task DatabaseMigrationService_PrepareMigration_GeneratesValidPlan()
{
    var config = new DatabaseConfiguration
    {
        LiteDBPath = CreateTestLiteDBWithData(),
        PostgreSQLConnectionString = GetTestPostgreSQLConnectionString()
    };

    var migrationService = new DatabaseMigrationService(
        Mock.Of<ILogger<DatabaseMigrationService>>(), 
        config);

    var result = await migrationService.PrepareMigrationAsync();

    Assert.AreEqual("Success", result.Status);
    Assert.IsTrue(result.MigrationPlan.Count > 0);
    Assert.IsTrue(result.TotalRecords >= 0);
    Assert.IsTrue(result.EstimatedMigrationTime > TimeSpan.Zero);
}

[Test]
public async Task DataValidationService_LiteDB_ReturnsValidationResult()
{
    var validationService = new DataValidationService(Mock.Of<ILogger<DataValidationService>>());
    
    var result = await validationService.ValidateDataIntegrityAsync(DatabaseProvider.LiteDB);

    Assert.AreEqual("LiteDB", result.Provider);
    Assert.IsNotNull(result.Status);
    Assert.IsNotNull(result.ValidationChecks);
}

[Test]
public async Task MigrationAnalysisService_RunsAtStartup_LogsAnalysis()
{
    var mockLogger = new Mock<ILogger<MigrationAnalysisService>>();
    var mockMigrationService = new Mock<DatabaseMigrationService>();
    var mockValidationService = new Mock<DataValidationService>();
    var mockProviderService = new Mock<DatabaseProviderService>();
    var mockFeatureFlags = new Mock<FeatureFlagService>();
    
    mockFeatureFlags.Setup(x => x.IsEnabled("MigrationTools", true)).Returns(true);
    mockProviderService.Setup(x => x.GetActiveProvider()).Returns(DatabaseProvider.LiteDB);
    
    var migrationResult = new MigrationPreparationResult { Status = "Success", TotalRecords = 100 };
    mockMigrationService.Setup(x => x.PrepareMigrationAsync()).ReturnsAsync(migrationResult);
    
    var validationResult = new DataValidationResult { Provider = "LiteDB", Status = "Healthy" };
    mockValidationService.Setup(x => x.ValidateDataIntegrityAsync(It.IsAny<DatabaseProvider>()))
        .ReturnsAsync(validationResult);
    
    var service = new MigrationAnalysisService(
        mockMigrationService.Object,
        mockValidationService.Object,
        mockProviderService.Object,
        mockFeatureFlags.Object,
        mockLogger.Object);
    
    await service.StartAsync(CancellationToken.None);
    
    // Verify migration analysis was logged
    mockLogger.Verify(
        x => x.Log(
            LogLevel.Information,
            It.IsAny<EventId>(),
            It.Is<It.IsAnyType>((v, t) => v.ToString().Contains("Migration analysis completed")),
            It.IsAny<Exception>(),
            It.IsAny<Func<It.IsAnyType, Exception, string>>()),
        Times.Once);
}

Manual Validation:

1. Review startup logs for migration analysis information
2. Test migration preparation with existing LiteDB data
3. Test migration preparation with empty database
4. Verify PostgreSQL connectivity validation in logs
5. Test with invalid PostgreSQL configuration and check error logs
6. Verify migration plan generation logic through log output

Success Criteria:

• Migration preparation analysis works correctly
• Data structure analysis provides accurate information and logs details
• Migration plan generated with realistic time estimates
• Prerequisites validation identifies potential issues and logs them
• Comprehensive migration information logged at startup
• No impact on existing application functionality

---

Step 8: Performance Monitoring and Benchmarking

Duration: 2-3 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Implement comprehensive performance monitoring and benchmarking to establish baselines and detect regressions.

Implementation

// 8.1: Performance monitoring service
public class PerformanceMonitoringService
{
    private readonly ILogger<PerformanceMonitoringService> _logger;
    private readonly Counter _requestCounter;
    private readonly Histogram _requestDuration;
    private readonly Histogram _databaseOperationDuration;
    private readonly Gauge _activeConnections;

    public PerformanceMonitoringService(ILogger<PerformanceMonitoringService> logger)
    {
        _logger = logger;
        
        _requestCounter = Metrics.CreateCounter(
            "motovault_http_requests_total",
            "Total HTTP requests",
            new[] { "method", "endpoint", "status_code" });

        _requestDuration = Metrics.CreateHistogram(
            "motovault_http_request_duration_seconds",
            "HTTP request duration in seconds",
            new[] { "method", "endpoint" });

        _databaseOperationDuration = Metrics.CreateHistogram(
            "motovault_database_operation_duration_seconds",
            "Database operation duration in seconds",
            new[] { "operation", "provider" });

        _activeConnections = Metrics.CreateGauge(
            "motovault_active_connections",
            "Number of active connections");
    }

    public void RecordHttpRequest(string method, string endpoint, int statusCode, double durationSeconds)
    {
        _requestCounter.WithLabels(method, endpoint, statusCode.ToString()).Inc();
        _requestDuration.WithLabels(method, endpoint).Observe(durationSeconds);
    }

    public void RecordDatabaseOperation(string operation, string provider, double durationSeconds)
    {
        _databaseOperationDuration.WithLabels(operation, provider).Observe(durationSeconds);
    }

    public void SetActiveConnections(double count)
    {
        _activeConnections.Set(count);
    }
}

// 8.2: Performance monitoring middleware
public class PerformanceMonitoringMiddleware
{
    private readonly RequestDelegate _next;
    private readonly PerformanceMonitoringService _monitoring;
    private readonly ILogger<PerformanceMonitoringMiddleware> _logger;

    public async Task InvokeAsync(HttpContext context)
    {
        var stopwatch = Stopwatch.StartNew();
        var endpoint = GetEndpointName(context);

        try
        {
            await _next(context);
        }
        finally
        {
            stopwatch.Stop();
            
            _monitoring.RecordHttpRequest(
                context.Request.Method,
                endpoint,
                context.Response.StatusCode,
                stopwatch.Elapsed.TotalSeconds);

            // Log slow requests
            if (stopwatch.ElapsedMilliseconds > 1000)
            {
                _logger.LogWarning("Slow request detected: {Method} {Endpoint} took {ElapsedMs}ms",
                    context.Request.Method, endpoint, stopwatch.ElapsedMilliseconds);
            }
        }
    }

    private string GetEndpointName(HttpContext context)
    {
        var endpoint = context.GetEndpoint();
        if (endpoint?.DisplayName != null)
        {
            return endpoint.DisplayName;
        }

        var path = context.Request.Path.Value;
        
        // Normalize common patterns
        if (path.StartsWith("/Vehicle/"))
        {
            return "/Vehicle/*";
        }
        if (path.StartsWith("/api/"))
        {
            return "/api/*";
        }

        return path ?? "unknown";
    }
}

// 8.3: Database operation interceptor
public class DatabaseOperationInterceptor : DbCommandInterceptor
{
    private readonly PerformanceMonitoringService _monitoring;
    private readonly ILogger<DatabaseOperationInterceptor> _logger;

    public DatabaseOperationInterceptor(
        PerformanceMonitoringService monitoring,
        ILogger<DatabaseOperationInterceptor> logger)
    {
        _monitoring = monitoring;
        _logger = logger;
    }

    public override ValueTask<InterceptionResult<DbDataReader>> ReaderExecutingAsync(
        DbCommand command,
        CommandEventData eventData,
        InterceptionResult<DbDataReader> result,
        CancellationToken cancellationToken = default)
    {
        var stopwatch = Stopwatch.StartNew();
        eventData.Context.ContextId.ToString(); // Use for correlation
        
        return base.ReaderExecutingAsync(command, eventData, result, cancellationToken);
    }

    public override ValueTask<DbDataReader> ReaderExecutedAsync(
        DbCommand command,
        CommandExecutedEventData eventData,
        DbDataReader result,
        CancellationToken cancellationToken = default)
    {
        var duration = eventData.Duration.TotalSeconds;
        var operation = GetOperationType(command.CommandText);
        var provider = GetProviderName(eventData.Context);

        _monitoring.RecordDatabaseOperation(operation, provider, duration);

        // Log slow queries
        if (eventData.Duration.TotalMilliseconds > 500)
        {
            _logger.LogWarning("Slow database query detected: {Operation} took {ElapsedMs}ms. Query: {CommandText}",
                operation, eventData.Duration.TotalMilliseconds, command.CommandText);
        }

        return base.ReaderExecutedAsync(command, eventData, result, cancellationToken);
    }

    private string GetOperationType(string commandText)
    {
        if (string.IsNullOrEmpty(commandText))
            return "unknown";

        var upperCommand = commandText.Trim().ToUpper();
        
        if (upperCommand.StartsWith("SELECT")) return "SELECT";
        if (upperCommand.StartsWith("INSERT")) return "INSERT";
        if (upperCommand.StartsWith("UPDATE")) return "UPDATE";
        if (upperCommand.StartsWith("DELETE")) return "DELETE";
        
        return "other";
    }

    private string GetProviderName(DbContext context)
    {
        return context?.Database?.ProviderName?.Contains("Npgsql") == true ? "PostgreSQL" : "Unknown";
    }
}

// 8.4: Performance benchmarking service
public class PerformanceBenchmarkService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<PerformanceBenchmarkService> _logger;

    public async Task<BenchmarkResult> RunBenchmarkAsync(BenchmarkOptions options)
    {
        var result = new BenchmarkResult
        {
            TestName = options.TestName,
            StartTime = DateTime.UtcNow
        };

        _logger.LogInformation("Starting benchmark: {TestName}", options.TestName);

        try
        {
            switch (options.TestType)
            {
                case BenchmarkType.DatabaseRead:
                    await BenchmarkDatabaseReads(result, options);
                    break;
                case BenchmarkType.DatabaseWrite:
                    await BenchmarkDatabaseWrites(result, options);
                    break;
                case BenchmarkType.HttpEndpoint:
                    await BenchmarkHttpEndpoint(result, options);
                    break;
            }

            result.Status = "Completed";
        }
        catch (Exception ex)
        {
            result.Status = "Failed";
            result.Error = ex.Message;
            _logger.LogError(ex, "Benchmark failed: {TestName}", options.TestName);
        }
        finally
        {
            result.EndTime = DateTime.UtcNow;
            result.Duration = result.EndTime - result.StartTime;
        }

        _logger.LogInformation("Benchmark completed: {TestName}, Duration: {Duration}ms, Status: {Status}",
            options.TestName, result.Duration.TotalMilliseconds, result.Status);

        return result;
    }

    private async Task BenchmarkDatabaseReads(BenchmarkResult result, BenchmarkOptions options)
    {
        using var scope = _serviceProvider.CreateScope();
        var vehicleAccess = scope.ServiceProvider.GetRequiredService<IVehicleDataAccess>();

        var durations = new List<double>();
        
        for (int i = 0; i < options.Iterations; i++)
        {
            var stopwatch = Stopwatch.StartNew();
            
            try
            {
                var vehicles = await vehicleAccess.GetVehiclesAsync(1); // Test user ID
                stopwatch.Stop();
                durations.Add(stopwatch.Elapsed.TotalMilliseconds);
            }
            catch (Exception ex)
            {
                stopwatch.Stop();
                result.Errors.Add($"Iteration {i}: {ex.Message}");
            }
        }

        if (durations.Count > 0)
        {
            result.Metrics["AverageMs"] = durations.Average();
            result.Metrics["MinMs"] = durations.Min();
            result.Metrics["MaxMs"] = durations.Max();
            result.Metrics["P95Ms"] = durations.OrderBy(x => x).Skip((int)(durations.Count * 0.95)).First();
            result.Metrics["SuccessfulIterations"] = durations.Count;
            result.Metrics["FailedIterations"] = options.Iterations - durations.Count;
        }
    }

    private async Task BenchmarkDatabaseWrites(BenchmarkResult result, BenchmarkOptions options)
    {
        // Similar implementation for write operations
        result.Metrics["WriteOperationsCompleted"] = options.Iterations;
    }

    private async Task BenchmarkHttpEndpoint(BenchmarkResult result, BenchmarkOptions options)
    {
        // HTTP endpoint benchmarking implementation
        result.Metrics["HttpRequestsCompleted"] = options.Iterations;
    }
}

// 8.5: Benchmark models
public class BenchmarkOptions
{
    public string TestName { get; set; }
    public BenchmarkType TestType { get; set; }
    public int Iterations { get; set; } = 10;
    public string TargetEndpoint { get; set; }
    public Dictionary<string, object> Parameters { get; set; } = new();
}

public enum BenchmarkType
{
    DatabaseRead,
    DatabaseWrite,
    HttpEndpoint
}

public class BenchmarkResult
{
    public string TestName { get; set; }
    public string Status { get; set; }
    public DateTime StartTime { get; set; }
    public DateTime EndTime { get; set; }
    public TimeSpan Duration { get; set; }
    public Dictionary<string, double> Metrics { get; set; } = new();
    public List<string> Errors { get; set; } = new();
    public string Error { get; set; }
}

Testing Plan

Automated Tests:

[Test]
public void PerformanceMonitoringService_RecordHttpRequest_UpdatesMetrics()
{
    var monitoring = new PerformanceMonitoringService(Mock.Of<ILogger<PerformanceMonitoringService>>());

    // Record some test requests
    monitoring.RecordHttpRequest("GET", "/test", 200, 0.5);
    monitoring.RecordHttpRequest("POST", "/test", 201, 0.8);

    // Verify metrics are updated (would need to access metrics collector in real implementation)
    Assert.IsTrue(true); // Placeholder - would verify actual metrics
}

[Test]
public async Task PerformanceBenchmarkService_DatabaseReadBenchmark_ReturnsValidResults()
{
    var serviceCollection = new ServiceCollection();
    // Add required services for benchmark
    var serviceProvider = serviceCollection.BuildServiceProvider();

    var benchmarkService = new PerformanceBenchmarkService(serviceProvider, Mock.Of<ILogger<PerformanceBenchmarkService>>());

    var options = new BenchmarkOptions
    {
        TestName = "Database Read Test",
        TestType = BenchmarkType.DatabaseRead,
        Iterations = 5
    };

    var result = await benchmarkService.RunBenchmarkAsync(options);

    Assert.AreEqual("Database Read Test", result.TestName);
    Assert.IsTrue(result.Duration > TimeSpan.Zero);
    Assert.IsNotNull(result.Status);
}

[Test]
public async Task PerformanceMonitoringMiddleware_SlowRequest_LogsWarning()
{
    var mockLogger = new Mock<ILogger<PerformanceMonitoringMiddleware>>();
    var monitoring = Mock.Of<PerformanceMonitoringService>();
    
    var middleware = new PerformanceMonitoringMiddleware(
        async context => await Task.Delay(1100), // Simulate slow request
        monitoring,
        mockLogger.Object);

    var context = new DefaultHttpContext();
    await middleware.InvokeAsync(context);

    // Verify warning was logged for slow request
    mockLogger.Verify(
        x => x.Log(
            LogLevel.Warning,
            It.IsAny<EventId>(),
            It.Is<It.IsAnyType>((v, t) => v.ToString().Contains("Slow request detected")),
            It.IsAny<Exception>(),
            It.IsAny<Func<It.IsAnyType, Exception, string>>()),
        Times.Once);
}

Manual Validation:

1. Run application and verify Prometheus metrics are collected
2. Access /metrics endpoint and verify metric format
3. Perform operations and verify metrics are updated
4. Test performance monitoring middleware with various request types
5. Run database operation benchmarks
6. Verify slow query logging functionality

Success Criteria:

• HTTP request metrics collected accurately
• Database operation metrics recorded
• Slow requests and queries properly logged
• Benchmark service provides realistic performance data
• Prometheus metrics endpoint functional
• Performance overhead < 5% of request time

---

Step 9: Feature Flags and Configuration Controls

Duration: 2-3 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Implement feature flags to safely enable/disable functionality during Phase 1 implementation and future migrations.

Implementation

// 9.1: Feature flag service
public class FeatureFlagService
{
    private readonly IConfiguration _configuration;
    private readonly ILogger<FeatureFlagService> _logger;
    private readonly Dictionary<string, bool> _cachedFlags = new();
    private readonly object _cacheLock = new();

    public FeatureFlagService(IConfiguration configuration, ILogger<FeatureFlagService> logger)
    {
        _configuration = configuration;
        _logger = logger;
    }

    public bool IsEnabled(string flagName, bool defaultValue = false)
    {
        lock (_cacheLock)
        {
            if (_cachedFlags.TryGetValue(flagName, out var cachedValue))
            {
                return cachedValue;
            }

            var value = GetFlagValue(flagName, defaultValue);
            _cachedFlags[flagName] = value;
            
            _logger.LogDebug("Feature flag {FlagName} = {Value}", flagName, value);
            return value;
        }
    }

    private bool GetFlagValue(string flagName, bool defaultValue)
    {
        // Check environment variable first (highest priority)
        var envValue = Environment.GetEnvironmentVariable($"FEATURE_{flagName.ToUpper()}");
        if (!string.IsNullOrEmpty(envValue))
        {
            if (bool.TryParse(envValue, out var envResult))
            {
                _logger.LogDebug("Feature flag {FlagName} set via environment variable: {Value}", flagName, envResult);
                return envResult;
            }
        }

        // Check configuration
        var configKey = $"Features:{flagName}";
        var configValue = _configuration[configKey];
        if (!string.IsNullOrEmpty(configValue))
        {
            if (bool.TryParse(configValue, out var configResult))
            {
                _logger.LogDebug("Feature flag {FlagName} set via configuration: {Value}", flagName, configResult);
                return configResult;
            }
        }

        _logger.LogDebug("Feature flag {FlagName} using default value: {Value}", flagName, defaultValue);
        return defaultValue;
    }

    public void InvalidateCache()
    {
        lock (_cacheLock)
        {
            _cachedFlags.Clear();
            _logger.LogInformation("Feature flag cache invalidated");
        }
    }

    public Dictionary<string, bool> GetAllFlags()
    {
        var flags = new Dictionary<string, bool>();
        
        // Get all known feature flags
        var knownFlags = new[]
        {
            "StructuredLogging",
            "HealthChecks",
            "PerformanceMonitoring",
            "DatabaseDiagnostics",
            "MigrationTools",
            "PostgreSQLOptimizations",
            "ConfigurationValidation",
            "DebugEndpoints"
        };

        foreach (var flag in knownFlags)
        {
            flags[flag] = IsEnabled(flag);
        }

        return flags;
    }
}

// 9.2: Feature flag startup logging service
public class FeatureFlagStartupService : IHostedService
{
    private readonly FeatureFlagService _featureFlags;
    private readonly IWebHostEnvironment _environment;
    private readonly ILogger<FeatureFlagStartupService> _logger;

    public FeatureFlagStartupService(
        FeatureFlagService featureFlags,
        IWebHostEnvironment environment,
        ILogger<FeatureFlagStartupService> logger)
    {
        _featureFlags = featureFlags;
        _environment = environment;
        _logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        try
        {
            var allFlags = _featureFlags.GetAllFlags();
            var enabledFlags = allFlags.Where(kvp => kvp.Value).ToList();
            var disabledFlags = allFlags.Where(kvp => !kvp.Value).ToList();

            _logger.LogInformation("Feature flags initialized. Environment: {Environment}, Total: {Total}, Enabled: {Enabled}, Disabled: {Disabled}",
                _environment.EnvironmentName,
                allFlags.Count,
                enabledFlags.Count,
                disabledFlags.Count);

            foreach (var flag in enabledFlags)
            {
                _logger.LogInformation("Feature flag ENABLED: {FlagName}", flag.Key);
            }

            foreach (var flag in disabledFlags)
            {
                _logger.LogDebug("Feature flag disabled: {FlagName}", flag.Key);
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to log feature flag status at startup");
        }
    }

    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

// 9.3: Feature-aware service registration
public static class FeatureAwareServiceExtensions
{
    public static IServiceCollection AddFeatureAwareServices(this IServiceCollection services, IConfiguration configuration)
    {
        services.AddSingleton<FeatureFlagService>();
        
        var featureFlags = new FeatureFlagService(configuration, null);

        // Register services based on feature flags
        if (featureFlags.IsEnabled("StructuredLogging", true))
        {
            services.AddSingleton<CorrelationIdService>();
        }

        if (featureFlags.IsEnabled("HealthChecks", true))
        {
            services.AddSingleton<DatabaseHealthCheck>();
            services.AddSingleton<ApplicationHealthCheck>();
        }

        if (featureFlags.IsEnabled("PerformanceMonitoring", true))
        {
            services.AddSingleton<PerformanceMonitoringService>();
            services.AddSingleton<PerformanceBenchmarkService>();
        }

        if (featureFlags.IsEnabled("DatabaseDiagnostics", true))
        {
            services.AddSingleton<DatabaseDiagnosticsService>();
        }

        if (featureFlags.IsEnabled("MigrationTools", true))
        {
            services.AddSingleton<DatabaseMigrationService>();
            services.AddSingleton<DataValidationService>();
        }

        return services;
    }
}

// 9.4: Feature flag configuration options
public class FeatureFlagOptions
{
    public const string SectionName = "Features";
    
    public bool StructuredLogging { get; set; } = true;
    public bool HealthChecks { get; set; } = true;
    public bool PerformanceMonitoring { get; set; } = true;
    public bool DatabaseDiagnostics { get; set; } = true;
    public bool MigrationTools { get; set; } = true;
    public bool PostgreSQLOptimizations { get; set; } = true;
    public bool ConfigurationValidation { get; set; } = true;
    public bool DebugEndpoints { get; set; } = false; // Disabled by default in production
}

// 9.5: Feature-gated components
public class FeatureGatedHealthCheckService
{
    private readonly FeatureFlagService _featureFlags;
    private readonly DatabaseHealthCheck _databaseHealthCheck;
    private readonly ApplicationHealthCheck _applicationHealthCheck;

    public async Task<object> GetHealthStatusAsync()
    {
        var healthInfo = new Dictionary<string, object>();

        if (_featureFlags.IsEnabled("HealthChecks"))
        {
            if (_databaseHealthCheck != null)
            {
                var dbHealth = await _databaseHealthCheck.CheckHealthAsync(null);
                healthInfo["Database"] = new
                {
                    Status = dbHealth.Status.ToString(),
                    Description = dbHealth.Description
                };
            }

            if (_applicationHealthCheck != null)
            {
                var appHealth = await _applicationHealthCheck.CheckHealthAsync(null);
                healthInfo["Application"] = new
                {
                    Status = appHealth.Status.ToString(),
                    Description = appHealth.Description
                };
            }
        }
        else
        {
            healthInfo["Message"] = "Health checks are disabled";
        }

        return healthInfo;
    }
}

Testing Plan

Automated Tests:

[Test]
public void FeatureFlagService_EnvironmentVariable_TakesPrecedence()
{
    Environment.SetEnvironmentVariable("FEATURE_TESTFLAG", "true");
    
    var config = new ConfigurationBuilder()
        .AddInMemoryCollection(new[] { new KeyValuePair<string, string>("Features:TestFlag", "false") })
        .Build();

    try
    {
        var service = new FeatureFlagService(config, Mock.Of<ILogger<FeatureFlagService>>());
        var result = service.IsEnabled("TestFlag", false);

        Assert.IsTrue(result); // Environment variable should override config
    }
    finally
    {
        Environment.SetEnvironmentVariable("FEATURE_TESTFLAG", null);
    }
}

[Test]
public void FeatureFlagService_Configuration_UsedWhenNoEnvironmentVariable()
{
    var config = new ConfigurationBuilder()
        .AddInMemoryCollection(new[] { new KeyValuePair<string, string>("Features:TestFlag", "true") })
        .Build();

    var service = new FeatureFlagService(config, Mock.Of<ILogger<FeatureFlagService>>());
    var result = service.IsEnabled("TestFlag", false);

    Assert.IsTrue(result);
}

[Test]
public void FeatureFlagService_DefaultValue_UsedWhenNoConfiguration()
{
    var config = new ConfigurationBuilder().Build();

    var service = new FeatureFlagService(config, Mock.Of<ILogger<FeatureFlagService>>());
    var result = service.IsEnabled("NonExistentFlag", true);

    Assert.IsTrue(result);
}

[Test]
public async Task FeatureFlagStartupService_RunsAtStartup_LogsFeatureFlags()
{
    var mockLogger = new Mock<ILogger<FeatureFlagStartupService>>();
    var mockFeatureFlags = new Mock<FeatureFlagService>();
    var mockEnvironment = new Mock<IWebHostEnvironment>();
    
    mockEnvironment.Setup(x => x.EnvironmentName).Returns("Testing");
    
    var flags = new Dictionary<string, bool>
    {
        ["StructuredLogging"] = true,
        ["HealthChecks"] = true,
        ["PerformanceMonitoring"] = false
    };
    
    mockFeatureFlags.Setup(x => x.GetAllFlags()).Returns(flags);
    
    var service = new FeatureFlagStartupService(
        mockFeatureFlags.Object,
        mockEnvironment.Object,
        mockLogger.Object);
    
    await service.StartAsync(CancellationToken.None);
    
    // Verify feature flags were logged
    mockLogger.Verify(
        x => x.Log(
            LogLevel.Information,
            It.IsAny<EventId>(),
            It.Is<It.IsAnyType>((v, t) => v.ToString().Contains("Feature flags initialized")),
            It.IsAny<Exception>(),
            It.IsAny<Func<It.IsAnyType, Exception, string>>()),
        Times.Once);
}

[Test]
public void FeatureAwareServiceExtensions_RegistersServicesBasedOnFlags()
{
    var config = new ConfigurationBuilder()
        .AddInMemoryCollection(new[]
        {
            new KeyValuePair<string, string>("Features:PerformanceMonitoring", "true"),
            new KeyValuePair<string, string>("Features:DatabaseDiagnostics", "false")
        })
        .Build();

    var services = new ServiceCollection();
    services.AddFeatureAwareServices(config);

    var serviceProvider = services.BuildServiceProvider();

    Assert.IsNotNull(serviceProvider.GetService<PerformanceMonitoringService>());
    Assert.IsNull(serviceProvider.GetService<DatabaseDiagnosticsService>());
}

Manual Validation:

1. Test feature flags via environment variables
2. Test feature flags via configuration file
3. Review startup logs for feature flag status
4. Toggle feature flags and verify services are enabled/disabled
5. Test feature flag cache invalidation
6. Verify feature flags work in different environments and log appropriately

Success Criteria:

• Feature flags correctly control service registration
• Environment variables override configuration values
• Feature flag status logged comprehensively at startup
• Cache invalidation works properly
• No performance impact when features are disabled
• Feature flags properly logged for debugging

---

Step 10: Final Integration and Validation

Duration: 2-3 days
Risk Level: Low
Rollback Complexity: Simple

Objective

Integrate all Phase 1 components and perform comprehensive validation of the Kubernetes-ready application.

Implementation

// 10.1: Integration validation service
public class IntegrationValidationService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<IntegrationValidationService> _logger;

    public async Task<IntegrationValidationResult> ValidateIntegrationAsync()
    {
        var result = new IntegrationValidationResult();
        _logger.LogInformation("Starting comprehensive integration validation");

        try
        {
            // Validate all Phase 1 components
            await ValidateStructuredLogging(result);
            await ValidateHealthChecks(result);
            await ValidateConfigurationFramework(result);
            await ValidateDatabaseProvider(result);
            await ValidatePerformanceMonitoring(result);
            await ValidateFeatureFlags(result);

            // Overall integration test
            await ValidateEndToEndWorkflow(result);

            result.OverallStatus = result.ComponentResults.All(c => c.Value.Status == "Healthy") ? "Healthy" : "Degraded";
        }
        catch (Exception ex)
        {
            result.OverallStatus = "Failed";
            result.GeneralIssues.Add($"Integration validation failed: {ex.Message}");
            _logger.LogError(ex, "Integration validation failed");
        }

        _logger.LogInformation("Integration validation completed. Status: {Status}", result.OverallStatus);
        return result;
    }

    private async Task ValidateStructuredLogging(IntegrationValidationResult result)
    {
        var componentResult = new ComponentValidationResult { ComponentName = "StructuredLogging" };
        
        try
        {
            var logger = _serviceProvider.GetService<ILogger<IntegrationValidationService>>();
            var correlationService = _serviceProvider.GetService<CorrelationIdService>();

            if (logger != null)
            {
                logger.LogInformation("Testing structured logging functionality");
                componentResult.Status = "Healthy";
                componentResult.Details["LoggerAvailable"] = true;
                componentResult.Details["CorrelationIdService"] = correlationService != null;
            }
            else
            {
                componentResult.Status = "Failed";
                componentResult.Issues.Add("Logger service not available");
            }
        }
        catch (Exception ex)
        {
            componentResult.Status = "Failed";
            componentResult.Issues.Add($"Structured logging validation failed: {ex.Message}");
        }

        result.ComponentResults["StructuredLogging"] = componentResult;
    }

    private async Task ValidateHealthChecks(IntegrationValidationResult result)
    {
        var componentResult = new ComponentValidationResult { ComponentName = "HealthChecks" };
        
        try
        {
            var databaseHealthCheck = _serviceProvider.GetService<DatabaseHealthCheck>();
            var applicationHealthCheck = _serviceProvider.GetService<ApplicationHealthCheck>();

            if (databaseHealthCheck != null && applicationHealthCheck != null)
            {
                var dbHealth = await databaseHealthCheck.CheckHealthAsync(null);
                var appHealth = await applicationHealthCheck.CheckHealthAsync(null);

                componentResult.Status = (dbHealth.Status == HealthStatus.Healthy && appHealth.Status == HealthStatus.Healthy) 
                    ? "Healthy" : "Degraded";
                
                componentResult.Details["DatabaseHealth"] = dbHealth.Status.ToString();
                componentResult.Details["ApplicationHealth"] = appHealth.Status.ToString();
            }
            else
            {
                componentResult.Status = "Failed";
                componentResult.Issues.Add("Health check services not available");
            }
        }
        catch (Exception ex)
        {
            componentResult.Status = "Failed";
            componentResult.Issues.Add($"Health check validation failed: {ex.Message}");
        }

        result.ComponentResults["HealthChecks"] = componentResult;
    }

    private async Task ValidateConfigurationFramework(IntegrationValidationResult result)
    {
        var componentResult = new ComponentValidationResult { ComponentName = "ConfigurationFramework" };
        
        try
        {
            var configValidation = _serviceProvider.GetService<ConfigurationValidationService>();
            var configMapping = _serviceProvider.GetService<ConfigurationMappingService>();

            if (configValidation != null && configMapping != null)
            {
                var validationResult = configValidation.ValidateConfiguration();
                
                componentResult.Status = validationResult.HasErrors ? "Failed" : "Healthy";
                componentResult.Details["ValidationErrors"] = validationResult.Errors.Count;
                componentResult.Details["ValidationWarnings"] = validationResult.Warnings.Count;
                
                if (validationResult.HasErrors)
                {
                    componentResult.Issues.AddRange(validationResult.Errors.Select(e => e.Message));
                }
            }
            else
            {
                componentResult.Status = "Failed";
                componentResult.Issues.Add("Configuration services not available");
            }
        }
        catch (Exception ex)
        {
            componentResult.Status = "Failed";
            componentResult.Issues.Add($"Configuration validation failed: {ex.Message}");
        }

        result.ComponentResults["ConfigurationFramework"] = componentResult;
    }

    private async Task ValidateDatabaseProvider(IntegrationValidationResult result)
    {
        var componentResult = new ComponentValidationResult { ComponentName = "DatabaseProvider" };
        
        try
        {
            var providerService = _serviceProvider.GetService<DatabaseProviderService>();
            var diagnosticsService = _serviceProvider.GetService<DatabaseDiagnosticsService>();

            if (providerService != null && diagnosticsService != null)
            {
                var provider = providerService.GetActiveProvider();
                var diagnostics = await diagnosticsService.PerformDiagnosticsAsync();

                componentResult.Status = diagnostics.OverallStatus;
                componentResult.Details["ActiveProvider"] = provider.ToString();
                componentResult.Details["ConnectionTime"] = diagnostics.ConnectionTime;
                componentResult.Details["IssueCount"] = diagnostics.Issues.Count;
                
                if (diagnostics.Issues.Count > 0)
                {
                    componentResult.Issues.AddRange(diagnostics.Issues);
                }
            }
            else
            {
                componentResult.Status = "Failed";
                componentResult.Issues.Add("Database services not available");
            }
        }
        catch (Exception ex)
        {
            componentResult.Status = "Failed";
            componentResult.Issues.Add($"Database provider validation failed: {ex.Message}");
        }

        result.ComponentResults["DatabaseProvider"] = componentResult;
    }

    private async Task ValidatePerformanceMonitoring(IntegrationValidationResult result)
    {
        var componentResult = new ComponentValidationResult { ComponentName = "PerformanceMonitoring" };
        
        try
        {
            var performanceService = _serviceProvider.GetService<PerformanceMonitoringService>();
            var benchmarkService = _serviceProvider.GetService<PerformanceBenchmarkService>();

            if (performanceService != null)
            {
                // Test metrics recording
                performanceService.RecordHttpRequest("GET", "/test", 200, 0.1);
                performanceService.RecordDatabaseOperation("SELECT", "Test", 0.05);

                componentResult.Status = "Healthy";
                componentResult.Details["PerformanceServiceAvailable"] = true;
                componentResult.Details["BenchmarkServiceAvailable"] = benchmarkService != null;
            }
            else
            {
                componentResult.Status = "Failed";
                componentResult.Issues.Add("Performance monitoring service not available");
            }
        }
        catch (Exception ex)
        {
            componentResult.Status = "Failed";
            componentResult.Issues.Add($"Performance monitoring validation failed: {ex.Message}");
        }

        result.ComponentResults["PerformanceMonitoring"] = componentResult;
    }

    private async Task ValidateFeatureFlags(IntegrationValidationResult result)
    {
        var componentResult = new ComponentValidationResult { ComponentName = "FeatureFlags" };
        
        try
        {
            var featureFlagService = _serviceProvider.GetService<FeatureFlagService>();

            if (featureFlagService != null)
            {
                var allFlags = featureFlagService.GetAllFlags();
                var enabledCount = allFlags.Count(f => f.Value);

                componentResult.Status = "Healthy";
                componentResult.Details["TotalFlags"] = allFlags.Count;
                componentResult.Details["EnabledFlags"] = enabledCount;
                componentResult.Details["DisabledFlags"] = allFlags.Count - enabledCount;
            }
            else
            {
                componentResult.Status = "Failed";
                componentResult.Issues.Add("Feature flag service not available");
            }
        }
        catch (Exception ex)
        {
            componentResult.Status = "Failed";
            componentResult.Issues.Add($"Feature flag validation failed: {ex.Message}");
        }

        result.ComponentResults["FeatureFlags"] = componentResult;
    }

    private async Task ValidateEndToEndWorkflow(IntegrationValidationResult result)
    {
        var componentResult = new ComponentValidationResult { ComponentName = "EndToEndWorkflow" };
        
        try
        {
            // Test a complete workflow that uses multiple components
            using var scope = _serviceProvider.CreateScope();
            var vehicleAccess = scope.ServiceProvider.GetService<IVehicleDataAccess>();

            if (vehicleAccess != null)
            {
                // Test basic database operation
                var vehicles = await vehicleAccess.GetVehiclesAsync(1);
                
                componentResult.Status = "Healthy";
                componentResult.Details["DatabaseOperationSuccessful"] = true;
                componentResult.Details["VehicleCount"] = vehicles?.Count ?? 0;
            }
            else
            {
                componentResult.Status = "Failed";
                componentResult.Issues.Add("Vehicle data access not available for end-to-end test");
            }
        }
        catch (Exception ex)
        {
            componentResult.Status = "Failed";
            componentResult.Issues.Add($"End-to-end workflow validation failed: {ex.Message}");
        }

        result.ComponentResults["EndToEndWorkflow"] = componentResult;
    }
}

// 10.2: Integration validation models
public class IntegrationValidationResult
{
    public string OverallStatus { get; set; } = "Unknown";
    public DateTime ValidationTime { get; set; } = DateTime.UtcNow;
    public Dictionary<string, ComponentValidationResult> ComponentResults { get; set; } = new();
    public List<string> GeneralIssues { get; set; } = new();
    public Dictionary<string, object> Summary { get; set; } = new();
}

public class ComponentValidationResult
{
    public string ComponentName { get; set; }
    public string Status { get; set; } = "Unknown";
    public Dictionary<string, object> Details { get; set; } = new();
    public List<string> Issues { get; set; } = new();
}

// 10.3: Integration validation startup service
public class IntegrationValidationStartupService : IHostedService
{
    private readonly IntegrationValidationService _validationService;
    private readonly ILogger<IntegrationValidationStartupService> _logger;

    public IntegrationValidationStartupService(
        IntegrationValidationService validationService,
        ILogger<IntegrationValidationStartupService> logger)
    {
        _validationService = validationService;
        _logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        try
        {
            _logger.LogInformation("Starting comprehensive integration validation");
            var result = await _validationService.ValidateIntegrationAsync();
            
            _logger.LogInformation("Integration validation completed. Overall Status: {OverallStatus}", result.OverallStatus);

            foreach (var component in result.ComponentResults)
            {
                var componentResult = component.Value;
                _logger.LogInformation("Component {ComponentName}: {Status}", 
                    componentResult.ComponentName, componentResult.Status);

                foreach (var detail in componentResult.Details)
                {
                    _logger.LogDebug("Component {ComponentName} detail - {Key}: {Value}", 
                        componentResult.ComponentName, detail.Key, detail.Value);
                }

                foreach (var issue in componentResult.Issues)
                {
                    _logger.LogWarning("Component {ComponentName} issue: {Issue}", 
                        componentResult.ComponentName, issue);
                }
            }

            foreach (var issue in result.GeneralIssues)
            {
                _logger.LogError("Integration validation general issue: {Issue}", issue);
            }

            if (result.OverallStatus != "Healthy")
            {
                _logger.LogWarning("Application integration validation indicates issues. Review component logs for details.");
            }
            else
            {
                _logger.LogInformation("All integration components are healthy and ready for Kubernetes deployment");
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Integration validation failed during startup");
        }
    }

    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

Testing Plan

Automated Tests:

[Test]
public async Task IntegrationValidationService_AllComponentsHealthy_ReturnsHealthyStatus()
{
    var services = new ServiceCollection();
    
    // Add all required services
    services.AddSingleton<DatabaseProviderService>();
    services.AddSingleton<FeatureFlagService>();
    services.AddSingleton<ConfigurationValidationService>();
    // ... add other services
    
    var serviceProvider = services.BuildServiceProvider();
    var validationService = new IntegrationValidationService(serviceProvider, Mock.Of<ILogger<IntegrationValidationService>>());

    var result = await validationService.ValidateIntegrationAsync();

    Assert.AreEqual("Healthy", result.OverallStatus);
    Assert.IsTrue(result.ComponentResults.Count > 0);
}

[Test]
public async Task IntegrationValidationStartupService_RunsAtStartup_LogsValidationResults()
{
    var mockLogger = new Mock<ILogger<IntegrationValidationStartupService>>();
    var mockValidationService = new Mock<IntegrationValidationService>();
    
    var validationResult = new IntegrationValidationResult
    {
        OverallStatus = "Healthy",
        ComponentResults = new Dictionary<string, ComponentValidationResult>
        {
            ["StructuredLogging"] = new ComponentValidationResult
            {
                ComponentName = "StructuredLogging",
                Status = "Healthy"
            }
        }
    };
    
    mockValidationService.Setup(x => x.ValidateIntegrationAsync())
        .ReturnsAsync(validationResult);
    
    var service = new IntegrationValidationStartupService(
        mockValidationService.Object,
        mockLogger.Object);
    
    await service.StartAsync(CancellationToken.None);
    
    // Verify integration validation was logged
    mockLogger.Verify(
        x => x.Log(
            LogLevel.Information,
            It.IsAny<EventId>(),
            It.Is<It.IsAnyType>((v, t) => v.ToString().Contains("Integration validation completed")),
            It.IsAny<Exception>(),
            It.IsAny<Func<It.IsAnyType, Exception, string>>()),
        Times.Once);
}

Manual Validation:

1. Review startup logs to verify all components are healthy
2. Test with individual component failures and verify proper error logging
3. Verify all Phase 1 features work together correctly
4. Test application startup with all new components and review logs
5. Perform end-to-end user workflows
6. Verify Kubernetes readiness (health checks, configuration, etc.)

Success Criteria:

• All Phase 1 components integrate successfully
• Integration validation service reports accurate status and logs details
• End-to-end workflows function correctly
• Application ready for Kubernetes deployment
• Comprehensive logging provides visibility into all components
• Performance remains within acceptable limits

---

Summary

This detailed implementation plan provides a safe, step-by-step approach to Phase 1 with:

1. Incremental Changes: Each step is isolated and testable
2. Comprehensive Testing: Automated and manual validation at each step
3. Debugging Focus: Extensive logging and diagnostic capabilities
4. Risk Mitigation: Rollback procedures and thorough validation
5. Performance Monitoring: Baseline and continuous validation
6. Feature Control: Feature flags for safe rollout of new functionality

The plan ensures that any issues can be detected and resolved quickly before proceeding to the next step, making the overall Phase 1 implementation much safer and more reliable than the original approach. Each step builds upon the previous ones while maintaining full backward compatibility until the final integration.