create-monitoring

---
name: create-monitoring
description: Create API monitoring dashboard
shortcut: mon
---
# Create API Monitoring Dashboard

Build comprehensive monitoring infrastructure with metrics, logs, traces, and alerts for full API observability.

## When to Use This Command

Use `/create-monitoring` when you need to:

- Establish observability for production APIs
- Track RED metrics (Rate, Errors, Duration) across services
- Set up real-time alerting for SLO violations
- Debug performance issues with distributed tracing
- Create executive dashboards for API health
- Implement SRE practices with data-driven insights

DON'T use this when:

- Building proof-of-concept applications (use lightweight logging instead)
- Monitoring non-critical internal tools (basic health checks may suffice)
- Resources are extremely constrained (consider managed solutions like Datadog first)

## Design Decisions

This command implements a **Prometheus + Grafana stack** as the primary approach because:

- Open-source with no vendor lock-in
- Industry-standard metric format with wide ecosystem support
- Powerful query language (PromQL) for complex analysis
- Horizontal scalability via federation and remote storage

**Alternative considered: ELK Stack** (Elasticsearch, Logstash, Kibana)

- Better for log-centric analysis
- Higher resource requirements
- More complex operational overhead
- Recommended when logs are primary data source

**Alternative considered: Managed solutions** (Datadog, New Relic)

- Faster time-to-value
- Higher ongoing cost
- Less customization flexibility
- Recommended for teams without dedicated DevOps

## Prerequisites

Before running this command:

1. Docker and Docker Compose installed
2. API instrumented with metrics endpoints (Prometheus format)
3. Basic understanding of PromQL query language
4. Network access for inter-service communication
5. Sufficient disk space for time-series data (plan for 2-4 weeks retention)

## Implementation Process

### Step 1: Configure Prometheus

Set up Prometheus to scrape metrics from your API endpoints with service discovery.

### Step 2: Create Grafana Dashboards

Build visualizations for RED metrics, custom business metrics, and SLO tracking.

### Step 3: Implement Distributed Tracing

Integrate Jaeger for end-to-end request tracing across microservices.

### Step 4: Configure Alerting

Set up AlertManager rules for critical thresholds with notification channels (Slack, PagerDuty).

### Step 5: Deploy Monitoring Stack

Deploy complete observability infrastructure with health checks and backup configurations.

## Output Format

The command generates:

- `docker-compose.yml` - Complete monitoring stack configuration
- `prometheus.yml` - Prometheus scrape configuration
- `grafana-dashboards/` - Pre-built dashboard JSON files
- `alerting-rules.yml` - AlertManager rule definitions
- `jaeger-config.yml` - Distributed tracing configuration
- `README.md` - Deployment and operation guide

## Code Examples

### Example 1: Complete Node.js Express API with Comprehensive Monitoring

```javascript
// metrics/instrumentation.js - Full-featured Prometheus instrumentation
const promClient = require('prom-client');
const { performance } = require('perf_hooks');
const os = require('os');

class MetricsCollector {
  constructor() {
    // Create separate registries for different metric types
    this.register = new promClient.Registry();
    this.businessRegister = new promClient.Registry();

    // Add default system metrics
    promClient.collectDefaultMetrics({
      register: this.register,
      prefix: 'api_',
      gcDurationBuckets: [0.001, 0.01, 0.1, 1, 2, 5]
    });

    // Initialize all metric types
    this.initializeMetrics();
    this.initializeBusinessMetrics();
    this.initializeCustomCollectors();

    // Start periodic collectors
    this.startPeriodicCollectors();
  }

  initializeMetrics() {
    // RED Metrics (Rate, Errors, Duration)
    this.httpRequestDuration = new promClient.Histogram({
      name: 'http_request_duration_seconds',
      help: 'Duration of HTTP requests in seconds',
      labelNames: ['method', 'route', 'status_code', 'service', 'environment'],
      buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    });

    this.httpRequestTotal = new promClient.Counter({
      name: 'http_requests_total',
      help: 'Total number of HTTP requests',
      labelNames: ['method', 'route', 'status_code', 'service', 'environment']
    });

    this.httpRequestErrors = new promClient.Counter({
      name: 'http_request_errors_total',
      help: 'Total number of HTTP errors',
      labelNames: ['method', 'route', 'error_type', 'service', 'environment']
    });

    // Database metrics
    this.dbQueryDuration = new promClient.Histogram({
      name: 'db_query_duration_seconds',
      help: 'Database query execution time',
      labelNames: ['operation', 'table', 'database', 'status'],
      buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5]
    });

    this.dbConnectionPool = new promClient.Gauge({
      name: 'db_connection_pool_size',
      help: 'Database connection pool metrics',
      labelNames: ['state', 'database'] // states: active, idle, total
    });

    // Cache metrics
    this.cacheHitRate = new promClient.Counter({
      name: 'cache_operations_total',
      help: 'Cache operation counts',
      labelNames: ['operation', 'cache_name', 'status'] // hit, miss, set, delete
    });

    this.cacheLatency = new promClient.Histogram({
      name: 'cache_operation_duration_seconds',
      help: 'Cache operation latency',
      labelNames: ['operation', 'cache_name'],
      buckets: [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1]
    });

    // External API metrics
    this.externalApiCalls = new promClient.Histogram({
      name: 'external_api_duration_seconds',
      help: 'External API call duration',
      labelNames: ['service', 'endpoint', 'status_code'],
      buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30]
    });

    // Circuit breaker metrics
    this.circuitBreakerState = new promClient.Gauge({
      name: 'circuit_breaker_state',
      help: 'Circuit breaker state (0=closed, 1=open, 2=half-open)',
      labelNames: ['service']
    });

    // Rate limiting metrics
    this.rateLimitHits = new promClient.Counter({
      name: 'rate_limit_hits_total',
      help: 'Number of rate limited requests',
      labelNames: ['limit_type', 'client_type']
    });

    // WebSocket metrics
    this.activeWebsockets = new promClient.Gauge({
      name: 'websocket_connections_active',
      help: 'Number of active WebSocket connections',
      labelNames: ['namespace', 'room']
    });

    // Register all metrics
    [
      this.httpRequestDuration, this.httpRequestTotal, this.httpRequestErrors,
      this.dbQueryDuration, this.dbConnectionPool, this.cacheHitRate,
      this.cacheLatency, this.externalApiCalls, this.circuitBreakerState,
      this.rateLimitHits, this.activeWebsockets
    ].forEach(metric => this.register.registerMetric(metric));
  }

  initializeBusinessMetrics() {
    // User activity metrics
    this.activeUsers = new promClient.Gauge({
      name: 'business_active_users',
      help: 'Number of active users in the last 5 minutes',
      labelNames: ['user_type', 'plan']
    });

    this.userSignups = new promClient.Counter({
      name: 'business_user_signups_total',
      help: 'Total user signups',
      labelNames: ['source', 'plan', 'country']
    });

    // Transaction metrics
    this.transactionAmount = new promClient.Histogram({
      name: 'business_transaction_amount_dollars',
      help: 'Transaction amounts in dollars',
      labelNames: ['type', 'status', 'payment_method'],
      buckets: [1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 10000]
    });

    this.orderProcessingTime = new promClient.Histogram({
      name: 'business_order_processing_seconds',
      help: 'Time to process orders end-to-end',
      labelNames: ['order_type', 'fulfillment_type'],
      buckets: [10, 30, 60, 180, 300, 600, 1800, 3600]
    });

    // API usage metrics
    this.apiUsageByClient = new promClient.Counter({
      name: 'business_api_usage_by_client',
      help: 'API usage segmented by client',
      labelNames: ['client_id', 'tier', 'endpoint']
    });

    this.apiQuotaRemaining = new promClient.Gauge({
      name: 'business_api_quota_remaining',
      help: 'Remaining API quota for clients',
      labelNames: ['client_id', 'tier', 'quota_type']
    });

    // Revenue metrics
    this.revenueByProduct = new promClient.Counter({
      name: 'business_revenue_by_product_cents',
      help: 'Revenue by product in cents',
      labelNames: ['product_id', 'product_category', 'currency']
    });

    // Register business metrics
    [
      this.activeUsers, this.userSignups, this.transactionAmount,
      this.orderProcessingTime, this.apiUsageByClient, this.apiQuotaRemaining,
      this.revenueByProduct
    ].forEach(metric => this.businessRegister.registerMetric(metric));
  }

  initializeCustomCollectors() {
    // SLI/SLO metrics
    this.sloCompliance = new promClient.Gauge({
      name: 'slo_compliance_percentage',
      help: 'SLO compliance percentage',
      labelNames: ['slo_name', 'service', 'window']
    });

    this.errorBudgetRemaining = new promClient.Gauge({
      name: 'error_budget_remaining_percentage',
      help: 'Remaining error budget percentage',
      labelNames: ['service', 'slo_type']
    });

    this.register.registerMetric(this.sloCompliance);
    this.register.registerMetric(this.errorBudgetRemaining);
  }

  startPeriodicCollectors() {
    // Update active users every 30 seconds
    setInterval(() => {
      const activeUserCount = this.calculateActiveUsers();
      this.activeUsers.set(
        { user_type: 'registered', plan: 'free' },
        activeUserCount.free
      );
      this.activeUsers.set(
        { user_type: 'registered', plan: 'premium' },
        activeUserCount.premium
      );
    }, 30000);

    // Update SLO compliance every minute
    setInterval(() => {
      this.updateSLOCompliance();
    }, 60000);

    // Database pool monitoring
    setInterval(() => {
      this.updateDatabasePoolMetrics();
    }, 15000);
  }

  // Middleware for HTTP metrics
  httpMetricsMiddleware() {
    return (req, res, next) => {
      const start = performance.now();
      const route = req.route?.path || req.path || 'unknown';

      // Track in-flight requests
      const inFlightGauge = new promClient.Gauge({
        name: 'http_requests_in_flight',
        help: 'Number of in-flight HTTP requests',
        labelNames: ['method', 'route']
      });

      inFlightGauge.inc({ method: req.method, route });

      res.on('finish', () => {
        const duration = (performance.now() - start) / 1000;
        const labels = {
          method: req.method,
          route,
          status_code: res.statusCode,
          service: process.env.SERVICE_NAME || 'api',
          environment: process.env.NODE_ENV || 'development'
        };

        // Record metrics
        this.httpRequestDuration.observe(labels, duration);
        this.httpRequestTotal.inc(labels);

        if (res.statusCode >= 400) {
          const errorType = res.statusCode >= 500 ? 'server_error' : 'client_error';
          this.httpRequestErrors.inc({
            ...labels,
            error_type: errorType
          });
        }

        inFlightGauge.dec({ method: req.method, route });

        // Log slow requests
        if (duration > 1) {
          console.warn('Slow request detected:', {
            ...labels,
            duration,
            user: req.user?.id,
            ip: req.ip
          });
        }
      });

      next();
    };
  }

  // Database query instrumentation
  instrumentDatabase(knex) {
    knex.on('query', (query) => {
      query.__startTime = performance.now();
    });

    knex.on('query-response', (response, query) => {
      const duration = (performance.now() - query.__startTime) / 1000;
      const table = this.extractTableName(query.sql);

      this.dbQueryDuration.observe({
        operation: query.method || 'select',
        table,
        database: process.env.DB_NAME || 'default',
        status: 'success'
      }, duration);
    });

    knex.on('query-error', (error, query) => {
      const duration = (performance.now() - query.__startTime) / 1000;
      const table = this.extractTableName(query.sql);

      this.dbQueryDuration.observe({
        operation: query.method || 'select',
        table,
        database: process.env.DB_NAME || 'default',
        status: 'error'
      }, duration);
    });
  }

  // Cache instrumentation wrapper
  wrapCache(cache) {
    const wrapper = {};
    const methods = ['get', 'set', 'delete', 'has'];

    methods.forEach(method => {
      wrapper[method] = async (...args) => {
        const start = performance.now();
        const cacheName = cache.name || 'default';

        try {
          const result = await cachemethod;
          const duration = (performance.now() - start) / 1000;

          // Record cache metrics
          if (method === 'get') {
            const status = result !== undefined ? 'hit' : 'miss';
            this.cacheHitRate.inc({
              operation: method,
              cache_name: cacheName,
              status
            });
          } else {
            this.cacheHitRate.inc({
              operation: method,
              cache_name: cacheName,
              status: 'success'
            });
          }

          this.cacheLatency.observe({
            operation: method,
            cache_name: cacheName
          }, duration);

          return result;
        } catch (error) {
          this.cacheHitRate.inc({
            operation: method,
            cache_name: cacheName,
            status: 'error'
          });
          throw error;
        }
      };
    });

    return wrapper;
  }

  // External API call instrumentation
  async trackExternalCall(serviceName, endpoint, callFunc) {
    const start = performance.now();

    try {
      const result = await callFunc();
      const duration = (performance.now() - start) / 1000;

      this.externalApiCalls.observe({
        service: serviceName,
        endpoint,
        status_code: result.status || 200
      }, duration);

      return result;
    } catch (error) {
      const duration = (performance.now() - start) / 1000;

      this.externalApiCalls.observe({
        service: serviceName,
        endpoint,
        status_code: error.response?.status || 0
      }, duration);

      throw error;
    }
  }

  // Circuit breaker monitoring
  updateCircuitBreakerState(service, state) {
    const stateValue = {
      'closed': 0,
      'open': 1,
      'half-open': 2
    }[state] || 0;

    this.circuitBreakerState.set({ service }, stateValue);
  }

  // Helper methods
  calculateActiveUsers() {
    // Implementation would query your session store or database
    return {
      free: Math.floor(Math.random() * 1000),
      premium: Math.floor(Math.random() * 100)
    };
  }

  updateSLOCompliance() {
    // Calculate based on recent metrics
    const availability = 99.95; // Calculate from actual metrics
    const latencyP99 = 250; // Calculate from actual metrics

    this.sloCompliance.set({
      slo_name: 'availability',
      service: 'api',
      window: '30d'
    }, availability);

    this.sloCompliance.set({
      slo_name: 'latency_p99',
      service: 'api',
      window: '30d'
    }, latencyP99 < 500 ? 100 : 0);

    // Update error budget
    const errorBudget = 100 - ((100 - availability) / 0.05) * 100;
    this.errorBudgetRemaining.set({
      service: 'api',
      slo_type: 'availability'
    }, Math.max(0, errorBudget));
  }

  updateDatabasePoolMetrics() {
    // Get pool stats from your database driver
    const pool = global.dbPool; // Your database pool instance
    if (pool) {
      this.dbConnectionPool.set({
        state: 'active',
        database: 'primary'
      }, pool.numUsed());

      this.dbConnectionPool.set({
        state: 'idle',
        database: 'primary'
      }, pool.numFree());

      this.dbConnectionPool.set({
        state: 'total',
        database: 'primary'
      }, pool.numUsed() + pool.numFree());
    }
  }

  extractTableName(sql) {
    const match = sql.match(/(?:from|into|update)\s+`?(\w+)`?/i);
    return match ? match[1] : 'unknown';
  }

  // Expose metrics endpoint
  async getMetrics() {
    const baseMetrics = await this.register.metrics();
    const businessMetrics = await this.businessRegister.metrics();
    return baseMetrics + '\n' + businessMetrics;
  }
}

// Express application setup
const express = require('express');
const app = express();
const metricsCollector = new MetricsCollector();

// Apply monitoring middleware
app.use(metricsCollector.httpMetricsMiddleware());

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', metricsCollector.register.contentType);
  res.end(await metricsCollector.getMetrics());
});

// Example API endpoint with comprehensive tracking
app.post('/api/orders', async (req, res) => {
  const orderStart = performance.now();

  try {
    // Track business metrics
    metricsCollector.transactionAmount.observe({
      type: 'purchase',
      status: 'pending',
      payment_method: req.body.paymentMethod
    }, req.body.amount);

    // Simulate external payment API call
    const paymentResult = await metricsCollector.trackExternalCall(
      'stripe',
      '/charges',
      async () => {
        // Your actual payment API call
        return await stripeClient.charges.create({
          amount: req.body.amount * 100,
          currency: 'usd'
        });
      }
    );

    // Track order processing time
    const processingTime = (performance.now() - orderStart) / 1000;
    metricsCollector.orderProcessingTime.observe({
      order_type: 'standard',
      fulfillment_type: 'digital'
    }, processingTime);

    // Track revenue
    metricsCollector.revenueByProduct.inc({
      product_id: req.body.productId,
      product_category: req.body.category,
      currency: 'USD'
    }, req.body.amount * 100);

    res.json({ success: true, orderId: paymentResult.id });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

module.exports = { app, metricsCollector };
```

### Example 2: Complete Monitoring Stack with Docker Compose

```yaml
# docker-compose.yml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:v2.45.0
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alerting-rules.yml:/etc/prometheus/alerting-rules.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
      - '--storage.tsdb.retention.time=15d'
    ports:
      - "9090:9090"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:10.0.0
    container_name: grafana
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana-dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_ROOT_URL=http://localhost:3000
    ports:
      - "3000:3000"
    networks:
      - monitoring
    depends_on:
      - prometheus

  jaeger:
    image: jaegertracing/all-in-one:1.47
    container_name: jaeger
    environment:
      - COLLECTOR_ZIPKIN_HOST_PORT=:9411
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
      - "16686:16686"  # Jaeger UI
      - "14268:14268"
      - "14250:14250"
      - "9411:9411"
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager:v0.26.0
    container_name: alertmanager
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    ports:
      - "9093:9093"
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus-data:
  grafana-data:
```

### Example 3: Advanced Grafana Dashboard Definitions

```json
// grafana-dashboards/api-overview.json
{
  "dashboard": {
    "id": null,
    "uid": "api-overview",
    "title": "API Performance Overview",
    "tags": ["api", "performance", "sre"],
    "timezone": "browser",
    "schemaVersion": 16,
    "version": 0,
    "refresh": "30s",
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "templating": {
      "list": [
        {
          "name": "datasource",
          "type": "datasource",
          "query": "prometheus",
          "current": {
            "value": "Prometheus",
            "text": "Prometheus"
          }
        },
        {
          "name": "service",
          "type": "query",
          "datasource": "$datasource",
          "query": "label_values(http_requests_total, service)",
          "multi": true,
          "includeAll": true,
          "current": {
            "value": ["$__all"],
            "text": "All"
          },
          "refresh": 1
        },
        {
          "name": "environment",
          "type": "query",
          "datasource": "$datasource",
          "query": "label_values(http_requests_total, environment)",
          "current": {
            "value": "production",
            "text": "Production"
          }
        }
      ]
    },
    "panels": [
      {
        "id": 1,
        "gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 },
        "type": "graph",
        "title": "Request Rate (req/s)",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=~\"$service\",environment=\"$environment\"}[5m])) by (service)",
            "legendFormat": "{{service}}",
            "refId": "A"
          }
        ],
        "yaxes": [
          {
            "format": "reqps",
            "label": "Requests per second"
          }
        ],
        "lines": true,
        "linewidth": 2,
        "fill": 1,
        "fillGradient": 3,
        "steppedLine": false,
        "tooltip": {
          "shared": true,
          "sort": 0,
          "value_type": "individual"
        },
        "alert": {
          "name": "High Request Rate",
          "conditions": [
            {
              "evaluator": {
                "params": [10000],
                "type": "gt"
              },
              "operator": {
                "type": "and"
              },
              "query": {
                "params": ["A", "5m", "now"]
              },
              "reducer": {
                "type": "avg"
              },
              "type": "query"
            }
          ],
          "executionErrorState": "alerting",
          "frequency": "1m",
          "handler": 1,
          "noDataState": "no_data",
          "notifications": [
            {
              "uid": "slack-channel"
            }
          ]
        }
      },
      {
        "id": 2,
        "gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 },
        "type": "graph",
        "title": "Error Rate (%)",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=~\"$service\",environment=\"$environment\",status_code=~\"5..\"}[5m])) by (service) / sum(rate(http_requests_total{service=~\"$service\",environment=\"$environment\"}[5m])) by (service) * 100",
            "legendFormat": "{{service}}",
            "refId": "A"
          }
        ],
        "yaxes": [
          {
            "format": "percent",
            "label": "Error Rate",
            "max": 10
          }
        ],
        "thresholds": [
          {
            "value": 1,
            "op": "gt",
            "fill": true,
            "line": true,
            "colorMode": "critical"
          }
        ],
        "alert": {
          "name": "High Error Rate",
          "conditions": [
            {
              "evaluator": {
                "params": [1],
                "type": "gt"
              },
              "operator": {
                "type": "and"
              },
              "query": {
                "params": ["A", "5m", "now"]
              },
              "reducer": {
                "type": "last"
              },
              "type": "query"
            }
          ],
          "executionErrorState": "alerting",
          "frequency": "1m",
          "handler": 1,
          "noDataState": "no_data",
          "notifications": [
            {
              "uid": "pagerduty"
            }
          ],
          "message": "Error rate is above 1% for service {{service}}"
        }
      },
      {
        "id": 3,
        "gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 },
        "type": "graph",
        "title": "Response Time (p50, p95, p99)",
        "targets": [
          {
            "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\",environment=\"$environment\"}[5m])) by (le, service))",
            "legendFormat": "p50 {{service}}",
            "refId": "A"
          },
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\",environment=\"$environment\"}[5m])) by (le, service))",
            "legendFormat": "p95 {{service}}",
            "refId": "B"
          },
          {
            "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\",environment=\"$environment\"}[5m])) by (le, service))",
            "legendFormat": "p99 {{service}}",
            "refId": "C"
          }
        ],
        "yaxes": [
          {
            "format": "s",
            "label": "Response Time"
          }
        ]
      },
      {
        "id": 4,
        "gridPos": { "h": 6, "w": 6, "x": 0, "y": 8 },
        "type": "stat",
        "title": "Current QPS",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=~\"$service\",environment=\"$environment\"}[1m]))",
            "instant": true,
            "refId": "A"
          }
        ],
        "format": "reqps",
        "sparkline": {
          "show": true,
          "lineColor": "rgb(31, 120, 193)",
          "fillColor": "rgba(31, 120, 193, 0.18)"
        },
        "thresholds": {
          "mode": "absolute",
          "steps": [
            { "value": 0, "color": "green" },
            { "value": 5000, "color": "yellow" },
            { "value": 10000, "color": "red" }
          ]
        }
      },
      {
        "id": 5,
        "gridPos": { "h": 6, "w": 6, "x": 6, "y": 8 },
        "type": "stat",
        "title": "Error Budget Remaining",
        "targets": [
          {
            "expr": "error_budget_remaining_percentage{service=~\"$service\",slo_type=\"availability\"}",
            "instant": true,
            "refId": "A"
          }
        ],
        "format": "percent",
        "thresholds": {
          "mode": "absolute",
          "steps": [
            { "value": 0, "color": "red" },
            { "value": 25, "color": "orange" },
            { "value": 50, "color": "yellow" },
            { "value": 75, "color": "green" }
          ]
        }
      },
      {
        "id": 6,
        "gridPos": { "h": 6, "w": 12, "x": 12, "y": 8 },
        "type": "table",
        "title": "Top Slow Endpoints",
        "targets": [
          {
            "expr": "topk(10, histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\",environment=\"$environment\"}[5m])) by (le, route)))",
            "format": "table",
            "instant": true,
            "refId": "A"
          }
        ],
        "styles": [
          {
            "alias": "Time",
            "dateFormat": "YYYY-MM-DD HH:mm:ss",
            "type": "date"
          },
          {
            "alias": "Duration",
            "colorMode": "cell",
            "colors": ["green", "yellow", "red"],
            "thresholds": [0.5, 1],
            "type": "number",
            "unit": "s"
          }
        ]
      }
    ]
  }
}
```

### Example 4: Production-Ready Alerting Rules

```yaml
# alerting-rules.yml
groups:
  - name: api_alerts
    interval: 30s
    rules:
      # SLO-based alerts
      - alert: APIHighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service, environment)
            /
            sum(rate(http_requests_total[5m])) by (service, environment)
          ) > 0.01
        for: 5m
        labels:
          severity: critical
          team: api-platform
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "{{ $labels.service }} in {{ $labels.environment }} has error rate of {{ $value | humanizePercentage }} (threshold: 1%)"
          runbook_url: "https://wiki.example.com/runbooks/api-high-error-rate"
          dashboard_url: "https://grafana.example.com/d/api-overview?var-service={{ $labels.service }}"

      - alert: APIHighLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)
          ) > 0.5
        for: 10m
        labels:
          severity: warning
          team: api-platform
        annotations:
          summary: "High latency on {{ $labels.service }}"
          description: "P95 latency for {{ $labels.service }} is {{ $value | humanizeDuration }} (threshold: 500ms)"

      - alert: APILowAvailability
        expr: |
          up{job="api-services"} == 0
        for: 1m
        labels:
          severity: critical
          team: api-platform
        annotations:
          summary: "API service {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been down for more than 1 minute"

      # Business metrics alerts
      - alert: LowActiveUsers
        expr: |
          business_active_users{plan="premium"} < 10
        for: 30m
        labels:
          severity: warning
          team: product
        annotations:
          summary: "Low number of active premium users"
          description: "Only {{ $value }} premium users active in the last 30 minutes"

      - alert: HighTransactionFailureRate
        expr: |
          (
            sum(rate(business_transaction_amount_dollars_sum{status="failed"}[5m]))
            /
            sum(rate(business_transaction_amount_dollars_sum[5m]))
          ) > 0.05
        for: 5m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: "High transaction failure rate"
          description: "Transaction failure rate is {{ $value | humanizePercentage }} (threshold: 5%)"

      # Infrastructure alerts
      - alert: DatabaseConnectionPoolExhausted
        expr: |
          (
            db_connection_pool_size{state="active"}
            /
            db_connection_pool_size{state="total"}
          ) > 0.9
        for: 5m
        labels:
          severity: warning
          team: database
        annotations:
          summary: "Database connection pool near exhaustion"
          description: "{{ $labels.database }} pool is {{ $value | humanizePercentage }} utilized"

      - alert: CacheLowHitRate
        expr: |
          (
            sum(rate(cache_operations_total{status="hit"}[5m])) by (cache_name)
            /
            sum(rate(cache_operations_total{operation="get"}[5m])) by (cache_name)
          ) < 0.8
        for: 15m
        labels:
          severity: warning
          team: api-platform
        annotations:
          summary: "Low cache hit rate for {{ $labels.cache_name }}"
          description: "Cache hit rate is {{ $value | humanizePercentage }} (expected: >80%)"

      - alert: CircuitBreakerOpen
        expr: |
          circuit_breaker_state == 1
        for: 1m
        labels:
          severity: warning
          team: api-platform
        annotations:
          summary: "Circuit breaker open for {{ $labels.service }}"
          description: "Circuit breaker for {{ $labels.service }} has been open for more than 1 minute"

      # SLO burn rate alerts (multi-window approach)
      - alert: SLOBurnRateHigh
        expr: |
          (
            # 5m burn rate > 14.4 (1 hour of error budget in 5 minutes)
            (
              sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service)
              /
              sum(rate(http_requests_total[5m])) by (service)
            ) > (1 - 0.999) * 14.4
          ) and (
            # 1h burn rate > 1 (confirms it's not a spike)
            (
              sum(rate(http_requests_total{status_code=~"5.."}[1h])) by (service)
              /
              sum(rate(http_requests_total[1h])) by (service)
            ) > (1 - 0.999)
          )
        labels:
          severity: critical
          team: api-platform
          alert_type: slo_burn
        annotations:
          summary: "SLO burn rate critically high for {{ $labels.service }}"
          description: "{{ $labels.service }} is burning error budget 14.4x faster than normal"

      # Resource alerts
      - alert: HighMemoryUsage
        expr: |
          (
            container_memory_usage_bytes{container!="POD",container!=""}
            /
            container_spec_memory_limit_bytes{container!="POD",container!=""}
          ) > 0.9
        for: 5m
        labels:
          severity: warning
          team: api-platform
        annotations:
          summary: "High memory usage for {{ $labels.container }}"
          description: "Container {{ $labels.container }} memory usage is {{ $value | humanizePercentage }}"

# AlertManager configuration
# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'YOUR_SLACK_WEBHOOK_URL'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true
    - match:
        severity: warning
      receiver: 'slack-warnings'
    - match:
        team: payments
      receiver: 'payments-team'

receivers:
  - name: 'default'
    slack_configs:
      - channel: '#alerts'
        title: 'Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_SERVICE_KEY'
        description: '{{ .GroupLabels.alertname }}: {{ .CommonAnnotations.summary }}'
        details:
          firing: '{{ .Alerts.Firing | len }}'
          resolved: '{{ .Alerts.Resolved | len }}'
          labels: '{{ .CommonLabels }}'

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#warnings'
        send_resolved: true
        title: 'Warning: {{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'
        actions:
          - type: button
            text: 'View Dashboard'
            url: '{{ .CommonAnnotations.dashboard_url }}'
          - type: button
            text: 'View Runbook'
            url: '{{ .CommonAnnotations.runbook_url }}'

  - name: 'payments-team'
    email_configs:
      - to: 'payments-team@example.com'
        from: 'alerts@example.com'
        headers:
          Subject: 'Payment Alert: {{ .GroupLabels.alertname }}'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'service']
```

### Example 5: OpenTelemetry Integration for Distributed Tracing

```javascript
// tracing/setup.js - OpenTelemetry configuration
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');
const {
  ConsoleSpanExporter,
  BatchSpanProcessor,
  SimpleSpanProcessor
} = require('@opentelemetry/sdk-trace-base');
const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');

class TracingSetup {
  constructor(serviceName, environment = 'production') {
    this.serviceName = serviceName;
    this.environment = environment;
    this.sdk = null;
  }

  initialize() {
    // Create resource identifying the service
    const resource = Resource.default().merge(
      new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: this.serviceName,
        [SemanticResourceAttributes.SERVICE_VERSION]: process.env.VERSION || '1.0.0',
        [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: this.environment,
        'service.namespace': 'api-platform',
        'service.instance.id': process.env.HOSTNAME || 'unknown',
        'telemetry.sdk.language': 'nodejs',
      })
    );

    // Configure Jaeger exporter for traces
    const jaegerExporter = new JaegerExporter({
      endpoint: process.env.JAEGER_ENDPOINT || 'http://localhost:14268/api/traces',
      tags: {
        service: this.serviceName,
        environment: this.environment
      }
    });

    // Configure Prometheus exporter for metrics
    const prometheusExporter = new PrometheusExporter({
      port: 9464,
      endpoint: '/metrics',
      prefix: 'otel_',
      appendTimestamp: true,
    }, () => {
      console.log('Prometheus metrics server started on port 9464');
    });

    // Create SDK with auto-instrumentation
    this.sdk = new NodeSDK({
      resource,
      instrumentations: [
        getNodeAutoInstrumentations({
          '@opentelemetry/instrumentation-fs': {
            enabled: false, // Disable fs to reduce noise
          },
          '@opentelemetry/instrumentation-http': {
            requestHook: (span, request) => {
              span.setAttribute('http.request.body', JSON.stringify(request.body));
              span.setAttribute('http.request.user_id', request.user?.id);
            },
            responseHook: (span, response) => {
              span.setAttribute('http.response.size', response.length);
            },
            ignoreIncomingPaths: ['/health', '/metrics', '/favicon.ico'],
            ignoreOutgoingUrls: [(url) => url.includes('prometheus')]
          },
          '@opentelemetry/instrumentation-express': {
            requestHook: (span, request) => {
              span.setAttribute('express.route', request.route?.path);
              span.setAttribute('express.params', JSON.stringify(request.params));
            }
          },
          '@opentelemetry/instrumentation-mysql2': {
            enhancedDatabaseReporting: true,
          },
          '@opentelemetry/instrumentation-redis-4': {
            dbStatementSerializer: (cmdName, cmdArgs) => {
              return `${cmdName} ${cmdArgs.slice(0, 2).join(' ')}`;
            }
          }
        })
      ],
      spanProcessor: new BatchSpanProcessor(jaegerExporter, {
        maxQueueSize: 2048,
        maxExportBatchSize: 512,
        scheduledDelayMillis: 5000,
        exportTimeoutMillis: 30000,
      }),
      metricReader: new PeriodicExportingMetricReader({
        exporter: prometheusExporter,
        exportIntervalMillis: 10000,
      }),
    });

    // Start the SDK
    this.sdk.start()
      .then(() => console.l