Note: Grafana v12 (2025–2026) adds tighter integrations for managing dashboards and alerts as code (Git Sync, Terraform provider, CLI) and improved visualization suggestions — prefer these features for reproducible observability configurations (see Grafana docs: What’s new in v12).

Prometheus data model

Prometheus stores time-series data as metric name + label set + timestamp + value.

http_requests_total{method="GET", endpoint="/api/users", status="200"} 1234 1672531200

Structured logging

Structured logs are JSON objects with consistent fields. They replace console.log strings with queryable data.

{
  "level": "error",
  "message": "Payment processing failed",
  "service": "billing",
  "traceId": "abc123",
  "userId": "user_456",
  "error": "insufficient_funds",
  "duration_ms": 342
}

Distributed tracing

A trace is a tree of spans. Each span represents a unit of work (HTTP request, DB query, external API call) with timing, status, and metadata.

Note: OpenTelemetry made several platform-level changes in 2025–2026: declarative collector configuration is now stable (use it for Collector pipelines), Kubernetes semantic conventions are moving toward RC/stable attributes, and the Span Events API was deprecated (March 2026). Also, OpenTelemetry Profiles (continuous profiling) entered public alpha in 2026 — watch for a production-ready profiling option in the OTel ecosystem.

Workflow

Step 0 — Choose your collector

• Production recommendation: use a maintained OpenTelemetry Collector distribution (Grafana Alloy or upstream collector-contrib) to receive OTLP, run transforms, and forward metrics/logs/traces to backends.
• Note: Grafana Agent reached End-of-Life on 2025-11-01. Migrate Agent deployments to Grafana Alloy or another Collector distribution — Alloy provides Prometheus pipeline compatibility and remote_write support.
• For Prometheus ingestion at scale, prefer remote_write pipelines to long-term storage (Remote-Write v2 adoption is increasing). Ensure labels and external_labels are applied consistently.

Step 1 — Instrument with OpenTelemetry (SDK + Collector)

Use the OpenTelemetry SDK for application-level instrumentation and export via OTLP to your Collector.

// lib/telemetry.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + "/v1/traces",
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + "/v1/metrics",
    }),
    exportIntervalMillis: 15000,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: process.env.OTEL_SERVICE_NAME ?? "my-app",
});

sdk.start();

Notes:

• Prefer sending telemetry to an intermediate Collector (Alloy / collector-contrib) rather than directly to storage backends. Declarative collector configuration (stable) simplifies pipeline management.
• Follow current OpenTelemetry semantic conventions for attribute names; monitor the K8s semantic conventions changes if you rely on pod/node attributes.
• Because the Span Events API was deprecated in 2026, prefer attributes and structured logs for per-span annotations where possible (see OpenTelemetry blog posts on deprecations and declarative config).

Step 2 — Add structured logging

// lib/logger.ts
import pino from "pino";

export const logger = pino({
  level: process.env.LOG_LEVEL ?? "info",
  formatters: {
    level(label) {
      return { level: label };
    },
  },
  timestamp: pino.stdTimeFunctions.isoTime,
  base: {
    service: process.env.OTEL_SERVICE_NAME ?? "my-app",
    environment: process.env.NODE_ENV,
  },
});

// Usage in a request handler
export function createRequestLogger(traceId: string, userId?: string) {
  return logger.child({ traceId, userId });
}

Correlate logs and traces by including traceId/spanId in structured logs.

Step 3 — Configure Prometheus scraping / remote_write

Use Prometheus scrape targets for high-cardinality, low-latency metrics, and use remote_write (modern implementations) to forward to long-term storage or to Grafana Alloy/TSDB backends.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "my-app"
    metrics_path: "/metrics"
    static_configs:
      - targets: ["app:3000"]
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

Step 4 — Set up Sentry for error tracking (optional)

Sentry remains useful for rich error context and user-impact visibility. Use Sentry for crash reporting and correlate Sentry events with traces/logs via a shared traceId.

Step 5 — Build a Grafana dashboard

Key panels and queries remain the same; with Grafana v12 prefer using Git Sync and the new dashboard layout features to keep dashboards as code.

1. Request rate → rate(http_requests_total[5m])
2. Error rate → rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
3. P50/P95/P99 latency → histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
4. Active connections → pg_stat_activity_count{state="active"}
5. Memory usage → process_resident_memory_bytes
6. CPU usage → rate(process_cpu_seconds_total[5m])

Step 6 — Define alerting rules

Use managed alerts where available or store alerting rules as code in Git (Git Sync) to ensure repeatable escalation policies.

# alerts.yml
groups:
  - name: api-slos
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 1% for 5 minutes"
          dashboard: "https://grafana.example.com/d/api-overview"

      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency above 1s for 5 minutes"

Examples

Custom business metric

import { metrics } from "@opentelemetry/api";

const meter = metrics.getMeter("billing");
const paymentCounter = meter.createCounter("payments_total", {
  description: "Total payment attempts",
});
const paymentDuration = meter.createHistogram("payment_duration_ms", {
  description: "Payment processing duration",
});

async function processPayment(amount: number) {
  const start = performance.now();
  try {
    await chargeCard(amount);
    paymentCounter.add(1, { status: "success" });
  } catch (err) {
    paymentCounter.add(1, { status: "failure" });
    throw err;
  } finally {
    paymentDuration.record(performance.now() - start);
  }
}

Correlating logs with traces

import { trace } from "@opentelemetry/api";

function handleRequest(req: Request) {
  const span = trace.getActiveSpan();
  const traceId = span?.spanContext().traceId ?? "no-trace";
  const log = createRequestLogger(traceId);

  log.info({ path: req.url, method: req.method }, "Request received");
  // ... handle request
  log.info({ status: 200, duration_ms: elapsed }, "Request completed");
}

Decision tree

What do you need to know?
├── "Is the system healthy right now?" → Metrics (Prometheus + Grafana)
├── "What happened during this specific request?" → Logs (structured JSON + Loki)
├── "Where is the bottleneck across services?" → Traces (Tempo or OpenTelemetry)
└── "Why did this specific user see an error?" → Sentry + correlated logs

Which alerting severity?
├── Revenue impact or data loss risk → critical (pages on-call)
├── Degraded performance, no data loss → warning (Slack notification)
├── Anomaly, unclear impact → info (dashboard annotation)
└── Expected spike (deploy, marketing event) → suppress / silence window

Edge cases and gotchas

• Grafana Agent EOL: Grafana Agent reached End-of-Life on 2025-11-01. Migrating to a supported Collector distribution such as Grafana Alloy or upstream OpenTelemetry Collector is required for security fixes and ongoing support.
• OpenTelemetry deprecations: The Span Events API was deprecated in March 2026 — prefer attributes and structured logs for per-event data and follow the OpenTelemetry project guidance. Declarative Collector configuration has reached stability; use it to manage pipelines reliably.
• High-cardinality labels kill Prometheus: Never use userId, requestId, or email as a Prometheus label. Use logs/traces for per-request data.
• Sampling trades detail for cost: Use 100% sampling for errors and a lower rate for successful traces. Re-evaluate sample rates after enabling continuous profiling (OpenTelemetry Profiles alpha in 2026) to balance cost vs. investigative power.
• Log volume surprises: A single noisy endpoint can generate gigabytes of logs per hour. Set rate limits on log emission per endpoint and use Collector processors to reduce volume before storage.
• Grafana dashboard sprawl: Start with one golden-signals dashboard per service (rate, errors, latency, saturation). Use Git Sync/observability-as-code features to keep dashboards consistent across environments.
• Clock skew breaks traces: Ensure NTP is synchronized across all services. Span timestamps out of order make trace waterfalls unreadable.
• Sentry quota exhaustion: Noisy errors consume quota fast. Use beforeSend to filter known non-actionable errors and consider sampling.
• Alert fatigue: If an alert fires more than twice a week without action, it is noise. Fix the root cause or delete the alert.

Evaluation criteria