Observability: Metrics, Logs, Traces

Production is a black box unless you instrument it. “It’s slow” without traces, “it’s failing” without metrics, and “a user reported an error” without correlated logs is the modern developer nightmare. Observability is not optional in 2026 — it’s the difference between debugging in five minutes and debugging in five hours.

The three pillars — metrics, logs, traces — are still the model, but the tooling has converged. Micrometer for metrics, structured JSON logging for logs, OpenTelemetry for traces. Scrape into Prometheus + Grafana + Tempo/Jaeger + Loki, or send to a SaaS (Datadog, New Relic, Honeycomb). The wiring is the same.

---

1. Metrics with Micrometer

Micrometer is Spring Boot’s built-in metrics facade. Add the Prometheus registry and you get an endpoint scraped by Prometheus.

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

management:
  endpoints:
    web:
      exposure:
        include: health, info, prometheus, metrics
  metrics:
    tags:
      application: ${spring.application.name}
      env: ${ENV:local}
  observations:
    key-values:
      service: shortener

Out of the box you get:

• HTTP server metrics: http_server_requests_seconds (histogram, tagged by method, uri, status)

• JVM metrics: heap, threads, GC pauses, classloading

• HikariCP: hikaricp_connections_active, _idle, _pending, _max

• Tomcat: thread pool usage, sessions

• Logback: log event counts by level

That’s often enough for a first-pass dashboard. Add custom metrics for business signals.

Custom metrics

@Service
public class UrlService {

    private final Counter urlsCreated;
    private final Timer redirectLatency;
    private final DistributionSummary payloadSizes;

    public UrlService(MeterRegistry registry) {
        this.urlsCreated = Counter.builder("app.urls.created")
            .description("Total number of URLs shortened")
            .tag("source", "api")
            .register(registry);

        this.redirectLatency = Timer.builder("app.urls.redirect")
            .publishPercentileHistogram()   // enables p95, p99 in Prometheus
            .register(registry);

        this.payloadSizes = DistributionSummary.builder("app.urls.target_length")
            .baseUnit("chars")
            .register(registry);
    }

    public UrlResponse create(CreateUrlRequest req) {
        urlsCreated.increment();
        payloadSizes.record(req.targetUrl().length());
        // ...
    }

    public String resolve(String code) {
        return redirectLatency.record(() -> repo.findByCode(code).orElseThrow().getTargetUrl());
    }
}

Rules:

• Low cardinality tags only. Never tag with user ID, request ID, or anything unbounded — Prometheus will hate you. Use logs/traces for high-cardinality data.

• Prefer histograms over averages. p95 and p99 matter; the mean lies.

• Snake_case dot names. Micrometer converts app.urls.created → app_urls_created_total for Prometheus automatically.

Observations API (Spring 6+)

Micrometer’s Observation API is metrics + traces in one abstraction:

Observation.createNotStarted("url.create", observationRegistry)
    .contextualName("createUrl")
    .lowCardinalityKeyValue("source", "api")
    .observe(() -> {
        // your code
    });

This emits a Micrometer timer AND an OpenTelemetry span. One abstraction, two signals. Prefer it for new code.

---

2. Structured Logging

Human-readable log lines are for dev. In production, logs are structured JSON consumed by Loki/ELK/Datadog. Grep gives way to queries like level=ERROR AND service=shortener AND traceId=abc.

Logback with JSON encoder

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>7.4</version>
</dependency>

<!-- src/main/resources/logback-spring.xml -->
<configuration>
  <springProfile name="local | dev">
    <include resource="org/springframework/boot/logging/logback/base.xml"/>
  </springProfile>

  <springProfile name="prod">
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
      <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        <includeMdcKeyName>traceId</includeMdcKeyName>
        <includeMdcKeyName>spanId</includeMdcKeyName>
        <includeMdcKeyName>userId</includeMdcKeyName>
        <customFields>{"service":"shortener","env":"${ENV:-prod}"}</customFields>
      </encoder>
    </appender>
    <root level="INFO">
      <appender-ref ref="JSON"/>
    </root>
  </springProfile>
</configuration>

Output:

{"@timestamp":"2026-07-06T14:32:01.123Z","level":"INFO","logger":"com.example.UrlService","message":"Created url","traceId":"0af7651916cd43dd","spanId":"b9c7c989f97918e1","service":"shortener","env":"prod","code":"abc123"}

Log levels in production

• ERROR — something failed that a human needs to look at. Wakes the on-call in critical cases.

• WARN — something recoverable. Rate-limited retries, degraded fallback.

• INFO — request lifecycle, state transitions. Default for prod.

• DEBUG — turned on surgically per package via /actuator/loggers when investigating.

• TRACE — almost never in prod. Even in dev, prefer step-through debugging.

Default to INFO. DEBUG on org.hibernate.SQL in prod will fill your log storage in an hour.

MDC for correlation

Always add a request ID / trace ID to MDC so every log line during a request is joinable:

@Component
public class MdcFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
            throws ServletException, IOException {
        var requestId = Optional.ofNullable(req.getHeader("X-Request-Id"))
            .orElseGet(() -> UUID.randomUUID().toString());
        MDC.put("requestId", requestId);
        try {
            res.setHeader("X-Request-Id", requestId);
            chain.doFilter(req, res);
        } finally {
            MDC.clear();
        }
    }
}

With OpenTelemetry, traceId and spanId are populated automatically — you don’t need to manage those yourself.

---

3. Distributed Tracing with OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard. It replaces Zipkin/Jaeger client libraries, replaces Spring Cloud Sleuth, and is the only tracing story Spring recommends in 2026.

Wiring

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

management:
  tracing:
    sampling:
      probability: 1.0        # 100% in dev. 0.1 or lower in prod. Higher for critical paths.
  otlp:
    tracing:
      endpoint: http://tempo:4318/v1/traces

What you get for free:

• Incoming HTTP requests are traced (a span per request, with the URL, status, duration).

• Outgoing RestClient / WebClient calls are traced (parent-child spans).

• JDBC calls are traced (via datasource-micrometer-spring-boot).

• Kafka producer/consumer are traced (via Spring Kafka + OTel).

Adding custom spans

Use the Observation API (see §1 above) or the OTel API directly. Prefer Observation — it gives you metrics too.

Propagation

OTel injects traceparent and tracestate headers on outgoing HTTP calls automatically (W3C Trace Context). Ensure downstream services accept them (any OTel-instrumented service does).

---

4. Actuator: The Production Endpoints

management:
  endpoints:
    web:
      exposure:
        include: health, info, metrics, prometheus, loggers, env, threaddump, heapdump
  endpoint:
    health:
      probes:
        enabled: true                    # /health/liveness, /health/readiness
      show-details: when-authorized
    env:
      show-values: never                 # secrets safety
  info:
    git:
      mode: full
    build:
      enabled: true

Endpoints worth knowing:

• /actuator/health — aggregate liveness. Custom HealthIndicator beans add checks (DB reachable, upstream API alive, disk space).

• /actuator/health/liveness — is the JVM alive? (K8s livenessProbe)

• /actuator/health/readiness — can it serve traffic? (K8s readinessProbe) Fails during startup and shutdown.

• /actuator/prometheus — metrics scrape endpoint.

• /actuator/loggers/{package} — change log level at runtime without redeploy. Life-saver.

• /actuator/env — effective config sources. Never expose without auth.

• /actuator/heapdump — downloads an .hprof. Analyze with Eclipse MAT or VisualVM.

• /actuator/threaddump — same for thread state. Diagnose deadlocks.

Secure Actuator in production: either firewall the port (management.server.port: 8081 bound to internal-only) or require auth on /actuator/**.

---

5. Dashboards & Alerts — What to Watch

Minimum viable Grafana dashboard for a Spring Boot service:

Panel	PromQL sketch
Request rate (req/s)	sum(rate(http_server_requests_seconds_count[1m])) by (uri)
p95 latency	histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket[5m])) by (le, uri))
Error rate	sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) by (uri)
JVM heap used	jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"}
GC pause	rate(jvm_gc_pause_seconds_sum[1m])
Hikari active	hikaricp_connections_active
Tomcat threads busy	tomcat_threads_busy_threads

Alerts on: error rate > 1%, p95 > SLO, heap > 80% for 10min, Hikari active = pool max (pool saturation).

Don’t alert on CPU. It doesn’t correlate with user-visible pain often enough. Alert on the symptoms users feel: errors and latency.

---

6. What Most People Get Wrong

• Logging inside a loop. One INFO per item across 10k items = disaster. Log once, with a count.

• High-cardinality metric tags (user id, request id, url path with ids). Prometheus explodes.

• e.printStackTrace() — goes to stderr, bypasses your log config. Always log.error("context", e).

• Confusing tracing sampling. 100% is fine in dev, wrong in prod. Head-based sampling at ingress with dynamic increase on errors is ideal.

• Health check hitting the DB on every call. Under load, this itself becomes the bottleneck. Cache health results for a few seconds.

• Deploying without a dashboard. “We’ll add metrics later” = “we’ll debug blind later.”

---

7. Local Observability Stack (docker-compose)

For practice, run the whole stack locally:

# docker-compose.observability.yml
services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml"]

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment: ["GF_AUTH_ANONYMOUS_ENABLED=true"]

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    ports: ["3200:3200", "4318:4318"]
    volumes: ["./tempo.yaml:/etc/tempo.yaml"]

  loki:
    image: grafana/loki:latest
    ports: ["3100:3100"]

Point Prometheus at host.docker.internal:8080/actuator/prometheus. Add Prometheus, Tempo, and Loki as data sources in Grafana. You now have production-grade observability on your laptop.

---

Practice Exercises

1. Add a custom Counter for a business event in a Spring Boot service. Verify it appears at /actuator/prometheus.

2. Switch on JSON logging via a prod profile. Confirm each log line has traceId and spanId.

3. Wire OpenTelemetry with OTLP export to a local Tempo. Make a RestClient call to another local service and see the parent-child spans in Grafana.

4. Add a custom HealthIndicator that reports DEGRADED when a downstream URL is slow.

5. Stretch: Grafana dashboard with the 7 panels above, tuned to your service. Screenshot for portfolio.

---

Return to README.md · Previous: 05_security_the_practical_way.md · Next: 07_config_and_deployment.md