Event-Driven Systems with Kafka — The Java Standard

Event-driven architecture inverts the direction of coupling: instead of service A calling service B, A emits a fact (“link created”) and any interested service consumes it. Done well, this makes systems easier to evolve, easier to scale, and easier to debug in retrospect. Done badly, it produces silent data loss and race conditions you cannot reproduce.

Kafka is the default event backbone for Java shops in 2026. This file gives you the operational understanding to use it without shooting yourself.

Kafka Fundamentals in 6 Bullets

• Topic — an append-only log, retained by time or size.

• Partition — a topic is split into N partitions. Ordering is guaranteed within a partition, never across.

• Offset — the position of a record inside a partition. Consumers commit offsets to track progress.

• Consumer group — a set of consumers that share the partitions of a topic. Each partition is read by exactly one consumer in the group at a time.

• Replication factor — each partition has R replicas across brokers. RF=3 is the production standard.

• Broker — a Kafka server. A cluster is typically 3, 5, or 7 brokers (odd numbers, for quorum).

⚠️ What most people get wrong: they assume topic-wide ordering. Kafka only guarantees ordering per partition. If ordering across a business key matters (all events for userId=42 in order), you must partition by that key.

Partitioning — The One Decision That Matters

The partition key determines both parallelism and ordering. Choose it once, live with it forever (repartitioning is painful).

Scenario	Partition key
Per-user event stream where order matters	userId
Per-order lifecycle	orderId
Metrics / analytics where order doesn’t matter	Round-robin (null key)
Hot key problem (one user = 90% of traffic)	Composite key or a shard suffix

Default partition count: start with 12 (num.partitions=12). More partitions = more parallelism but also more open file handles and longer rebalance times. 200 partitions per topic is usually the ceiling before problems.

Spring Kafka — Producer

<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>

spring:
  kafka:
    bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all                     # wait for all in-sync replicas
      properties:
        enable.idempotence: true    # exactly-once producer (see below)
        max.in.flight.requests.per.connection: 5
        retries: 2147483647          # effectively infinite; idempotence prevents duplicates
        compression.type: zstd
        linger.ms: 20                # small batching window
        batch.size: 65536

@Service
@RequiredArgsConstructor
class LinkEventPublisher {
    private final KafkaTemplate<String, LinkCreatedEvent> kafka;

    public void publishCreated(LinkCreatedEvent event) {
        // Partition by userId so all events for one user land on the same partition
        kafka.send("link.created.v1", event.userId(), event)
             .whenComplete((res, ex) -> {
                 if (ex != null) log.error("publish failed for {}", event, ex);
                 else log.debug("published to {}-{} offset {}",
                     res.getRecordMetadata().topic(),
                     res.getRecordMetadata().partition(),
                     res.getRecordMetadata().offset());
             });
    }
}

Delivery Semantics — Say It Precisely

At an study, if you say “exactly once” without qualifying, you’re wrong. The precise picture:

Semantic	Producer config	Consumer behavior	Reality
At-most-once	acks=0 or 1, no retries	Auto-commit before processing	Fast, loses data on failure. Rarely wanted.
At-least-once	acks=all, retries>0	Manual commit after processing	Default. Handler MUST be idempotent.
Exactly-once (idempotent producer)	enable.idempotence=true, acks=all	Same as above	Producer won’t duplicate under retry. Consumer still needs idempotent handler.
Exactly-once transactions (EOS)	enable.idempotence=true + transactional.id + initTransactions()	isolation.level=read_committed	End-to-end EOS within Kafka, including reads and writes across topics. Higher latency.

The honest advice: aim for at-least-once + idempotent consumers. Reach for full EOS transactions only when you’re producing derived data from Kafka input to Kafka output (Kafka Streams territory).

Spring Kafka — Consumer

spring:
  kafka:
    consumer:
      group-id: link-analytics
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "com.shortly.events"
        isolation.level: read_committed
        max.poll.records: 100
      enable-auto-commit: false      # NEVER auto-commit in production
      auto-offset-reset: earliest
    listener:
      ack-mode: manual_immediate
      concurrency: 3                 # 3 threads = up to 3 partitions in parallel

@Component
@RequiredArgsConstructor
class LinkCreatedListener {
    private final AnalyticsService analytics;

    @KafkaListener(topics = "link.created.v1", groupId = "link-analytics")
    public void onLinkCreated(ConsumerRecord<String, LinkCreatedEvent> rec, Acknowledgment ack) {
        try {
            analytics.recordCreation(rec.value());   // MUST be idempotent (upsert by eventId)
            ack.acknowledge();
        } catch (TransientException e) {
            // Don't ack; Spring will redeliver based on the retry topic config
            throw e;
        } catch (PoisonPillException e) {
            log.error("poison pill at {}-{}@{}", rec.topic(), rec.partition(), rec.offset(), e);
            ack.acknowledge();   // skip; the error handler routes it to DLT
            throw e;
        }
    }
}

Retry Topic + Dead-Letter Topic (DLT)

One of the great Spring Kafka features. Failed messages route to topic.retry with backoff, then to topic.DLT after exhaustion.

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object,Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);
    var backoff = new ExponentialBackOff(1000L, 2.0);
    backoff.setMaxInterval(30_000L);
    var handler = new DefaultErrorHandler(recoverer, backoff);
    handler.addNotRetryableExceptions(DeserializationException.class,
                                      PoisonPillException.class,
                                      IllegalArgumentException.class);
    return handler;
}

Naming convention: link.created.v1 → link.created.v1.retry → link.created.v1.DLT. Everyone knows what they mean.

⚠️ What most people get wrong: they build sophisticated retry logic and forget to monitor the DLT. A DLT with 10 million messages in it is not resilience — it’s a leak. Alert on DLT lag > 0.

Schema — Avro or Protobuf, Not JSON

JSON is fine for the first month. As soon as producers and consumers evolve independently, you need a schema registry. Confluent Schema Registry (with Avro or Protobuf) is the standard.

Format	Pros	Cons
JSON	Human readable, zero setup	No compatibility guarantees, verbose on wire
Avro	Compact, strict schema evolution rules (BACKWARD/FORWARD/FULL compatibility)	Requires registry, slightly steeper Java DX
Protobuf	Very compact, familiar to gRPC users, ubiquitous tooling	Requires registry, nullability semantics are subtle

Compatibility modes — pick BACKWARD unless you know why not:

• BACKWARD: new consumers can read old data (add-only fields with defaults).

• FORWARD: old consumers can read new data.

• FULL: both.

Kafka Streams — Stateful Processing

Kafka Streams is a library (not a cluster) that turns a Java app into a stream processor with state stores backed by internal topics. Use it for:

• Aggregations — count clicks per link per hour

• Joins — enrich order events with user data from another topic

• Windowed operations — rolling averages, sessions

@Bean
KStream<String, LinkCreatedEvent> pipeline(StreamsBuilder builder) {
    KStream<String, LinkCreatedEvent> created = builder.stream("link.created.v1");

    created
        .groupBy((k, v) -> v.userId())
        .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)))
        .count(Materialized.as("links-per-user-5m"))
        .toStream()
        .to("link.aggregate.per_user.v1", Produced.with(WindowedSerdes.timeWindowedSerdeFrom(String.class), Serdes.Long()));

    return created;
}

For most microservice use cases, plain consumers are enough. Reach for Streams when you have real stream-processing needs — don’t use it as “a nicer consumer.”

Debezium — Change Data Capture

Debezium reads the write-ahead log of your database (Postgres, MySQL, MongoDB, SQL Server) and publishes row-level changes as Kafka events. This is the right way to build the “outbox pattern” without a custom polling job.

Two dominant patterns:

1. Outbox pattern: your service writes to a local outbox table in the same transaction as the domain change. Debezium ships the outbox rows to Kafka. Guarantees the event is published if and only if the data change committed.

2. Direct CDC: Debezium ships every row change on every table. Simpler to set up, but couples consumers to your table schema — a refactor breaks them.

Prefer outbox for public events; direct CDC is fine for internal analytics pipelines.

The Alternatives — Honest Take

Kafka is the default. But it is not the only tool, and it is over-provisioned for many workloads.

Broker	Best for	Avoid when
Kafka	Event streaming, high throughput, replay, analytics, CDC	You just need a work queue with 100 msg/s
RabbitMQ (with Quorum Queues on 3.13+)	Classic queue + complex routing (topic/fanout/headers), request/reply, RPC	You need long retention or replay
Apache Pulsar	Unified streaming + queuing, geo-replication, multi-tenancy	Small team; operational complexity is real
NATS / NATS JetStream	Low-latency real-time, edge / IoT, service mesh eventing	Long retention, heavy analytics
AWS SQS / SNS	Simple decoupling on AWS, no ops	Ordering guarantees (only FIFO queues), replay
Redis Streams	Small-scale event log, already have Redis	Anything with multiple consumers at scale

Community verdict as of 2026:

• Kafka remains the default for enterprise Java event streaming.

• RabbitMQ, hardened by Quorum Queues, is having a comeback for the boring-good queueing case.

• Pulsar is “for the brave” — real strengths, real ops cost.

• NATS is winning at the low-latency edge where Kafka’s setup would be overkill.

Operational Sins Checklist

• enable.auto.commit=true in production

• No monitoring on consumer lag

• No monitoring on DLT depth

• acks=1 because “acks=all was slow” (fix the batch size instead)

• Single-partition topic that later needs parallel processing

• JSON in production without a schema registry

• Non-idempotent consumer handler on an at-least-once pipeline

• Consumer processing time > max.poll.interval.ms (default 5min) — silently kicked from the group

Practice

1. Set up a local 3-broker Kafka cluster with docker-compose + Kafka UI.

2. Build a producer that emits 1 million LinkCreatedEvent records with userId as key. Confirm partitioning by inspecting partition sizes.

3. Build a consumer with manual ack, retry topic, and DLT. Poison one message and watch the DLT.

4. Kill a broker while the pipeline runs. Prove no data loss with acks=all + RF=3.

---

Return to README.md · Previous: 01_microservices_patterns.md · Next: 03_caching_and_state.md