The Outbox Pattern: Reliable Data Consistency in Microservices with Spring Boot and Kafka
- Thalles Vieira
- 6 de mai.
- 6 min de leitura
A practical look at how to solve the dual-write problem between your database and your message broker, without falling back on fragile homemade workarounds.
Introduction
Anyone who has worked with microservices for a while has lived through this nightmare. A service persists something to the database, then publishes an event to Kafka so other services can react to the change. Everything works locally. In production, the broker hiccups for two seconds, the application restarts mid-operation, or the network stutters, and you end up with the worst of both worlds: the database says one thing and Kafka says another.
This is the classic dual-write problem. Many teams try to fix it with fancier try/catch blocks, but that approach simply does not work. The correct solution has a name, and it has been around for years: the Transactional Outbox Pattern.
In this article I walk through the problem, the solution, a concrete implementation in Spring Boot with Kotlin, and the trade-offs you need to weigh before taking it to production.
1. The Problem: Dual-Write Is Not Atomic
Consider a common scenario. An endpoint creates an order and emits an OrderCreated event.
@Service
class OrderService(
private val repository: OrderRepository,
private val kafkaTemplate: KafkaTemplate<String, OrderCreatedEvent>
) {
@Transactional
fun create(command: CreateOrderCommand): Order {
val order = repository.save(Order.from(command)) // 1. write to the DB
kafkaTemplate.send("orders", OrderCreatedEvent.from(order)) // 2. publish to Kafka
return order
}
}
At first glance it looks correct. It is wrapped in @Transactional, after all. The catch is that Spring's @Transactional only controls the database transaction. Kafka is not part of it, and that is exactly where the problem lives.
The dual-write problem
The failure scenarios are real, and they happen in production:
The DB commits, Kafka fails. The order exists, but no other service knows about it. Did the customer get a confirmation email? No. Was the inventory reserved? Also no.
Kafka publishes, the DB rolls back. An OrderCreated event was emitted for an order that never existed. Otherwise-correct consumers will react to a phantom order.
The application crashes between the two operations. Completely undefined state.
You might think about reaching for XA transactions or two-phase commit. In theory, that works. In practice, almost nobody does it because the operational cost is enormous, most modern brokers do not support it well, and performance tanks. It is no accident that the community has converged on a different approach.
2. The Solution: Transactional Outbox
The idea is elegant and simple. If you cannot atomically commit to two systems, then perform both writes in the same system.
Instead of publishing to Kafka during the business operation, you write the event to an outbox table inside the same database transaction. A separate process, called the message relay, reads from that table and publishes to Kafka asynchronously.
Outbox Pattern architecture
The key points are:
The transaction that persists the order also persists the event. If one fails, both fail. ACID does the heavy lifting for you.
The message relay reads the outbox, publishes to Kafka, and marks the event as published. If it crashes before marking, the event will be republished, which is why idempotent consumers matter.
The delivery contract becomes at-least-once. An event may be delivered more than once, but never fewer.
3. Practical Implementation with Spring Boot and Kotlin
Let's get to the code. The solution has three pieces: the outbox entity, the business operation, and the relay.
3.1 The Table and the Entity
CREATE TABLE outbox_event (
id UUID PRIMARY KEY,
aggregate_type VARCHAR(100) NOT NULL,
aggregate_id VARCHAR(100) NOT NULL,
event_type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
published_at TIMESTAMP
);
CREATE INDEX idx_outbox_pending
ON outbox_event (created_at)
WHERE published_at IS NULL;
The partial index matters. It only covers pending events, so the index stays small even after millions of events have been processed.
@Entity
@Table(name = "outbox_event")
class OutboxEvent(
@Id val id: UUID = UUID.randomUUID(),
val aggregateType: String,
val aggregateId: String,
val eventType: String,
@Column(columnDefinition = "jsonb")
val payload: String,
val createdAt: Instant = Instant.now(),
var publishedAt: Instant? = null
)
3.2 The Domain Service
@Service
class OrderService(
private val orderRepository: OrderRepository,
private val outboxRepository: OutboxEventRepository,
private val objectMapper: ObjectMapper
) {
@Transactional
fun create(command: CreateOrderCommand): Order {
val order = orderRepository.save(Order.from(command))
val event = OutboxEvent(
aggregateType = "Order",
aggregateId = order.id.toString(),
eventType = "OrderCreated",
payload = objectMapper.writeValueAsString(OrderCreatedEvent.from(order))
)
outboxRepository.save(event)
return order
}
}
Notice that there is no KafkaTemplate in this class. The business service does not know about the broker at all. That aligns nicely with Clean Architecture and DDD: the domain layer speaks in domain events, not in messaging infrastructure.
3.3 The Message Relay (Polling Strategy)
@Component
class OutboxRelay(
private val outboxRepository: OutboxEventRepository,
private val kafkaTemplate: KafkaTemplate<String, String>
) {
@Scheduled(fixedDelay = 1_000)
@Transactional
fun publishPending() {
val pending = outboxRepository.findPendingForUpdate(PageRequest.of(0, 100))
pending.forEach { event ->
try {
kafkaTemplate.send(
topicFor(event.eventType),
event.aggregateId,
event.payload
).get(5, TimeUnit.SECONDS)
event.publishedAt = Instant.now()
} catch (ex: Exception) {
log.error("Failed to publish event ${'$'}{event.id}", ex)
// do not mark as published, so it gets retried on the next cycle
}
}
}
}
And the repository, with SELECT ... FOR UPDATE SKIP LOCKED, which is essential when you run multiple instances of the service and want to avoid them stepping on the same events.
interface OutboxEventRepository : JpaRepository<OutboxEvent, UUID> {
@Query("""
SELECT e FROM OutboxEvent e
WHERE e.publishedAt IS NULL
ORDER BY e.createdAt
""")
@Lock(LockModeType.PESSIMISTIC_WRITE)
@QueryHints(QueryHint(name = "jakarta.persistence.lock.timeout", value = "0"))
fun findPendingForUpdate(pageable: Pageable): List<OutboxEvent>
}
With SKIP LOCKED (configured here via the zero lock timeout on PostgreSQL), each instance grabs a different batch of events. No blocking, no duplicated work.
4. The Full Flow
Execution flow
The separation between the synchronous path (the client request) and the asynchronous path (the relay) is what makes the pattern robust. The client gets 201 Created as soon as the database transaction commits, and does not have to wait for Kafka. Even if Kafka is unavailable for minutes, events will be published once the broker comes back online.
5. Polling vs CDC: Which One Should You Pick?
There are two strategies for the relay. The one I showed uses polling, where a scheduler queries the table periodically. The alternative is Change Data Capture (CDC), typically with Debezium tailing the PostgreSQL WAL or the MySQL binlog.
Polling vs CDC
My practical recommendation, after deploying both in production, is this:
Start with polling. It solves 90% of cases, is trivial to operate, and does not add Kafka Connect or Debezium to your stack.
Move to CDC when volume justifies it. That usually happens when you are publishing thousands of events per second, or when polling's few-second latency starts to hurt.
If you are on AWS, it is worth remembering that Amazon MSK Connect already offers managed Debezium, which significantly reduces the friction of adoption.
6. Observability Is Not Optional
Outbox without observability is a ticking time bomb. At a minimum, expose these metrics in Prometheus and Grafana:
outbox_pending_count: number of unpublished events. Alert if it stays above N for more than M minutes.
outbox_oldest_pending_age_seconds: age of the oldest pending event. If it grows, the relay is stuck.
outbox_publish_duration_seconds: histogram of publish duration. Spikes indicate broker pressure.
outbox_publish_failures_total: failure counter, labeled by event type.
@Component
class OutboxMetrics(meterRegistry: MeterRegistry) {
private val publishLatency = Timer.builder("outbox.publish.duration")
.publishPercentileHistogram()
.register(meterRegistry)
fun recordPublish(block: () -> Unit) = publishLatency.recordCallable(block)
}
A simple Grafana dashboard with these four panels is enough to give you confidence to run this 24/7.
7. Gotchas That Save You from On-Call Pages
A few hard-earned lessons from running this in production.
Consumer idempotency. Because the pattern is at-least-once, events can arrive duplicated. Always include a unique eventId and have a deduplication strategy on the consumer side. That can be a processed-events table, or an INSERT ... ON CONFLICT DO NOTHING on the derived write.
Outbox cleanup. The table grows without bound. Run a job that removes events published more than N days ago. Something like 7 to 30 days is usually safe and gives you room to investigate.
Ordering via partition key. Use aggregateId as the Kafka message key. That way, all events for the same aggregate land in the same partition and are consumed in order.
Schema Registry. If you are in a serious environment, use Avro or Protobuf with a Schema Registry. JSON is fine to start with, but it tends to evolve poorly over time.
Don't publish JPA entities. Create explicit DTOs for your events. The entity may change,
but the event contract must not break external consumers.
Conclusion
The Outbox Pattern is not new. It has been discussed for over a decade. Even so, I keep finding production systems trying to "solve" the dual-write problem with retries, manual compensations, or worse, ignoring it until it explodes on a Sunday afternoon.
The good news is that the implementation is straightforward: one table, one transaction in the right place, and a scheduler. The ROI in stability is enormous, and the cost of adoption is low.
If you are building microservices and still use @Transactional followed by kafkaTemplate.send(), take this as a friendly warning. Refactor it before production teaches you the lesson at the worst possible moment.
Comentários