CDC, event sourcing, and the outbox pattern

Why this matters

Modern systems can't operate as monolithic transactional databases. The data flowing out of your transactional system feeds: search indexes, caches, analytics warehouses, audit logs, notification systems, downstream services. You need a reliable way to get data out without dual-writing.

The patterns here — outbox, CDC, event sourcing — are the foundation of modern data architectures. Almost every interesting product uses some combination of these. Staff engineers know when to reach for each.

The dual-write problem

You want to write to your DB and also publish an event (or update a cache, send a notification, write to another DB). Naively:

// PROBLEM: not atomic
db.save(order);
kafka.publish('order.placed', order);

Three failure modes:

1. DB write succeeds, Kafka publish fails → no event for a real DB write. Downstream consumers miss the event.
2. DB write fails, Kafka publish succeeded (because publishing happened first or the failure manifested late) → phantom event with no DB backing.
3. Both succeed but in opposite orders → other consumers see inconsistent state.

You cannot solve this by retrying — retries can cause duplicate events. You cannot solve this by 2PC (two-phase commit) between Postgres and Kafka — both support XA in theory but it's operationally fragile and slow.

The solution: the outbox pattern.

The outbox pattern in depth

sequenceDiagram
    participant Client
    participant App
    participant DB as Postgres
    participant Relay as Outbox Relay
    participant Kafka

    Client->>App: place order
    App->>DB: BEGIN
    App->>DB: INSERT INTO orders
    App->>DB: INSERT INTO outbox (event)
    App->>DB: COMMIT
    DB-->>App: ok
    App-->>Client: 200 OK

    loop poll outbox
        Relay->>DB: SELECT * FROM outbox WHERE sent=false
        Relay->>Kafka: publish event
        Kafka-->>Relay: ack
        Relay->>DB: UPDATE outbox SET sent=true
    end

Atomically write to the DB and to an "outbox" table in the same transaction. A separate worker reads the outbox and publishes to Kafka.

-- Schema
CREATE TABLE outbox (
  id              BIGSERIAL PRIMARY KEY,
  aggregate_id    UUID NOT NULL,
  aggregate_type  TEXT NOT NULL,
  event_type      TEXT NOT NULL,
  payload         JSONB NOT NULL,
  partition_key   TEXT,            -- for ordered publishing
  created_at      TIMESTAMPTZ DEFAULT now(),
  processed_at    TIMESTAMPTZ
);

CREATE INDEX idx_outbox_unprocessed ON outbox (id)
  WHERE processed_at IS NULL;

-- Business write + outbox insert in same transaction
BEGIN;
  INSERT INTO orders (id, user_id, amount) VALUES ($1, $2, $3);
  INSERT INTO outbox (aggregate_id, aggregate_type, event_type, payload, partition_key)
  VALUES ($1, 'order', 'order.placed', $payload, $1::text);
COMMIT;

-- Worker (separate process)
LOOP:
  rows = SELECT * FROM outbox
         WHERE processed_at IS NULL
         ORDER BY id
         LIMIT 100
         FOR UPDATE SKIP LOCKED;

  FOR row IN rows:
    kafka.publish(row.event_type, row.payload, partition_key=row.partition_key);
    UPDATE outbox SET processed_at = now() WHERE id = row.id;

  IF rows empty: sleep 100ms
END LOOP

Why this works:

• The DB write and outbox insert are atomic — same transaction.
• If publishing fails or the worker crashes, the row is still there for the next attempt.
• FOR UPDATE SKIP LOCKED lets multiple workers consume concurrently without blocking each other.

The remaining concern: what if the worker publishes to Kafka but crashes before marking processed_at? The next worker iteration will publish the same event again. So Kafka consumers must be idempotent — deduplicate by event_id.

This is the at-least-once delivery guarantee. Exactly-once requires significantly more complexity (transactional Kafka producers, idempotent consumers with persistent dedup state) and is rarely worth it.

Outbox variants and considerations

Per-aggregate ordering

Some consumers need events for the same aggregate to arrive in order. Example: order.placed must arrive before order.shipped for the same order.

Within Kafka: messages with the same key go to the same partition; messages within a partition are ordered. So set partition_key = aggregate_id. Within a partition, the worker processes in id order. Across partitions, no global order is guaranteed.

If multiple workers consume from the outbox concurrently, they might publish events for the same aggregate out of order. Solution: each worker owns a subset of aggregate_ids (consistent hashing on the worker). Or use only one worker (simple, may not scale).

Cleanup

The outbox grows forever unless cleaned. Two strategies:

• DELETE after publish: the worker deletes the row after successful publish. Outbox stays small.
• Mark processed, periodic cleanup: keep processed rows for some retention period (audit, debugging), then bulk delete. Easier to debug "did we publish this event?"

Schema evolution

The payload is JSONB or similar. When the schema evolves, consumers must handle old + new formats. Use schema versioning (include a version field in the payload, or use a schema registry like Confluent Schema Registry with Avro/Protobuf).

CDC (Change Data Capture)

An alternative to outbox: read the database's WAL or binlog directly, decode it into row-change events, publish to Kafka. No application code change needed — the database itself becomes the event source.

How it works

Most databases maintain a write-ahead log (Postgres WAL, MySQL binlog, MongoDB oplog). CDC tools read this log and emit row-change events.

Debezium is the leading open-source CDC framework. Architecture:

Postgres ──(logical replication slot)──> Debezium connector ──(produces)──> Kafka topics
                                                                                    │
                                              ┌─────────────────────────────────────┘
                                              │
                              [Consumers: Elasticsearch sink, cache invalidator,
                                          analytics ETL, audit log, etc.]

For each row change, Debezium produces a Kafka message with the before-state, after-state, operation (insert/update/delete), and metadata (timestamp, transaction ID).

CDC vs outbox — when to use which

Outbox is better when you want to emit domain events that don't map 1:1 to row changes. CDC is better when you want a complete change feed of a database with no app code change — typical for analytics pipelines and search index sync.

Many systems use both: outbox for app-emitted domain events, CDC for analytics replication.

Event sourcing

Event sourcing inverts the relationship between events and state: events are the source of truth, current state is derived by replaying events.

Conventional system:
  State table (orders) — the truth
  Events (optional, often denormalised from state changes)

Event-sourced system:
  Events (orders.placed, orders.shipped, orders.cancelled, ...) — the truth
  State (current orders view) — derived by replaying events

Benefits:

• Complete audit trail by construction
• Can replay events to reconstruct state at any past point
• Multiple derived views (read models) from the same event stream
• Natural fit for domains with rich temporal semantics (finance, scheduling)

Costs:

• Querying current state requires replay (or maintained read models)
• Schema evolution on events is hard — old events were emitted in old schemas; you must handle all versions
• Event design is harder than table design; events have semantic meaning that's harder to refactor
• Replay can be slow (event volume × time)

When event sourcing is right: ledger systems (financial transactions, audit logs), workflow engines, domain models with rich state machines, regulatory environments requiring complete audit.

When it's over-engineering: CRUD apps, simple state where "current value" is what matters, teams unfamiliar with the patterns. Many companies have tried full event sourcing and regretted the complexity.

Practical compromise: event-driven without full event sourcing

Most production systems do "event-driven architecture, not event-sourced":

• State lives in conventional tables (the source of truth for "current value")
• Events are emitted (via outbox or CDC) on every state change
• Other systems consume events to maintain their own derived state
• You don't replay events to derive state; the tables are authoritative

This gets you most of the benefits of event sourcing (downstream pipelines, audit trail) without the operational complexity of "replay 100M events to compute today's balance."

Real-world systems

Stripe's ledger

Financial transactions are inherently event-shaped. Stripe's transaction ledger uses an event-sourced approach for transactions, with derived read models for balances. They've written about this extensively: stripe.com/blog/ledger-data-stack.

LinkedIn's Kafka usage

LinkedIn invented Kafka specifically as a "unified log" — every state change in their systems emitted to Kafka, consumed by everything downstream (search index, recommendation engines, analytics, member activity). Their architecture document is foundational: The Log: What every software engineer should know about real-time data's unifying abstraction by Jay Kreps.

Notion's blocks

Notion stores documents as blocks. Edits emit events; downstream systems (search, sync, collaboration) consume them. Their writeup: Data model behind Notion.

Applied scenarios

Further reading