Design a payment system (Stripe-class)

The setup

"Build a payment processing system. APIs for charges, refunds, and transfers. 1000 transactions per second; never double-charge a customer; survive failures gracefully. Money is involved — correctness is non-negotiable."

This is the canonical case where correctness dominates everything else. Performance, throughput, even availability are secondary. A payment system that loses one $5 transaction is broken; one that handles 10× the spec but allows occasional double-charges is unshippable.

Clarifying questions

The non-negotiables

1. Idempotency: a retry must not double-charge. Required by every payments API.
2. Auditability: every state change traceable. Required by financial regulators.
3. Reconcilability: every charge must reconcile with the bank's view. Required for accounting.
4. Atomicity at business level: a "purchase" creates a charge AND deducts inventory AND ships. These don't atomic-DB-commit together but must be logically atomic via sagas.

Design Decision 1: the ledger

At the core of every serious payment system is the ledger — an append-only record of every transaction. Stripe, banks, accounting systems all use double-entry accounting principles:

ledger_entries (append-only):
  id, transaction_id, account_id, amount, side (debit/credit),
  currency, created_at, idempotency_key, related_transaction_id

Constraint: for any transaction_id,
  SUM(amount where side=debit) = SUM(amount where side=credit)

Every transaction has both a debit and a credit. Money doesn't appear or
disappear — it moves between accounts. This is foundational.

A charge: debit customer's payment source account, credit merchant's account. A refund: opposite direction.

The ledger is append-only. You never update or delete a row. Corrections are new transactions that net against earlier ones.

Design Decision 2: idempotency

Client sends a charge with an Idempotency-Key header (UUID generated client-side). Server:

BEGIN TRANSACTION;

  -- Atomic claim-or-fetch
  INSERT INTO idempotency_keys (
    key, request_hash, status, response, created_at, expires_at
  )
  VALUES ($key, $hash, 'pending', null, now(), now() + interval '24h')
  ON CONFLICT (key) DO UPDATE SET key = idempotency_keys.key
  RETURNING *;

  -- If status='complete': return stored response (replay).
  -- If status='pending' and request_hash matches: another request is processing.
  --   Either wait/poll or return 409 conflict.
  -- If new row created: we own this request; process it.

COMMIT;

-- Process the charge (call card network, etc.)
result = call_card_network(amount, source, merchant);

BEGIN TRANSACTION;
  -- Write ledger entries atomically
  INSERT INTO ledger_entries (transaction_id, ...) VALUES (...);  -- debit
  INSERT INTO ledger_entries (transaction_id, ...) VALUES (...);  -- credit
  -- Update idempotency key
  UPDATE idempotency_keys SET status='complete', response=$result WHERE key=$key;
COMMIT;

RETURN result;

Retries with the same key return the same response without re-charging.

Design Decision 3: handling external state — the card network

Design Decision 4: reconciliation

At end of each day, you receive settlement files from each bank — what they actually paid out, what your customers actually charged. The ledger must match.

Reconciliation runs nightly. Mismatches generate alerts; humans investigate. Mismatch rate < 0.001% is healthy; rising mismatch rate is a critical alarm.

The architecture

sequenceDiagram
    participant C as Client
    participant PS as Payment Service
    participant L as Ledger DB
    participant PG as Payment Gateway
    participant B as Bank / Card network

    C->>PS: POST /pay { idempotency_key=K, amount }
    PS->>L: lookup K
    alt already processed
        L-->>PS: cached result
        PS-->>C: same response (idempotent)
    else first time
        PS->>L: insert PENDING with K
        PS->>PG: authorize + capture
        PG->>B: charge
        B-->>PG: success / auth_id
        PG-->>PS: success
        PS->>L: UPDATE to SETTLED + auth_id
        PS-->>C: 200 OK
    end

Client ──API call──► [Charge Service]
                          │
                          ├──► [Idempotency check] (Postgres)
                          │
                          ├──► [Card network adapter] (Visa/Mastercard/etc.)
                          │
                          ├──► [Ledger writer] (Postgres / immutable store)
                          │
                          └──► Return result

Background:
  [Settlement reconciliation job] reads daily bank files,
  compares to ledger, alerts on discrepancies

  [Webhook dispatcher] sends payment events to merchant webhooks
  (signed, idempotent, retried)

Cross-examination round 1: failure modes

Cross-examination round 2: distributed atomicity

Takeaway

Payment systems are the canonical case where correctness dominates everything else. Idempotency, double-entry accounting, reconciliation aren't optional — they're the system's reason for being. The architectural lesson: when money is at stake, make every state transition idempotent and every record append-only. Mutability is the enemy.

The deeper insight: external systems aren't transactional. The card network can't participate in your DB transaction. You must design for the case where you've sent the request but don't know if it succeeded. Idempotency keys propagated all the way through are the only safe pattern.