Transactions - Atomicity Across Services, Two Ways

Helpful context:

• Databases & Indexes - The Structures That Make Queries Fast

A bank transfer sounds simple: debit $500 from Account A, credit $500 to Account B. Two operations. What if the server crashes between them?

Without any protection, you’ve just destroyed $500. Account A is down $500; Account B received nothing. The money vanished into an inconsistent state that no business rule anticipated and no error handler can easily fix.

This problem is not hypothetical. It’s the first question anyone building a financial system confronts, and the answer - transactions - is one of the most consequential abstractions in computing. But what happens when Account A and Account B live in different databases, run by different services, in different data centers?

ACID: Not Just an Acronym

ACID (Atomicity, Consistency, Isolation, Durability) is the set of guarantees a database transaction must provide. Each property addresses a specific failure mode:

Atomicity means the transaction is all-or-nothing. If you debit Account A and then the server crashes before crediting Account B, the debit is rolled back. The database returns to its pre-transaction state, as if nothing happened. The mechanism is the write-ahead log (WAL): before modifying any data, the database writes what it’s about to do to a durable log. On crash recovery, the database replays committed transactions from the log and rolls back uncommitted ones.

Consistency means transactions take the database from one valid state to another valid state. If a constraint says balance >= 0, no transaction can leave a negative balance - the transaction is rejected rather than committed. Consistency is partially enforced by the database (constraints, foreign keys, triggers) and partially by the application (business rules that the database doesn’t know about).

Isolation means concurrent transactions behave as if they ran serially. If two transactions both try to withdraw from the same account simultaneously, neither should see the intermediate state of the other. In practice, perfect isolation is expensive, so databases offer configurable isolation levels - a spectrum from “fast but buggy” to “correct but slow.”

Durability means once a transaction commits, the data survives any subsequent failure. The database has flushed the commit record to durable storage (disk, not just memory) before acknowledging the commit. This is why databases are slow to acknowledge writes under high load - they’re waiting for fsync.

Isolation Levels: The Correctness-Performance Tradeoff

Full serializability is expensive because it requires locking or conflict detection across all concurrent transactions. Databases offer weaker levels that allow specific anomalies in exchange for higher throughput.

Read Committed (PostgreSQL default): a query sees only committed data at the moment each row is read. Two reads of the same row within one transaction may see different values if another transaction committed between them. This is called a non-repeatable read.

Repeatable Read (MySQL InnoDB default): a transaction sees a consistent snapshot of the database as of its start time. Reads within the transaction always return the same data, regardless of concurrent commits. However, the transaction may see new rows appearing that match a range query if another transaction inserts them - a phantom read.

Serializable: transactions execute as if they ran one at a time in some serial order. No anomalies. PostgreSQL implements this with Serializable Snapshot Isolation (SSI), which detects dangerous read-write conflicts between concurrent transactions and aborts one of them. Applications must retry aborted transactions.

A concrete anomaly that bites real systems: lost update. Two sessions read an account balance (100), both compute new balance (100 + deposit), both write back. The first write is overwritten by the second. Both transactions committed, but one update was silently lost. Repeatable Read prevents this in PostgreSQL (it detects the write conflict). Read Committed does not.

-- Session 1                    -- Session 2
BEGIN;                           BEGIN;
SELECT balance FROM accounts     SELECT balance FROM accounts
WHERE id = 42; -- returns 100    WHERE id = 42; -- returns 100

UPDATE accounts SET balance = 150 WHERE id = 42;
COMMIT;
                                 -- Under Read Committed, Session 2
                                 -- doesn't know Session 1 committed
                                 UPDATE accounts SET balance = 130 WHERE id = 42;
                                 COMMIT; -- Session 1's update is lost

Under Repeatable Read in PostgreSQL, Session 2’s UPDATE would fail with “could not serialize access due to concurrent update” - the application retries and reads the correct value of 150.

Optimistic vs Pessimistic Locking

Pessimistic locking (SELECT ... FOR UPDATE) acquires a row-level lock at read time. No other transaction can modify the locked rows until the current transaction commits or rolls back. Safe against concurrent modifications but reduces throughput and risks deadlock.

Optimistic locking skips the lock and instead validates at commit time. The common implementation uses a version column:

-- Read the current version
SELECT balance, version FROM accounts WHERE id = 42;
-- balance=100, version=7

-- Update only if version hasn't changed
UPDATE accounts SET balance = 150, version = 8
WHERE id = 42 AND version = 7;
-- If 0 rows updated, another transaction already modified this row - retry

Optimistic locking works well when conflicts are rare (most of the time nobody else is modifying the same row). When conflicts are frequent, the retry overhead erases the throughput advantage. For high-contention resources - a shared inventory count, a popularity counter - pessimistic locking or an atomic counter operation is more appropriate.

The Distributed Transaction Problem

A single database gives you ACID for free. Distributed systems do not.

Consider an e-commerce checkout flow across three microservices:

1. Order Service: create an order record
2. Inventory Service: reserve the items
3. Payment Service: charge the credit card

Each service has its own database. If payment fails after inventory is reserved and the order is created, who is responsible for undoing the reservation and cancelling the order? There’s no single database to roll back.

This is the distributed transaction problem, and there are exactly two serious solutions: Two-Phase Commit (strong consistency) and the Saga pattern (eventual consistency).

Two-Phase Commit (2PC)

2PC introduces a coordinator that orchestrates all participants through two phases:

Phase 1 - Prepare: The coordinator sends PREPARE to all participants. Each participant writes the transaction to its durable log (it can commit if asked) and replies READY or ABORT. This vote is durable - the participant is promising that it can commit.

Phase 2 - Commit or Abort: If all participants voted READY, the coordinator writes a commit record to its own durable log (this is the point of no return) and sends COMMIT to all participants. If any participant voted ABORT, the coordinator sends ABORT to all. Participants apply or discard their prepared changes accordingly.

The appeal is strong: if 2PC completes, all participants either committed or all aborted. Global atomicity.

The catastrophic problem: 2PC is a blocking protocol. If the coordinator crashes after sending PREPARE but before sending COMMIT, participants are stuck. They voted READY - they promised to commit - but they can’t unilaterally decide whether to commit or abort because they don’t know how other participants voted. They must hold locks indefinitely, waiting for the coordinator to recover.

This is not a theoretical edge case. In a distributed system with hundreds of services, coordinator crashes and network partitions happen regularly. The blocking failure mode - where a crashed coordinator freezes all participants - is why 2PC is described as “the protocol that proves distributed transactions are fundamentally hard.”

2PC is implemented as XA transactions in most enterprise databases (PostgreSQL, MySQL, Oracle). It’s appropriate when you control all participants, all are databases that support XA, the operation completes in milliseconds, and coordinator failure is truly rare. It’s rarely the right choice for cross-service coordination over HTTP in microservices.

The Saga Pattern

A Saga breaks a distributed transaction into a sequence of local transactions, each within a single service, each individually atomic using that service’s local database. If a step fails, previously completed steps are reversed using compensating transactions.

The e-commerce checkout as a Saga:

1. Order Service: create order record (local transaction) → success
2. Inventory Service: reserve items (local transaction) → success
3. Payment Service: charge card (local transaction) → FAILS (card declined)
4. Inventory Service: release reservation (compensating transaction)
5. Order Service: mark order as failed (compensating transaction)

Compensating transactions are not rollbacks. They’re new forward transactions that undo the visible effects of a prior step. An inventory reservation was committed to the database; you can’t roll it back. You release it with a new transaction. This distinction matters: compensating transactions must account for the state changes that happened after the original transaction committed.

Critical requirement: compensating transactions must be idempotent. The saga orchestrator may crash and retry. If POST /inventory/release/reservation-123 is called twice, the second call must succeed without double-releasing. This is usually implemented by checking whether the compensation has already been applied before applying it.

Choreography: Decentralized Sagas

In choreography, each service reacts to events from the previous step and emits events for the next:

Order Service emits: OrderCreated
  → Inventory Service hears it, reserves items, emits: InventoryReserved
  → Payment Service hears it, charges card, emits: PaymentCharged or PaymentFailed
  → If PaymentFailed:
      → Inventory Service hears it, releases reservation, emits: InventoryReleased
      → Order Service hears it, marks order failed

No central coordinator. Each service is autonomous. The overall transaction flow is distributed across event handlers.

Pros: no single point of failure; services are loosely coupled; adding a new step only requires adding a new event listener.
Cons: the overall flow is implicit - it’s invisible in any one service’s code. Tracing a distributed flow requires correlating events across multiple services' logs. Testing requires simulating the entire event chain.

Orchestration: Centralized Sagas

An orchestrator (sometimes called a process manager or saga coordinator) explicitly commands each service and tracks the saga’s state:

Orchestrator persists state: {saga_id: "abc", step: "reserve_inventory", status: "in_progress"}

→ POST /inventory/reserve {order_id: 123, items: [...]}
← 200 OK: {reservation_id: "res-456"}
Orchestrator updates state: {step: "charge_payment", reservation_id: "res-456"}

→ POST /payments/charge {order_id: 123, amount: 99.99}
← 402 Payment Required: {error: "card_declined"}
Orchestrator updates state: {step: "compensating", failed_at: "charge_payment"}

→ DELETE /inventory/reservations/res-456
← 200 OK
→ PATCH /orders/123 {status: "failed"}
← 200 OK
Orchestrator updates state: {step: "complete", outcome: "failed"}

Pros: the flow is explicit and visible in one place; easy to add tracing, monitoring, and retry logic; failure modes are easier to reason about.
Cons: the orchestrator is a new service to build and operate; it becomes a partial coupling point (it knows about all participants); it’s a single point of failure unless itself made reliable.

AWS Step Functions is a managed orchestrator. You define state machine transitions in JSON; Step Functions handles persistence, retries, and compensation logic. Lambda functions implement each step. The managed infrastructure removes the need to build and operate the orchestrator yourself - at the cost of AWS lock-in and per-transition pricing.

2PC vs Saga: When to Use Which

Use 2PC when all participants are databases you control with XA support, the operation is short-lived, and you need true atomicity with no intermediate state visible. Use Saga when the transaction spans independent microservices, involves external APIs (payment processors, email providers) that can’t participate in XA, or represents a long-running business process.

The Critique: Saga’s Hidden Complexity

Sagas introduce complexity that’s easy to underestimate. Compensating transactions are business logic. Deciding how to compensate a failed payment requires understanding business rules: should inventory be released immediately or held in case the customer retries with a different card? Should the order be marked failed or pending-retry? These decisions don’t live in a library - they’re application code that must be written, tested, and maintained.

Eventual consistency creates user experience problems. Between the time an order is created and the time payment fails and compensation completes, the user might see an order in “processing” state. If the saga takes several minutes to compensate (perhaps an external API is slow to respond), the user sees an order that looks active but will eventually be cancelled. Designing UX around eventually consistent state requires careful thought about which states are meaningful to display and when.

Debugging a failed saga in production is non-trivial. Which step failed? Were compensating transactions applied? Were they idempotent-safe? Distributed tracing (OpenTelemetry, AWS X-Ray) that correlates events across services by saga ID is essentially mandatory for production sagas.

The idempotency requirement propagates through your entire service: every compensation endpoint must be idempotent, which means every downstream database operation must be idempotent, which often means adding “compensation already applied” checks everywhere. This is more code, more tests, more surface area for bugs.

Future Outlook

Google Spanner implements serializable isolation globally using TrueTime - hardware GPS and atomic clocks that provide a globally bounded clock with sub-10ms uncertainty. Spanner waits out the clock uncertainty before committing, ensuring that even in a globally distributed transaction, the commit timestamp is after all events that causally preceded it. This gives Spanner external consistency: if transaction A commits before transaction B starts (according to real time), B sees A’s writes. This is stronger than serializable - no distributed system without TrueTime can offer it cheaply.

CockroachDB achieves serializable isolation globally without TrueTime, using Hybrid Logical Clocks (HLC) and MVCC. It implements a distributed 2PC-like protocol but with a consensus layer (Raft) per partition that eliminates the blocking coordinator failure mode. CockroachDB can survive the loss of any single node without stalling transactions.

The long arc here is: distributed transactions are genuinely hard, 2PC is too fragile for most distributed systems, Saga is operationally complex, and NewSQL databases (Spanner, CockroachDB) are attempting to provide ACID across distributed nodes so application developers don’t have to implement Sagas at all. As these systems mature and become more widely available, the choice between 2PC and Saga may become less relevant - replaced by “use a database that handles it.”

Read Next:

• Consistency & CAP - The Tradeoff Distributed Systems Cannot Escape