5. A request traced

Four chapters of model and API. Now one request, followed all the way down:

table.put_item(Item={
    "PK": "CUSTOMER#42",
    "SK": "ORDER#2026-03-11#A917",
    "total": Decimal("48.20"),
    "status": "pending",
})

The item is about 300 bytes. The table is on-demand, in eu-west-1, with one global secondary index.

   THE WHOLE JOURNEY

   app ──► SDK ──► HTTPS ──► REQUEST ROUTER ──► LEADER
                             (auth, metadata)      │
                                                   ├─► replica AZ-b
                                                   └─► replica AZ-c
                             ◄── ack when leader + 1 durable

① The SDK

There is no connection to a database. The SDK builds an HTTP request:

   POST / HTTP/1.1
   Host: dynamodb.eu-west-1.amazonaws.com
   X-Amz-Target: DynamoDB_20120810.PutItem
   Authorization: AWS4-HMAC-SHA256 Credential=…, Signature=…
   Content-Type: application/x-amz-json-1.0

   {"TableName":"AppData","Item":{"PK":{"S":"CUSTOMER#42"}, …}}

   WHAT THE SDK DOES FOR YOU
     • marshals native types to the wire format (ch 2)
     • signs the request with SigV4
     • reuses a pooled HTTPS connection
     • retries throttles and 5xx with exponential
       backoff and jitter

That retry behaviour is not a detail. It is why a brief throttle looks like a small latency bump rather than an error — and why a sustained throttle looks like a latency cliff instead of a clean failure (chapter 14).

② The request router

The request lands on a fleet of stateless request routers. Each one:

   1. verifies the SigV4 signature
   2. evaluates the IAM policy — including any
      fine-grained conditions on keys or attributes
      (ch 12)
   3. looks up the table's PARTITION METADATA
   4. hashes PK = "CUSTOMER#42" to find the partition
   5. finds that partition's current LEADER
   6. checks throughput: is this request admissible?
   7. forwards it

Steps 2 and 6 are where a request fails before touching any data — AccessDeniedException and ProvisionedThroughputExceededException both come from here, which is why they are fast failures.

③ At the leader

   LEADER STORAGE NODE (AZ-a)

   • append the mutation to its write-ahead log
   • apply to the in-memory/B-tree structure for the
     partition
   • replicate to AZ-b and AZ-c
   • ACK once the leader and at least ONE replica have
     it durably

That acknowledgement rule is the durability contract: two availability zones have your write before you are told it succeeded. A single AZ failure loses nothing.

④ The index write

Because the table has a GSI, the work is not finished:

   TABLE WRITE           acknowledged to you now
        │
        │  asynchronously, within milliseconds
        ▼
   GSI WRITE             the index partition for this
                         item's index key gets its own
                         write, with its own capacity
                         cost and its own throttling
                         behaviour            (ch 7)

This is the mechanism behind one of the more alarming failure modes in the product: a throttled GSI eventually applies backpressure to the base table, because the pending index writes cannot drain. Chapter 14 covers the symptoms.

⑤ The response

{
  "ConsumedCapacity": {"TableName": "AppData", "CapacityUnits": 1.0}
}

   ASK FOR THIS, ALWAYS, IN DEVELOPMENT
     ReturnConsumedCapacity=TOTAL | INDEXES

   It turns "I think this is efficient" into a number,
   and it is the fastest way to catch a query that
   reads a megabyte to return one item.

Where the time went

   PutItem, 300 B, same region, warm connection

   ├─ SDK marshal + SigV4          ~0.1–0.3 ms
   ├─ network to endpoint          ~0.3–1   ms
   ├─ router: auth, metadata,      ~0.5–1   ms
   │  admission
   ├─ leader: log + replicate      ~2–4     ms   ◄ dominant
   └─ response                     ~0.3–1   ms
   ═══════════════════════════════════════════
   typically 5–10 ms end to end

   FIRST request on a cold Lambda: add TLS handshake,
   credential resolution and DNS — often 100 ms+.
   Reuse the client across invocations.

Tip

The single most effective latency fix in serverless code is creating the DynamoDB client outside the handler so the connection and credentials survive between invocations. It is a two-line change that routinely removes 50–150 ms from cold-path requests.

Now a Query

resp = table.query(
    KeyConditionExpression=Key("PK").eq("CUSTOMER#42")
                         & Key("SK").begins_with("ORDER#2026-"),
    ScanIndexForward=False, Limit=20,
    ReturnConsumedCapacity="TOTAL",
)

   ① router hashes PK → one partition, one leader
      (or any replica, if eventually consistent)

   ② storage node descends the B-tree to the first SK
      ≥ "ORDER#2026-" and reads sequentially backwards

   ③ stops at 20 items, 1 MB, or the end of the range

   ④ returns items + LastEvaluatedKey if more remain

   COST  total bytes READ ÷ 4 KB, halved for eventually
         consistent. 20 items × 300 B ≈ 6 KB → 2 RCU
         strongly consistent, 1 RCU eventually.

Note what did not happen: no scan of other partitions, no coordination between nodes, no dependence on table size. One hash, one tree descent, one sequential read.

The same query written badly

# no partition key: this is a Scan with extra steps
resp = table.scan(FilterExpression=Attr("PK").eq("CUSTOMER#42"))

   Scan reads the WHOLE TABLE, then filters.
   1 GB table → ~262,000 RCU, many pages, seconds to
   minutes — for the same 20 items.

Same result. Six orders of magnitude apart. ReturnConsumedCapacity makes the difference visible on the first run, which is why chapter 13 puts it in the development checklist.

A transaction traced

TransactWriteItems over two items in different partitions is a different shape of request, and worth seeing:

   client ──► request router
                 │  validates every item, checks IAM
                 │  for each, groups by partition
                 ▼
          ┌──────┴───────┐
          ▼              ▼
     LEADER (p3)     LEADER (p9)
       │  PREPARE: take the item, evaluate its
       │  condition, hold it
       └──────┬───────┘
              ▼
       all prepared? ──► COMMIT on both
       any refused?  ──► CANCEL everything
                         TransactionCanceledException
                         with a reasons array   (ch 9)

   COST      2× WCU per item — you are paying for the
             two-phase protocol
   LATENCY   roughly 2× a single write, since two
             rounds happen instead of one
   FAILURE   an item held in PREPARE that conflicts
             with another transaction causes
             TransactionConflict, not a queue

That last line is the operational point: DynamoDB cancels rather than waits. There is no lock queue to back up, and therefore no deadlock — but also no fairness, which is why one contended item turns into an endless retry loop (chapter 14).

Where retries happen, and where they do not

   THE SDK RETRIES
     ProvisionedThroughputExceededException
     ThrottlingException
     5xx / InternalServerError
     connection resets and timeouts

   THE SDK DOES NOT RETRY
     ConditionalCheckFailedException   ← your logic
     ValidationException               ← your bug
     AccessDeniedException             ← your IAM
     ResourceNotFoundException

   WHY IT MATTERS FOR CORRECTNESS

   A retried PutItem is harmless. A retried
   "ADD balance :amount" is a double credit — because
   the first attempt may have SUCCEEDED and only the
   response was lost.

   → make writes idempotent, or carry a
     ClientRequestToken on transactions      (ch 9)

The “response was lost” case is the one people forget. From the client’s point of view a timeout is indistinguishable from a failure, and only the server knows which it was.

What failure looks like mid-request

   ProvisionedThroughputExceeded   router rejected it;
                                   SDK retries       (ch 6)
   ConditionalCheckFailed          leader evaluated the
                                   condition and said no;
                                   still billed      (ch 4)
   ItemCollectionSizeLimitExceeded an LSI item collection
                                   passed 10 GB      (ch 7)
   InternalServerError (5xx)       retry with backoff;
                                   the SDK already does
   RequestLimitExceeded            account-level limit,
                                   not table-level

Distinguishing the retryable ones from the ones that will never succeed is what separates a resilient client from one that hammers a wall (chapter 12).

What to take from this chapter

• There is no connection — every operation is a signed HTTPS request carrying its whole intent, which is why the SDK’s pooling and retry behaviour matters so much.
• Stateless request routers authenticate, evaluate IAM (including key-level conditions), hash the partition key, find the leader and enforce throughput admission — so auth and throttle failures happen before any data is touched.
• The leader logs, applies and replicates, acknowledging once two availability zones hold the write durably.
• GSI writes happen asynchronously after the acknowledgement, with their own capacity and their own throttling — and a throttled index eventually backpressures the base table.
• A typical same-region request is 5–10 ms, dominated by log-and-replicate; cold serverless clients add 100 ms+, so create the client outside the handler.
• A Query is one hash, one tree descent, one sequential read, costing bytes read ÷ 4 KB, halved for eventual consistency — independent of table size.
• The same result via Scan + filter can cost six orders of magnitude more. ReturnConsumedCapacity makes that visible immediately.
• Learn which errors are retryable (throughput, 5xx) and which are final (conditional check, validation, access denied).

Next: capacity and cost — RCUs and WCUs, on-demand versus provisioned, throttling, and how to predict the bill from the access patterns.