13. Tuning and operations

There is no server to tune. What you have instead are modelling decisions, client settings and a bill — and in DynamoDB those three are the same subject, because cost is a direct function of how you access data.

The diagnostic order

   ① IS IT A HOT KEY?          one partition key taking
                               a disproportionate share
                               → Contributor Insights  (ch 3)

   ② IS IT AN INDEX?           GSI throttling, which
                               backpressures the table
                               → per-index metrics     (ch 7)

   ③ IS IT CAPACITY?           sustained consumption at
                               the provisioned ceiling
                               → raise it, or go on-demand

   ④ IS IT THE ACCESS PATTERN? scans, filters, huge
                               items, N+1 requests
                               → ScannedCount vs Returned

   ⑤ IS IT THE CLIENT?         cold connections, no
                               pooling, bad retries
                               → chapter 5, chapter 12

Only ③ is solved by money, and it is the one people reach for first.

Item size is a lever

   WRITES round up to 1 KB, READS to 4 KB   (ch 6)

   1,000,000 events × 200 bytes
     as separate items  → 1,000,000 WCU
     batched 5 per item → 200,000 WCU   (5× cheaper)

   one 300 KB item
     read in full       → 75 RCU per read
     split into parts,
     read one part      → a few RCU

   RULES OF THUMB
     • keep items under ~4 KB if they are read often —
       one RCU per read
     • split rarely-read attributes into a separate
       item with the same PK, fetched only when needed
     • store big blobs in S3 and keep a pointer  (ch 2)
     • shorten attribute NAMES — they count toward the
       item size, on every item, forever

That last one looks like a micro-optimisation and is not: "description" versus "d" across a hundred million items is real money and real latency.

Sharding a hot key

# write: spread over N shards
shard = random.randint(0, 9)
table.put_item(Item={"PK": f"PRODUCT#123#{shard}", "SK": f"VIEW#{ts}", …})

# read: fan out and merge
results = []
for shard in range(10):
    results += table.query(KeyConditionExpression=Key("PK")
                           .eq(f"PRODUCT#123#{shard}"))["Items"]

   COST         10× the read requests for a full fan-out
   BENEFIT      10× the write ceiling on that logical key
   CHOOSE N     the smallest number that clears the
                1,000 WCU per-partition ceiling (ch 3)

   CALCULATED SHARDS beat random ones when you can
   derive the shard from something the reader knows
   (a user id, a date), because then a read needs ONE
   query rather than N.

Pagination, done correctly

kwargs = {"KeyConditionExpression": Key("PK").eq(pk), "Limit": 50}
while True:
    resp = table.query(**kwargs)
    yield from resp["Items"]
    if "LastEvaluatedKey" not in resp:
        break
    kwargs["ExclusiveStartKey"] = resp["LastEvaluatedKey"]

   THREE BUGS THIS PREVENTS
     • stopping on an empty page — filters can produce
       empty pages with more data behind them   (ch 4)
     • assuming Limit means "give me 50 results" —
       it means "read at most 50 ITEMS", before
       filtering
     • losing data when the 1 MB boundary falls
       mid-collection

For user-facing pagination, pass an encoded LastEvaluatedKey back to the client as an opaque cursor. Offset pagination (page=7) does not exist here and cannot be emulated cheaply.

Batching and parallelism

   BatchGetItem      100 items, one round trip
   BatchWriteItem    25 items, one round trip
   parallel Scan     N segments, for migrations only

   Batches do not reduce CAPACITY cost — they reduce
   LATENCY and connection overhead. Always retry
   UnprocessedItems with backoff.             (ch 4)

   THE N+1 PROBLEM, DYNAMODB EDITION
     query 50 orders  → 1 request
     then GetItem the customer for each  → 50 requests

   FIX: denormalise the customer name onto the order
        (ch 8), or BatchGetItem the 50 customers in
        one call.

Choosing capacity mode, practically

   START                 on-demand. you do not know
                         your traffic yet.

   AFTER 2–4 WEEKS       look at ConsumedCapacity.
                         steady? switch to provisioned
                         with auto-scaling at ~70%
                         target utilisation.

   PREDICTABLE PEAKS     scheduled scaling ahead of
                         the peak, not reactive.

   STEADY AND LARGE      reserved capacity on the
                         baseline, auto-scaling above.

   ANY SPIKY WORKLOAD    stay on-demand. the premium
                         is cheaper than the throttles.

Reducing cost without changing behaviour

   □ eventually consistent reads wherever correctness
     allows                       → 50% off those reads
   □ TTL instead of DeleteItem    → free deletions
   □ drop unused GSIs             → every write pays
                                    for each index
   □ KEYS_ONLY or INCLUDE
     projections                  → smaller index writes
   □ batch small items together   → fewer 1 KB roundings
   □ shorter attribute names      → smaller everything
   □ export to S3 for analytics   → no table capacity
   □ provisioned + reserved for
     steady load                  → the largest single
                                    saving available

Tip

Run the list above as an actual review, with numbers from ReturnConsumedCapacity and CloudWatch, twice a year. Teams routinely find 40–70% savings on mature tables, and almost all of it comes from those eight lines rather than from anything clever.

Migrations

   ADDING AN ACCESS PATTERN      new GSI, backfills in
                                 background         (ch 7)

   CHANGING ITEM SHAPE           write new attributes
                                 alongside old; migrate
                                 readers; backfill via
                                 a scan or stream;
                                 remove old

   CHANGING THE KEY SCHEMA       new table + dual write
                                 + backfill + cutover.
                                 the expensive one.

   BACKFILL TOOL OF CHOICE       a parallel Scan feeding
                                 BatchWriteItem, rate
                                 limited so it does not
                                 starve production

# rate-limited backfill: leave headroom for real traffic
for segment in range(TOTAL_SEGMENTS):
    for page in paginate_scan(segment=segment, total_segments=TOTAL_SEGMENTS):
        write_batch(transform(page))
        time.sleep(pace)      # keep consumed capacity under target

Capacity planning, worked

An API with a known shape, planned before launch.

   TRAFFIC
     200,000 daily active users
     each: 20 reads (1 KB), 4 writes (0.5 KB)
     peak hour holds 15% of the day's traffic

   READS
     4,000,000/day. peak hour = 600,000
     → 167/s average in the peak hour
     1 KB eventually consistent = 0.5 RCU
     → ~84 RCU at peak

   WRITES
     800,000/day. peak hour = 120,000
     → 33/s, at 1 WCU each (0.5 KB rounds up)
     → ~33 WCU at peak
     plus ONE GSI, projection INCLUDE
     → ~66 WCU total                        (ch 6)

   PROVISION
     auto-scaling, target 70%:
       reads  min 20  max 200
       writes min 20  max 150
     …and check the numbers again after two weeks of
     real traffic.

   SANITY CHECKS BEFORE BELIEVING ANY OF IT
     □ is any single partition key taking more than
       ~1,000 WCU or 3,000 RCU of that?      (ch 3)
     □ does the peak assumption match reality, or a
       marketing plan?
     □ have you costed the GSI writes separately?
     □ what happens on the day traffic is 10×?

Load testing that tells the truth

   ① RAMP, do not step. a cold table splits as it
      learns; instant full load throttles for reasons
      that will not recur.                    (ch 14)
   ② use REALISTIC KEYS. a load test with random
      UUIDs will never reproduce the hot key your
      real traffic has.
   ③ measure ConsumedCapacity, not just latency.
      latency is guaranteed; cost is the variable.
   ④ test the FAILURE path — what does your
      application do when throttled for 30 seconds?
   ⑤ run long enough for auto-scaling to act, if you
      are testing provisioned capacity.

Point ② is where most load tests go wrong: uniformly random keys produce a beautifully flat capacity graph that tells you nothing about the celebrity user who will arrive on launch day.

The pre-launch checklist

   MODEL
   □ access patterns written down beside the table
   □ no request path performs a Scan
   □ partition keys have high cardinality and even
     access
   □ item sizes known; nothing near 400 KB

   CAPACITY
   □ capacity mode chosen deliberately
   □ alarms on throttles, per table AND per index
   □ peak traffic estimated and costed

   RESILIENCE
   □ PITR on, deletion protection on          (ch 11)
   □ table names from configuration
   □ a restore practised and timed
   □ SDK retries adaptive; timeouts short     (ch 12)
   □ writes idempotent                        (ch 9)

   OPERATIONS
   □ TTL configured where data has a lifetime
   □ streams enabled if anything downstream needs them
   □ Contributor Insights on for the first weeks
   □ dashboards: latency p99, consumed vs provisioned,
     throttles, ScannedCount vs ReturnedItemCount

What to take from this chapter

• Diagnose in order: hot key, index, capacity, access pattern, client. Only capacity is fixed by money, and it is the first thing people try.
• Item size is a lever in both directions: batch small items to avoid per-item rounding, split or offload large ones, and shorten attribute names.
• Shard hot keys with the smallest suffix count that clears the per-partition ceiling, and prefer calculated shards so reads need one query rather than a fan-out.
• Paginate until LastEvaluatedKey is absent; Limit counts items read, not results returned, and offset pagination does not exist.
• Batches cut latency, not capacity, and always require retrying UnprocessedItems. Watch for the N+1 pattern and fix it by denormalising or batching.
• Capacity mode: start on-demand, move to provisioned plus auto-scaling at ~70% once traffic is understood, add reserved capacity for steady baselines, stay on-demand for spiky traffic.
• The eight-line cost review — eventual reads, TTL, unused GSIs, projections, batching, attribute names, S3 exports, reserved capacity — routinely finds large savings.
• Key-schema changes mean a new table, dual writes and a rate-limited backfill. Plan access patterns before launch to avoid it.

Next: failure modes — throttles, hot keys, silent truncation, cost explosions, and what each one looks like from the outside.