14. Failure modes

Symptom, cause, fix. These are the failures that actually happen, roughly in order of frequency.

Throttling

   ProvisionedThroughputExceededException

   symptom: p99 latency climbs (the SDK is retrying),
            then requests fail
   causes:  ① table capacity genuinely exceeded
            ② ONE partition key over the per-partition
              ceiling — 1,000 WCU / 3,000 RCU   (ch 3)
            ③ a GSI out of capacity, backpressuring
              the table                        (ch 7)

   TELLING THEM APART
     table ConsumedCapacity ≈ Provisioned     → ①
     table consumption well below provisioned → ② or ③
     per-INDEX consumed capacity high         → ③
     Contributor Insights names one key       → ②

Only ① is solved by raising capacity. ② is a key-design problem (shard it, chapter 13), ③ is an index-capacity problem.

The hot key

   symptom: throttling while most of your capacity is
            unused; one key dominates Contributor
            Insights
   causes:  a low-cardinality partition key; a
            celebrity user; a single global counter;
            "today's date" as a key

   FIXES
     shard the key with a suffix           (ch 13)
     move the counter out of the hot path and
       aggregate from the stream           (ch 10)
     use on-demand, which adapts faster — but note
       it does NOT raise the per-partition ceiling

Silent truncation at 1 MB

resp = table.query(KeyConditionExpression=Key("PK").eq(pk))
items = resp["Items"]        # ← up to 1 MB. NOT all of them.

   symptom: "some records are missing" — usually
            reported weeks later, by a user
   cause:   a query stopping at the 1 MB page boundary
            with LastEvaluatedKey ignored     (ch 4)

Danger

This is the quietest and most damaging bug in the whole product. No error is raised, the code looks correct, and it works in every test where the data is small. Every query and scan must loop on LastEvaluatedKey — put it in a shared helper and ban the raw call in review.

Filters that cost a fortune and return nothing

   symptom: high consumed capacity, empty or tiny
            results, slow queries
   cause:   FilterExpression doing an index's job.
            you pay for every item READ, not returned.

   ScannedCount     10,000
   ReturnedItemCount     3     ◄ the tell

Fix. A GSI (chapter 7), a better sort-key prefix (chapter 8), or a sparse index that contains only the items you were filtering for.

Item size limit

   ValidationException: Item size has exceeded the
   maximum allowed size of 400 KB

   causes:  an unbounded list or map on an item —
            comments, events, history, appended
            forever by list_append
   fixes:   • each element becomes its own item under
              the same PK (that is what sort keys are
              FOR)
            • large blobs to S3, pointer in the item
            • compress, if the data is genuinely one
              opaque value

The general lesson: any attribute that grows without bound is a modelling error, and it will fail at 400 KB rather than degrading gracefully.

Unbounded item collections with an LSI

   ItemCollectionSizeLimitExceededException

If the table has any local secondary index, all items sharing a partition key are capped at 10 GB (chapter 7). Without an LSI there is no such limit.

Fix. Almost always: do not use LSIs. Where one is genuinely needed, ensure partition keys are narrow enough that no collection approaches 10 GB — and know that recovering from this requires moving items to new keys.

Transaction conflicts

   TransactionCanceledException
     reasons: [… {Code: "TransactionConflict"} …]

   symptom: rising transaction failures under load
   cause:   many transactions touching one shared item
            — a counter, a tenant summary, a status
            record
   fix:     shard the contended item, or take it out
            of the transaction and maintain it from a
            stream                          (ch 9, 10)

Retries help with brief contention and make sustained contention worse. Read CancellationReasons before retrying — a failed condition will never succeed on retry.

Read-your-own-write surprises

   symptom: an item written, then immediately not
            found
   causes:  • the read used a GSI, which is ALWAYS
              eventually consistent          (ch 7)
            • the read was eventually consistent and
              hit a lagging replica          (ch 4)
            • the read went to another region of a
              global table                   (ch 11)

Fix. Read the base table by primary key with ConsistentRead=true for anything the user expects to see immediately.

The cost explosion

   symptom: the bill is 10× last month, traffic is not
   causes, in order of how often they happen:
     ① a Scan added to a request path
     ② a new GSI multiplying every write
     ③ strongly consistent reads turned on globally
     ④ a retry storm — failures causing retries
        causing more failures
     ⑤ on-demand meeting a traffic spike
     ⑥ a backfill or migration left running

   PREVENTION
     AWS Budgets alarm on the DynamoDB line item
     ReturnConsumedCapacity in load tests
     a review of ScannedCount vs ReturnedItemCount
     adaptive retry mode, so storms damp rather than
       amplify                              (ch 12)

Reserved-word and type errors

   ValidationException: Invalid UpdateExpression:
     Attribute name is a reserved keyword; name: status

   ValidationException: Type mismatch for key PK

   ValidationException: The provided key element does
     not match the schema

   FIXES
     alias every attribute name with #      (ch 4)
     use the document client; hand-built {"N": …}
       values are how numbers become strings
     GetItem needs BOTH key parts on a composite-key
       table

Accidental table deletion

   symptom: the table is gone
   causes:  a terraform apply on the wrong workspace,
            a console mis-click, a delete-and-recreate
            triggered by a GSI change       (ch 12)

Prevention is the whole story. Deletion protection on, IAM denying dynamodb:DeleteTable outside a break-glass role, PITR enabled so a restore is at least possible, and careful reading of infrastructure plans.

Throughput on a new table

   symptom: throttling in the first minutes of a load
            test on a brand-new table
   cause:   the table starts with few partitions and
            splits as it learns your traffic (ch 3);
            auto-scaling reacts in minutes   (ch 6)
   fix:     ramp load gradually, or start on-demand,
            or provision high for the test and lower
            afterwards

This one produces a lot of false conclusions about DynamoDB’s performance, drawn from the first ninety seconds of a benchmark.

Stream consumer falling behind

   symptom: IteratorAge climbing; downstream data
            increasingly stale; eventually, records
            expire unprocessed at 24 hours
   causes:  • slow processing per record
            • one shard's worth of work exceeding one
              Lambda's throughput
            • the consumer writing back to a throttled
              table — a feedback loop
            • a poison record blocking the shard (ch 10)

   FIXES
     ParallelizationFactor up to 10 (costs per-key
       ordering across lanes)
     larger batch size + batching window
     ReportBatchItemFailures, so one bad record does
       not re-run the whole batch
     make the consumer's own writes cheaper — batch
       them, or aggregate in memory per batch

Lambda concurrency exhaustion

   symptom: unrelated functions in the account start
            throttling
   cause:   a stream with many shards, a high
            parallelisation factor, and no reserved
            concurrency — the consumer eats the
            account's concurrency budget
   fix:     set reserved concurrency on stream
            consumers. always.

TTL confusion

   symptom: "expired" items still being returned
   causes:  • TTL is BEST EFFORT — deletion can lag by
              hours                            (ch 10)
            • expiresAt written in milliseconds
              instead of SECONDS, so the timestamp is
              in the year 56,000
            • expiresAt written as a string, not a
              number — TTL silently ignores it

   FIXES
     • always filter on expiresAt in queries that must
       not see expired items
     • assert the attribute type and magnitude in
       tests: epoch SECONDS, as N

The wrong-units bug is common enough to check first: Date.now() in JavaScript returns milliseconds, and TTL wants seconds.

Clock and ordering assumptions

   symptom: items appear out of order; "latest wins"
            picks the wrong one
   causes:  • sort keys built from client clocks,
              which disagree
            • global tables' last-writer-wins using
              service timestamps you do not control
              (ch 11)
   fix:     derive ordering from a server-side value
            where it matters, or accept that ordering
            is per-item and approximate across items

What to take from this chapter

• Throttling has three causes — table capacity, the per-partition ceiling, and GSI backpressure. Distinguish them with per-index metrics and Contributor Insights before spending money.
• Hot keys are modelling failures: shard them, or move the contended aggregate to a stream consumer.
• Silent 1 MB truncation is the most damaging bug available. Always loop on LastEvaluatedKey, behind a shared helper.
• ScannedCount ≫ ReturnedItemCount means a filter is doing an index’s job; add a GSI or a sparse index.
• 400 KB item limits and 10 GB LSI item collections are hard walls — any unbounded attribute or collection is a modelling error.
• Transaction conflicts come from shared items; read CancellationReasons and never blind-retry a failed condition.
• Read-your-own-write requires the base table with ConsistentRead=true — never a GSI, never another region.
• Cost explosions come from scans on request paths, new indexes, global strong reads, retry storms, spikes on on-demand and forgotten backfills. Budget alarms and adaptive retries are the defences.
• Enable deletion protection and PITR, and read infrastructure plans before applying index changes.
• A brand-new table throttles while it splits — ramp load, or benchmark on-demand.

Next: when to use it — the honest comparison against relational databases, Cassandra, MongoDB and the rest, including the lock-in question.