S3 Eventual Consistency Edge Cases in Data Pipelines

>GLB_ // tools · Amazon S3

First, a correction to the title, because a lot of blog posts and Stack Overflow answers are still out of date: Amazon S3 is no longer eventually consistent for object operations. Since December 2020, S3 provides strong read-after-write consistency for PUT and DELETE of objects, and for LIST, in all regions, at no extra cost.

So S3Guard, emrfs consistent view and “sleep 30 then list” hacks exist to solve a problem that’s gone.

But pipelines on S3 still break in ways that look like eventual consistency. The cause is usually one of the semantics below. Strong consistency means a read sees the latest successful write. It doesn’t mean S3 is a filesystem, a transaction manager, or a queue.

1. There is no atomic rename

S3 has no rename. A “rename” in Hadoop/Spark is COPY + DELETE, per object, non-atomic, O(data size).

Why it matters: the classic Hadoop FileOutputCommitter writes to _temporary/ and renames on commit. On S3 that means:

• Commit time grows with output size.
• A job killed mid-commit leaves partial output in the final location.
• Readers can observe a half-committed directory.

Fixes:

• Use a table format (Delta, Iceberg, Hudi). The commit is a single metadata write; data files that aren’t referenced by the log/metadata are invisible to readers.
• For plain Parquet output from Spark on S3A, use the S3A committers (magic, directory, partitioned), which use multipart upload completion instead of rename.
• Never treat “the directory exists” or “files are present” as “the job finished”. Use a _SUCCESS marker at minimum, a table commit ideally.

2. Concurrent writers: last writer wins

Two writers PUT the same key at the same time → both succeed, the later one (by S3’s internal timestamp) wins. No error, no merge.

Classic victim: Delta Lake with multiple clusters writing to the same table on S3. Delta commits by creating _delta_log/00000000000000000042.json and requires that the create fails if the file already exists (“put-if-absent”). Historically S3 couldn’t guarantee that, so Delta’s default S3 LogStore is only safe for a single writer (single driver). Concurrent writers from different clusters can silently overwrite each other’s commit → lost data.

Options:

• S3DynamoDBLogStore (Delta OSS): uses a DynamoDB table to arbitrate commits across clusters.

  spark.delta.logStore.s3.impl=io.delta.storage.S3DynamoDBLogStore
  spark.delta.logStore.s3a.impl=io.delta.storage.S3DynamoDBLogStore
  spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName=delta_log
  spark.io.delta.storage.S3DynamoDBLogStore.ddb.region=us-east-1

• A managed platform/catalog that coordinates commits for you.

• Or: architect for one writer per table.

S3 now supports conditional writes: If-None-Match: * (create only if the key doesn’t exist, since August 2024) and If-Match: <etag> (compare-and-swap, since November 2024). This is the primitive table formats need. Check your table format/engine version for whether it uses it; don’t assume.

Your own code can use it directly:

aws s3api put-object --bucket lake --key locks/job-2026-10-08 \
  --body /dev/null --if-none-match '*'
# 412 Precondition Failed if another run already created it

3. LastModified is not “when the data became visible”

Incremental loaders often do: list objects with LastModified > last_watermark. Two problems:

• For multipart uploads, LastModified reflects when the upload was initiated, not completed. A 20 GB file that starts uploading at 10:00 and completes at 10:12 is listed with 10:00. If your watermark advanced to 10:05 at 10:06, you’ll skip that file forever.
• Clocks of producers and your watermark logic are different clocks.

Fixes:

• Track processed keys (a manifest/state table), not just a timestamp.
• Or re-scan with an overlap window (watermark - N minutes) and dedupe by key + ETag. Reading and snapshotting ETags with boto3, and the multipart caveats, are covered in tracking S3 file changes with ETags and LastModified.
• Better: consume S3 Event Notifications or an upstream commit log instead of polling LIST.

4. Event notifications are at-least-once and unordered

S3 Event Notifications (to SQS, SNS, Lambda, EventBridge):

• Can be delivered more than once.
• Are not guaranteed to be in order.
• Usually arrive within seconds, but can take a minute or longer.

So: ObjectRemoved can arrive before the ObjectCreated for the same key, and a duplicate ObjectCreated can trigger a second load.

Fixes:

• Make consumers idempotent (dedupe on bucket + key + versionId/ETag).
• Use the sequencer field to order events for the same key. It’s only comparable within one key.
• Don’t use notifications as an exactly-once source of truth; reconcile periodically with a LIST.

5. Bucket-level configuration is still eventually consistent

Strong consistency covers objects. Bucket configuration does not. AWS documents that:

• After deleting a bucket, it may still appear in ListBuckets for a while.
• After enabling versioning on a bucket for the first time, AWS recommends waiting 15 minutes before issuing write operations.

Infra-as-code that creates a bucket, enables versioning and immediately starts writing in the same pipeline run can hit this.

6. Replication is asynchronous

Cross-Region (CRR) and Same-Region Replication (SRR) copy objects asynchronously. A reader in the replica region can see an older version or no object at all. S3 Replication Time Control (RTC) gives an SLA (99.99% of objects within 15 minutes), not immediacy.

If a downstream job in another region is triggered by the source write, it must handle “not replicated yet” — check x-amz-replication-status or retry with backoff.

7. Overwrites + caching layers

S3 itself is consistent, but things in front of it might not be:

• CloudFront or other CDNs caching objects.
• Engine-side file/metadata caches (e.g. Spark caching a file listing, Trino/Presto metastore caches, local SSD caches).
• Clients that hold a stale listing in memory between steps.

The rule: don’t overwrite keys in place in a data pipeline. Write new keys (versioned paths or table-format commits) and switch pointers atomically.

Checklist

• Output committed via a table format or S3A committer, not rename.
• One writer per Delta table on S3, or a coordinated LogStore / catalog.
• Incremental loads track keys, not just LastModified.
• Event consumers are idempotent and order-tolerant.
• No “create bucket → write immediately” in the same run.
• Cross-region consumers handle replication lag.
• No in-place overwrites of data files.

References

• AWS — Amazon S3 now delivers strong read-after-write consistency (Dec 2020): https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-now-delivers-strong-read-after-write-consistency-automatically-for-all-applications/
• Amazon S3 — Data consistency model: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel
• Amazon S3 — Conditional writes (If-None-Match, If-Match): https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html
• Amazon S3 — Event notification types and destinations (delivery, sequencer): https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html
• Amazon S3 — Event message structure: https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html
• Amazon S3 — Object metadata (Last-Modified for multipart uploads = initiation date): https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html
• Amazon S3 — Multipart upload overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• Amazon S3 — Using versioning (15-minute recommendation): https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html
• Amazon S3 — Replication and S3 RTC: https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html
• Delta Lake — Storage configuration on S3 (single vs. multi-cluster, S3DynamoDBLogStore): https://docs.delta.io/latest/delta-storage.html#amazon-s3
• Apache Hadoop — S3A committers: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/committers.html