On a bucket without versioning, S3 keeps no history of an object. You can still tell whether a file changed between two runs by comparing two pieces of metadata that every list_objects_v2 call returns: the ETag and the LastModified timestamp. This guide shows how to read both with boto3, how to snapshot ETags to a CSV for later comparison, and where ETags mislead you.
What ETag and LastModified tell you
- ETag (entity tag): an identifier S3 generates for each object version, based on its content.
- For single-part uploads, the ETag is typically the MD5 hash of the object content.
- For multipart uploads, it is derived from the MD5 hashes of the parts, followed by
-and the number of parts (for example,33a01f6c513ec334bbdfbc606ad2cbe1-1).
- LastModified: the timestamp of the last write to the object, useful for judging how fresh a file is.
If the ETag of a key changes, the content changed. If it stays the same, the content is very likely unchanged.
Read ETag and LastModified with boto3
import boto3
def list_s3_objects(bucket_name, folder_prefix):
# Initialize the S3 client
s3_client = boto3.client('s3')
# List the objects in the given bucket and folder
response = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=folder_prefix)
# Check if the response contains 'Contents' key with object metadata
if 'Contents' in response:
for obj in response['Contents']:
object_key = obj['Key']
last_modified = obj['LastModified']
etag = obj['ETag'].strip('"') # ETag comes with extra double quotes
print(f"Object: {object_key}")
print(f"Last Modified: {last_modified}")
print(f"ETag: {etag}\n")
else:
print(f"No objects found in {bucket_name}/{folder_prefix}")
bucket_name = "your-bucket-name"
folder_prefix = "your-folder/year/month/day/"
list_s3_objects(bucket_name, folder_prefix)boto3.client('s3')creates the S3 client.list_objects_v2lists the objects underPrefix, which acts as a folder filter.- Each entry in
response['Contents']carriesKey,LastModifiedandETag.
Example output:
Object: your-folder/2024/09/12/data-file.csv
Last Modified: 2024-09-12 13:45:10+00:00
ETag: c699f24dbb44ff83b22df8b6bb41f5e0Snapshot ETags to a CSV for change tracking
To detect changes over time, store the ETags somewhere and compare the next run against them. This version walks several folders with a folder/year/month/day/ layout and writes one consolidated CSV:
import boto3
import csv
def list_s3_objects_with_etag(bucket_name, folder, year, month, day):
s3_client = boto3.client('s3')
# Construct the prefix (folder path)
prefix = f"{folder}/{year}/{month}/{day}/"
response = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=prefix)
# Extract object names and ETags
object_details = []
if 'Contents' in response:
for obj in response['Contents']:
object_name = obj['Key']
etag = obj['ETag'].strip('"') # Remove surrounding quotes
object_details.append({'Object Name': object_name, 'ETag': etag})
return object_details
def create_consolidated_etag_csv(bucket_name, folders, year, month, day, output_file):
all_object_details = []
# Iterate over each folder to gather object details
for folder in folders:
object_details = list_s3_objects_with_etag(bucket_name, folder, year, month, day)
if object_details: # Check if the object details list is not empty
all_object_details.extend(object_details)
# Write the consolidated data into a CSV file
with open(output_file, mode='w', newline='') as file:
writer = csv.DictWriter(file, fieldnames=['Object Name', 'ETag'])
writer.writeheader()
writer.writerows(all_object_details)
print(f"Consolidated ETag data saved to {output_file}")
bucket_name = "your-bucket-name"
folders = ["folder1", "folder2", "folder3"]
year = "2024"
month = "08"
day = "28"
output_file = "etag_report.csv"
create_consolidated_etag_csv(bucket_name, folders, year, month, day, output_file)list_s3_objects_with_etagreturns the key and ETag of every object under one dated prefix.create_consolidated_etag_csvloops over the folders and writes all rows toetag_report.csv.
Run it on a schedule and diff the CSVs by Object Name: a different ETag means the object was rewritten with different content, and a missing or new row means a delete or a new file.
Why this is useful:
- Simple: the ETag is already in the listing, so no extra request per object.
- No versioning required: it works on buckets where versioning is disabled.
Gotchas
- Multipart ETags are not the file’s MD5. The ETag of a multipart upload is built from the part hashes, so you can’t compare it with an
md5sumof the local file. Two uploads of identical content with different part sizes (or one single-part, one multipart) produce different ETags. That is a false “changed” signal, not a missed change. - Encrypted objects. For objects encrypted with SSE-KMS or SSE-C, the ETag is not an MD5 digest of the content, even for single-part uploads (S3
ObjectAPI reference). It still changes when the content changes, but don’t validate it against a local hash. - ETags only show the current state. Without versioning or your own stored snapshots, you can’t tell how many times a file changed, only whether it differs from the last snapshot you kept.
- 1,000 keys per call.
list_objects_v2returns at most 1,000 objects per response. Both functions above read only the first page. For larger prefixes, followNextContinuationToken, as in the pagination loop for listing S3 objects by date. - The ETag string includes quotes. Strip the surrounding
"before storing or comparing it. LastModifiedis timezone-aware UTC. Compare it only with aware datetimes (naive vs aware datetimes in Python).
Related
- Efficiently listing and filtering S3 objects by date lists a dated prefix with full pagination, and pairs well with the ETag snapshot above.
- S3 consistency edge cases in data pipelines explains why
LastModifiedwatermarks miss multipart uploads and how to dedupe by key + ETag. - Data encryption at AWS S3 covers the server-side encryption options that also affect what the ETag contains.