Tag: S3
All the articles with the tag "S3".
-
Tracking S3 file changes with ETags and LastModified
· UpdatedDetect changed S3 objects without versioning: read ETag and LastModified with boto3, export an ETag CSV per date prefix, and handle multipart ETags.
-
S3 Eventual Consistency Edge Cases in Data Pipelines
S3 has been strongly consistent since 2020. The edge cases that still break data pipelines: no atomic rename, concurrent writers, LastModified, events.
-
Trino in Modern Architectures: SQL Queries on S3 and MinIO
The rise of cloud object storage has transformed how organizations build data platforms. Hadoop Distributed File System (HDFS) once dominated, but today services like Amazon S3, Google Cloud Storage
-
How Dynamo Reshaped the Internal Architecture of Amazon S3
Introduction Amazon S3 launched in 2006 as a scalable, durable object storage system. It avoided hierarchical file systems and used flat key-based addressing from day one. However, early versions of
-
What’s Behind Amazon S3?
When you upload a file to the cloud using an app or service, there's a good chance it's being stored on Amazon S3 (Simple Storage Service). But what powers it under the hood? What is Amazon S3? Amazon
-
From HDFS to S3: The Evolution of Data Lakes in the Cloud
For years, HDFS (Hadoop Distributed File System) was the default choice for building data lakes in on-premises and Hadoop-based environments. But as cloud computing gained momentum, a new player took
-
Is S3 the New HDFS? Comparisons and Use Cases in Big Data
Over the past decade, the way organizations store and manage big data has shifted dramatically. Once dominated by the Hadoop Distributed File System (HDFS) , the field is now led by Amazon S3 and
-
The History and Evolution of Amazon S3: Was It Ever Based on HDFS?
When discussing cloud storage today, Amazon S3 is almost synonymous with scalable, reliable object storage. However, a common question among those familiar with big data technologies like Hadoop is: