Posts
All the articles I've posted.
-
Fixing Spark Ivy Error in Docker: "basedir must be absolute"
If you're running Apache Spark inside Docker using Bitnami's images and suddenly encounter an Ivy error that says: Exception in thread "main" java.lang.IllegalArgumentException: basedir must be
-
How Dynamo Reshaped the Internal Architecture of Amazon S3
Introduction Amazon S3 launched in 2006 as a scalable, durable object storage system. It avoided hierarchical file systems and used flat key-based addressing from day one. However, early versions of
-
What’s Behind Amazon S3?
When you upload a file to the cloud using an app or service, there's a good chance it's being stored on Amazon S3 (Simple Storage Service). But what powers it under the hood? What is Amazon S3? Amazon
-
How HDFS Achieves Fault Tolerance Through Replication
One of the core strengths of the Hadoop Distributed File System (HDFS) is its fault tolerance . In a world of distributed computing, failures are not rare—they're expected. HDFS tackles this by using
-
Summary: Teaching HDFS Concepts to New Learners
Introducing Hadoop Distributed File System (HDFS) to newcomers can be both exciting and challenging. To make the learning experience structured and impactful, it’s helpful to break down the core
-
How Clients Know Where to Read or Write in HDFS
Hadoop Distributed File System (HDFS) is designed to decouple metadata management from actual data storage . But how does a client—like a Spark job or command-line tool—know where to read or write the
-
How HDFS Avoids Understanding File Content
One of the defining features of Hadoop Distributed File System (HDFS) is that it doesn’t understand the contents of the files it stores . This is not a limitation—it's an intentional design choice
-
How Spark and MapReduce Handle Partial Records in HDFS
When working with large-scale data processing frameworks like Apache Spark or Hadoop MapReduce, one common question arises: What happens when a record (e.g., a line of text or a JSON object) is split