Tag: Hadoop
All the articles with the tag "Hadoop".
-
The History of Hive and Trino: From Hadoop to Lakehouses
The evolution of Big Data architectures is deeply tied to the history of two projects born at Facebook: Hive and Trino . Both emerged from real engineering pain points, but at different times and for
-
How Hadoop Made Specialized Storage Hardware Obsolete
In the early 2000s, enterprise data processing was dominated by high-end hardware. Organizations relied heavily on centralized storage systems such as SAN (Storage Area Networks) and NAS (Network
-
How Google Changed Big Data: The Story of GFS, MapReduce, and Bigtable
In the early 2000s, Google faced a unique challenge: how to store, process, and query massive amounts of data across thousands of unreliable machines. The traditional systems of the time—designed for
-
How Network Topology Shapes Distributed Computing and Big Data Systems
When discussing distributed systems and Big Data, people often focus on storage, processing frameworks, and scalability—but one foundational concept underlies it all: network topology . It’s the
-
How HDFS Achieves Fault Tolerance Through Replication
One of the core strengths of the Hadoop Distributed File System (HDFS) is its fault tolerance . In a world of distributed computing, failures are not rare—they're expected. HDFS tackles this by using
-
How Clients Know Where to Read or Write in HDFS
Hadoop Distributed File System (HDFS) is designed to decouple metadata management from actual data storage . But how does a client—like a Spark job or command-line tool—know where to read or write the
-
How HDFS Avoids Understanding File Content
One of the defining features of Hadoop Distributed File System (HDFS) is that it doesn’t understand the contents of the files it stores . This is not a limitation—it's an intentional design choice
-
How Spark and MapReduce Handle Partial Records in HDFS
When working with large-scale data processing frameworks like Apache Spark or Hadoop MapReduce, one common question arises: What happens when a record (e.g., a line of text or a JSON object) is split