HDFS (Hadoop Distributed File System) was built to support the reliable storage and access of large datasets distributed across commodity hardware. To make this possible, HDFS relies on a master/slave architecture composed of two main types of nodes: the NameNode and the DataNodes.
1. The NameNode (Master)
The NameNode is the brain of HDFS. It manages:
- Metadata: Keeps track of the filesystem namespace (file names, directories, permissions).
- Block mapping: Records which DataNode holds which block of a file.
- Cluster status: Monitors the health of DataNodes through regular heartbeats.
However, the NameNode does not store the file data itself—only metadata about the data.
Example:
If you store a 300MB file (with 128MB block size), the NameNode will know:
- The file is divided into 3 blocks.
- Block 1 is on DataNode A, Block 2 on B, and Block 3 on C.
- Each block has 3 replicas (e.g., A, D, and F).
2. The DataNodes (Workers)
DataNodes are responsible for:
- Storing the actual data blocks.
- Serving read/write requests from HDFS clients.
- Sending heartbeat and block reports to the NameNode.
They don’t know what they’re storing—just that they hold a block identified by a block ID.
3. The Client
The client interacts with both:
- The NameNode, to get metadata (e.g., block locations).
- The DataNodes, to read/write file blocks directly.
This design reduces the load on the NameNode and allows for high-throughput data transfer.
4. Block-Based Storage
Files in HDFS are split into large blocks (usually 128MB or 256MB). These blocks are:
- Stored independently.
- Distributed across multiple DataNodes.
- Replicated (default replication factor is 3) for fault tolerance.
5. How Metadata Is Stored
The NameNode stores metadata in memory for fast access, and persists it to disk in:
- A namespace image (
fsimage): a snapshot of the filesystem. - An edit log (
edits): a transaction log of recent changes.
On restart, the NameNode combines these to restore its state.
Summary of HDFS Architecture
| Component | Role |
|---|---|
| NameNode | Stores metadata and controls the system |
| DataNodes | Store actual file data (blocks) |
| Client | Reads/writes data by talking to both |
This architecture allows HDFS to scale horizontally and handle very large volumes of data reliably, even in the face of hardware failures.