Overview
Hadoop is an open-source distributed storage and computing framework led by the Apache Foundation, specifically designed for the storage, processing, and analysis of massive structured, semi-structured, and unstructured data. The core goal of Hadoop is to build a highly available, highly scalable distributed system by deploying commodity servers in clusters, breaking through the limitations of stand-alone storage capacity and computing performance, and achieving cost-effective, high-efficiency big data processing.The technical prototype of Hadoop originated from two classic papers published by Google: the 2003 paper "The Google File System (GFS)" and the 2004 paper "MapReduce: Simplified Data Processing on Large Clusters." In 2006, the Apache Foundation officially incubated Hadoop as a separate project. After years of iteration, it has evolved into a complete ecosystem comprising multiple sub-projects covering storage, computing, scheduling, and management, becoming the core foundation of the global big data technology stack.
2. Core Components of Hadoop: The Pillars of the Ecosystem
The Hadoop ecosystem is not a single tool but is composed of multiple complementary core components. The most fundamental and essential among them are HDFS (distributed file system), MapReduce (distributed computing model), and YARN (resource scheduling system), together with Common (basic utility library), which collectively support the core processes of big data processing.
2.1 HDFS: The "Data Warehouse" for Distributed Storage
HDFS (Hadoop Distributed File System) is the distributed file storage component of Hadoop. It stores massive amounts of data across multiple nodes (servers) in a cluster. Through a master-slave architecture, it ensures data reliability and access efficiency:
- NameNode (Master Node): Acts as the "scheduling hub," storing file metadata (such as file names, storage paths, and data block locations), managing the file system namespace, and coordinating read/write operations from data nodes.
- DataNode (Slave Node): Functions as a "storage node," actually storing data blocks (each block is 128 MB by default) and periodically reporting their status to the NameNode.
- Core mechanism: Uses a multi-replica storage policy (3 replicas by default) to distribute data blocks across different nodes and even different racks. Even if a single node fails, data can be recovered through replicas, ensuring high reliability.
2.2 MapReduce: The "Task Engine" for Distributed Computing
MapReduce is Hadoop's distributed computing model that follows a "divide and conquer" approach. It splits large computing tasks into smaller sub-tasks, distributes them to cluster nodes for parallel processing, and finally aggregates the results:
- Map Phase (Mapping): Splits input data into independent key-value pairs, and each node processes local data in parallel, outputting intermediate results.
- Reduce Phase (Reduction): Collects intermediate results from the Map phase, aggregates them by key, and outputs the final computed result.
- Core advantage: No manual intervention required for task splitting and scheduling. It automatically adapts to cluster node status. Even if some nodes fail, tasks can be reassigned to ensure stable completion.
2.3 YARN: The "Intelligent Manager" for Resource Scheduling
YARN (Yet Another Resource Negotiator) is Hadoop's resource management and scheduling system. It coordinates hardware resources such as CPU and memory in the cluster, allocates resources to various computing tasks, and monitors their execution status:
- ResourceManager: The global resource scheduling hub, receiving client-submitted tasks and allocating resource queues based on cluster resource availability.
- NodeManager: Runs on each slave node, monitoring local resource usage and executing tasks assigned by the ResourceManager.
- Core value: Decouples resources from computing tasks, supports multiple computing frameworks (such as MapReduce and Spark) sharing cluster resources, improving resource utilization, and adapting to diverse big data processing scenarios.
2.4 Common: The "Basic Utility Library" for the Ecosystem
Common (Hadoop Common) is the foundational support component of the Hadoop ecosystem. It provides common utilities and APIs for file I/O, serialization, communication protocols, and security authentication. It offers a unified underlying foundation for core components like HDFS, MapReduce, and YARN, ensuring compatibility across the entire ecosystem.
