2.3 YARN Resource Management

YARN, the resource management layer of Hadoop, revolutionizes cluster utilization. It separates resource management from data processing, enabling diverse workloads to run efficiently. YARN's flexible architecture allows for dynamic resource allocation, optimizing cluster performance.

The YARN architecture consists of key components working together seamlessly. The Resource Manager oversees cluster-wide resources, while Node Managers handle individual nodes. Application Masters negotiate resources and manage task execution, ensuring efficient application lifecycle management.

YARN Architecture and Components

Role of YARN in resource management

Top images from around the web for Role of YARN in resource management

• 

  [Data Science 到底是什麼從一個完全外行角度來看][07]更深入看看Hadoop裡面的YARN和HDFS @ Alan Tsai 的學習筆記｜A Microsoft MVP ... View original

  Is this image relevant?

• 

  An introduction to Apache Hadoop for big data | Opensource.com View original

  Is this image relevant?

• 

  An introduction to Hadoop - Mayflower Blog View original

  Is this image relevant?

• 

  [Data Science 到底是什麼從一個完全外行角度來看][07]更深入看看Hadoop裡面的YARN和HDFS @ Alan Tsai 的學習筆記｜A Microsoft MVP ... View original

  Is this image relevant?

• 

  An introduction to Apache Hadoop for big data | Opensource.com View original

  Is this image relevant?

1 of 3

Top images from around the web for Role of YARN in resource management

• 

  [Data Science 到底是什麼從一個完全外行角度來看][07]更深入看看Hadoop裡面的YARN和HDFS @ Alan Tsai 的學習筆記｜A Microsoft MVP ... View original

  Is this image relevant?

• 

  An introduction to Apache Hadoop for big data | Opensource.com View original

  Is this image relevant?

• 

  An introduction to Hadoop - Mayflower Blog View original

  Is this image relevant?

• 

  [Data Science 到底是什麼從一個完全外行角度來看][07]更深入看看Hadoop裡面的YARN和HDFS @ Alan Tsai 的學習筆記｜A Microsoft MVP ... View original

  Is this image relevant?

• 

  An introduction to Apache Hadoop for big data | Opensource.com View original

  Is this image relevant?

1 of 3

• Manages and allocates resources (CPU, memory) across a Hadoop cluster
• Separates resource management from data processing enabling better scalability and flexibility
• Allows running diverse workloads (batch processing, interactive queries, real-time streaming)
• Dynamically allocates resources to applications based on their requirements optimizing cluster utilization

Components of YARN architecture

• Resource Manager (RM)
  • Master daemon manages and allocates resources across the cluster
  • Scheduler allocates resources to running applications based on requirements and policies (capacity, fair)
  • Application Manager accepts job submissions, negotiates first container for Application Master, provides fault-tolerance
• Node Manager (NM)
  • Per-node daemon manages containers, monitors resource usage
  • Registers with RM providing node's available resources
  • Launches and manages containers based on RM instructions
• Application Master (AM)
  • Per-application process negotiates resources from RM, works with NMs to execute tasks
  • Manages application lifecycle including resource allocation, task scheduling, monitoring

YARN Resource Management and Monitoring

Resource allocation for YARN applications

• Configure resource allocation using XML files (yarn-site.xml, capacity-scheduler.xml)
• Set properties for minimum and maximum resource allocations for containers
  1. yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb for memory
  2. yarn.scheduler.minimum-allocation-vcores and yarn.scheduler.maximum-allocation-vcores for CPU
• Configure queue properties for capacity or fair scheduling
  • Set queue capacity, maximum capacity, user limits
  • Define queue hierarchies and resource allocation policies
• Submit applications with resource requirements using command-line options or application-specific configurations
  • Specify memory and vCore requirements for containers
  • Set priority and queue for application execution

Monitoring and troubleshooting YARN

• YARN Web UI
  • Access ResourceManager web interface (default port 8088) to monitor cluster status, running applications, resource usage
  • View application-specific information (containers allocated, resource usage, logs)
• YARN application logs
  • Access logs using YARN web UI or command-line tools
  • Logs aggregated and stored on HDFS by NodeManager
  • Use

    yarn logs

    command to view or download application logs
• Troubleshooting steps
  1. Check YARN service status and logs (ResourceManager, NodeManager) for errors or warnings
  2. Investigate application-specific issues by examining application logs and container exit codes
  3. Monitor resource usage and identify bottlenecks or contention using YARN web UI and metrics