Loading page...

Hops-YARN introduces a distributed Resource Manager, whose the shared cluster state is migrated to MySQL Cluster. MySQL Cluster is a replicated, partitioned, in-memory NewSQL database. This enables our YARN architecture to have no down-time, with failover of a ResourceManager happening in a few seconds. Since the ground truth cluster state is now in an external database, this opens up new possibilities for analyzing the performance of both the ResourceManager and applications submitted to it. We have implemented the FIFO scheduler, and the Capacity Scheduler, and we have distributed the ResourceTracker service, where any ResourceManager can act as a ResourceTracker, enabling YARN clusters to scale even better.

Project categories

HA
Customize
Scale-Out

Share the love