Install Hops using Chef Orchestration
We promise to help you make it from running Spark locally to running Spark on YARN/HDFS by giving you Chef Orchestration - reliable and repeatable deployments every time.
Using Karamel, a Chef orchestration service and Chef Solo, we have automated the deployment of both Hops and services such as Spark, reducing both the time to install Hadoop and the probability of manually introducing build or configuration errors.
Our Chef cookbooks encapsulate our long operations experience in configuring Hops and its services. You can also add other Chef cookbooks for other services, from the thousands available online.
In Karamel, a Hops cluster is defined in YAML, and in contrast to other orchestation services, it is not platform or cloud-specific. Karamel does the following:
- creates VMs or sets up bare-metal machines,
- runs the chef recipes in the correct order using only ssh (agentless),
- recovers from failures in VMs or recipes.
Other ways of installing Hadoop are Apache BigTop (built on Master-less Puppet), Apache Brooklyn and Cloudbreak (both built on Ambari), and proprietary solutions such as Cloudera Manager. None of these provides a SPA for installing and managing your Hadoop Cluster along with a single file to describe your cluster. Brooklyn tries to provide an orchestration layer above Chef/Puppet/Ambari-installers, but it's highly coupled to Hadoop. We're not.