At Continuuity, we needed a fast, self-service way to provision clusters because testing against real clusters is such a critical and frequent component of our development cycle. Our vision was to enable all Continuuity developers to be able to quickly create a cluster with the push of a button.
We started off with simple scripts. But scripts had multiple issues - they required a lot of setup, there was no central management keeping track of available clusters, and the scripts had to be changed whenever we moved from one cloud to another, or one software deployment to another. Next, we started looking at different SCM (software configuration management) technologies and picked Chef. Combining scripts and Chef recipes, we were able to get to one-button push, but it became extremely complex for two reasons - there were an increasing number of customizations needed for each new type of cluster and a significant amount of setup was needed on every developer’s box.
When we launched our Reactor product, we wanted to enable developers to deploy our single-node Sandbox Reactor to a public cloud directly from the website. This forced us to think harder about designing a complete production system while still providing an easy and flexible solution for our developers. We wrote a combination of scripts and Chef recipes, designed a system that can templatize the Hadoop cluster, added capabilities to make it extendable to other services and providers, and made it very developer and ops friendly - that’s when Continuuity Loom was born.
Today, Continuuity Loom is used to provision Sandbox Reactors on the Continuuity Cloud and as an internal DevOps tool to test new and incremental features on an ongoing basis. We’ve been developing and using Continuuity Loom for almost a year and we’re noticing the benefits: our developer and DevOps productivity have doubled, and our development and test cycles have become faster.
We built Continuuity Loom for ourselves. But we believe that the broader developer community can benefit from it, so we're making it available for everyone to use.
Continuuity Loom is cluster management software that manages clusters on public and private clouds. Clusters created with Continuuity Loom utilize templates of any hardware and software stack, from simple standalone LAMP-stack servers and traditional application servers like JBoss, to full Apache Hadoop clusters comprised of thousands of nodes. Clusters can be deployed across many cloud providers (Rackspace, Joyent, and OpenStack) while utilizing common SCM tools (Chef and scripts). Continuuity Loom is pluggable so integration with new providers like AWS and new SCM tools like Puppet can be added by implementing plugins.
Continuuity Loom simplifies the installation and configuration of any software stack, including Hadoop, and ensures that all installations are verified before a cluster is made available. Developers can create custom cluster types ranging from single VM application servers to large-scale, distributed, heterogeneous clusters at the push of a button.
Continuuity Loom enables the creation of custom cluster templates ranging from Hadoop to LAMP to Solr search clusters. Templates define rules describing how hardware, images, and services should be laid out in a cluster, leaving Continuuity Loom to solve and coordinate what tasks need to be done in what order to create a cluster. Services are automatically placed on the correct nodes and are configured, initialized, and started in the correct order. In this way, the template used to create a 3 node Hadoop cluster with only hdfs and YARN can also be used to create a 300 node cluster with hdfs, YARN, and HBase.
Continuuity Loom integrates with various IaaS providers (currently Rackspace, Joyent and OpenStack) and SCM tools (currently just Chef) out of the box. Continuuity Loom has a pluggable architecture, so adding a provider or SCM tool can be done by writing a plugin. If your provider or SCM tool is not supported out of the box, simply implement a plugin for it (and contribute back!). Switching between supported providers is even simpler; you simply select a different provider from a drop down menu.
Continuuity Loom from your laptop
Continuuity Loom has been tested on Linux and Mac. It uses little resources and can be used to provision and decommission a cluster directly from a laptop. Continuuity Loom creates a very simple user API that allows the developer to choose the type of cluster, how many nodes, and which cloud. Any developer can go from zero to Hadoop, from their laptop to any cloud, in minutes.
Run it from your Mac
A standalone version of the current snapshot can be downloaded here, or built from the source code as described further below. After downloading it, unzip it and follow the instructions in standalone/README.md to start up Continuuity Loom on your own machine. It comes pre-packaged with templates for Hadoop and LAMP. In order to create an actual cluster, you will need an account with a supported provider (Rackspace, Openstack, or Joyent), and to perform a couple steps to setup up Continuuity Loom to use the right credentials to integrate with your provider. Follow the Quickstart Guide, which steps through an example of creating a Hadoop cluster.
$ git clone http://github.com/continuuity/loom.git $ cd loom/standalone $ mvn clean package assembly:single
The build places a zip of the standalone version of Continuuity Loom into the target directory.
The User Guide describes what Continuuity Loom is and how to use it from the point of view of an end user and of an administrator. The Tech Guide describes how Continuuity Loom is implemented, and is a good overview for those who wish to contribute to the project.
Are you interested in making Continuuity Loom better? Our development model is a simple pull-based model with a consensus building phase, similar to the Apache's voting process. If you want to help make Continuuity Loom better, by adding new features, fixing bugs, or even suggesting improvements to something that's already there, here's how you can contribute:
- Fork Continuuity Loom into your own GitHub repository
- Create a topic branch with an appropriate name
- Work on your favorite feature to your content
- Once you are satisifed, create a pull request by going to the continuuity/loom project.
- Address all the review comments
- Once addressed, the changes will be committed to the continuuity/loom repo.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.