For those of you still following along, we have come a long way in setting up the foundation for a personal virtualization platform. We finished the last post, Building a Developer Virtualization Lab - Part 2, by configuring Vagrant to provision a single container in our Proxmox cluster. Having done that, we now have set the stage for actually using our infrastructure to host multiple containers.
I concluded the last post by saying that this article would be around deploying a multi-node SolrCloud cluster. Well, this is not exactly true. Instead, we will lay the foundation by installing a multi-node ZooKeeper ensemble. We might as well deploy everything following an architecture that resembles a production deployment. My other motivation for starting with ZooKeeper first is that I also intend to run Hadoop and HBase on this infrastructure and HBase also has a dependency on ZooKeeper.
Before we get started, let's review the basic approach and assumptions:
- Proxmox will continue to be our target deployment virtualization environment.
- Chef, specifically, Chef Zero, will be used as the provisioner for Vagrant.
- Our Vagrantfile will be updated to support multiple machines using the multi-machine support in Vagrant.
Note: This was my first exposure using Chef. I have always used the shell provisioner with Vagrant as I started my career as a UNIX Systems Administrator maintaining Sun Solaris, FreeBSD and various Linux distributions. I must say, there is a reason why it's called "DevOps" and not systems administration. My life would have been much simpler if Chef was around in the mid 90's through early 2000's.
ZooKeeper Architecture
Apache ZooKeeper is an open source solution designed to provide services to support distributed computing applications such as SolrCloud and HBase. While ZooKeeper can be run in standalone mode for local development and testing, it is typically deployed in a cluster known as an ensemble. Usually, an ensemble consists of three or more nodes with an odd number of nodes (e.g., 3, 5, 7, etc.) used in order to achieve an operating quorum. For this example, we will create a 3-node ensemble.
To begin, carve out 3 unused IPv4 addresses and add them to DNS. For example, I added the following to my pfSense DNS forward service:
- 192.168.5.150 zk1.localdomain
- 192.168.5.151 zk2.localdomain
- 192.168.5.152 zk3.localdomain
Chef and Vagrant Tooling
We'll start by updating our local tool chain. As before, I hope you're on a Mac. I haven't used a Windows machine for over a decade.
1. Begin by downloading and installing the Chef SDK from https://downloads.chef.io/chefdk on your workstation.
2. Install the vagrant-omnibus and vagrant-berkshelf plugins. The former is a Vagrant plugin that bootstraps a specific version of Chef on to the virtual machine. The latter is a Vagrant plugin that adds Berkshelf integration (i.e. dependency management) to the Chef provisioner and will install the referenced cookbooks directly on the machine being provisioned.
$ vagrant plugin install vagrant-omnibus
$ vagrant plugin install vagrant-berkshelf3. Install knife-solo. Since we are not using a Vagrant server, knife-solo provides some of the conveniences of Chef server without the need for a full server while providing more powerful tooling than Chef Solo.
$ chef gem install knife-solo4. Create a local Chef repository and define the basic directory structure using knife-solo.
$ mkdir blog-virt-lab3
$ cd blog-virt-lab3
$ touch Berksfile
$ chef exec knife solo init .5. The directory structure should look as follows when complete.
$ tree
.
├── Berksfile
├── cookbooks
├── data_bags
├── environments
├── nodes
├── roles
└── site-cookbooks
6 directories, 1 fileChef Cookbook
I explored Chef Supermarket and tried to find an appropriate, well-maintained ZooKeeper cookbook. After all, why re-invent the wheel? Initially, I started with the zookeeper community cookbook, but found that it required a few modifications to make it work with an ensemble. I did, however, succeed, but it did not feel terribly clean. Finally, I settled on zookeeper-cluster which had everything I needed.
1. Begin by editing Berksfile and reference our dependencies. We will have two dependencies: one that refers to a ZooKeeper community cookbook (zookeeper-cluster) and a second wrapper cookbook that will contain our customizations. In my research, I found that a wrapper cookbook is preferred over forking a community book and modifying it.
2. Create a wrapper cookbook for ZooKeeper called zookeeper_virtlab. There are two best practices that I settled on. The first was around the naming convention for the cookbook. I used a hyphen the first time and according to the style guide it is discouraged. Instead, and underscore is preferred. The second best practice is to put our custom cookbook under site-cookbooks and not the cookbooks directory.
$ cd site-cookbooks
$ chef generate cookbook -b --copyright "Gaston Gonzalez" --license apachev2 zookeeper_virtlab
$ cd ..3. Begin work on our wrapper cookbook by specifying our dependency on the zookeeper-cluster community cookbook.
$ vi site-cookbooks/zookeeper_virtlab/metadata.rb4. Define a default recipe in our wrapper cookbook. First, we will get a reference to a data bag that has not been defined yet. It will be responsible for storing the configuration information for our ensemble. Then, we will expose our data bag configuration to the zookeeper-cluster cookbook and lastly include the default recipe from the zookeeper-cluster cookbook.
$ vi site-cookbooks/zookeeper_virtlab/recipes/default.rb5. By default, OpenJDK is used by the community cookbook. We’ll update our wrapper cookbook to use Oracle JDK 8. The second change I made was to downgrade ZooKeeper to the latest stable release. By default, the current community cookbook uses version 3.5.1-alpha. Both customizations can be defined in site-cookbooks/zookeeper_virtlab/attributes/default.rb.
$ mkdir -p site-cookbooks/zookeeper_virtlab/attributes
$ vi site-cookbooks/zookeeper_virtlab/attributes/default.rbThe author of the zookeeper-cluster uses one of his community library cookbooks for fetching artifacts called libartifact. This is significant as it uses a SHA-256 hash to confirm the integrity of the ZooKeeper binary. Apache software projects do not publish SHA-256 hashes, so you will need to calculate it. If you need to use a different version, here’s how you can calculate the correct binary_checksum.
Download the desired Apache ZooKeeper tarball and then calculate a SHA-256 hash.
$ shasum -a 256 zookeeper-3.4.9.tar.gz
e7f340412a61c7934b5143faef8d13529b29242ebfba2eba48169f4a8392f535 zookeeper-3.4.9.tar.gz6. Earlier I mentioned a data bag that will be used to define the nodes in our ZooKeeper ensemble. We will create a data bag manually since we are not running a Chef server. The data bag below defines the ZooKeeper ensemble for each environment. In my case, I am calling my environment glab. This must match the value defined in site-cookbooks/zookeeper_virtlab/recipes/default.rb. (In a future post, I will add support for Chef environments).
$ mkdir data_bags/configs
$ vi data_bags/configs/zookeeper.jsonMulti-Machine Vagrant File
In this section we will create a multi-machine Vagrantfile. It will be similar to the Vagrantfile used in the previous post, however, this version will:
- Define multiple VMs (in our case LXC containers). In fact, there will be three VMs: zk1, zk2 and zk3.
- Support for using Chef Zero as the provisioner.
At this point our cookbook and Vagrantfile are fully defined and we can provision our ensemble. Your workspace should look as follows:
$ tree
.
├── Berksfile
├── Berksfile.lock
├── Vagrantfile
├── data_bags
│ └── configs
│ └── zookeeper.json
├── environments
├── nodes
├── roles
└── site-cookbooks
└── zookeeper_virtlab
├── Berksfile
├── README.md
├── attributes
│ └── default.rb
├── chefignore
├── metadata.rb
├── recipes
│ └── default.rb
├── spec
│ ├── spec_helper.rb
│ └── unit
│ └── recipes
│ └── default_spec.rb
└── test
└── recipes
└── default_test.rb
The key difference when provisioning new VMs in a multi-machine configuration, is whether we spin them up all at once using vagrant up or one at a time using vagrant up <vm>. In my case, I opted to run them one at a time just in case there was a failure.
$ vagrant up zk1
$ vagrant up zk2
$ vagrant up zk3Assuming there are no errors, we can run a few tests to validate our deployment.
1. SSH into the first ZooKeeper node and review the ZooKeeper configuration. Note, that we need to specify the VM when using a multi-node configuration. Also notice that all the nodes for our ZooKeeper ensemble are listed. Again, these were all pulled from the data bag.
$ vagrant ssh zk1
[vagrant@zk1 ~]$ cat /etc/zookeeper/zoo.properties2. Verify the node ID in the cluster. It should match the ID in server.<ID> for the current host as defined in /etc/zookeper/zoo.properties.
[vagrant@zk1 ~]$ cat /var/lib/zookeeper/myid
13. Invoke the ZooKeeper client and connect to the ensemble.
[vagrant@zk1 ~]$ /srv/zookeeper/current/zookeeper-3.4.9/bin/zkCli.sh4. Check the Proxmox dashboard.
Git Project
If you experienced difficulty following along, you may want to checkout the code and configuration used for this project on GitHub.
Conclusion
In summary, we have taken one more step in fully leveraging our investment in a personal virtualization platform. In the past, setting up environments that resembled production environments were not easily feasible on a single developer workstation. However, now armed with a more than capable virtualization infrastructure (i.e. Proxmox) and a method for automatically provisioning multiple machines (i.e. Vagrant multi-machine + Chef) we are now on our way to experimenting with larger scale, production-like deployments at home that can be easily managed from a modest developer workstation.
Well, I hope you've enjoyed this installment. In the next post, we'll take on a nearly identical provisioning process and spin up a multi-node SolrCloud cluster.