Pallet & SmartOS example

Intro

I love functional programming and have been taking quite a liking to Clojure.  At a previous job of mine, we were using Joyent as the cloud provider, and I was disappointed to find out that I could not use pallet with this cloud provider and SmartOS.  So I decided to add support (with the help of others).

First let's discuss some terminology.  Pallet is a platform for provisioning and configuring of servers in the cloud.  Pallet is written in Clojure.  It is built on top of jclouds.  Joyent is a cloud provider whose default OS is SmartOS, whose roots are from Solaris.  It is a very interesting OS from a couple of angles including ZFS, zones, and DTrace.  I definitely would use it for a LAMP type stack where the PHP could be changed out to NodeJS as well.

The jclouds and pallet team was very, very helpful.  In addition, the community provides good help as well.  They all made my job a lot easier by adding Joyent support to jclouds and helping me through the code base.

So I have created a project on github that contains the source code for this example that you can clone.  But the point of this blog post is to explain some things in more detail.

Dependencies & Set-up

First of all you need to have at least the following dependencies if you don't want to follow along with this code example exactly.

• > 1.5.6 of jclouds (fixes a couple bugs in the Joyent/SmartOS support.
• 0.7.3 of pallet (SmartOS support has not yet been ported to 0.8.0.)
• 1.5.1 of pallet-jclouds.  (This provides providers (like AWS, ec2, etc) from jclouds to be accessible by Pallet )

I point out the above dependencies to make it clear that at least these are needed.  There are also other dependencies but will let you look at the project.clj.  Initially, I had a somewhat difficult time finding the right dependencies needed for this version of Pallet so I hope this documentation helps someone else.

Next I highly recommend creating a ~/.pallet/config.clj file.  It is a much better place to put credential related items than in the .clj files themselves.

My config.clj looks like the below

(defpallet
  :services
  {:joyent-service
   {:provider "joyent-cloudapi" :identity <identity> :credential <password>
    :jclouds.zones "us-east-1"
    :endpoint "https://api.joyentcloud.com"
    :environment
    {:user {:username "root"
            :private-key-path <private key>
            :public-key-path <public key>}}}})

Playing in the REPL

After doing the above, we can start fooling around in the repl.

To get the list of supported providers (you will want to see joyent in the list)...

 user=> (use 'pallet-smartos-example.core)  
 user=> (pallet.compute/supported-providers)  
 ("hybrid" "stub" "joyent-cloudapi" "joyentcloud" "node-list")  

To get the list of nodes running in Joyent (in my case this is 0)

 ;; Please note that the :joyent-service is the key looked to in the map of services  
 ;; from your config.clj  
 user=> (pallet.compute/nodes (pallet.compute/service :joyent-service))  
 ()  

To create a call my-converge with a parameter of 1 which says to start 1 server.

 user=> (my-converge 1)  

Now get the list of nodes

 user=> (pallet.compute/nodes (pallet.compute/service :joyent-service))  
 (smartos-machines      ZONE/us-east-1.PROVIDER/joyent-cloudapi null  
            smartos null base64 sdc:sdc:base64:1.8.4  
            RUNNING  
            public: XXX.XX.XXX.X private: XX.XXX.X.XXX)  

Now shut down the node

 user=> (my-converge 0)  

(defn my-converge [num]  
                  (pallet.core/converge  
                      {smartos-machines-group num}
                      :compute my-compute-service ))

The above says to add/subtract nodes until nodes in the smartos-machines-group = num and to use the provider defined by :compute.

(def smartos-machines-group
  (pallet.core/group-spec "smartos-machines"
                          :extends [with-play]
                          :node-spec mynodes
                          ;:packager :pkgin))

The above defines a group that contains a definition for the nodes and a configuration of with-play to apply to the group.

(def mynodes
  (pallet.core/node-spec
   :image {:image-id image
           :os-family :smartos
           }
   :location {:location-id "us-east-1"}
   :hardware {:smallest true}
   :network {:inbound-ports [22]}))

The above says to create an image with the image id, in the us-east-1, on the smallest hardware. Please note I don't believe the inbound-ports does anything at this point in time for Joyent machines. I have not yet had time to verify this belief.

(defn play-with-packages
  "Function to play with pkgin and see if it really works"
  [session]
  (->
   session
   (package/packages :pkgin ["zsh"])))

(def with-play
  (pallet.core/server-spec
   :phases {:configure play-with-packages}))

The above specifies the configuration for the server.  In this case, it is using the pkgin manager (smartos default) to install the zsh package.  

Test on the VM

Finally, please understand if you want to test out the node configuration on a VM install of SmartOS you can do that as well.  Currently, you can't do provisioning so you will have to set-up the machine to begin with.  So this means you need to set-up VirtualBox with SmartOS which will create a global zone.   Then you need to install an image into a local zone in that global zone.  After that you can then configure your machines with pallet using pallet's existing infrastructure provider called the node-list.  This is what is used when you have existing infrastructure that you want to configure.  The difference is that you will then have your config.clj file like the following...

(defpallet
  :services
  {:data-center
   {:provider "node-list"
    :node-list [["smartos-test" "smartos-testing" 
                   "<public-ip-address>" :smartos]]}})

Then you will reference the :data-center in your compute service like so

 user=> (pallet.compute/nodes (pallet.compute/service :data-center))  

The pallet smartos supports the commands for pkgin and svcs.  If you need something to work with smf then you can use use my smf-crate as a starting point.
I hope this is useful for someone else.