Automated Infrastructure with Pallet and Clojure

Monday, May 05, 2014

At Ona we are rebuilding our data management platform. We are starting with a light weight front-end that will serve up content pulled from the REST API of our current application. We are aiming to have the back-end in Clojure, the front-end in ClojureScript, and the infrastructure in Clojure using Pallet. We are excited to have a single (and a great) language handle all of these responsibilities.

We are still at a very early stage but we are a distributed team and like to have our apps on development boxes as we go. This allows us to share a common reference point, give mini-demos, and QA each other’s changes. Like Fabric for Python and Capistrano for Ruby, Pallet let’s us do quick deploys of the latest master or branch code.

Even better, Pallet let’s us write Clojure to bring up new clusters, similarly to Puppet, Chef, or Ansible – but in Clojure. We deploy to EC2 on AWS and are glad to avoid spending time mucking around in the AWS GUI. A succinct pallet file specifies the instance, the web application, and the deployment. Putting the current code online and bringing up a server (if one doesn’t already exist) is a single command:

lein do uberjar, with-profile +pallet pallet up \
--phases install,configure,deploy

This tells Leiningen to first create an uberjar, which puts all of our app’s dependencies in a single jar file. It then uses the pallet profile to install, configure, and deploy our application. This command is idempotent, making it easy to push the latest jar up.

A nuance we did not anticipate is that you cannot output logs to stdout in a Jetty app. This is not particularly surprising, but using stdout was a development configuration that we had not yet bothered to abstract.

For now we are handling this with the below middleware wrapper:

(defn wrap-with-logger [handler verbose?]
  (if verbose?
      (logger/wrap-with-logger handler "/dev/stdout")
          (fn [request] (handler request))))

This does the normal logging if verbose? is true and otherwise does nothing. When you run lein ring server-headless a handler is called which sets verbose? to true. When you run the app through java -jar ..., as in our pallet configuration, verbose? is set to false.

The ona-viewer project is a work-in-progress and we would welcome any feedback. Check it out on github.

Cascalog at Intent Media

Monday, April 28, 2014

While I was at Intent Media I led the data engineering team in rebuilding and extending the Intent Media data platform. To structure and simplify queries we relied on Cascalog, a Clojure DSL built on top of the Cascading library that is built on top of Apache Hadoop.

Cascalog is inspired by Datalog and uses logic programming to simplify query expression. It is similar to Datomic for Clojure and the recent DataScript for ClojureScript. This allows simple and concise queries, e.g. to compute the average age per country:

(?<- (stdout) [?country ?avg] 
   (location ?person ?country _ _) (age ?person ?age)
   (c/count ?count) (c/sum ?age :> ?sum)
   (div ?sum ?count :> ?avg))

Jon Sondag, a data scientist at Intent Media, recently gave a presentation at the NYC Clojure Meetup about Cascalog in production. His slides are embedded below.

It is great to see Cascalog being used in production data platforms.

Ona builds tally software for the Libyan elections

Wednesday, March 12, 2014

Ona, a company I co-founded, recently built the tallying software used to aggregate votes in the Libyan constitutional assembly elections. These votes were cast throughout the country, on off-shore oil rigs, and at international voting centers throughout the world.

The Libyan High National Election Commission has generously made the tally system software open source. All application source code is on github, there is an overview of the tallying process, and additional code documentation. A description of the technologies used is posted on the Ona blog.

Mobile Technologies presentation to the Mount Sinai Global Health Program

Friday, January 17, 2014 Benedetta Simeonidis, Roger Wong, and I gave a lecture on Wednesday discussing mobile technologies and their intersection with global health. We demoed Formhub and talked about Drishti. We also talked about the importance of user centric design in mobile technology.

The slides from our lecture are below:

Introduction to Clojure Presenation at Intent Media

Wednesday, November 27, 2013

Ivan Willig and I gave a presentation on Monday introducing Clojure and discussing some of the production Clojure code in the Intent Media data platform.

Rich Hickey

The full presentation is available on github and as interactive slides.