Tuesday, April 1, 2014

Reactive Mongo pattern: Serialize Joda DateTime

I really like ReactiveMongo. That said there are not enough code examples out there. So I'm gong to try to help. Here is a simple snippet which will allow you to serialize Joda DateTime instances to MongoDB.

Grunt recipe: Deploy to remote server with SSH

I finally moved my AngularJS apps over to Grunt mostly by following a great blog post here:

Kickstart Your AngularJS Development with Yeoman, Grunt and Bower

I have several sites that I host on a DigitalOcean micro instance and I wanted to make it as easy as possible to deploy new code. First I just tried using GitHub to pull the code on my server and build it there. Unfortunately I ran into problems because my server didn't have enough memory. That's not really a great solution anyway. Instead I added a Grunt task which uses SSH to copy my dist folder to the new server and take care of the deploy steps. Here is what it looks like:

This makes use of the Grunt Shell plugin. I create a tarball of the dist directory to save space. I also modified my Grunt tasks to only package files from the bower-components directory that I needed which made the dist much smaller. I would recommend setting up public key authentication to the servers in question.

Wednesday, January 8, 2014

Scala/Spray ReactiveMongo Recipe: Spray Path Matcher for BSONObjectID

This is a pretty simple one. Spray is implemented such that it is very easy to extend with nothing more than Scala code. In this case I wanted to provide an endpoint that could take a Mongo ObjectID as a path parameter. I had some other endpoint that were taking string patterns as the path parameter so I couldn't just use a String matcher without having special logic to tell which specific endpoint the user really wanted to hit. So I created a new Spray PathMatcher which will try to parse the input as an ObjectID and create a rejection if it hits a parse error. This way Spray can continue looking for another endpoint that might be a better match for this particular request.

Sunday, December 29, 2013

Android Recipe: Reusable Android Gson HTTP Client

Here's a simple implementation of a reusable HTTP client for android using the AndroidHttpClient. It takes a ton of ugly code to accomplish the simple task of sending HTTP requests in Android so at least you only have to do it in one place. Unfortunately type erasure makes this slightly more difficult because we can't create a pure generic implementation because the unmarshalling code doesn't know what type to use. If you create a generic implementation you won't get compile errors but instead will get nasty runtime errors like:

ClassCastException: java.util.LinkedHashMap cannot be cast to com.your.class.Name

We can get around this by providing a concrete implementation for the unmarshalling code whe we instantiate our HttpClient like so:

Verbose, ugly, and full of boilerplate. Pretty much par for the course for Android code.

There's no error handling to speak of so you need to add that bit in. The way I do it is to throw custom exceptions for whatever kind of error I come across.

Tuesday, December 17, 2013

ReactiveMongo Recipe: Awesome macros with Scala 2.10

Serialization is always a pain. There is always the trade-off between high performance, but manual static mapping code and more flexible libraries based on reflection. I usually go with the second approach because I think the potential of mistakes creeping in to manual mapping implementations far outweight the performance hit in many cases.

Scala macros now available in Scala 2.10 seem to provide us the best of both worlds by automatically generating the static mapping code at compile time. I got my first taste of this today using the 0.10.0-SNAPSHOT version of ReactiveMongo. To use them you can simply create a package.scala class that defines your implicit conversions from BSONDocument to your case classes and use Reactive's macros to generate this code: This should be highly performant and easy to manage. Just make sure you have any nested classes define their converters above their parent. Then in your DAO you can convert to your case class with no special code: This is super cool. I suspect there will be lots of really interesting uses for macros once Scala 2.10 gets more common.

British Library releases a fascinating data set

I recently saw an article about the British Library releasing over a million images from historic texts to the public domain. Their stated goal was to have the public slice and dice and remix the data to see what we can get out of it. How could I resist?

I wanted to do something with this data so I figured I would start with what I know the best. Create a RESTful interface for the data to allow myself and others to easily build on top of the data using any language that provides some HTTP library (all of them).

The first step was to take the image metadata in TSV files the British Library kindly put up on GitHub and import it into a database to allow easy integration. MongoDB is my go to database and it fits well for this kind of data so I wrote a handy dandy python script to consume the TSV files and upload them to MongoDB. I love python for tasks like this. Clean and powerful code and I was able to get it written and tested in a couple hours.

Next I tackled the actual REST app to expose this data. There are countless options for implementing REST apps these days. I am hoping this app can serve as the backend for lots of people to create lots of interesting apps of their own so I need something performant that can handle concurrency and scale elegantly. The desire for modern concurrency eliminates about 99% of the REST development frameworks built on top of Python, Ruby, Node etc. Pretty much my only option is something running on the JVM. I'm not a masochist so a pure Java implementation was not an option. Scala and Clojure are really the best fit for the kind of app I wanted to create. Since Clojure is still fairly new to me and I wanted to get this up and running as soon as possible I went with the familiar stack of Scala using the fantastic Spray REST library which leverages Akka for its sophisticated concurrency model. To integrate with MongoDB I decided to use ReactiveMongo for the obvious benefit of non-blocking IO and as an opportunity to try a new tool.

I was able to implement this project with minimal code and I am hugely impressed with all of the tools I went with. The entire Scala ecosystem has matured so much since I began using it several years ago. You can see the code in its current form along with API docs here:


I have this app deployed to Heroku so feel free to integrate with this for your own apps. As of this writing I only have two simple endpoints implemented but I will add more as necessary. I want to create a simple webapp front end to allow people to tag the photos by answering simple questions. I am thinking of storing the raw data in a Graph datastore like Neo4J which is new to me so if you have any experience or ideas on that front please get in touch.

Monday, December 16, 2013

Scala/MongoDB Recipe: Extract connection info from MongoDB URI with regex

I am deploying a Scala app on Heroku with MongoLab addon which means my MongoDB connection URI will be exposed as a system variable. Unfortunately I am using ReactiveMongo and as far as I can tell there is no way to create a connection using a URI. So I need to chop the URI into its component parts. Here is the regex I used: