SeatSync Engineering: Choosing Platforms

May 8, 2012

Welcome to the very first installment of SeatSync Engineering, where we discuss the software, technology, and general nerdiness behind SeatSync. Not only do we enjoy writing software for SeatSync, we also immensely enjoy talking about how it works; we invite the interested to read on.

This is a repost of an article originally published here.

Any software project that isn't written completely from scratch requires decisions on which types of platforms they will build upon. For web-apps (which is a large part of SeatSync's code portfolio), the number of toolkits, frameworks, and data-stores at one's disposal is ridiculous. The breadth of choices is expansive enough to induce paralysis, I say. To make things tougher, choosing the components of a platform implies at least some degree of lock-in for your software. Your decision better be good, or you will be stuck with crappy tools that make your job harder and drastically increase the quantity and nastiness of your curses. In this post we present to you, dear reader, a choice selection from the buffet of third-party software which composes the SeatSync platform, and how each piece was chosen.

The Inevitability of Choice
You almost have to choose a platform, because the alternative is to redo what's already been done; often times with less-than-good results. Sometimes you have the benefit of experience with a framework, in which case it's often best to stick with what you know. But having to choose a platform without tons of foreknowledge or experience is a hazard of the trade.

Web Framework
We're using a very minimalistic Python WSGI utility called Werkzeug.

Coming from a Django background, we knew we couldn't use the Django ORM (more on that below), and we never before put Django's template engine to serious use, so instead of cherry-picking from a full-service framework, we wanted something resembling Django but without as much cruft. We have for years replaced Django's template engine in new web projects with Jinja2, written by the lovely organization Pocoo. Pocoo is also the author of Werkzeug which is how we originally stumbled across it.

We have a very loose definition of what a so-called web framework entails; to us a web framework is software which answers HTTP requests with HTTP responses and provides tools to make things easy on the developer.  Here were the qualities we needed in a web framework:

  • Python-based. We're a Python shop, plain and simple.
  • Non-MVC. We need flexibility when it comes to the M and the V. 
  • Sessions. I mean, who doesn't need sessions?
  • As small and light as possible. Being able to easily browse and grok the source is a plus.
  • Good URL routing.
  • Good documentation. 
Werkzeug fills these requirements quite nicely. It's essentially just a fairly thin wrapper over WSGI, with some optional extra goodies. One extra is Werkzeug's interactive stack trace and debugger
Data Store
MongoDB is the data store we're using in the SeatSync platform. It's no secret that the landscape of database options is awash with alternatives. Though the hype over NoSQL seems to be dying down (somewhat at least) and traditional relational databases aren't going anywhere, MongoDB was chosen as it's features are a good fit for the characteristics for our data storage and accessibility needs. When looking for a data store, we had the following in mind:
  • We absolutely need flexibly. We're growing and adding features constantly and thus don't want to deal with schema evolution hell
  • We have the need to implement recursive data structures
  • Our data is not overly relational; we can live without joins by doing a small amount of sensible denormalization
  • We don't need transaction support 
  • Immediate consistency isn't necessary
  • We need support for geospatial indexes
  • We'd like our data store to scale horizontally and vertically with reasonable ease
  • Our data has the potential to change frequently
MongoDB addresses these concerns nicely (in our estimation, at least).  An ancillary benefit is that MongoDB deals exclusively in JSON. All our HTTP APIs speak JSON, and it's like Python's second language so that certainly isn't something we don't like. Because we're ok with eventual consistency, we also can get away with asynchronous writes in most situations, which can make our mobile APIs a bit speedier.
We wrote a tiny layer to sit atop pymongo. It implements a model/manager pattern much like the one present in Django's ORM. More on this later :)
Message Queue
In a search for a message queue, we landed on RabbitMQ.

Just a quick background on the utility of message queues... In running a web app, there are a variety of secondary tasks which might need to occur as the result of a client's request that're irrelevant to the return value of said request. A decent-ish concrete example is sending email; if a client's action necessitates the sending of email, instead of gathering the relevant data, rendering the email body, and doing a bunch of network i/o while the client waits, we can make a note that this type of email needs to be sent, slap that note at the tail of a queue, and return the client's return value right away. Some other process can then read our note off the head of the queue and do all the latent processing and i/o without holding up a busy human being.

We recognize the following benefits of a message queue, specifically RabbitMQ:
  • Asynchronously doing hard stuff lets you to complete requests faster
  • Load is alleviated from the application servers
  • Presents a great opportunity for horizontal scaling
  • Though we're only making simple use of RabbitMQ, it's offerings include more complex workflows which might be useful later on
  • RabbitMQ provides varying degrees of message persistence
The software of SeatSync has quite a few more than three dependencies, but these three are the biggest and most critical. We've already had lots of fun making use of these tools but it will be most interesting (and perhaps nerve-wracking) to see if the fun continues as SeatSync grows up.

Thanks for reading our first SeatSync Engineering post!