The Fuel of Interstate: One Year Later

One year ago, we wrote a blog post explaining Interstate’s stack and architecture after we were just two months in to development. Since then a lot has changed on both the product and business sides of Interstate (we’ve released 3 new major versions since then and we recently took part in the Summer 2011 cycle of Y Combinator) and so I thought this would be a good time to explain what’s changed in the past year in terms of our stack.

Hosting

At the time of our last post, we were running Interstate on a single server from the guys at MediaTemple. Whilst MediaTemple were a great host, they didn’t really support our expansion plans as well as other hosts could. Because of this, we moved our web app over to Amazon Web Services in June/July 2011. At the current moment in time, we’re running just one large (7.5GBs memory, 4 EC2 Compute Units) 64bit EC2 instance (with CentOS) in the US East (Virginia) availability zone. Over the next few weeks/months we’ll be looking to spread Interstate across more EC2 instances and availability zones.

Database

At Interstate, we use MongoDB as our primary data store and before our switch to EC2, we managed our own MongoDB instance. Due to Interstate’s growth though, it was becoming a large effort to keep monitoring our MongoDB setup whilst still building our product. The main issue being that for the better part of a year, we hosted our MongoDB instance on the same server as our web app. This quickly became an issue as our Mongo instance was using too much of our server’s resources and so our web app quickly began to suffer.

Because of this, we made the decision to move our database to a group of guys who know a lot more about hosting MongoDB than we did, those guys being MongoHQ. We moved to MongoHQ around March 2011 and since then the experience has been pretty great.

HTTP

Just as we were a year ago - we’re still using NGiNX to handle all of our requests and to serve our content. Here’s what we said in our original post regarding our choice of NGiNX..

We have decided to use nginx to serve all HTTP requests over the other popular alternatives such as Apache. We chose to use nginx after benchmarking other sites of ours that run on the same PHP framework but use Apache as the HTTP server. Both the speed difference and usage of system resources (e.g. memory) between powering these sites on nginx rather than Apache made it an easy decision for us to use nginx.

Programming

The language which Interstate is written in is pretty much always a surprise and shock for people. When we were originally meeting the other founders in our Y Combinator class, we often showed them what we were working on and 99% of the time, their first reaction was: “So what did you write this in? Rails, right?”. As we told them what language we actually used, letter by letter their face turned from interested to surprised, shocked and then somewhat horrified.

As you may have guessed (or previously read), we primarily use PHP to build Interstate. The reason behind this is not really all that strange, we’ve simply built a lot of PHP powered apps in the past and know how to build clean and efficient apps in the language. We believe you can write ugly and bad code in any language and so just because PHP has a history of badly written code doesn’t make it an inherently bad language. Yes, there are a lot of silly problems with the language itself (such as the infamous namespace separator) but most of the problems don’t really affect you (or at least they don’t affect me) day-to-day.

All that aside, the main Interstate app runs on PHP 5.3.6 using a custom (MVC) framework, which we hope to open-source in full at some point. To handle our PHP processes with NGiNX we’re still using PHP-FPM (which is now bundled with PHP as of v5.3.3).

Node.js

As of recently, we’ve also began to start experimenting with Node.js for some new and exciting real-time features. Most notably, our new Streaming API (which will be opened up to developers soon) which powers pretty much all real-time updates within Interstate (such as our new Growl/Toast-esque notifications) and soon our Mac App. Our second biggest use of Node.js is our upcoming real-time team chat system, called Discuss - which Node.js has been particularly great for.

When developing apps using Node.js, we often use a mix of open-source modules made by other great developers. These modules generally include: Socket.io, node-mongodb-native, node-memcached and node-redis to name just a few.

To manage our Node apps we also use forever.

Caching

We still use APC for PHP bytecode caching and Memcached for all in-memory caching. Our use of Redis for caching has somewhat reduced and we’re now using Memcached a lot more aggressively than we were a year ago.

Redis

As said above, our usage of Redis as a cache has reduced and our main use case has shifted from caching to instead using it as a pub/sub backend for our Streaming API. So far Redis has been working really well for this use case. Once the Streaming API is made public, we’ll try and take some time to write about how the stream works in more detail.

3rd Party Services

To help us build Interstate we also use several third party web services..

Version Control

At Interstate, we use git for version control. For our git hosting we tried out several popular services but we eventually settled for the most popular choice, GitHub. We chose GitHub simply because all of our open source and public projects are already hosted there and so to host our private repositories there as well just made things easier.

Email Delivery

For the small bits of email marketing (newsletters) which we do, we have started using Mailchimp. We’ve only sent out one campaign so far but the service has been great.

For transactional email delivery (e.g. forgotten password emails, notifications, etc) we still use Postmark. Postmark makes transactional email delivery super easy (just as it should be), they provide great and simple analytics and an easy (and pretty cheap!) pay-per-email pricing model. We’re more than happy to keep giving these guys our money.

Software

Finally, to build Interstate we use a bunch of great apps, including..

If you have any questions about our particular use of any software in our stack or anything else, please just write a comment and we’d be happy to answer.

TL;DR: We’re hosted on Amazon EC2, running CentOS, NGiNX, PHP (+ PHP-FPM) and Node.js. We use Mongodb as our data store, Memcached for memory caching and Redis as a pub/sub backend.

Interstate-CLI

Interstate CLI began on a whim, starting with my (Alec’s) recent conversion to Git from SVN. As you do, I started using Terminal so often that I couldn’t help but consider other tasks and ask whether they could be reasonably accomplished from the command line. This is not a new story, so I’ve gathered: this is why libraries like Thor, purpose-built for crafting shell scripts, exist in the first place.

We have happily discovered that, in fact, Interstate lends itself quite well to the command line. The Roadmap > Route > Road pathway is easily traversed with simple commands, and providing method options feels as natural here as it does in Git. 

Some aspects, however, needed reconsidering. For example, we made the choice to remove Interstates as the point of departure. In terms of navigating to a road, this works perfectly well with a mouse, but is slightly fatiguing on a keyboard. Instead, we’ve made roadmaps top-level, whereby typing “interstate maps” presents you with a list of roadmaps.

The second big thing was that we didn’t want to type either a 22 character identifier or the full title of whatever we were navigating to. The API doesn’t provide shortcodes, so we generate them locally. We’ve kept them short and easy to read: just one letter and two numbers. Finding what you want in a list is easy and the shortcode seems to stick in your mind quite easily.

Without further ado, here is what interstate-cli 0.0.4 can do so far:

interstate help

Show a list of tasks

interstate maps

Show a list of roadmaps associated with your API key

interstate map x##

Switch to a roadmap

interstate roads -r

Show a list of roads for the current roadmap. Because we have to cache the roads locally, occasionally run it with -r to see if you’ve got latest.

interstate road x##

Show a detail view of a road

interstate update x## -m -s

Update a road with optional —message (-m) and —status (-s) parameters. Note: the status parameter can take either a status code (1-14), the human readable status (e.g. “10%”, or “Started”), or if left blank, you’ll see a table of options and are prompted at that point.

interstate add -t -d

Add a road to the current roadmap with required —title (-t) and —description (-d) parameters.

interstate delete x##

Delete a road

interstate log

Display a log of recent activity for the current roadmap

interstate reset

Resets all interstate-cli local config 

Interstate-cli requires at least Ruby 1.87 and a reasonably current version of Rubygems. 

Install it with

sudo gem install interstate

You may need to install Apple’s developer tools on certain machines, although this issue is not currently reproducable and I haven’t had time to look into it in any depth. Also, some machines experience write permissions with certain interstate-cli config files. We are also looking into that.

For now, if you have problems setting it up, please email alec@bakedcode.com

Interstate-cli is by no means a finished product. If you like the concept and want to see it mature into a more useful, fully-rounded tool, please send us feedback! We see this as an experiment for now, but if it evolves into something bigger, that’s cool too!

Work has begun on a mobile version of Interstate. First up is the iPhone (which is a favourite here at BakedCode) and if you visit your Interstate via Safari on your iPhone you will see it in action. It is still in the early stages as it was more of a test of user experience using our current code. The entire design was merely done by defining iPhone CSS files, we aim to make a more native mobile site in the future. At the moment various pages aren’t available by navigation such as theme, config and the various pages related to roadmaps. We’d love to hear your feedback!
View Separately

Work has begun on a mobile version of Interstate. First up is the iPhone (which is a favourite here at BakedCode) and if you visit your Interstate via Safari on your iPhone you will see it in action.

It is still in the early stages as it was more of a test of user experience using our current code. The entire design was merely done by defining iPhone CSS files, we aim to make a more native mobile site in the future. At the moment various pages aren’t available by navigation such as theme, config and the various pages related to roadmaps.

We’d love to hear your feedback!

We’ve just pushed out an update to those BETA testing the third release of Interstate! The update includes most notably file sharing and roadmap exporting - two features which have been highly requested since we launched our original BETA last year. If you’re using the latest version of Interstate, go ahead - try out these new features now! (If you’re not using the new version then tweet us at @interstateapp with your Interstate subdomain and we’ll hook you up!). Check out the above video for a preview of how file sharing works within Interstate.

NB: File sharing currently requires a modern browser (such as Chrome or Firefox) with HTML5 Drag and Drop capabilities. To share a file, simply drag a file from your desktop and drop it on the road of your choice. The file will then be uploaded to Amazon S3 and attached to the road for your colleagues to see (they will also receive an email notification if they have chosen to).

Once this feature is out of BETA testing we will begin to add backwards compatibility by also offering a simple file upload form for those folks using older browsers (if that’s you, come on - upgrade! It’s easy!).

The Fuel of Interstate

Interstate has been in full-on development for just around 2 months now (and in private BETA for just over 1 month) and in that small amount of time we’ve had to make some pretty quick yet equally important choices when it comes to our site’s technical setup.

We’ve had to decide things from simply where to host our servers to questions like which HTTP server we should use. The great thing about the Interstate project is that because we’re just 2 guys building something we love to both make and use, we get to make some pretty interesting choices when it comes to the site’s architecture.

Because of this I’ve decided to do a post explaining what we use to power and build Interstate. So, without further ado, here we go!

Development

To develop Interstate, Greg and I both use Apple Macbooks and take advantage of some pretty sweet Mac software. The software we find ourselves using on a daily basis is:

  • Adobe Photoshop CS5 (Greg uses this bad boy to create the fantastic design you lucky BETA testers get to use)
  • Coda by Panic (We both use this amazing IDE to code both the frontend and the backend of Interstate)
  • Transmit by Panic (Greg uses this to upload the bulk of the site’s static files such as images, etc)
  • Terminal (Simon uses this to keep control of Interstate’s server)
  • Spotify (What is website development without some kick ass music in the background?)
  • Google Chrome (Possibly the best browser out there at the moment)
  • MongoHub (Simon uses this to monitor MongoDB)
  • Forrst (O.K., this isn’t exactly software but it’s an awesome community where we try and share as much stuff as we can whilst we build Interstate).

Hardware

Up until the latest release of Interstate (“Frank”) both the frontend and the backend was hosted on a single RapidSwitch server which also hosted about 7 other websites of ours as well. This really did not help when it came to making the site as fast as it could be. At this point we also hosted our databases on a separate server over at MediaTemple. This really did not help as it meant we had to make thousands of requests to a server that was on a completely different network to the backend.

With the latest release of Interstate we decided that we needed to fix this right away. As of writing this post we are currently hosted on a single (MediaTemple based) dedicated server with around 2GB of RAM (more spec details can be found on MediaTemple’s website). We are looking to separate our backend from our database in the very near future (both in to individual MediaTemple (ve) servers).

Software

Interstate uses PHP (PHP 5.3.3 to be exact) and it runs on its own custom framework. We have decided to skip the conventional route of using MySQL for our primary datastore and we have instead opted to use MongoDB and so far it is performing extremely well. We also use a combination of Redis and Memcached for our currently rather small caching layer that will most probably be used a lot more in the near future.

We have decided to use nginx to serve all HTTP requests (and PHP-FPM as the PHP process manger) over the other alternatives such as Apache. We chose to use nginx after benchmarking other sites of ours that run on the same PHP framework but use Apache as the HTTP server. The speed difference between powering these sites on nginx rather than Apache made it an easy decision for us to use nginx.

To make sure Interstate is always performing as well as it should be, we use a great piece of server monitoring software called munin.

When it comes to 3rd party web services we currently only use one: Postmark (aka Postmarkapp). We use Postmark to simply deliver all on-site emails (such as invite codes, notifications, etc) and we really can’t recommend it enough.

I hope this gives you a little better idea of how the site works. If you have any specific questions regarding Interstate’s development/architecture feel free to send me an email via simon@interstateapp.com.