One year ago, we wrote a blog post explaining Interstate's stack and architecture after we were just two months in to development. Since then a lot has changed on both the product and business sides of Interstate (we've released 3 new major versions since then and we recently took part in the Summer 2011 cycle of Y Combinator) and so I thought this would be a good time to explain what’s changed in the past year in terms of our stack.
At the time of our last post, we were running Interstate on a single server from the guys at MediaTemple. Whilst MediaTemple were a great host, they didn’t really support our expansion plans as well as other hosts could. Because of this, we moved our web app over to Amazon Web Services in June/July 2011. At the current moment in time, we’re running just one large (7.5GBs memory, 4 EC2 Compute Units) 64bit EC2 instance (with CentOS) in the US East (Virginia) availability zone. Over the next few weeks/months we’ll be looking to spread Interstate across more EC2 instances and availability zones.
At Interstate, we use MongoDB as our primary data store and before our switch to EC2, we managed our own MongoDB instance. Due to Interstate’s growth though, it was becoming a large effort to keep monitoring our MongoDB setup whilst still building our product. The main issue being that for the better part of a year, we hosted our MongoDB instance on the same server as our web app. This quickly became an issue as our Mongo instance was using too much of our server’s resources and so our web app quickly began to suffer.
Because of this, we made the decision to move our database to a group of guys who know a lot more about hosting MongoDB than we did, those guys being MongoHQ. We moved to MongoHQ around March 2011 and since then the experience has been pretty great.
Just as we were a year ago - we’re still using NGiNX to handle all of our requests and to serve our content. Here’s what we said in our original post regarding our choice of NGiNX..
We have decided to use nginx to serve all HTTP requests over the other popular alternatives such as Apache. We chose to use nginx after benchmarking other sites of ours that run on the same PHP framework but use Apache as the HTTP server. Both the speed difference and usage of system resources (e.g. memory) between powering these sites on nginx rather than Apache made it an easy decision for us to use nginx.
The language which Interstate is written in is pretty much always a surprise and shock for people. When we were originally meeting the other founders in our Y Combinator class, we often showed them what we were working on and 99% of the time, their first reaction was: “So what did you write this in? Rails, right?”. As we told them what language we actually used, letter by letter their face turned from interested to surprised, shocked and then somewhat horrified.
As you may have guessed (or previously read), we primarily use PHP to build Interstate. The reason behind this is not really all that strange, we’ve simply built a lot of PHP powered apps in the past and know how to build clean and efficient apps in the language. We believe you can write ugly and bad code in any language and so just because PHP has a history of badly written code doesn’t make it an inherently bad language. Yes, there are a lot of silly problems with the language itself (such as the infamous namespace separator) but most of the problems don’t really affect you (or at least they don’t affect me) day-to-day.
All that aside, the main Interstate app runs on PHP 5.3.6 using a custom (MVC) framework, which we hope to open-source in full at some point. To handle our PHP processes with NGiNX we’re still using PHP-FPM (which is now bundled with PHP as of v5.3.3).
As of recently, we’ve also began to start experimenting with Node.js for some new and exciting real-time features. Most notably, our new Streaming API (which will be opened up to developers soon) which powers pretty much all real-time updates within Interstate (such as our new Growl/Toast-esque notifications) and soon our Mac App. Our second biggest use of Node.js is our upcoming real-time team chat system, called Discuss - which Node.js has been particularly great for.
When developing apps using Node.js, we often use a mix of open-source modules made by other great developers. These modules generally include: Socket.io, node-mongodb-native, node-memcached and node-redis to name just a few.
To manage our Node apps we also use forever.
We still use APC for PHP bytecode caching and Memcached for all in-memory caching. Our use of Redis for caching has somewhat reduced and we’re now using Memcached a lot more aggressively than we were a year ago.
As said above, our usage of Redis as a cache has reduced and our main use case has shifted from caching to instead using it as a pub/sub backend for our Streaming API. So far Redis has been working really well for this use case. Once the Streaming API is made public, we’ll try and take some time to write about how the stream works in more detail.
To help us build Interstate we also use several third party web services..
At Interstate, we use git for version control. For our git hosting we tried out several popular services but we eventually settled for the most popular choice, GitHub. We chose GitHub simply because all of our open source and public projects are already hosted there and so to host our private repositories there as well just made things easier.
For the small bits of email marketing (newsletters) which we do, we have started using Mailchimp. We’ve only sent out one campaign so far but the service has been great.
For transactional email delivery (e.g. forgotten password emails, notifications, etc) we still use Postmark. Postmark makes transactional email delivery super easy (just as it should be), they provide great and simple analytics and an easy (and pretty cheap!) pay-per-email pricing model. We’re more than happy to keep giving these guys our money.
Finally, to build Interstate we use a bunch of great apps, including..
Coda to write all code for the web app.
Photoshop CS5 for all the UI.
Xcode to build the Mac App.
MongoHub for quick querying of our database.
Spotify for the great tunes.
Gaug.es for web analytics.
Interstate for project management ;)
If you have any questions about our particular use of any software in our stack or anything else, please just write a comment and we’d be happy to answer.
TL;DR: We’re hosted on Amazon EC2, running CentOS, NGiNX, PHP (+ PHP-FPM) and Node.js. We use Mongodb as our data store, Memcached for memory caching and Redis as a pub/sub backend.