One year ago, we wrote a blog post explaining Interstate's stack and architecture after we were just two months in to development. Since then a lot has changed on both the product and business sides of Interstate (we've released 3 new major versions since then and we recently took part in the Summer 2011 cycle of Y Combinator) and so I thought this would be a good time to explain what’s changed in the past year in terms of our stack.
At the time of our last post, we were running Interstate on a single server from the guys at MediaTemple. Whilst MediaTemple were a great host, they didn’t really support our expansion plans as well as other hosts could. Because of this, we moved our web app over to Amazon Web Services in June/July 2011. At the current moment in time, we’re running just one large (7.5GBs memory, 4 EC2 Compute Units) 64bit EC2 instance (with CentOS) in the US East (Virginia) availability zone. Over the next few weeks/months we’ll be looking to spread Interstate across more EC2 instances and availability zones.
At Interstate, we use MongoDB as our primary data store and before our switch to EC2, we managed our own MongoDB instance. Due to Interstate’s growth though, it was becoming a large effort to keep monitoring our MongoDB setup whilst still building our product. The main issue being that for the better part of a year, we hosted our MongoDB instance on the same server as our web app. This quickly became an issue as our Mongo instance was using too much of our server’s resources and so our web app quickly began to suffer.
Because of this, we made the decision to move our database to a group of guys who know a lot more about hosting MongoDB than we did, those guys being MongoHQ. We moved to MongoHQ around March 2011 and since then the experience has been pretty great.
Just as we were a year ago - we’re still using NGiNX to handle all of our requests and to serve our content. Here’s what we said in our original post regarding our choice of NGiNX..
We have decided to use nginx to serve all HTTP requests over the other popular alternatives such as Apache. We chose to use nginx after benchmarking other sites of ours that run on the same PHP framework but use Apache as the HTTP server. Both the speed difference and usage of system resources (e.g. memory) between powering these sites on nginx rather than Apache made it an easy decision for us to use nginx.
The language which Interstate is written in is pretty much always a surprise and shock for people. When we were originally meeting the other founders in our Y Combinator class, we often showed them what we were working on and 99% of the time, their first reaction was: “So what did you write this in? Rails, right?”. As we told them what language we actually used, letter by letter their face turned from interested to surprised, shocked and then somewhat horrified.
As you may have guessed (or previously read), we primarily use PHP to build Interstate. The reason behind this is not really all that strange, we’ve simply built a lot of PHP powered apps in the past and know how to build clean and efficient apps in the language. We believe you can write ugly and bad code in any language and so just because PHP has a history of badly written code doesn’t make it an inherently bad language. Yes, there are a lot of silly problems with the language itself (such as the infamous namespace separator) but most of the problems don’t really affect you (or at least they don’t affect me) day-to-day.
All that aside, the main Interstate app runs on PHP 5.3.6 using a custom (MVC) framework, which we hope to open-source in full at some point. To handle our PHP processes with NGiNX we’re still using PHP-FPM (which is now bundled with PHP as of v5.3.3).
As of recently, we’ve also began to start experimenting with Node.js for some new and exciting real-time features. Most notably, our new Streaming API (which will be opened up to developers soon) which powers pretty much all real-time updates within Interstate (such as our new Growl/Toast-esque notifications) and soon our Mac App. Our second biggest use of Node.js is our upcoming real-time team chat system, called Discuss - which Node.js has been particularly great for.
When developing apps using Node.js, we often use a mix of open-source modules made by other great developers. These modules generally include: Socket.io, node-mongodb-native, node-memcached and node-redis to name just a few.
To manage our Node apps we also use forever.
We still use APC for PHP bytecode caching and Memcached for all in-memory caching. Our use of Redis for caching has somewhat reduced and we’re now using Memcached a lot more aggressively than we were a year ago.
As said above, our usage of Redis as a cache has reduced and our main use case has shifted from caching to instead using it as a pub/sub backend for our Streaming API. So far Redis has been working really well for this use case. Once the Streaming API is made public, we’ll try and take some time to write about how the stream works in more detail.
To help us build Interstate we also use several third party web services..
At Interstate, we use git for version control. For our git hosting we tried out several popular services but we eventually settled for the most popular choice, GitHub. We chose GitHub simply because all of our open source and public projects are already hosted there and so to host our private repositories there as well just made things easier.
For the small bits of email marketing (newsletters) which we do, we have started using Mailchimp. We’ve only sent out one campaign so far but the service has been great.
For transactional email delivery (e.g. forgotten password emails, notifications, etc) we still use Postmark. Postmark makes transactional email delivery super easy (just as it should be), they provide great and simple analytics and an easy (and pretty cheap!) pay-per-email pricing model. We’re more than happy to keep giving these guys our money.
Finally, to build Interstate we use a bunch of great apps, including..
Coda to write all code for the web app.
Photoshop CS5 for all the UI.
Xcode to build the Mac App.
MongoHub for quick querying of our database.
Spotify for the great tunes.
Gaug.es for web analytics.
Interstate for project management ;)
If you have any questions about our particular use of any software in our stack or anything else, please just write a comment and we’d be happy to answer.
TL;DR: We’re hosted on Amazon EC2, running CentOS, NGiNX, PHP (+ PHP-FPM) and Node.js. We use Mongodb as our data store, Memcached for memory caching and Redis as a pub/sub backend.
Interstate has been in full-on development for just around 2 months now (and in private BETA for just over 1 month) and in that small amount of time we’ve had to make some pretty quick yet equally important choices when it comes to our site’s technical setup.
We’ve had to decide things from simply where to host our servers to questions like which HTTP server we should use. The great thing about the Interstate project is that because we’re just 2 guys building something we love to both make and use, we get to make some pretty interesting choices when it comes to the site’s architecture.
Because of this I’ve decided to do a post explaining what we use to power and build Interstate. So, without further ado, here we go!
To develop Interstate, Greg and I both use Apple Macbooks and take advantage of some pretty sweet Mac software. The software we find ourselves using on a daily basis is:
Adobe Photoshop CS5 (Greg uses this bad boy to create the fantastic design you lucky BETA testers get to use)
Coda by Panic (We both use this amazing IDE to code both the frontend and the backend of Interstate)
Transmit by Panic (Greg uses this to upload the bulk of the site’s static files such as images, etc)
Terminal (Simon uses this to keep control of Interstate’s server)
Google Chrome (Possibly the best browser out there at the moment)
MongoHub (Simon uses this to monitor MongoDB)
Forrst (O.K., this isn’t exactly software but it’s an awesome community where we try and share as much stuff as we can whilst we build Interstate).
Up until the latest release of Interstate (“Frank”) both the frontend and the backend was hosted on a single RapidSwitch server which also hosted about 7 other websites of ours as well. This really did not help when it came to making the site as fast as it could be. At this point we also hosted our databases on a separate server over at MediaTemple. This really did not help as it meant we had to make thousands of requests to a server that was on a completely different network to the backend.
With the latest release of Interstate we decided that we needed to fix this right away. As of writing this post we are currently hosted on a single (MediaTemple based) dedicated server with around 2GB of RAM (more spec details can be found on MediaTemple's website). We are looking to separate our backend from our database in the very near future (both in to individual MediaTemple (ve) servers).
Interstate uses PHP (PHP 5.3.3 to be exact) and it runs on its own custom framework. We have decided to skip the conventional route of using MySQL for our primary datastore and we have instead opted to use MongoDB and so far it is performing extremely well. We also use a combination of Redis and Memcached for our currently rather small caching layer that will most probably be used a lot more in the near future.
We have decided to use nginx to serve all HTTP requests (and PHP-FPM as the PHP process manger) over the other alternatives such as Apache. We chose to use nginx after benchmarking other sites of ours that run on the same PHP framework but use Apache as the HTTP server. The speed difference between powering these sites on nginx rather than Apache made it an easy decision for us to use nginx.
To make sure Interstate is always performing as well as it should be, we use a great piece of server monitoring software called munin.
When it comes to 3rd party web services we currently only use one: Postmark (aka Postmarkapp). We use Postmark to simply deliver all on-site emails (such as invite codes, notifications, etc) and we really can’t recommend it enough.
I hope this gives you a little better idea of how the site works. If you have any specific questions regarding Interstate’s development/architecture feel free to send me an email via firstname.lastname@example.org.