One year ago, we wrote a blog post explaining Interstate's stack and architecture after we were just two months in to development. Since then a lot has changed on both the product and business sides of Interstate (we've released 3 new major versions since then and we recently took part in the Summer 2011 cycle of Y Combinator) and so I thought this would be a good time to explain what’s changed in the past year in terms of our stack.
At the time of our last post, we were running Interstate on a single server from the guys at MediaTemple. Whilst MediaTemple were a great host, they didn’t really support our expansion plans as well as other hosts could. Because of this, we moved our web app over to Amazon Web Services in June/July 2011. At the current moment in time, we’re running just one large (7.5GBs memory, 4 EC2 Compute Units) 64bit EC2 instance (with CentOS) in the US East (Virginia) availability zone. Over the next few weeks/months we’ll be looking to spread Interstate across more EC2 instances and availability zones.
At Interstate, we use MongoDB as our primary data store and before our switch to EC2, we managed our own MongoDB instance. Due to Interstate’s growth though, it was becoming a large effort to keep monitoring our MongoDB setup whilst still building our product. The main issue being that for the better part of a year, we hosted our MongoDB instance on the same server as our web app. This quickly became an issue as our Mongo instance was using too much of our server’s resources and so our web app quickly began to suffer.
Because of this, we made the decision to move our database to a group of guys who know a lot more about hosting MongoDB than we did, those guys being MongoHQ. We moved to MongoHQ around March 2011 and since then the experience has been pretty great.
Just as we were a year ago - we’re still using NGiNX to handle all of our requests and to serve our content. Here’s what we said in our original post regarding our choice of NGiNX..
We have decided to use nginx to serve all HTTP requests over the other popular alternatives such as Apache. We chose to use nginx after benchmarking other sites of ours that run on the same PHP framework but use Apache as the HTTP server. Both the speed difference and usage of system resources (e.g. memory) between powering these sites on nginx rather than Apache made it an easy decision for us to use nginx.
The language which Interstate is written in is pretty much always a surprise and shock for people. When we were originally meeting the other founders in our Y Combinator class, we often showed them what we were working on and 99% of the time, their first reaction was: “So what did you write this in? Rails, right?”. As we told them what language we actually used, letter by letter their face turned from interested to surprised, shocked and then somewhat horrified.
As you may have guessed (or previously read), we primarily use PHP to build Interstate. The reason behind this is not really all that strange, we’ve simply built a lot of PHP powered apps in the past and know how to build clean and efficient apps in the language. We believe you can write ugly and bad code in any language and so just because PHP has a history of badly written code doesn’t make it an inherently bad language. Yes, there are a lot of silly problems with the language itself (such as the infamous namespace separator) but most of the problems don’t really affect you (or at least they don’t affect me) day-to-day.
All that aside, the main Interstate app runs on PHP 5.3.6 using a custom (MVC) framework, which we hope to open-source in full at some point. To handle our PHP processes with NGiNX we’re still using PHP-FPM (which is now bundled with PHP as of v5.3.3).
As of recently, we’ve also began to start experimenting with Node.js for some new and exciting real-time features. Most notably, our new Streaming API (which will be opened up to developers soon) which powers pretty much all real-time updates within Interstate (such as our new Growl/Toast-esque notifications) and soon our Mac App. Our second biggest use of Node.js is our upcoming real-time team chat system, called Discuss - which Node.js has been particularly great for.
When developing apps using Node.js, we often use a mix of open-source modules made by other great developers. These modules generally include: Socket.io, node-mongodb-native, node-memcached and node-redis to name just a few.
To manage our Node apps we also use forever.
We still use APC for PHP bytecode caching and Memcached for all in-memory caching. Our use of Redis for caching has somewhat reduced and we’re now using Memcached a lot more aggressively than we were a year ago.
As said above, our usage of Redis as a cache has reduced and our main use case has shifted from caching to instead using it as a pub/sub backend for our Streaming API. So far Redis has been working really well for this use case. Once the Streaming API is made public, we’ll try and take some time to write about how the stream works in more detail.
To help us build Interstate we also use several third party web services..
At Interstate, we use git for version control. For our git hosting we tried out several popular services but we eventually settled for the most popular choice, GitHub. We chose GitHub simply because all of our open source and public projects are already hosted there and so to host our private repositories there as well just made things easier.
For the small bits of email marketing (newsletters) which we do, we have started using Mailchimp. We’ve only sent out one campaign so far but the service has been great.
For transactional email delivery (e.g. forgotten password emails, notifications, etc) we still use Postmark. Postmark makes transactional email delivery super easy (just as it should be), they provide great and simple analytics and an easy (and pretty cheap!) pay-per-email pricing model. We’re more than happy to keep giving these guys our money.
Finally, to build Interstate we use a bunch of great apps, including..
Coda to write all code for the web app.
Photoshop CS5 for all the UI.
Xcode to build the Mac App.
MongoHub for quick querying of our database.
Spotify for the great tunes.
Gaug.es for web analytics.
Interstate for project management ;)
If you have any questions about our particular use of any software in our stack or anything else, please just write a comment and we’d be happy to answer.
TL;DR: We’re hosted on Amazon EC2, running CentOS, NGiNX, PHP (+ PHP-FPM) and Node.js. We use Mongodb as our data store, Memcached for memory caching and Redis as a pub/sub backend.
Just over a month ago we pushed out a minor update to Interstate which included a few important features such as file sharing and roadmap exporting, which we hope you loved. “Yeah, that was cool but what have you guys been doing since then?”, I hear you ask. Well, I can sure promise you that we haven’t just been sitting around watching Reno 911 and It’s Always Sunny in Philadelphia all day.. Well, O.K., maybe we have, but we’ve also been hard at work working on the next Interstate release! Which by the way, has just been launched!
This one’s a biggy. Ever since we shipped the original Interstate BETA, back in August 2010, there has been one feature which almost everyone has asked for: sub-roads. You lovely BETA testers have been telling us that you’d like a way to add roads inside of roads (I N T E R C E P T I O N) so that you can add many tasks to one individual feature or group lots of small features together. Well if you were someone who wanted this functionality, today is your lucky day. We have just released our implementation of sub-roads which we’re calling “Routes”.
Routes allow you to create a parent road (which can have a title and description only) which you can then assign roads to. A route’s progress is completely based off its sub-roads. For example, if you had a route which let’s say had two sub-roads, with one set to “10% complete” and another set to “50% complete”, the route itself would be set to “30% complete”. This means you can see, at a glance, how far along each individual task is and how far away the feature in general is from shipping. Handy, huh?!
We’ve tried to keep our implementation as simple as possible (without losing important functionality) so that it should meet everyone’s needs and can be used in almost any situation.
For some time now we’ve felt as though our frontend site didn’t really show off what exactly Interstate offers. For a company that’s charging for its product and relies on revenue to keep its service afloat, having a descriptive homepage is something that is majorly important. But as you know, we here at Interstate do not yet charge for our service as we’re in a BETA testing phase and so this hasn’t been something that we have had to rush to fix.
But as we’re getting ever closer to a full release of Interstate that we’re both 100% happy with and willing to charge for, we thought it was a good time to come up with a new frontend site and so we did just that.
The new frontend, which was launched today has a brand new design, improved tour page, re-designed roadmap page and lots more. Hopefully this’ll make things easier when explaining what exactly Interstate offers to your colleagues, etc.
Since the second release of Interstate we’ve had the simple functionality of allowing you to have a public-facing roadmap using our http://roadma.ps/ URL. Well, since we originally released this feature we’ve had some feedback on how to improve it.
The main comment we received regarding this feature was making the short URL actually a short URL. You see, up until now every roadma.ps link looked something like this: http://roadma.ps/4c2d3b5f8ead0ec070010000. Which, well, isn’t very short. For this release of Interstate we went back to the drawing board and we were able to make our short URLs well and truly short and look something like this: http://roadma.ps/8d. Much better, no?
The second idea we often received was to add more privacy settings to public roadmaps. We listened to this feedback and in the new release of Interstate, you are now able to set your public roadmaps to be password protected. This means you are now able to add an extra step of security to your public roadmaps by adding a password which people must first enter before they can see your roads.
The next feature is something which hasn’t been suggested that often but we feel as though some of our users will find it useful. Even if you’re using the roadma.ps short URL to share your roadmaps (instead of embedding them on your website), you may still want to link back to your website or provide a way for your users to get in touch regarding a road on your roadmap. Because of this we’ve added three new roadmap settings:
This input lets you link back to your website so that people viewing your roadmap know its source.
This URL is a way for your users to get in touch with you. We generally recommend to fill this input with an email or a Get Satisfaction page (or something similar).
If you’d like to give a quick update/message to people viewing your roadmap or just link to some other places (such as your Twitter, Facebook page, etc), you can now do so by entering a message which will appear at the top of your roadma.ps page.
From using Interstate ourselves, we know that when you set a road to “Launched”, you often want to share this change with your users to let them know the feature is now available. In our case, we do this via Twitter. So we thought to ourselves, why not make things easier for everyone else who works like this?
In the new release of Interstate you are now able to connect with a Twitter account (1 per roadmap) and as you set roads to “Launched”, you will be asked (unobtrusively of course) if you’d like to tweet this change. You don’t have to tweet the change straight away and as you continue to change more roads to “Launched”, the pre-generated Tweet will change also, to include the newly launched roads.
Roadmap exporting is a feature which was added in the last release of Interstate and whilst the feature works as we’d like it to, the design used when exporting roadmaps wasn’t, in our opinion, the best it could be.
Because of this we decided to redesign the PDF version of roadmaps to look a little simpler, show a lot more data and look more like the regular roadmaps view on Interstate.
It’s an important part of Interstate yet it just didn’t seem to live up to its job. We originally wanted overview to provide you with a great experience for getting a general idea of what’s been going on. Unfortunately, it just didn’t seem to live up to this requirement. It was rather dull and lacking on the descriptive front.
To rectify this we’ve improved the overall look of the feeds which includes display people’s avatars and including a more detailed view when a road is updated.
On top of that we’ve included a little section to see who is currently active or when they were last active. This should allow you to work more closely with other people on your team, even when you are on different sides of the world.
Most importantly though, commenting has arrived. A lot of people had questioned whether road updates were truly the best way to allow a team of people to communicate what’s happening with a particular road. Commenting was something which was asked for however we always said it just wouldn’t fit into the requirements we set out for Interstate a long time ago. This meant we needed a clever way to allow interaction without over complicating roadmaps, activity commenting was the way to go. What this means is when you are actively working on a road with someone you can spark a discussion based on what has actually happen rather than what you think has happened or plan to do. Time will tell if we’ve got this one right!
The people section of Interstate has always seemed like the lost child. I didn’t get much attention on the original release of Interstate and was neglected ever since. We’ve had a little refresh of the page to make it more appealing as well as bring some important features to the foreground. Previously you were required to visit a rather empty page in order to see who has requested access to your Interstate, this now appears to the right hand side on the main page. Why make it more difficult or time consuming than it needs to be?
Over the past few months we have been part of the Yoggrt advertising network, this was to try and provide us with funds whilst we built Interstate. The income from the adverts helped somewhat towards the running costs. We’re grateful to have been accepted onto the Yoggrt network and wish the network luck for the future. Moving onwards we will now be displaying adverts courtesy of Carbon, an advertising network aimed at various circles of development and culture. We look forward to a happy relationship with Carbon and it’s team.