Look ma, no internets! Yopa!

If you’ve been following this blog, you’ve heard about our ongoing efforts to break our monolithic Rails application into smaller, more manageable pieces. You’ve also heard about our efforts around a better usage of the messaging features of the Amazon Web Services (AWS) cloud platform. These efforts are obviously linked: evolving towards an event-driven message-passing share-nothing architecture is our goal and we’re hard at work to make it happen.

Why are we doing this? Here are a couple of good reasons:

• Monolithic applications are expensive to maintain and hard to evolve: having smaller applications to work with will allow us to roll out new features faster. Our recent release of Reputator, a standalone application that we’ve built and launched in a few iterations, proved that we’re on the right track.
• Event-driven architectures open the door to new and unexpected features: when applications broadcast generic events that represent outcomes of their activities, it becomes possible to graft new business features to these events without any change. We expect this to percolate up to the front-end, where it would materialize in a more dynamic user experience.

We’ve defined the following architectural blueprint for our applications:

This translates in the following principles that each application should abide by:

• It jealously encapsulates its state and only shares it via well defined endpoints,
• Synchronous (request-response) interactions are performed over HTTP/S endpoints, using JSON-schema defined entities and RAML for the API specification.
• Asynchronous (one-way) requests are performed over queue endpoints, using messages defined with JSON schema.
• Events are broadcast over topic endpoints, again using messages defined with JSON schema.

We’re using Amazon SQS and SNS for our queues and topics, which is great because they are very solid, highly available services. As our applications grow, so do their dependencies, as represented in the diagram below:

This is not a problem by itself but it makes our developers lives more complex because, when they want to set-up their personal environment, they need to create and configure personal copies of these queues, topics and topic subscriptions. All this set-up could be scripted or automated quite easily, but one very important feature would still be missing, namely the capacity to trace the flow of messages and events through topic subscriptions.

Hence yopa (aka Your Own Personal AWS), a tool that we’ve built to simulate AWS queues and topics locally and that we’ve just open sourced.

yopa makes it very easy to define queues, topics and subscriptions (SQS and HTTP/S) via a simple configuration file and also the standard AWS client SDK (we’ve tested with the Ruby and JVM SDKs). Currently yopa focuses on messaging only but we’re already considering adding support for EC2 and S3 features as well, as it may further improve our developers capacity to work using the AWS APIs but without needing to actually connect to it over the Internet.

If you’re using SQS and SNS, please give yopa a try! Bug reports and pull requests are more than welcome.

-David Dossot
Director of Software Architecture 

PS. Unbounce <3 open source. Check our other projects here: http://unbounce.github.io/