Shiny Engine! by Terrence Tak-Shing Tam on Flickr

Until last week, the software architecture powering our core site at outside.in was a standard Ruby on Rails application stack sitting on top of a set of PostgreSQL databases, with various aggressive caching strategies in play on the front end. It was no surprise that the setup began showing its limitations as site traffic ramped up. True to our aspirations to become a high-traffic destination, we were becoming victims of our own success, and as a result our scaling challenge was real. Page load times, overall throughput, and latency for new content all began to degrade as databases grew. And we faced a queue of new products and enhancements our over-caffeinated product team was excited to launch, each a big driver of additional traffic. We endeavored to meet the challenge with a carefully planned re-imagining of the OI platform. Here’s a high-level look at what it entailed.

Earlier this year we had the opportunity to build a new product and business, Outside.in for Publishers, from scratch.  We maximized that opportunity by rethinking the core data models and the way we handle geometric calculations, with primary goals of speed for users and minimal latencies for content acquisition. With the OIP launch, we also unleashed a new framework for content acquisition, code-named Feed Monster, which can happily overpower any mere mortal relational database with its light-speed content analysis and row insert pressure. We were very effectively adding capacity for crawling and content analysis, but not keeping pace publishing new content because of relational bottle-necking. After the launch of OIP, we plotted a road map to bring the speed optimizations of OIP to the core site and handle the Feed Monster volume without breaking a sweat…with a few other tricks in the mix.

So, on to the specifics. The knowns we started with were:

• Relational databases are slow when they become large.
• For all its elegance and productivity, Ruby is slow to execute.
• Geometry, even with a mature system like PostGIS, is slow and hugely resource-intensive at scale.

• Denormalize content and metadata into a search-based structure
• Move the heavy-lifting out of Ruby and into a faster stack by building a cluster of “datajoiners.”
• Intelligently cache long-lived data in the datajoiner, where it is most flexibly utilized for various output types.

Denormalizing data into a set of search indices in a master/slave clustered environment enables very fast content retrieval without the overhead of relational integrity. We apply our computing resources at content processing time to make locating and displaying content very lightweight. We have accomplished this by embracing Apache Lucene and building some clever code to shard indexing across the farm.

Datajoiners are a set of servers that power content delivery to our internal APIs and front-end applications. The new middleware tier is built on the Java Virtual Machine in Scala, where we can take full advantage of multiple cores for parallelization with minimal effort. The datajoiner’s purpose is to take data from disparate sources, like PostgreSQL, Lucene, key value stores, and memory caches, and to speedily produce an output suitable for any of our consumers.

In the area of language performance, we knew we would be introducing a compromise between the development speed of a dynamic language, Ruby, and the execution speed and threading of a JVM-based one. We chose Scala for the datajoiner because it offers a bit, although I won’t say the best, of both worlds. Scala is a young language with its share of warts, but shows tremendous promise. With type inference, its myriad syntactic conveniences, and fast run-time performance, Scala served us well in this area of our architecture.  And ultimately performance handily exceeded our goals. Under massive load scenarios, we are able to service 8 to 10 front-end Rails web servers with each of these datajoiners without compromising page load time or requests per second.

Finally, we revisited caching and baked it into the datajoiner layer. Typically caching is done as close to the front-end as possible to reduce resource usage by serving stale data. In our case, caching policies can be applied at a very granular level such that content is served in near realtime without staleness, while long-lived data structures like region containment are efficiently cached for long periods. With the data assembly work being done in Scala, we are able to push tons of data into the system and serve it out without sacrificing freshness.

So, as Lauren says, welcome to the new Outside.in! I hope this post provides a useful peek behind the scenes of our most recent effort. I’m extremely proud of the team for bringing this to life, and it is incredibly rewarding for us to see it working flawlessly in the wild. The only thing that excites me more right now is the bright future of possibility our new platform represents. Stay tuned for a host of new stuff that will be powered by our new engine!