It has been a particularly busy and exciting couple of weeks here at Outside.in HQ. CEO Mark Josephson and co-founder/Chairman Steven Johnson both spoke about the future of news and media to key audiences; Mark at NAA and Steven at South by Southwest. Steven’s speech is available to read here. They each addressed the problems the news industry is facing head on, and not surprisingly our company has a lot to do with the future they portrayed in their messages. In this post, I’ll share some of the efforts the OI engineering team is executing to bring Steven’s and Mark’s notion of Aggregation/Curation/Networks to life.
Software development in a technology startup is a constant juggling act. One aspect you can always count on is the need to balance infrastructure/scaling work with development of new features and evolution of old ones. We have key initiatives on all of these fronts going on right now.
Keeping infrastructure humming under loads that increase week over week as our user base grows is ideally what I call a “dollar solvable” problem, meaning we simply bring up some additional machines, watch work distribution kick in, and get back to our software engineering work. The reality is that in any large system, bottlenecks and hotspots develop such that adding hardware incrementally is not always effective, and at those junctures code is refactored or replaced. In both cases, it is critical to have a computing environment that is as agile as possible; it must be simple and cheap to bring new machines into clusters, even simpler to remove them, and trivial to spin up small networks and environments for testing and experimentation. To achieve this, we have moved to a “cloud” environment for much of our live product presence. About 2 weeks ago, on a calm sunny Sunday in New York, we seamlessly migrated from a fixed data center model for hosting to a very flexible virtual environment. By the end of that afternoon we officially flipped the switch and watched our traffic migrate to the new network, and we haven’t looked back. Virtual hosting isn’t a panacea, and it is certainly no substitute for good architecture. But it does make many of the tasks around managing growth inexpensive and much faster.
Architecture is a daily focus. I’ll detail some of the innovations we’re making in subsequent posts, but the summary is we’re constructing new data models to account for curation of content. Data structures can generally be made fast for data capture or retrieval, but not for both simultaneously. We have some techniques in mind for optimizing the user experience for our partners and readers of content aggregated with our platform. It is a data design and transformation problem, and with our cloud environment we will be benchmarking and honing our approach. The goal is fast, responsive UI for all of our products built on top of an engine that maximizes throughput for very fast processing and publishing of news and local information. I think we have it all figured out, and you’ll be able to grade us in short order.
And this brings us to the salient point for you, our user. The product that brings the aggregation/curation/networks vision to the forefront of what we do is in active development, and it will be announced very soon. “Very soon” means the engineering and product teams are gunning for an aggressive release date, and the tone in the office shows that to be the case; average caffeine intake is on the rise and the whiteboards are full. The release will feature our best work on all fronts, and that work will also raise the bar for the central Outside.in website, our iPhone application, the Geo Toolkit (which just got a slick facelift), and our partner offerings.
Lastly in this installment, I want to applaud OI co-founder and VP of Engineering Cory Forsyth who is off to the sold-out Scotland on Rails conference at the end of this month. Cory is presenting a talk on image processing in Ruby (read more here). Outside.in has become a fixture in the NYC Ruby community, fueled by our monthly Ruby Happy Hour meetups and events like Cory’s talks. Look for more from us in the Ruby and Open Source realms in the coming months.
Now, back to the juggling act! I’ll post again soon.
