AddThis Blog

This is the second in a three part series on the processing and infrastructure of AddThis. Make sure to check out part one and part three.

Fast forward to today. Our data infrastructure supports AddThis code deployments on more than 14 million domains, ingesting billions of events and terabytes of logs daily.  Scores of machines has grown to around a thousand totalling tens of terabytes of RAM and petabytes of disk space.

The majority of our internal compute capacity is dedicated to our custom data processing platform.  This platform is highly dependent on large IO capacity and low latency more than CPU cycles.  Virtualization, generally speaking, can be manageable for many CPU and memory-bound tasks, but is almost always destructive to the performance of network and disk subsystems.  As a result, we employ no virtualization save for a few test machines.  And while we run relatively commodity network gear, our decisions around topology and infrastructure have been made deliberately and carefully to maximize performance.  In our experience, the compounding effects of latency and IO bottlenecks quickly become a non-linear problem.

• even with the best of intentions, virtualization kills performance
• compress everything, keep it compressed even in RAM until it’s needed
• all data must stream (compressed!), remote procedure calls are evil
• data must be sharded for locality of processing, not physical locality (see previous)
• model around simple tasks that can checkpoint and replicate
• anything that provides transactional safety between checkpoints works against you
• design for (constant) failure whether in a public or private cloud

AddThis employs top-tier CDNs for most static web-asset serving, but we have still chosen to internalize our http-based APIs.  Even though cloud offerings excel at many web-service scenarios, they mainly rely on traditional LAMP stacks with limited back-end commitments beyond scripting and modest databases.  In these environments, virtualization is usually OK because most instances vastly under-utilize system resources. But when contention rears its head in the cloud, there is no recourse other than take the penalty or provision random new hardware.

Our “edge” and eventing API are http services that require maniacal tuning for performance and reliability.  With billions of requests a day spread across millions of domains, our code is trusted by site owners to respond quickly.  Latency *is* supreme.  Flowing from our edge and eventing APIs to the backend are terabytes of real-time data.  This live stream is forked into log storage and real-time systems.  Proximity to our compute clusters enables both faster and more economical processing of data by avoiding repeating data across high-cost, high-latency links to remote data centers.

• commodity network gear with the right topology and software can work wonders
• optimization of networks, proxies and kernels remains a high art
• chrome does unspeakable things (for another day)

AddThis’ compute clusters are constantly processing and re-processing hundreds of terabytes of data, applying new algorithms, generating new metrics and finding deeper insights.  Our current model supports complete visibility into optimizing our stack.  With a top to bottom view on hardware and software, we can squeeze out more than 10 times the performance per dollar than we can get from cloud-based offerings.  Equally important, we can do this reliably and predictably.

Continue to Part III: A Simple Decision – Performance, Predictability, Price

This is the first in a three part series on the processing and infrastructure of AddThis. Be sure to check out part two and part three.

Sailing Past the Clouds

The emergence and maturation of multi-tenant, virtualized, commodity computing (“the cloud”) has enabled entirely new business models by bringing large scale distributed computing at an affordable price point.  But by designing systems flexible enough to function well in most scenarios, but tackling no specific problem, cloud vendors have made architectural trade-offs that are particularly challenging to us.  Over the last five years, we’ve built a lower cost, more effective, more scalable solution that is closely tailored to the needs of a large scale data, low latency businesses like ours.  And in doing so we have gained control over performance, predictability and price.

A Little History

AddThis’ success in the Widget business, starting in 2006 / 2007, catapulted us directly into scale problems that early clouds of that time were not ready to handle.  At that time, our platform was providing real-time analytics for hundreds of thousands of widgets being viewed hundreds of millions of times a day.  It doesn’t seem like much compared to the billions of events we’re tracking today, but it was on par with the social giant of that time: MySpace.

Early decisions, which were formative to our thinking, were based on pragmatism.  We were small and the price to upgrade databases at the scale we needed was out of the question.  The performance of scripted / hosted commodity web stacks could not predictably meet our needs.  Before long, we were forced to replace our SQL databases with a much faster (and simpler) in-house developed noSQL store just to keep up with massive data growth (the term noSQL, like Hadoop itself, didn’t exist yet). We found we could get better scale with operational safety and huge cost savings by architecting around a few key learnings:

• Databases can be fast, but network connections and serialization are slow
• Transactional safety is untenable overhead for real-time anything
• In-process local data CRUD is at least 1000 times faster
• Have hot spares/replicas, stagger checkpoints
• Backup only logs, not databases
• Design around replay from logs

Continue to Part II: AddThis Today

Google Chrome 19 landed yesterday, bringing with it Web Intents, a new method for web sites/apps to communicate with one another. We wrote about Web Intents earlier this month, but now that it’s here we are making good on our promise and rolling out a suite of AddThis features to help publishers and users take advantage of Web Intents right off the bat:

Web Intents Support

Want to add Web Intent support to your site? If you use AddThis, you already have. Just drop this line into your addthis_toolbox:

<a class=”addthis_button_intent_share_url”></a>

Web Intents users will be able to share your page through their favorite intent handler just like any other service.

Intents support for non-Chrome users

Web Intents are great for Chrome users, but what about the other two-thirds of visitors? AddThis provides another one-liner to make web intents work no matter what. Set:

addthis_config.webintents = true;

and AddThis will handle Intent events in the absence of native support. Implement Web Intents however you like—in the AddThis toolbox or not—and rest assured that your visitors will have access to their preferred service.

The AddThis Intent Handler

For those of you testing out your brand-spanking-new Chrome installs, remember that not all of your favorite services have an intent handler in the Chrome Web Store yet. Not to worry—AddThis does! Install it as you would a normal Chrome extension and give it a spin on a demo intent.

Watch this Space

As Web Intents grows, so will our support for it. Meanwhile, we’re developing tools to enable OExchange providers to handle Web Intents and integrating intent handling into our browser extensions.

Web Intents is a new tool in your sharing arsenal, and AddThis is thrilled to support it! Your feedback on our Web Intents support is invaluable, both now and in the coming months as your implementation of intents evolves alongside Web Intents itself.