Category: Engineering

Building a Distributed System with Akka Remote Actors

At AddThis, we deployed our first production system written in Scala almost two years ago. Since then, a growing stack of new applications are built using this exciting language. Among the many native Scala libraries we have tried and adopted, Akka stands out as the most indispensable.

Akka is a library for building concurrent scalable applications using the Actor Model. Its fault-tolerance model is heavily influenced by Erlang. In this post, I will talk about our experience with using Akka (2.0.x) remote actors to build a distributed system, SAM. Continue…

An Introduction to AddThis Analytics Profiles

One of the most powerful features of AddThis analytics is our profiles. These allow you to separate analytics collected under your account however you want. In this post, I’ll explain how to create new profiles in your account, and then give a couple examples of how you can use them.

Creating a new profile

Creating profiles is really easy. Just go to and click the “Add Profile” button.

Profile Manager Screenshot

Then, enter the name for your new profile and click “Add”

Profile Creator Screen

This will create a profile with that name and create a random profile ID that’s associated with that profile. A couple things to note:

  • Profiles can have the same names
  • Profile IDs can’t be changed

Now that you’ve got a profile you can click on the name of the profile to edit its settings.

Profile settings page

Here’s a quick explanation of these settings:

  • Email Reports: When checked AddThis will send you weekly analytics reports for this profile
  • Alerts: This allows you to set email alerts for when shares, or clicks change for the entire profile, a specific domain that this profile tracks, or for a specific URL tracked by this profile, as well as adjust the frequency.
  • Sharing: You can share the profile with anyone who has an email address and give them either full control over it or just allow them to view the analytics.
  • Blocked Websites: If your profile ID is somehow used on a domain that you don’t want reports for you can manage that domain here.
  • Email Templates: This section lets you create specific email templates that are used when someone shares using our email system. More information on email templates is available here.
  • Content Feeds: These allow you to create RSS or JSON feeds for shares, clicks, and trending content for this profile.
  • Applications: This allows you to manage which iOS or Android apps can use this profile.
  • Labels: These allow you to group profiles together and see a roll-up of the analytics collected with these profiles. More on these later.
  • Shortener Settings: This allows you to set a specific login and API key on our servers. Then you’ll just have to configure your page to shorten using
  • Delete Profile: Just what it says – you can delete the profile if you don’t need it anymore.

Some ways you can use profiles

As a consultant

Profiles provide consultants with a way to both monitor the effectiveness of their clients’ sites and give their clients a way to keep an eye on things.

First, create profiles for each of your clients. Then, share those profiles with the relevant stakeholders for each client. If you want, you can group these clients by using labels based on their industry, the type of client, or the level of support they’ve purchased. Then you can generate reports for each one of these labels.

As an editor

If you run a site with multiple contributors, profiles can give you a way to get detailed information about how well each contributor’s content is performing, as well as seeing an overall view of what links are generating the most social traffic.

Just create profiles for each one of your editors. Then adjust your content management system to put that editor’s profile ID on their stories. You could also share these profiles with the editors so they can make use of the analytics to increase social sharing.

As an online store owner

If you run an ecommerce site selling multiple types of product you can get more granular information about sharing by using profiles. Create profiles for each product or group of products and use your theming system to set the profile ID.

One Caveat

Unfortunately our application plugins don’t currently support assigning different profiles for different criteria. To set a different profile you’ll have to add this to the page:

<script type="text/javascript">
var addthis_config = addthis_config||{};
addthis_config.pubid = 'THE PROFILE ID';

Just replace THE PROFILE ID with the relevant profile ID.

If you’ve used multiple profiles on your site, let us know how! If you have any other questions about profiles, don’t hesitate to reach out.

Even More Metrics!

We are big fans of Coda Hale’s Metrics library. Virtually all of our applications emit copious metrics that we use to monitor application state, optimize performance and resource allocation, and keep tabs on business metrics. In short: to know what is going on. We like it so much for our Java applications that we made it easy for any language with a network stack to participate with MetricCatcher and Phetric.

All of these metrics are sent to Ganglia and Graphite for dashboards, big screens filled with graphs, and fault detection. Ganglia has a solid architecutre of distributed metrics collection out of the box, and Graphite is known for its sweet graphs. We have contributed to both the Metrics Ganglia reporter and one of the available Ganglia-Graphite bridges.

Unfortunately, we ended up with a messy slew of ways to configure where we were sending metrics–Java properties files, Spring XML, even the occasional hard coded in-app configuration. It also annoyed our engineers that the Ganglia configuration was repeated both in each app and in the local gmond.conf.

Shocking though it may be, it’s also possible to have too many metrics. We have on occasion, for example, had a distributed service keep a Timer for every other service it talked through. This was totally awesome for in situ performance debugging, but the combinatorial explosion of unique host-metric-service tuples was too expensive to keep around.

Metrics-Reporter-Config tries to solve both of these problems:

  • Configuration of multiple reporters is in a single yaml file. This should fit particularly nicely into any Dropwizard services you might have written.
  • Ganglia udp_send_channels can be extracted from a gmond.conf instead of repeating them for each app.
  • It’s easy to add white or black lists for which metrics go to which reporters. We usually use this to expose some of those timing metrics only via JMX.

There are a bunch of options there, but usually we just use a simple config that looks like this:

   period: 60
   timeunit: 'SECONDS'
   gmondConf: '/etc/ganglia/gmond.conf'
     color: "black"
     - ".*JMXONLY$"

Jars are already on their way to Maven Central. Check out the README and get started. We hope that others find this code as useful as we have. Feedback, comments, patches, bugs, and forks are all welcome.

Debugging CSS & JavaScript for AddThis

Most of the support issues I see involve AddThis styles or scripts that are conflicting with the AddThis code. Debugging these sorts of conflicts is pretty easy with the right tools, and these are usually built into every browser. In this post I’ll be using Safari’s Web Developer Tools, but Google Chrome’s Web Inspector or Firebug for Firefox can do similar things.

Debugging CSS Conflicts

Since AddThis code is built from DOM elements, instead of Flash objects or similar non-standard elements, other scripts CSS rules can affect them. This is both a good thing, for instance when you want to center the toolbox, but can be a problem if your stylesheets have overly-broad rules in them. For example, this rule:

<style> a { color:white !important; } </style> 

Will result in something like this:

The counter text is white

It looks like the counter isn’t loaded, but it’s there, and you can see it if you select it:

Like magic!

To debug overly broad CSS rules, control – or right-click – on the element you wish to inspect, then select “Inspect Element”:

Inspect Element Selection

This will take you to a window where you can see all the CSS rules that apply to that element. Search through them to find the one that’s not from AddThis and then click the check box to the left to disable it. You’ll notice that it also gives you the filename and line number of the rule you’re disabling, so you can go straight to that rule in your CSS file and fix it.

Debugging JavaScript Problems

Debugging JavaScript can be a bit more difficult. The first thing to check if AddThis isn’t showing up at all is if the proper variables are set on the page. First, right click anywhere on the page and select “Show Page Source.” Then, type ‘addthis_config’ into the small area at the bottom of the window, next to the blue greater-than sign and then push enter. If the code is set up properly on your page you should see something like this:

addthis_config Output

This will let you check whether your configuration values are correct or if they’re possibly getting overridden by re-declaring the addthis_config variable elsewhere on the page. Other good ones to check are addthis_share, which lets you check the sharing configuration, and just addthis, which will verify that our code is being loaded properly.

Another problem I see a lot is errors in other JavaScript on the page causing the buttons not to render. AddThis buttons are rendered, or put on the page, using JavaScript. If there is an error in other JavaScript on the page that halts the execution of scripts on the page then our buttons won’t show up:

They're not there

This page has an onload attribute in the body that isn’t a valid function:

<body onload="AFunctionThatDoesntExist()"> 

To find out where this error is, right-click on the page and select “Show Page Source” again. Then, select the error icon in the upper left corner of the Developer Tools window. (You may have to display the window’s left sidebar to see it.)

Onload Error Window

Now that we know what the problem is, we can fix it. In this case, all we have to do is define a function named AFunctionThatDoesntExist, like this:

<script type="text/javascript"> function AFunctionThatDoesntExist(){ var docBody = document.getElementsByTagName('body'); var mySubheading = document.createElement('h2'); var myText = document.createTextNode('That function does exist!'); mySubheading.appendChild(myText); docBody[0].appendChild(mySubheading); } </script> 

Then the AddThis buttons will show up after the function runs:

AddThis displaying after the function appends text to the document

Debugging Bigger Problems

Note: This is for advanced users who don’t mind editing HTML or using a command line

Sometimes you’re not sure what file has the CSS or JavaScript conflict in it. In these case you’ll have to disable the CSS and/or JavaScript on the page until you get the buttons to work, then figure out what conflicts are occurring.

Here’s how I’d do it on a Mac, but the steps are similar for a PC. First, get a web server. For Mac users this is built in. Just go into the Sharing preference pane in System Preferences and check “Web Sharing.”

Then, open up the Terminal app in your Applications folder, which will give you something like this:

A Terminal Window

At the prompt, type this command (without the dollar sign.)

$ curl > Sites/example.html 

(Replace with the URL of the page you want to debug.) This will download the source of the page to your Sites folder in a file example.html. Open this file in a text editor and add the following just after the <head> tag:

<base href=""/> 

(Again, replace with the URL of the page you’re debugging.) Now open your browser and go to: http://localhost/~[user]/example.html where [user] is replaced with your username. Your page should show up with all the images and scripts.

Example page from my local server

Once you’ve got that, edit the page in the same text editor to start removing JavaScript and CSS, refreshing the page after each edit to see if it fixed the issue. Once you find the AddThis buttons behaving like you should, replace all but the last item you removed to see if that’s the culprit.

Once you’ve found the culprit it’s up to you to decide whether to keep using it, upgrade to a newer version that might not have a bug (if it’s a third-party library), or contact us to see if we have a workaround. Please note that we don’t support any external JavaScript beyond helping you find out that there’s a problem.

Looking for errors like this before you contact our support team might help you save time and frustration when AddThis is acting up.

October Hackathon 2012: Robotics for the Remote Worker and Plenty of Plants

Another successful AddThis Hackathon is in the books! We measure the success of a Hackathon in a few different ways:

For those not familiar with hackathons, they are our 24-hour sprints of innovation. A-Teamers group up, or work alone, on a new project outside of what they are currently working on in their day-to-day. The energy surrounding a hackathon is always fun and familial. They are a very important part of the AddThis culture.

WINNER: Telepresence Roomba Robot

The winning project of this installment of all-night coding was quite impressive. The team went out to solve a common challenge we face at AddThis. Since we have multiple offices across the country, sometimes communicating and collaborating requires extra effort. So, the team created a telepresence robot using a Roomba, a MacBook Air, foamcore and a rocket launcher. And the entire thing was controlled from a Google Hangout! See it in action:

Congratulations to the winning team! It was well deserved.

Honorable Mention: AddThis Garden

The unofficial “runners-up” also deserve Earth Day Award 2012. “AddThis Plant” went on a mission to “green” our new HQ office in Virginia. It is an extensive project of plant growing, caring and loving with the end goal to have a nice array of beautiful oxygen-producing decorations for our work space. Check them out:

And those are just the official winners! There were many other brilliant and noteworthy projects that did not go unnoticed or under-appreciated. Some of which will be used to make everyday AddThis processes more efficient — a win all by itself!

Like this? Check out more posts about AddThis Engineering.


Service Outage

We had a network outage for several hours yesterday afternoon (US time).  Our buttons continued to serve into web pages but sharing was down and our website, analytics, and web services were unavailable.  We truly apologize for the outage.  We want to be a critical part of your social infrastructure and we can’t keep your trust unless our service is always available.  We take this responsibility very seriously.  The root cause was an administrative error by one of our service providers and we’ve made changes to reduce the risk of further network outages.

PennApps 2012 Wrapup

Sunrise at PennApps
6:33am Sunday: the sun rises on the final day of PennApps

This past weekend Matt Keesan and I traveled down to Philadelphia for PennApps, America’s largest student-run hackathon. 48 hours, 91 teams, 320 hackers, $10K+ in prizes–any way you look at it, the event was huge. So before getting into some of our favorite hacks of the weekend, congratulations to the whole PennApps team, including frontman Pulak Mittal, for keeping the wifi, food, and energy flowing all weekend. AddThis was a proud sponsor and we’re looking forward to next semester!

Vivek Panyam, Facebook for Grandma

Vivek Panyam with his AddThis-infused Facebook for Grandma app
Advice for solo-hackers-to-be: Vivek worked in high-traffic areas of the hackathon and was accordingly flush with advice and attention from the 40 sponsors at the event. What’s more, he took the advice seriously.

Solo-hacker and UPenn freshman Vivek Panyam won the “most delicious UI” award for Facebook for Grandma, a pared-down Facebook app offering a compelling subset of features in an even more compelling package. Seriously, your grandmother just got 30 clicks closer to your photos; you’ve been warned.

Devon Peticulas and Dan Mundy, bbcat

PennApps veterans Devon Peticulas and Dan Mundy won big style points for scoping out and executing on a deceptively simple tamagotchi-inspired pet. Get your own at, then share it with your significant other to either test his or prove your own commitment. What impressed me most about the duo is that at the 11th-hour, when most teams were tracing syntax errors, Devon and Dan were sitting pretty, chatting about node.js design patterns and that perfect shade of beige found on the Atari 800. What’s more, they are coordinating HackRU at Rutgers next month. Maybe they’ll throw us dog-lovers a bone.

Devon Peticulas of team bbCat models AddThis shades
Devon Peticulas wears AddThis sunglasses at night. And a Venmo t-shirt for three consecutive days.

Raj, Dave, and Shiva: BuddyHack, BuddyMeme

Were there an award for the most roguish team at PennApps, the entire UMich contingent would have had to split it. Instead, we gave our own “Most Viral” award to BuddyHack, the only project that was developed, deployed, HackerNews’d, and banned before the demos even began! Luckily the team is prolific and their BuddyMeme project is still up from the Michigan Facebook Hackathon held this March. By creating a dead-simple meme generator (open-sourced, by the way) and leveraging the tight-knit nature of Facebook’s network, BuddyMeme allows memes to thrive within small social groups.

My humble self with the BuddyHack crew
I can has internship? Me and the BuddyHack team.

There were too many cool projects for me to list. AddThis was super happy to share the weekend with so many talented developers.

PennApps 2012

Joe Sullivan and I are headed down to Philadelphia today for Penn Apps 2012, an awesome bi-annual 48-hour hackathon for university students, run by Penn students themselves. There’s thousands of dollars in prizes, not to mention mentors from sponsors like us, Facebook, Google, Dropbox, Mongo, Mozilla, the New York Times, Quora, Zappos, and more. They’ll even reimburse your bus fare (which is extra rad since the price of a ticket from New York has risen substantially from the $5 I paid as an undergrad).

This session is already completely full, but there’s a waiting list for student hackers, and the next PennApps is just six months away. (All developers in the area are invited to come by to help advise teams in their areas of expertise.)

If you’re a student looking for help, or live in Philly and just want to meet your friendly local AddThis developers, please drop me a line or send a tweet (@itsjoesullivan or @keesan)! We’re excited to be sponsoring our first university hackathon.