Let’s focus our attention to concurrent data structures in a shared-memory environment where multiple threads are concurrently reading and/or writing. For a thorough background into concurrent data structures, I recommend reading The Art Of Multiprocessor Programming by Herlihy and Shavit (Morgan Kaufmann 2008, revised printing 2012). An assumption for this discussion will be that all concurrent data structures under consideration are linearizable.

First question: How do we construct concurrent data-structures? There are at least three common techniques used today, plus a fourth technique that I’ll briefly go over.

Coarse-Grained Locking

The first method is coarse-grained locking. Take your precious data, and wrap a giant lock around it.

Several large open source projects have relied on course-grained locking. The Linux kernel used a big kernel lock when it first introduced support for simultaneous multithreading in Linux 2.0. Work began in 2008 to remove the big kernel lock, and it was carefully replaced with fine-grained locks until it was removed in Linux 2.6.39. The default Python interpreter (CPython) uses a global interpreter lock to prevent multiple native threads from executing Python bytecode concurrently.

Fine-Grained Locking

The second method is fine-grained locking. Smash that coarse-grained lock into multiple locks! Each fine-grained lock is responsible for protecting a region of the data. Operations on the data are responsible for acquiring one or more of these locks in order to read, modify, and/or write.

Fine-grained locking can improve the overall throughput of a concurrent system. However, one must be careful to avoid the perils of concurrency including deadlock, livelock, starvation, preemption, priority inversion, convoying, etc. You must also have a strategy for releasing shared resources when a thread unexpectedly dies.

Lock-Free Programming

The third method extends the concept of fine-grained locking its logical extreme, which is to have lock-free programming.

Lock‑free programming is programming without locks. Or more formally, a lock‑free implementation of a concurrent data structure is one that guarantees that some thread can complete an operation in a finite number of steps regardless of the execution of the other threads. Lock‑free programming uses atomic operations, such as atomic swap, test-and-set, fetch-and-add, compare-and-swap, load-link/store-conditional, etc., to maintain consistent state. Lock‑free programming can improve system throughput and has desirable liveness properties. It is also tricky to design and implement correctly. The term wait‑free was introduced in (Herlihy, 1991). One of the earliest formal definitions of lock‑free is found in (Massalin and Pu, 1991). Jeff Preshing has a clever introduction to lock‑free programming.

Transactional Memory

The fourth method is to use transactional memory. Transactional memory allows several memory operations to be grouped together, and execute atomically. Either the entire sequence of operations appear to occur atomically, or else the system state is unchanged.

Transactions are a very familiar concept in a database management system. Transactional memory is still somewhat of an exotic concept. It can be implemented either in software or in hardware, or as a hybrid in software with hardware support. The Intel Haswell microarchitecture will have support for transactional synchronization. Scala, Haskell, and Clojure have support for transactional memory semantics (as do other languages). There are active con- and pro- discussions on the benefits of transactional memory.

More is Easy, Less is Hard

What about the secret life of concurrent data structures? Imagine you are building your own concurrent data structure. Let’s imagine this data structure as an abstract blob. There are three basic operations that we will want to support: get(), insert(), and delete(). Make a prediction on the relative effort necessary to design and implement each of the tree operations.

My ballpark estimate is that the relative effort for these operations amounts to 10% get(), 30% insert(), and 60% delete(). Common sense dictates that the modify operations are more complex than the read operation. Common sense might also lead us to think that the insert() and delete() operations are approximately equal in complexity. This is almost never the case!

It is almost always easier to correctly move from a state of less information to a state of more information than it is to move from a state of more information to a state of less information. This is even ignoring the extra complexity of ensuring that you have hygienic access to objects in memory. In other words I’m assuming a runtime which cleans up memory references, and (mostly) eliminates the ABA problem.

The relative difficulty of the delete() operation increases as one moves from coarse-grained locking to fine-grained locking to lock-free programming. To illustrate the delete() complexity phenomenon let’s take a look into lock-free linked lists. If only get() and insert() operations are necessary, then the lock-free linked list is identical in structure to the sequential linked list.

Lock-free Linked Lists

A linked-list is a collection of nodes. Each node is composed of a value, and a possibly null link reference to another node. For simplicity, lets assume that values in the linked list are immutable. The get() operation performs an atomic read on each link reference as it traverses the list in search of the target element. The insert() operation locates the predecessor node where the new node is to be inserted and attempts a compare-and-swap (CAS) operation on the link reference on the predecessor node. If the CAS operation fails then rinse and repeat until successful.

What happens when we add support for the delete() operation? It is illustrative to look at the literature ,and see how the design of the lock-free linked list has evolved over time.

Valois (1995) introduced the first lock-free linked list algorithm. Consistency is maintained by using auxiliary nodes which are defined as nodes that do not store values. Every normal node in the list is required to have an auxiliary node as its predecessor and its successor. With this design, consistency is maintained at the cost of a x2 storage overhead.

Greenwald (1999) constructed a lock-free linked list that does not rely on auxiliary nodes. The design relies on the double compare-and-swap operation (the ability to compare-and-swap two disjoint memory locations) which is not available on modern hardware.

The modern lock-free linked list design was introduced by Harris (2001) and Michael (2002). In this design two CAS operations are used to perform deletion. The first CAS operation marks the node as logically deleted. The second CAS operation physically removes the node. The linearization point is the first CAS operation. This separation of concerns between logical deletion and physical deletion is a powerful technique. It forms the basis of many modern lock-free data structures. The get() and insert() implementations need to be revised in light of this technique. The other operations need to either ignore or eliminate logically deleted nodes.

The secret messiness of deletion is indirectly exposed throughout the literature. The seminal book Purely Functional Data Structures introduces immutable red-black trees that support get() and insert() but not delete() operations.

The recent paper on the design and implementation of Bw-trees includes the following innocent sentence: “[Node] merges are decidedly more complicated than splits, and we need more atomic actions to accomplish them.”

The Bw-tree paper is definitely worth a read. I have seen other papers in the literature on concurrent data structures that simply leave out the delete() operation. Possibly these papers have sufficient research contributions such that they stand on their own merit without this operation, or possibly the data structure is designed for those contexts in which information is never deleted.

Conclusions

When you are designing and/or implementing a concurrent data structure, the final task in your workflow is typically the deletion operation. But this is the most complex of the operations you will be supporting. You must plan ahead in your design with an eye on the deletion problem. Use the technique of separation of concerns to distinguish between logical deletion and physical deletion. To my knowledge, this is the most common approach used today in supporting lock-free deletions.

Caveat emptor

My personal experience with concurrent data structures has been primarily with ordered data structures. I have a hunch that the relative complexity of deletion for unordered concurrent data structures may not be as high. But I don’t have the evidence to back up that claim. A good place to start reading up on concurrent hash tables is “Hopscotch Hashing.”

Akka is a library for building concurrent scalable applications using the Actor Model. Its fault-tolerance model is heavily influenced by Erlang. In this post, I will talk about our experience with using Akka (2.0.x) remote actors to build a distributed system, SAM.

SAM is an internal tool for managing the data processing pipeline for billions of records daily. Much of the big data heavy lifting is done with our home-grown Hydra system. SAM functions as the command and control center that submits jobs and runs queries to advance the workflow. It also needs to do intermediary and post-processing of a non-trivial amount of data returned by Hydra.

SAM is designed to be horizontally scalable, such that it can scale out when the underlying Hydra cluster scales out, or when its own workload requires more processing power. It consists of a master and multiple slaves. The master is responsible for task allocation, bookkeeping (DB access), and providing a RESTful API. The slaves are responsible for task execution, such as running hydra queries and aggregating results.

A typical workflow starts when the master receives a user request as illustrated in the sequence diagram below (the participants are Actors, and interactions are message exchanges between them):

• TaskCreator creates a task based on the user request and sends it to the TaskQueue
• TaskQueue enqueues the task as well as the sender–the TaskCreator’s ActorRef
• TaskQueue dequeues a task and sends it to a Slave’s TaskReceiver, which has registered itself as a task consumer. TaskQueue actually doesn’t know or care about the locality of a consumer–it is just an ActorRef
• TaskReceiver receives the task and sends it to the appropriate TaskProcessor, with the original sender as the result receiver
• TaskProcessor executes the task. When processing is done, it sends the result back to the receiver–the TaskCreator that created the task at the beginning. Here again, the locality of the receiver is transparent to TaskProcessor

Remote message passing is enabled by a few lines of configuration:

akka {
...
	actor {
		provider = "akka.remote.RemoteActorRefProvider"
	}
	remote {
		transport = "akka.remote.netty.NettyRemoteTransport"
		netty {
			port = 2552
		}
	}
...
}

Akka makes it easy to send messages to local or remote Actors. The code is exactly the same from the applications perspective. The Akka libraries automatically route the message to the correct actor.

Akka provides a very easy way to lookup remote actors:

val actor = actorSystem.actorFor("akka://actorSystemName@server:2552/user/actorName")

To minimize explicit knowledge of actors’ locality, however, we try to limit the use of such lookup in our code. The only place we use it is in the slaves for finding the MasterService actor defined by the master that allows the slaves to register themselves as TaskReceivers.

To detect cluster node connection/disconnection, we take advantage of Akka remote events.

On the slave side, we created a MasterMonitor actor dedicated to monitoring the master connection status. It tries to reconnect when the master is disconnected. On startup, the actor subscribes to the global RemoteClientLifeCycleEvent, and pings MasterService:

override def preStart = {
	context.system.eventStream.subscribe(self, classOf[RemoteClientLifeCycleEvent])
	reconnect(Duration.Zero)
}

The response of the ping message is not important–as a matter of fact, MasterService doesn’t even handle “ping”–it is a way to trigger the RemoteClientLifeCycleEvent that MasterMonitor is interested in:

def receive = {
	case _: RemoteClientConnected   => updateMasterStatus(newValue = true)
	case _: RemoteClientWriteFailed => updateMasterStatus(newValue = false)
	case _: RemoteClientShutdown    => updateMasterStatus(newValue = false)
}

If the outbound connection is established, the actor will receive RemoteClientConnected, and it sets the master connection status to true. When the master is disconnected for whatever reason, this actor will receive either RemoteClientShutdown or RemoteClientWriteFailed, and it will set the master connection status to false. Based on the master status, other slave actors such as TaskReceiver can change its behavior accordingly.

The MasterService has a similar approach to handling status messages. Slave disconnection triggers a RemoteClientLifeCycleEvent:

case RemoteClientShutdown(_, addr)          => unregisterSlave(addr)

case RemoteClientWriteFailed(_, _, _, addr) => unregisterSlave(addr)

Because the master must maintain statuses of multiple slaves, it uses the remote address value of the event to tell which slave is disconnected. This simple implementation for detecting cluster node status has proven very robust in practice. It works equally well in case of normal shutdown/reconnect as well as abnormal exits (e.g. kill -9).

Even though Akka makes it easy to use remote actors it is still a good practice to understand how that distribution works. Specifically one should be aware that:

• Akka uses Java object serialization by default to serialize remote messages. Make sure the message objects you use can be serialized/deserialized properly (case class is always a good choice for message)

• Default max message size is 1M, so be mindful about data transfer between remote systems via message passing. When the size of a message exceeds the limit, the error is not always obvious. (For example, if Master tried to send too large a message to Slave, a RemoteClientError would be triggered on the slave side, and RemoteClientShutdown on the master side. Had we not subscribed to RemoteClientLifeCycleEvent, it would appear that the message was simply lost.)

As we build out SAM and gain more knowledge about Akka, we realize that a few things are less than ideal in the current design and implementation:

• In most master-slave setups, master tends to become the bottleneck/single point of failure. It’s no different for SAM. Heavy usage of the RESTful API sometimes puts the master under load, and even a short downtime of the master may affect other systems that depend on it
• Master and slave have different code bases; this makes remoting more difficult. For example, we have to take extra care to make sure the classes we are using as messages between the master and slave are available in both. We ended up putting all those in a common module, which adds some complexity to the whole project
• We designed the master to be the sole location for storing data files. When a task requires files to be exchanged between the master and slave, we can’t send the content of the files as messages because they are usually too large. To work around that we make the sender send a url that the receiver can use to download the file. That becomes tedious quickly. Using a distributed file system such as HDFS can be helpful here

Going forward, we’d like to change the master-slave setup to a cluster of identical nodes with dynamic leader election. A major addition in Akka 2.1.x over 2.0.x is cluster support, including a gossip protocol for cluster membership. That can potentially simplify things quite a bit for us. We can’t wait to try it out!

SAM is not the type of high throughput low latency systems that Akka is usually used for, nonetheless we still find Akka an invaluable tool. Its actor model implementation makes building correct concurrent applications easy. In this post, we covered our usage of remote actors, which is only one of many wonderful things Akka has to offer. We look forward to sharing more of our experiences with you.

Did you get this far? Are you interested in the type of scalable work we’re doing at AddThis?  If so, why not check out the engineering openings on our careers page?

Can you connect to AddThis?

AddThis buttons are generated using JavaScript and CSS that’s stored on our servers, so if your computer doesn’t have access to these servers, you won’t be able to see the buttons. This can happen for a number of reasons, but is usually related to adware or malware blocking software. Verify you have access by going to these URLs:

• http://www.addthis.com/
• http://s7.addthis.com/js/300/addthis_widget.js
• http://ct1.addthis.com/static/r07/widget110.css

If you experience errors visiting any of those URLs, or see something that doesn’t look like AddThis, you don’t have access to AddThis and therefore the buttons won’t show up. If you can identify which software is blocking AddThis, follow the prompts to unblock us, and the buttons should show up.

Do you have all the code on the page?

If you’ve recently made changes to your site another problem might be that you didn’t put everything back on the page. View the source of your page (by right-clicking or Control-click your page, and select “View Source” or “View Page Source”) and use the find command to make sure you’ve got the following items on the page:

• The AddThis toolbox with buttons in it. (Note that the buttons inside of this might be different depending on which ones you use.)

  <div class="addthis_toolbox addthis_default_style "> <a class="addthis_button_preferred_1"></a> <a class="addthis_button_preferred_2"></a> <a class="addthis_button_preferred_3"></a> <a class="addthis_button_preferred_4"></a> <a class="addthis_button_compact"></a> <a class="addthis_counter addthis_bubble_style"></a> </div> 

• Also, make sure you have our addthis_widget.js on the page. This script loads all the other code that shows our buttons. If it’s not on the page then none of the buttons will show up:

  <script src="http://s7.addthis.com/js/300/addthis_widget.js" type="text/javascript"></script> 

Are you using WordPress?

First, make sure the plugin hasn’t somehow been deactivated. If you recently changed your theme, make sure it uses the the_content() function instead of get_the_content() as the latter doesn’t include the callback that adds the AddThis buttons to your posts.

Finally, if you’ve got a caching plugin like WP Super Cache make sure that you flush these to ensure the buttons get added to your posts.

Still don’t see them?

If you’re still not seeing the buttons after checking the steps above, you may be having another issue. In that case, contact our support team, and we’ll be happy to help!

The Find Profiles feature helps you find all of your Twitter and Facebook profiles (usernames and pages), and then allows you to select which profile(s) to use for your follow buttons.

This feature currently works with both Facebook and Twitter, and will be integrated with more social networks in future releases.

It is important to note that you do not need to create an AddThis account to use the Find Profiles feature.  But, if you do already have an AddThis account (and have linked your Facebook and/or Twitter accounts), then sign into your AddThis account for your follow buttons to automatically use your social profiles.

Create your AddThis follow buttons today!

Creating a new profile

Creating profiles is really easy. Just go to http://www.addthis.com/settings/publisher and click the “Add Profile” button.

Then, enter the name for your new profile and click “Add”

This will create a profile with that name and create a random profile ID that’s associated with that profile. A couple things to note:

• Profiles can have the same names
• Profile IDs can’t be changed

Now that you’ve got a profile you can click on the name of the profile to edit its settings.

Here’s a quick explanation of these settings:

• Email Reports: When checked AddThis will send you weekly analytics reports for this profile
• Alerts: This allows you to set email alerts for when shares, or clicks change for the entire profile, a specific domain that this profile tracks, or for a specific URL tracked by this profile, as well as adjust the frequency.
• Sharing: You can share the profile with anyone who has an email address and give them either full control over it or just allow them to view the analytics.
• Blocked Websites: If your profile ID is somehow used on a domain that you don’t want reports for you can manage that domain here.
• Email Templates: This section lets you create specific email templates that are used when someone shares using our email system. More information on email templates is available here.
• Content Feeds: These allow you to create RSS or JSON feeds for shares, clicks, and trending content for this profile.
• Applications: This allows you to manage which iOS or Android apps can use this profile.
• Labels: These allow you to group profiles together and see a roll-up of the analytics collected with these profiles. More on these later.
• Bit.ly Shortener Settings: This allows you to set a specific Bit.ly login and API key on our servers. Then you’ll just have to configure your page to shorten using Bit.ly
• Delete Profile: Just what it says – you can delete the profile if you don’t need it anymore.

Some ways you can use profiles

As a consultant

Profiles provide consultants with a way to both monitor the effectiveness of their clients’ sites and give their clients a way to keep an eye on things.

First, create profiles for each of your clients. Then, share those profiles with the relevant stakeholders for each client. If you want, you can group these clients by using labels based on their industry, the type of client, or the level of support they’ve purchased. Then you can generate reports for each one of these labels.

As an editor

If you run a site with multiple contributors, profiles can give you a way to get detailed information about how well each contributor’s content is performing, as well as seeing an overall view of what links are generating the most social traffic.

Just create profiles for each one of your editors. Then adjust your content management system to put that editor’s profile ID on their stories. You could also share these profiles with the editors so they can make use of the analytics to increase social sharing.

As an online store owner

If you run an ecommerce site selling multiple types of product you can get more granular information about sharing by using profiles. Create profiles for each product or group of products and use your theming system to set the profile ID.

One Caveat

Unfortunately our application plugins don’t currently support assigning different profiles for different criteria. To set a different profile you’ll have to add this to the page:

<script type="text/javascript">
var addthis_config = addthis_config||{};
addthis_config.pubid = 'THE PROFILE ID';
</script>

Just replace THE PROFILE ID with the relevant profile ID.

If you’ve used multiple profiles on your site, let us know how! If you have any other questions about profiles, don’t hesitate to reach out.

All of these metrics are sent to Ganglia and Graphite for dashboards, big screens filled with graphs, and fault detection. Ganglia has a solid architecutre of distributed metrics collection out of the box, and Graphite is known for its sweet graphs. We have contributed to both the Metrics Ganglia reporter and one of the available Ganglia-Graphite bridges.

Unfortunately, we ended up with a messy slew of ways to configure where we were sending metrics–Java properties files, Spring XML, even the occasional hard coded in-app configuration. It also annoyed our engineers that the Ganglia configuration was repeated both in each app and in the local gmond.conf.

Shocking though it may be, it’s also possible to have too many metrics. We have on occasion, for example, had a distributed service keep a Timer for every other service it talked through. This was totally awesome for in situ performance debugging, but the combinatorial explosion of unique host-metric-service tuples was too expensive to keep around.

Metrics-Reporter-Config tries to solve both of these problems:

• Configuration of multiple reporters is in a single yaml file. This should fit particularly nicely into any Dropwizard services you might have written.
• Ganglia udp_send_channels can be extracted from a gmond.conf instead of repeating them for each app.
• It’s easy to add white or black lists for which metrics go to which reporters. We usually use this to expose some of those timing metrics only via JMX.

There are a bunch of options there, but usually we just use a simple config that looks like this:

ganglia:
 - 
   period: 60
   timeunit: 'SECONDS'
   gmondConf: '/etc/ganglia/gmond.conf'
   predicate:
     color: "black"
     patterns:
     - ".*JMXONLY$"

Jars are already on their way to Maven Central. Check out the README and get started. We hope that others find this code as useful as we have. Feedback, comments, patches, bugs, and forks are all welcome.

Debugging CSS Conflicts

Since AddThis code is built from DOM elements, instead of Flash objects or similar non-standard elements, other scripts CSS rules can affect them. This is both a good thing, for instance when you want to center the toolbox, but can be a problem if your stylesheets have overly-broad rules in them. For example, this rule:

<style> a { color:white !important; } </style> 

Will result in something like this:

It looks like the counter isn’t loaded, but it’s there, and you can see it if you select it:

To debug overly broad CSS rules, control – or right-click – on the element you wish to inspect, then select “Inspect Element”:

This will take you to a window where you can see all the CSS rules that apply to that element. Search through them to find the one that’s not from AddThis and then click the check box to the left to disable it. You’ll notice that it also gives you the filename and line number of the rule you’re disabling, so you can go straight to that rule in your CSS file and fix it.

Debugging JavaScript Problems

Debugging JavaScript can be a bit more difficult. The first thing to check if AddThis isn’t showing up at all is if the proper variables are set on the page. First, right click anywhere on the page and select “Show Page Source.” Then, type ‘addthis_config’ into the small area at the bottom of the window, next to the blue greater-than sign and then push enter. If the code is set up properly on your page you should see something like this:

This will let you check whether your configuration values are correct or if they’re possibly getting overridden by re-declaring the addthis_config variable elsewhere on the page. Other good ones to check are addthis_share, which lets you check the sharing configuration, and just addthis, which will verify that our code is being loaded properly.

Another problem I see a lot is errors in other JavaScript on the page causing the buttons not to render. AddThis buttons are rendered, or put on the page, using JavaScript. If there is an error in other JavaScript on the page that halts the execution of scripts on the page then our buttons won’t show up:

This page has an onload attribute in the body that isn’t a valid function:

<body onload="AFunctionThatDoesntExist()"> 

To find out where this error is, right-click on the page and select “Show Page Source” again. Then, select the error icon in the upper left corner of the Developer Tools window. (You may have to display the window’s left sidebar to see it.)

Now that we know what the problem is, we can fix it. In this case, all we have to do is define a function named AFunctionThatDoesntExist, like this:

<script type="text/javascript"> function AFunctionThatDoesntExist(){ var docBody = document.getElementsByTagName('body'); var mySubheading = document.createElement('h2'); var myText = document.createTextNode('That function does exist!'); mySubheading.appendChild(myText); docBody[0].appendChild(mySubheading); } </script> 

Then the AddThis buttons will show up after the function runs:

Debugging Bigger Problems

Note: This is for advanced users who don’t mind editing HTML or using a command line

Sometimes you’re not sure what file has the CSS or JavaScript conflict in it. In these case you’ll have to disable the CSS and/or JavaScript on the page until you get the buttons to work, then figure out what conflicts are occurring.

Here’s how I’d do it on a Mac, but the steps are similar for a PC. First, get a web server. For Mac users this is built in. Just go into the Sharing preference pane in System Preferences and check “Web Sharing.”

Then, open up the Terminal app in your Applications folder, which will give you something like this:

At the prompt, type this command (without the dollar sign.)

$ curl http://www.example.com/ > Sites/example.html 

(Replace http://www.example.com/ with the URL of the page you want to debug.) This will download the source of the page to your Sites folder in a file example.html. Open this file in a text editor and add the following just after the <head> tag:

<base href="http://www.example.com/"/> 

(Again, replace http://www.example.com with the URL of the page you’re debugging.) Now open your browser and go to: http://localhost/~[user]/example.html where [user] is replaced with your username. Your page should show up with all the images and scripts.

Once you’ve got that, edit the page in the same text editor to start removing JavaScript and CSS, refreshing the page after each edit to see if it fixed the issue. Once you find the AddThis buttons behaving like you should, replace all but the last item you removed to see if that’s the culprit.

Once you’ve found the culprit it’s up to you to decide whether to keep using it, upgrade to a newer version that might not have a bug (if it’s a third-party library), or contact us to see if we have a workaround. Please note that we don’t support any external JavaScript beyond helping you find out that there’s a problem.

Looking for errors like this before you contact our support team might help you save time and frustration when AddThis is acting up.

• Lots of extra cross-team collaboration
• Many amazing and brilliant project presentations
• Developers passed out across the office

For those not familiar with hackathons, they are our 24-hour sprints of innovation. A-Teamers group up, or work alone, on a new project outside of what they are currently working on in their day-to-day. The energy surrounding a hackathon is always fun and familial. They are a very important part of the AddThis culture.

WINNER: Telepresence Roomba Robot

The winning project of this installment of all-night coding was quite impressive. The team went out to solve a common challenge we face at AddThis. Since we have multiple offices across the country, sometimes communicating and collaborating requires extra effort. So, the team created a telepresence robot using a Roomba, a MacBook Air, foamcore and a rocket launcher. And the entire thing was controlled from a Google Hangout! See it in action:

Congratulations to the winning team! It was well deserved.

Honorable Mention: AddThis Garden

The unofficial “runners-up” also deserve Earth Day Award 2012. “AddThis Plant” went on a mission to “green” our new HQ office in Virginia. It is an extensive project of plant growing, caring and loving with the end goal to have a nice array of beautiful oxygen-producing decorations for our work space. Check them out:

And those are just the official winners! There were many other brilliant and noteworthy projects that did not go unnoticed or under-appreciated. Some of which will be used to make everyday AddThis processes more efficient — a win all by itself!

Like this? Check out more posts about AddThis Engineering.

This past weekend Matt Keesan and I traveled down to Philadelphia for PennApps, America’s largest student-run hackathon. 48 hours, 91 teams, 320 hackers, $10K+ in prizes–any way you look at it, the event was huge. So before getting into some of our favorite hacks of the weekend, congratulations to the whole PennApps team, including frontman Pulak Mittal, for keeping the wifi, food, and energy flowing all weekend. AddThis was a proud sponsor and we’re looking forward to next semester!

Vivek Panyam, Facebook for Grandma

Advice for solo-hackers-to-be: Vivek worked in high-traffic areas of the hackathon and was accordingly flush with advice and attention from the 40 sponsors at the event. What’s more, he took the advice seriously.

Solo-hacker and UPenn freshman Vivek Panyam won the “most delicious UI” award for Facebook for Grandma, a pared-down Facebook app offering a compelling subset of features in an even more compelling package. Seriously, your grandmother just got 30 clicks closer to your photos; you’ve been warned.

Devon Peticulas and Dan Mundy, bbcat

PennApps veterans Devon Peticulas and Dan Mundy won big style points for scoping out and executing on a deceptively simple tamagotchi-inspired pet. Get your own at bbcat.co, then share it with your significant other to either test his or prove your own commitment. What impressed me most about the duo is that at the 11th-hour, when most teams were tracing syntax errors, Devon and Dan were sitting pretty, chatting about node.js design patterns and that perfect shade of beige found on the Atari 800. What’s more, they are coordinating HackRU at Rutgers next month. Maybe they’ll throw us dog-lovers a bone.

Devon Peticulas wears AddThis sunglasses at night. And a Venmo t-shirt for three consecutive days.

Raj, Dave, and Shiva: BuddyHack, BuddyMeme

Were there an award for the most roguish team at PennApps, the entire UMich contingent would have had to split it. Instead, we gave our own “Most Viral” award to BuddyHack, the only project that was developed, deployed, HackerNews’d, and banned before the demos even began! Luckily the team is prolific and their BuddyMeme project is still up from the Michigan Facebook Hackathon held this March. By creating a dead-simple meme generator (open-sourced, by the way) and leveraging the tight-knit nature of Facebook’s network, BuddyMeme allows memes to thrive within small social groups.

I can has internship? Me and the BuddyHack team.

There were too many cool projects for me to list. AddThis was super happy to share the weekend with so many talented developers.

This session is already completely full, but there’s a waiting list for student hackers, and the next PennApps is just six months away. (All developers in the area are invited to come by to help advise teams in their areas of expertise.)

If you’re a student looking for help, or live in Philly and just want to meet your friendly local AddThis developers, please drop me a line or send a tweet (@itsjoesullivan or @keesan)! We’re excited to be sponsoring our first university hackathon.

We’re happy to announce major updates to the AddThis sharing tool suite: the new design we’ve been testing in AddThis Labs graduates today, with a slew of new features. Go grab the new code!

We’ve written before about the mind-boggling amount of data every day, and how we use it to provide insights both to our publishers and into trends across our network. But we also use data to drive and refine our own processes and designs. Today, we’ve used what we’ve learned to make sharing even easier, and more personal. That means more engagement from your visitors, more sharing, and more traffic for your site.

Simpler design.

First, we have a new look. We’ve streamlined and simplified the compact menu. We’ve made our email form a lot easier to use, like the email clients you’re already familiar with. Our expanded menu’s been optimized for faster sharing to the most popular services in your region, and easier searching for your personal favorites. (As always, we’ll highlight the services you use for even easier sharing next time.) We’ll take advantage of jQuery on pages that already include it for simple but lovely transitions. And while our new platform takes advantage of HTML5 and loads of modern browser features, we won’t leave you out in the cold if you’re stuck on IE6–though, seriously guys, it’s time.

In Labs, the updated design increased sharing by 11%. In production, it’s going to do even better–due to our second major update.

Instant sharing.

Sharing to Facebook or Twitter used to mean popping a new window. That’s fast, but we figured if we made it even faster, more people would be sharing your content. To that end, we’re introducing “instant” inline sharing to the most popular social networks. If your users link their social accounts with AddThis, they’ll be able to share directly to Facebook or Twitter without even leaving your page. The new inline interface means there’s no extra page load–that means more unimpeded sharing. Even better, once a user authenticates, they’ll remain logged in on every single site with AddThis, until they log out. Everybody wins.

Adding social sign in to our sharing tools doesn’t just mean it’s easier to share. It also means it’s a lot easier to personalize your AddThis experience–easily choose the services you want to see once, and you’ll see them across all the sites that use AddThis.

Pain-free pinning.

Our recent updates for Pinterest are fully integrated with the new design. We’ve created the simplest and easiest way to integrate with Pinterest: almost no work necessary! Put AddThis on any page you want to make pinnable, and we’ll show a simple, elegant lightbox to showcase all the images on your page for your users to pick from. You can always specify your own image, too, if you want! We’ve found this feature can boost sharing to Pinterest by 20% or more.

You can add all these capabilities to your site with one simple code snippet. Take a look and try it out!

Our plugins for WordPress, Magento, and Shopify will be updated this week to support the new surfaces.

Questions? Comments? As always, we’d love your feedback.

We’re stoked to announce the newest addition to AddThis Labs:
Social Sign In.

You’ve probably noticed that many websites now allow you to register or sign in using one of your existing social network accounts, like your Facebook username or your Twitter ID. It’s not just a startup fad; in fact, there’s solid science behind it. Blue Research studied the holiday shopping habits of US consumers and noticed that the vast majority of us are annoyed by having to create a new account to use a site, and would generally rather use one of our existing social network accounts.

If you’ve ever had to register on an obscure website just to buy one perfect gift, you probably know exactly how this feels! Even outside of retail applications, users are much more likely to register for and engage with your site if the process is bootstrapped with their existing online identity–some research has shown users authenticating socially spend up to 50% more time on site than the rest.

Seeing the value of social authentication is easy. Building it, though, hasn’t been. Most existing social authentication solutions have either required a lot of complicated server-side programming; mastering the ins and outs of OpenID, OAuth, and other specifications; or paying for a hosted solution and thus giving up control over your users’ data, not to mention some of your budget!

Our approach is simpler. And it might be the right one for you.

Use their standard buttons, or design your own.

Social Sign In by AddThis is a straightforward JavaScript library that makes it easy to strengthen your relationship to your users by adding Facebook, Twitter or Google authentication to your site.You provide the application keys for each service you’d like to integrate. We take care of rendering the buttons and managing the authentication flow. When a user successfully authenticates, we provide the data directly to you, via JavaScript.

Some of the social data you’d get,
if I logged into your site using Social Sign In.

From then on, it’s your show, whether you want to pre-fill a registration form, directly create a new account, or store the data for later analysis. Our goal is just to make it easy to bring your users’ social data to your site, whether for registration, authentication, or personalization. You can try it out live on our Labs page.

Want to offer as many services as possible? No problem. Regardless of the network used, the data comes back in the same format — so you only have to write code once.

Like all our tools, you’ll get data in our analytics dashboard, too. We’re starting by keeping track of what networks your users are logging in with, but we’d love your thoughts on other information that might be relevant.

To make integration even easier, we’ve open sourced a fully functional demo in JavaScript, PHP and MySQL, with a simple user registration system based on Social Sign In. If you’ve always wanted a user system, feel free to start with ours! Or you can use our code as a basis for integrating Social Sign In into your existing registration and authentication system.

Questions? Comments? We’d love your feedback!

Over a dozen amazing presentations were given this morning from individuals and teams, showing off some of the most impressive projects to date! Fun new products, feature overhauls and internal tools for improved efficiency – our talented crew covered all the bases.

While the grand prize — a fancy bottle of scotch, some gift certificates, and bragging rights until the next hackathon — could only go to one project, every participant is a winner in teamwork, creativity and dedication.

We’ll be sharing more information about the various projects as we roll some of them out in the coming months but to give you an idea, topics and tools ranged from new Pinterest solutions for publishers to usage of gestural and voice control technology. It was an impressive array of presentations and they made one thing very clear: the A-Team is proving that the future is now!

Here is a look back at the past 24 hours. Now excuse us while we go catch some Zzz’s!

The winners with their most important prize.

Shhh! Sleepy hackers! ...

Cats hacking?! Only at AddThis.

Hungry, hungry hackers enjoying our early morning omelete bar!

If this looks like fun to you, check out our careers page. We’re hiring hackers and other great minds to join the party!

What is a Hackathon exactly? Hackathons are our 24-hour full-team sprints of innovation.  Individuals can work alone or as part of teams. The only real rules are:

1. The projects have to have something loosely to do with AddThis.
2. They should be compelling.
3. And finally, they must be built and demoed within 24 hours!

Not only are Hackathons a great way to reinforce the importance of teamwork, but some amazing products and tools have been born out of the tradition.

Keep checking in on our Twitter account, where we’ll be live tweeting photos and other fun  behind-the-scenes Hackathon action! Here are some early snapshots:

Look at all that collaboration! We love it!

In the end, our decisions rest on cost and control.  For our uses, CPU cycles in the cloud are about 3x more expensive than the fully loaded cost of dedicated cycles in our own data center.  Data stored in the cloud is at least 10x more expensive than in our own clusters.  Even if data storage were free, we would still be paying for CPU cycles to process the data and network costs to expose it.  But the killer is still latency and IO bottlenecks.  At our scale, there aren’t meaningful pricing options to overcome these in the cloud.

As both our business and cloud offerings have matured, we would still make the same choices around internal build-out vs shared cloud.  If there is one reason that drives this decision, it’s the fact that we have a business in which our needs can be modeled and anticipated.  And with the ability to predict comes the ability to optimize resources for both cost and performance.  With our own build-outs, we are able to regain control and apply it to the bottom line.  But to achieve this goodness requires a clear understanding of your needs, a skilled dev/ops team, a hacker mentality and focus.

• clouds remain king for spot capacity, typical web apps, development
• they do not solve complex management problems or magically scale apps
• operations at even modest scale still requires skilled, creative staff
• once a baseline of capacity is known, dedicated hardware can’t be beat

Depending on the profile of your business, cloud computing could be a game changer.  Our decision to bypass the traditional cloud was game changing for us;  we are noticeably stronger, faster and larger than our competitive set.  And with our developed abilities in large scale data and processing, we are in control of our destiny, ready for what comes next.

If you have any questions or want to hear more about our data processing capabilities, please feel free to reach out to us.

Fast forward to today. Our data infrastructure supports AddThis code deployments on more than 14 million domains, ingesting billions of events and terabytes of logs daily.  Scores of machines has grown to around a thousand totalling tens of terabytes of RAM and petabytes of disk space.

The majority of our internal compute capacity is dedicated to our custom data processing platform.  This platform is highly dependent on large IO capacity and low latency more than CPU cycles.  Virtualization, generally speaking, can be manageable for many CPU and memory-bound tasks, but is almost always destructive to the performance of network and disk subsystems.  As a result, we employ no virtualization save for a few test machines.  And while we run relatively commodity network gear, our decisions around topology and infrastructure have been made deliberately and carefully to maximize performance.  In our experience, the compounding effects of latency and IO bottlenecks quickly become a non-linear problem.

• even with the best of intentions, virtualization kills performance
• compress everything, keep it compressed even in RAM until it’s needed
• all data must stream (compressed!), remote procedure calls are evil
• data must be sharded for locality of processing, not physical locality (see previous)
• model around simple tasks that can checkpoint and replicate
• anything that provides transactional safety between checkpoints works against you
• design for (constant) failure whether in a public or private cloud

AddThis employs top-tier CDNs for most static web-asset serving, but we have still chosen to internalize our http-based APIs.  Even though cloud offerings excel at many web-service scenarios, they mainly rely on traditional LAMP stacks with limited back-end commitments beyond scripting and modest databases.  In these environments, virtualization is usually OK because most instances vastly under-utilize system resources. But when contention rears its head in the cloud, there is no recourse other than take the penalty or provision random new hardware.

Our “edge” and eventing API are http services that require maniacal tuning for performance and reliability.  With billions of requests a day spread across millions of domains, our code is trusted by site owners to respond quickly.  Latency *is* supreme.  Flowing from our edge and eventing APIs to the backend are terabytes of real-time data.  This live stream is forked into log storage and real-time systems.  Proximity to our compute clusters enables both faster and more economical processing of data by avoiding repeating data across high-cost, high-latency links to remote data centers.

• commodity network gear with the right topology and software can work wonders
• optimization of networks, proxies and kernels remains a high art
• chrome does unspeakable things (for another day)

AddThis’ compute clusters are constantly processing and re-processing hundreds of terabytes of data, applying new algorithms, generating new metrics and finding deeper insights.  Our current model supports complete visibility into optimizing our stack.  With a top to bottom view on hardware and software, we can squeeze out more than 10 times the performance per dollar than we can get from cloud-based offerings.  Equally important, we can do this reliably and predictably.

Continue to Part III: A Simple Decision – Performance, Predictability, Price

Sailing Past the Clouds

The emergence and maturation of multi-tenant, virtualized, commodity computing (“the cloud”) has enabled entirely new business models by bringing large scale distributed computing at an affordable price point.  But by designing systems flexible enough to function well in most scenarios, but tackling no specific problem, cloud vendors have made architectural trade-offs that are particularly challenging to us.  Over the last five years, we’ve built a lower cost, more effective, more scalable solution that is closely tailored to the needs of a large scale data, low latency businesses like ours.  And in doing so we have gained control over performance, predictability and price.

A Little History

AddThis’ success in the Widget business, starting in 2006 / 2007, catapulted us directly into scale problems that early clouds of that time were not ready to handle.  At that time, our platform was providing real-time analytics for hundreds of thousands of widgets being viewed hundreds of millions of times a day.  It doesn’t seem like much compared to the billions of events we’re tracking today, but it was on par with the social giant of that time: MySpace.

Early decisions, which were formative to our thinking, were based on pragmatism.  We were small and the price to upgrade databases at the scale we needed was out of the question.  The performance of scripted / hosted commodity web stacks could not predictably meet our needs.  Before long, we were forced to replace our SQL databases with a much faster (and simpler) in-house developed noSQL store just to keep up with massive data growth (the term noSQL, like Hadoop itself, didn’t exist yet). We found we could get better scale with operational safety and huge cost savings by architecting around a few key learnings:

• Databases can be fast, but network connections and serialization are slow
• Transactional safety is untenable overhead for real-time anything
• In-process local data CRUD is at least 1000 times faster
• Have hot spares/replicas, stagger checkpoints
• Backup only logs, not databases
• Design around replay from logs

Continue to Part II: AddThis Today

This month we sponsored the Second Annual Open Source Software BBQ in Washington, DC.  Four local open source meetups, WordPress DC, DC PHP, jQuery DC, and DC Droids came together to celebrate open source software with some delicious pulled pork and an array of sides. In the past we have also sponsored individual open source meetups that focus on other open source projects including Cassandra and regularly host and sponsor the Big Data DC meetup which often discusses Open Source Software.

Members of the AddThis team eating BBQ and chatting about Open Source

In addition to supporting the local open source communities, the engineers at AddThis contribute to many open source projects, both the popular and the obscure.  WordPress, GNU Emacs, CLISP, and Apache Kafka are some of the best known projects that we have contributed to.

If that isn’t enough, the Stream-Lib, MetricCatcher, Phetric, and TracBoard projects all have grown from work we do at AddThis and have been made available to the larger development community.  Checkout our github account to see some of the things we are working on now.

If working on open source software and open standards interests you, checkout out our job openings.

Web Intents Support

Want to add Web Intent support to your site? If you use AddThis, you already have. Just drop this line into your addthis_toolbox:

<a class=”addthis_button_intent_share_url”></a>

Web Intents users will be able to share your page through their favorite intent handler just like any other service.

Intents support for non-Chrome users

Web Intents are great for Chrome users, but what about the other two-thirds of visitors? AddThis provides another one-liner to make web intents work no matter what. Set:

addthis_config.webintents = true;

and AddThis will handle Intent events in the absence of native support. Implement Web Intents however you like—in the AddThis toolbox or not—and rest assured that your visitors will have access to their preferred service.

The AddThis Intent Handler

For those of you testing out your brand-spanking-new Chrome installs, remember that not all of your favorite services have an intent handler in the Chrome Web Store yet. Not to worry—AddThis does! Install it as you would a normal Chrome extension and give it a spin on a demo intent.

Watch this Space

As Web Intents grows, so will our support for it. Meanwhile, we’re developing tools to enable OExchange providers to handle Web Intents and integrating intent handling into our browser extensions.

Web Intents is a new tool in your sharing arsenal, and AddThis is thrilled to support it! Your feedback on our Web Intents support is invaluable, both now and in the coming months as your implementation of intents evolves alongside Web Intents itself.