Category Archives: Developers

Hydra: Storage of Nested Hierarchical Data

Hydra is a distributed data processing and storage system developed at AddThis, which we recently released as open source. It ingests streams of data and builds hierarchical tree structures that are aggregates, summaries, or transformations of the data. Sibling nodes in the tree are stored in lexicographic sorted order. This ordering is often used explicitly by the human when writing queries or implicitly by the query system to optimize the execution of queries. Continue reading

Open-Sourcing Ssync: An Out-of-the-Box Distributed Rsync

We periodically have to transfer files to a collection of machines in a cluster. Without a distributed filesystem we rely on user level processes to move these files to their target destinations. Previously we had been making a sequence of N rsync calls to populate a collection of N machines. When looking for an preexisting solution that would improve our workflow, I could not find one that met the following requirements:

Continue reading

JavaScript Design Patterns Used in Smart Layers

JavaScript design patterns are important for the maintainability and scalability of web applications. For Smart Layers, we focused on writing DRY (Don’t Repeat Yourself), consistent, and cross-browser compliant code. Before we talk about the specific techniques that we used, let’s first understand the Smart Layers use case.

Continue reading

Performance Optimizing for Smart Layers

It is a competitive advantage for websites to be fast and responsive, so we made performance a priority when building Smart Layers. Let’s take a look at some of the mobile and desktop performance best practices that help make Smart Layers blazing fast.


Continue reading

The Secret Life of Concurrent Data Structures

concurrent data structure is “a particular way of storing and organizing data for access by multiple computing threads (or processes) on a computer.” In this blog entry, we’ll be covering one of the hidden sides of concurrent data structures that are not so documented in the literature. We’ll be looking at insertion and deletion operations, and comparing the relative complexity of implementing these two operations.

Let’s focus our attention to concurrent data structures in a shared-memory environment where multiple threads are concurrently reading and/or writing. Continue reading