AddThis Blog

Even More Metrics!

We are big fans of Coda Hale’s Metrics library. Virtually all of our applications emit copious metrics that we use to monitor application state, optimize performance and resource allocation, and keep tabs on business metrics. In short: to know what is going on. We like it so much for our Java applications that we made it easy for any language with a network stack to participate with MetricCatcher and Phetric.

All of these metrics are sent to Ganglia and Graphite for dashboards, big screens filled with graphs, and fault detection. Ganglia has a solid architecutre of distributed metrics collection out of the box, and Graphite is known for its sweet graphs. We have contributed to both the Metrics Ganglia reporter and one of the available Ganglia-Graphite bridges.

Unfortunately, we ended up with a messy slew of ways to configure where we were sending metrics–Java properties files, Spring XML, even the occasional hard coded in-app configuration. It also annoyed our engineers that the Ganglia configuration was repeated both in each app and in the local gmond.conf.

Shocking though it may be, it’s also possible to have too many metrics. We have on occasion, for example, had a distributed service keep a Timer for every other service it talked through. This was totally awesome for in situ performance debugging, but the combinatorial explosion of unique host-metric-service tuples was too expensive to keep around.

Metrics-Reporter-Config tries to solve both of these problems:

There are a bunch of options there, but usually we just use a simple config that looks like this:

ganglia:
 - 
   period: 60
   timeunit: 'SECONDS'
   gmondConf: '/etc/ganglia/gmond.conf'
   predicate:
     color: "black"
     patterns:
     - ".*JMXONLY$"

Jars are already on their way to Maven Central. Check out the README and get started. We hope that others find this code as useful as we have. Feedback, comments, patches, bugs, and forks are all welcome.