New Release of Treasure Agent: Improved Portability, Enhanced Performance

Treasure Agent (td-agent) is a distribution package of Fluentd — the popular open source unified data collection software — that adds extra support within Treasure Data Service.

Since September of last year, we have been working on a new version of the Treasure Agent streaming data collector. After extensive testing with production workloads, we’re proud to announce the new version of Treasure Agent (td-agent2) is now available.

The new version provides even more reliability, portability, performance, ease of use and available features. The configurations and protocols are completely compatible with older versions.

How to install td-agent2?

We’ll continue to maintain and provide support for the previous version over the next 12 months, but we recommend that all existing users upgrade to this release. To do so, simply upgrade the package and it’s done.

Summary of Changes

 td-agent2 (New Stable)td-agent (Old Stable)
Package System
  • yum
  • deb
  • pkg (NEW)
  • yum
  • deb
Platform
  • RHEL/CentOS 5
  • RHEL/CentOS 6
  • RHEL/CentOS 7 (New)
  • Ubuntu 10
  • Ubuntu 12
  • Ubuntu 14 (New)
  • Debian Squeeze (New)
  • Debian Wheezy (New)
  • Mac OS X (New)
  • Amazon Linux (Coming)
  • RHEL/CentOS 5
  • RHEL/CentOS 6
  • Ubuntu 10
  • Ubuntu 12
Ruby Interpreterv2.1v1.9
Fluentdv0.12 or laterv0.10
Fluentd-UIv0.3N/A
MessagePackv0.5v0.4
Cool.iov1.2 or laterv1.1
jemallocv2.2v2.2
Installation Directory/opt/td-agent//usr/{lib,lib64}/
Plugin Install/usr/sbin/td-agent-gemManual
Packaging ScriptOmnibusManual

Portability

Treasure Agent now additionally supports Ubuntu 14, Debian Squeeze/Wheezy Mac OS X, and Amazon Linux. We have switched our build system from in-house scripts to Omnibus. This enables us to generate installers for various environments much easier.

Get Treasure Data blogs, news, use cases, and platform capabilities.

Thank you for subscribing to our blog!

Performance

We have made substantial performance optimizations in various layers.

Ruby:  Ruby was upgraded from 1.9 to 2.1, which boosts the performance.

MessagePack: We have rewritten MessagePack Ruby, the highly efficient binary serialization format used internally. (MessagePack was invented by Treasure Data co-founder Sada Furuhashi ).

Specifically, we introduced an intelligent buffer management system, which yields higher performance with lower memory footprint.

The chart below contains the performance comparisons of various serialization libraries, including new/old MessagePack Ruby, using a Twitter JSON data set. As you can see, the new version is 4x faster for serialization and 10% faster for deserialization.

Multi-Process Mode: One of the common inquiries we receive is are how to leverage multiple cores. Now we support a multi-process mode, which leverages multiple CPU cores on the node.

Overall, we have seen 20-40% performance improvements depending on the traffic pattern.

Ease of Use

Treasure Agent now supports configuration done via Web UI, by Fluentd-UI gem. This makes it significantly easier to configure the system. Because Fluentd-UI is simply a config generator, you’re still able to edit the configuration via command line if you prefer.

Reliability

Although Treasure Agent is known for its reliable data collection, as with any software we continually work to advance the reliability and fix various bugs, such as UNIX signal handling race condition. The changes are listed here.

Features

Treasure Agent now supports various features in the core, as well as plugins like S3. We’ll spotlight these feature enhancements in a separate blog post in the near future.

Try it Now!

Now it’s time to take advantage of advantages within this upgrade! And because the new version is completely compatible with the previous version, you can simply remove old td-agent package and install td-agent2 package. Try it today!

Kazuki Ota
Kazuki Ota
Kaz is no stranger to big data. He co-founded the Hadoop User Group Japan in 2009. A long time open source advocate, Kaz was a proponent of KDE at age 18 and has made numerous contributions to open source software. Follow @kzk_mover on Twitter.
Related Posts