Fluentd + Hadoop: Instant Big Data Collection

Fluentd + Hadoop: Instant Big Data Collection

This post describes how to use Fluentd's newly released WebHDFS plugin to aggregate semi-structured logs into Hadoop HDFS.


Fluentd is a JSON-based, open-source log collector originally written at Treasure Data. Fluentd is specifically designed for solving big data collection problem.

Many companies choose Hadoop Distributed Filesystem (HDFS) for big data storage. [1] Until recently, however, the only API interface was Java. This changed with the new WebHDFS interface, which allows users to interact with HDFS via HTTP. [2]

This post shows you how to set up Fluentd to receive data over HTTP and upload it to HDFS via WebHDFS.


The figure below shows the high-level architecture.


For simplicity, this post shows the one-node configuration. You should have the following software installed on the same node.

Get Treasure Data blogs, news, use cases, and platform capabilities.

Thank you for subscribing to our blog!

Fluentd’s most recent version of deb/rpm package (v1.1.10 or later) includes the WebHDFS plugin. If you want to use Ruby Gems to install the plugin, gem install fluent-plugin-webhdfs does the job.

Fluentd Configuration

Let’s configure Fluentd. If you use deb/rpm, the Fluentd’s config file is located at /etc/td-agent/td-agent.conf. Otherwise, it is located at /etc/fluentd/fluentd.conf.

HTTP Input

For input, let’s set up Fluentd to accept data from HTTP. This is what the Fluentd configuration looks like.

  type http
  port 8080

WebHDFS Output

The output configuration should look like this:

  type webhdfs
  host namenode.your.cluster.local
  port 50070
  path /log/%Y%m%d_%H/access.log.${hostname}
  flush_interval 10s

The match section specifies the regexp to match the tags. If the tag is matched, then the config inside it is used.

flush_internal indicates how often data is written to HDFS. Append operation is used to append the incoming data to the file specified by the path parameter.

For the value of path, you can use the placeholders for time and hostname (notice how %Y%m%d_%H and ${hostname} are used above). This prevents multiple Fluentd instances to append the data into the same file, which must be avoided for append operation.

The other two options, host and port, specify HDFS’s NameNode host and port respectively.

HDFS Configuration

Append is disabled by default. Please put these configurations into your hdfs-site.xml and restart the whole cluster.




Also, please make sure that path specified in Fluentd’s WebHDFS output is configured to be writable by hdfs user.


To test the setup, just post a JSON to Fluentd. This example users curl command to do so.

$ curl -X POST -d 'json={"action":"login","user":2}' 

Then, let’s access HDFS and see the stored data.

 $ sudo -u hdfs hadoop fs -lsr /log/
drwxr-xr-x   - 1 supergroup          0 2012-10-22 09:40 /log/20121022_14/access.log.dev



Fluentd + WebHDFS make real-time log collection easy, robust and scalable! @tagomoris has been using this plugin to collect 100,000 msgs/sec for a couple of months to help NHN Japan analyze big data.

Further Readings


Satoshi Tagomori contributed the WebHDFS plugin and battle-tested it in a super large-scale production environment. Thanks Satoshi!

  • Some of Fluentd users have been using fluent-plugin-mongo with MongoDB quite successfully.
  • WebHDFS is supported for Apache 1.0.0 (and later), CDH3u5 (and later) and CDH4 (and later).

Kazuki Ohta
Kazuki Ohta
Kazuki Ohta is the CEO and co-founder of Treasure Data. He also founded the Japanese Hadoop User Group, the world’s largest such group. Kaz is an acknowledged expert on distributed and parallel computing, and combines his knowledge of these technologies and Hadoop with the conviction that the service model is the only way to bring big data analytics to the mass market.