Enabling Facebook’s Log Infrastructure with Fluentd
This post shows how you can replace Scribe with Fluentd.
What is Scribe?
Facebook uses Scribe as its core log aggregation service. The description of Github reads, “Scribe is a server for aggregating log data streamed in real time from a large number of servers.”
A network of Scribe servers forms a directed graph. Each server is a node and directed edges represent lines of communication. Usually, Scribe is installed on every node, and logs are collected to one giant “aggregator” node. The collected logs are written into HDFS (Hadoop Distributed File System) and later analyzed by Hadoop MapReduce or Hive.
Scribe is quite popular. In addition to Facebook, Twitter and Zynga use Scribe in production.
Scribe is solid. It has been effectively deployed at several web powerhouses with serious scalability challenges. So, why would you switch to Fluentd? The answer is threefold: 1) Ease of management, 2) Flexibility, and 3) Compatibility.
1) Ease of Management
Scribe is insanely difficult to install correctly. Not only do you need to build Boost, Thrift, and libhdfs from source, you must pick the correct versions of the software or the build would fail. In contrast, installing and deploying Fluentd is a breeze. It comes with rpm/deb packages maintained by Treasure Data, Inc. (That’s us!). If you use Chef (systems integration framework), you can use the cookbook we have authored, too.
Scribe is fast because it’s written in C++. But C++’s hairiness makes Scribe difficult to modify or extend. On the other hand, Fluentd is written in ~3,000 lines of Ruby, and you can easily customize and extend its behavior. In terms of performance, Scribe definitely beats Fluentd, but Fluentd is quite competent: it supports a multi-process mode and can handle upto 20,000 messages per second on a single host. If that’s not good enough, go ahead and choose Scribe. I hope you don’t get stuck in the versioning hell.
Thanks to its extendable design, Fluentd already has a Scribe plug-in that supports log aggregation via Thrift. This plug-in is 100% compatible with Scribe and can replace an existing instance of Scribe out of the box.
Just to show off Fluentd’s versatility…Fluentd also has a plug-in that can output to Hoop, a REST HTTP gateway with full support for HDFS operations. For the list of all the officially supported plug-ins, please check out the Fluent Github repo.
These plug-ins are assumed to be installed with Fluentd.
deb/rpm packages are by far the easiest way to install all three. Here are the relevant links:
This section walks you through how to replace a Scribe-based system with a Fluentd-based system. Don’t worry, it really is a drop-in replacement.
Configuring Fluentd on Front-end Nodes
For front-end nodes, The Scribe Input and Output plug-in are used (see below). If you have multiple aggregator nodes, you can use the [Copy plug-in].(https://fluentd.org/doc/plugin.html#copy)
# Scribe Input
# Scribe Output
Configuring Fluentd on Log-Aggregator Nodes
The aggregator nodes receive the requests from the Scribe Input plug-in, and output to HDFS with the Hoop plugin. The received logs are buffered, and periodically appended to the existing log files on HDFS.
Fluentd brings Facebook-like log aggregation infrastructure to your servers. The only difference is your system is a lot more flexible and does not require an army of engineers to maintain.
- Fluentd Scribe Plugin
- Fluentd Hoop Plugin
- Fluentd Documentation
- Fluentd Plugins List
- Fluentd Source Code
- Tutorial: Store Apache Logs into Amazon S3
- Tutorial: Store Apache Logs into MongoDB
- Tutorial: Fluentd + HDFS: Instant Big Data Collection