Elastic 2016 and Fluentd FAQs

Elastic 2016 and Fluentd FAQs

Fluentd at {ON} ‘16

Atlasssian presented on their logging as a (internal) service, using Logstash and Fluentd. Atlasssian presenting on their logging as a service that uses Fluentd among others.

Last week, Elastic, the company behind Elasticsearch, Logstash, Kibana and Beats, put together their annual user conference in San Francisco. Because Elasticsearch is a popular backend for Fluentd, the open source log collector that we created and continue to sponsor and help maintain, Kaz and I decided to join Elasticon to meet with Fluentd users and learn what Elastic folks are up to.

Especially exciting for us personally was meeting with Fluentd users. We were pleasantly surprised how many people have heard of Fluentd, tried Fluentd, and managed Fluentd in production. It was our moment of “Geez, I didn’t know <household name> used Fluentd. You are totally making my day!”

As we talked to more users, it became clear that many of them weren’t aware of our more recent updates and couldn’t find answers to their questions online. Hence, I decided to compile quick Fluentd FAQs.

Fluentd FAQs

Q. Does Fluentd support Windows?

A. Yes! It’s already supported in the master branch and the upcoming v0.14 release will include this change. Our plan is to release v0.14 in the coming months.

Q. I’ve deployed Fluentd at scale and need to squeeze more performance. Any tips?

Get Treasure Data blogs, news, use cases, and platform capabilities.

Thank you for subscribing to our blog!

A. Our blog article “5 Tips to Optimize Fluentd Performance” is a good place to start. If you are seeing throughput issues, also make sure that your output destinations are properly configured (MongoDB, Elasticsearch, MySQL, Redshift, etc.).

Q. Does Fluentd have commercial support?

A. Yes.

Q. How is Fluentd fast if it’s written in Ruby?

A. First of all, we strongly believe how you architect your software has just as much to do with language choice. That said, all other things equal, some languages and data formats make it easier to write performant software. For this reason, Fluentd makes extensive use of CRuby’s C extensions and uses MessagePack (and its fast Ruby binding) as its wire protocol. Full disclosure: Sada Furuhashi, the original creator of Fluentd, also created MessagePack.

Q. Does Fluentd talk to Kafka?

A. Yes. To post data into Kafka, you want to use the out_kafka plugin. To get data from Kafka and send them to various systems, we recommend our fluentd-kafka-consumer.

Q. Does Fluentd integrate with Elastic’s Beats?

A. Yes. You can send data from Beats to Fluentd directly.

Q. Wait, then what does Treasure Data do?

A. In addition to providing enterprise support for Fluentd, Treasure Data provides analytics infrastructure as a service. It means that you can collect data from multiple sources and analyze data with familiar SQL both in an ad hoc and repeatable manner. If you are interested in learning more about Treasure Data, request a demo to learn more!

Thanks Elastic and Fluentd Fans!

In short, it was a GREAT event. Food was tasty, the party was fun and quirky, and many talks were fascinating. Kudos to Elasticon staff: it must have been a ton of work!

Finally, a quick announcement: If you are an existing Treasure Data user trying to bring query results into Elasticsearch or an existing Elasticsearch user trying to extend its analytic capabilities, I recommend you check out our new Elasticsearch integration.

Kiyoto Tamura
Kiyoto Tamura
Kiyoto began his career in quantitative finance before making a transition into the startup world. A math nerd turned software engineer turned developer marketer, he enjoys postmodern literature, statistics, and a good cup of coffee.