Meetup: Logging & Docker in Production
On June 16, Treasure Data hosted a meetup focusing on Logging and Docker in production. With around 65 enthusiastic attendees, and two of the best known authorities speaking on their respective technologies, Docker and Fluentd, this meetup was definitely a hit!
Logging for Production Systems
“Sada” Sadayuki Furuhashi
Sada, one of the creators and original committers to the Fluentd project, kicked off the sessions with a talk about Fluentd, and how it works in production systems.
Speaking at first on containers, and how logging and metrics should work within them, Sada then took a step back to discuss traditional logrotate and rsync logging. With downsides that include being short-lived (note: ephemerality is ALSO a feature of logging tools such as Fluentd when run inside a container, rather than on a host), high-latency and hard to analyze, the problem is confounded even more when there are many containers.
Many small containers running logging can quickly overload available storage with log files, and too many connections from microcontainers can create connection pool problems while overloading available processors and memory.
Fortunately, Fluentd solves these and other container-logging issues. By using an aggregate instance of Fluentd on the host to collect logs from container-level instances, and thus replacing a point-to-point integration with a single interface for logging and message transmission, configuration is decoupled from applications and is scalable.
Sada then went on to discuss Fluentd’s plug-in architecture, error handling, buffering and recovery, as well as some sample use-cases.
Fluentd can be started within a docker container with the simple command switch: –log-driver=fluentd .
Fluentd is configurable with a simple fluentd.conf file containing configuration, parsing and routing directives and can be started and stopped as any unix service. To learn more about Fluentd’s unified logging layer, go to Fluentd.org
Open source recipes to build, ship and run production apps with Docker
Jérôme was part of the team that built, scaled, and operated the dotCloud PaaS, before it became Docker, and continues to be involved actively in Docker development today. His talk, which was peppered with demos, approached Docker and Docker configuration from multiple perspectives. After touching on best practices and container-based pragmatism, we heard about provisioning hosts, followed by key-value stores and their usefulness in a container environment; he then touched on overlay networks, and we learned how to distribute container images. After briefly discussing how to deploy applications on Swarm, we learned about load balancing. Finally, he tied all of the above together with logging.
Jérôme’s demos and configuration examples aren’t normally found in Docker’s standard docs. Here, for example, is how you’d configure your docker-compose.yml if you were setting up and EFK (Elastic, Fluentd, Kibana) stack:
This is how you’d provision a swarm cluster with a generic driver:
For more of Jérôme’s examples and configurations, you can find his repo here.
And you can find more on Docker here.
Would you like to build the easiest possible logging infrastructure you can? Get Fluentd!
- There are more than two hundred input, output, and other plugins here. Here you can see them sorted in descending order by popularity at fluentd.org/plugins/all
- If you are interested in seeing the plug-ins by category, go to fluentd.org/plugins
If you are interested in deploying Fluentd + Kubernetes/Docker at scale, check out our Fluentd Enterprise offering.