At Treasure Data, we’ve kicked off August with a new podcast: The Data Drive. In our series, we survey cutting-edge technologies in the data analytics space, and interview their inventors, contributors, power users and evangelists, with a view to data integrations and analytics.
In this bi-weekly episode, we interview Baruch Sadugorsky of JFrog, the makers of Artifactory, a well-known binary repository manager and Bintray, a social software distribution platform, among other related products.
We propose a Fluentd plugin for monitoring each.
Back in earlier days days of software engineering, distributed, remote, and open source projects were still in the (somewhat) distant future; and our projects churned and bumped along a predictable, deterministic, yet clunky waterfall process.
Fast forward to today, and you can imagine that the tools we used yesterday couldn’t come even close to today’s requirements. Among other things, a typical modern, distributed, open-source (or partially open-source) project needs to support:
- Offline access to artifacts and metadata;
- Fault tolerance and reliability for the repository – which may be distributed across several nodes and locations;
- Extensive and network-heavy build and CI processes;
- Unified logging and analytics across the entire version control and continuous integration stack (that’s where Treasure Data and Fluentd come in!)
How would you monitor your continuous integration? Repository commits? Build processes? Query those results? Search those results, if it’s a large set? Develop and run custom analytics on your repository, build, or CI analytics?
While we’ll be talking largely from JFrog’s perspective, including taking a wide sweep through Baruch’s own experience; we believe that there are takeaways that can be applied to data engineering concerns generally, particularly when it comes to new and innovative ways to monitor our repositories, builds, and CI processes using Fluentd and Treasure Data.
Our Podcast theme music today is Odin, by Peter Coon and The New Ones, by John Hammink and the Mr. Chaos Trio. Used with Creative Commons Attribution license.