Jenkins World 2016 Wrap-up - Scaling
| This is a guest post by Liam Newman, Technical Evangelist at CloudBees. |
One of the great features of Jenkins is how far it can scale, not only from a software perspective, but also from an organizational one. From a single Jenkins controller with one or two agents to a multiple controller with thousands of agents, from a team of only a few people to a whole company with multiple disparate departments and organizations, you’ll find space where Jenkins is used.
Like any software or organization, there are common challenges for increasing scale with Jenkins and some common best practices, but there are also some unique solutions. A big conference like Jenkins World brings users from all scales together to see how people in other organizations at similar or greater scale are solving similar problems.
Here’s a recap of the some of the sessions on scaling Jenkins, with links to slides and videos shared by CloudBees:
Paul Miles and Grant Dickie of NPR talked about JenkinsOps: An Initiative to Streamline and Automate Jenkins. They shared ways their team has used Jenkins to automate many of the administrative tasks related to managing feature code branches, handling deployments, running tests, and properly configuring their environments. They also showed code samples and talked about future challenges in their quest to achieve continuous deployment.
Maxfield F Stewart of Riot Games showed how they built an integrated Docker solution using Jenkins in Thinking Inside the Container: A Continuous Delivery Story He showed how their system allows engineers around the company to submit Docker images as build environments. This has let their containerized farm now create over 10,000 containers per week and handles nearly 1,000 jobs at a rate of about 100 jobs per hour. And they have done this using readily available, open source Jenkins plugins. He also talked about how they settled on this design, lessons learned, best practices, and how to build and scale other similar system.
How to Do Continuous Delivery with Jenkins Pipeline, Docker and Kubernetes, presented by James Strachan of Red Hat, showed how to use Jenkins Pipeline with Docker and Kubernetes to implement a complete end-to-end continuous delivery and continuous improvement system using open source software for both microservices and monolithic applications. He demonstrated how to create or import projects, and have them automatically build, run system and integration tests, stage, and finally deploy. He also showed to manage and update those deployed applications using continuous delivery practices.
Carlos Sanchez of CloudBees discussed Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos? He compared various Docker Swarm, Apache Mesos, and Kubernetes in terms of their ability to dynamically scale in Jenkins by running jobs inside containers. He also discussed the pros and cons, best practices, level of Jenkins support for each of these technologies.
Stephen Connolly of CloudBees asked "So, You Want to Build the World’s Biggest Jenkins Cluster?" and explained how to do so. He started with real world results realized by Jenkins users who have built large clusters. Next, he showed experiments around scaling some individual sub-components of Jenkins in isolation to see what challenges have been faced when integrated. Finally, he arrived at recipes for building Jenkins clusters with different scaling capabilities and making existing Jenkins clusters more efficient.
Bill Houston and Ali Raza of Splunk gave a talk in two parts, Jenkins at Splunk and Splunking Jenkins In the first part, Bill showed how Splunk uses Jenkins to implement their end-to-end CI system. They discussed features and design goals, challenges they encountered, and how they addressed these challenges. In the second part, Ali showed how to use the Jenkins Splunk plugin. Using plugin, he gathered test results and Jenkins environment data, and delivered it to a Splunk indexer for analysis and presentation.
David Hoover of Google talked about Jenkins inside Google. Last year, they presented their initial investigations and stress testing as they prepared to deploy a large-scale Jenkins installation at Google. Now, with a year of real-world use under their belts, they returned to present on how their expectations held up, what new issues they encountered, how they have addressed those issues, and the challenges and opportunities they see ahead.
About the author