Speaker blogpost: A Cloud Native Jenkins

Pluggable storage is one of the major areas we have been working on over the last few months. There are a number of parallel stories which are summarized on this page. There has been significant progress in the areas of artifact storage, build logging and configuration storage. A number of Jenkins Enhancement Proposals were submitted and accepted, and there are plugin releases and prototypes for these stories.

During our talk we will discuss the current status of these stories and future plans. In particular, we will cover the following areas and reference implementations:

There are existing plugins for the areas above, but there is a difference in approach we have taken. Instead of creating new custom steps we extend Jenkins architecture in a way that the storage becomes transparent to users. For example, with Artifact Manager for S3 Plugin common Archive Artifacts steps work transparently with Remote storage, as well as Jenkins Pipeline’s stash()/unstash() steps.

The reference implementations intentionally use different technologies so that we cover more scenarios. We regularly discuss the implementations in the Cloud Native SIG, and we would appreciate your feedback.

Want something new? Several days ago Kohsuke Kawaguchi, the creator of Jenkins, posted the Jenkins: Shifting Gears article to summarize the plan for Jenkins evolution. Cloud Native Jenkins is a critical part of this plan, and it is not “just Jenkins X”. There are various architectural changes in Jenkins required to make this vision happen, and we plan to work on these changes in the Cloud Native SIG.

In our presentation, we will talk about our experiment with ephemeral Jenkins and single-shot masters. In this story we are creating a headless single-shot master which starts in a container, executes a Pipeline build and pushes all the results to remote storage so that the container can just be deleted after completion. Such a master bundles plugins and self-configuration logic using “Configuration as Code”, so that it can start executing Pipelines in just a few seconds. Once packaged, it can be invoked from CLI as simply as…​

or, in Kubernetes:

Such a single-shot master could also be made a part of a Cloud Native Jenkins system. Standard event handlers like Prow can invoke the builds on webhooks and report results back, so that the single-shot master can be used to build pull requests or to run Continuous Delivery flows. Extra agents could also be connected to the master on-demand, using the Kubernetes plugin or sidecar containers.

In order to make this experiment possible, we used a toolchain based on Docker, Jenkinsfile Runner, Configuration as Code Plugin (JCasC), and a Custom WAR Packager tool which glues all the things together.

Custom WAR Packager is a new tool which takes various configurations (YAML specification defining core version, list of plugins, system properties, Groovy Hooks, JCasC YAMLs)…​ and then bundles everything as a ready-to-fly WAR file or Docker image. Starting from version 1.2, Custom WAR Packager also supports packaging Jenkinsfile Runner images as an experimental feature. I will do a separate blogpost about this new tool later, but there is already some documentation a number of demos in the project’s repo.

Yes, we will have a demo! We will show a single-shot master running with Pluggable storage implementations for AWS environments (Amazon S3, AWS CloudWatch, EKS, etc.), which executes Jenkins Pipelines for Maven projects and provisions agents in Kubernetes on-demand.

The demo has to be published yes, but you can already find a more simple Jenkinsfile Runner demo here.

The upcoming DevOps World | Jenkins World conferences are heavily packed with talks related to Cloud Native Jenkins, including war stories and presentations on projects like Jenkins X and Jenkins Evergreen. It is a great chance to get more information about using Jenkins in cloud environments.

If you are a Jenkins contributor or just want to become a contributor, also join the Contributor Summit (Sep 17 in US and Oct 23 in Nice) or visit the Jenkins community booth in the Exhibition hall. At the Contributor Summit on Sep 17 we will also have a face-to-face Cloud Native SIG meeting. Feel free to contribute to the agenda here.