Introducing the Pipeline Configuration History Plugin

It all started 10 years ago — with classical job types (e.g. Freestyle, Maven, etc.). Every once in a while users contacted us because their jobs failed to build overnight. Why did the job fail? Was the failure related to a job configuration change? The users' typical answer was: "We didn’t change anything!", but is that really true? We thought about this and decided to develop a plugin that helped us solve this problem. This was the idea and the beginning of Job Configuration History.

Now it was possible to view changes of job configurations (like other branches, JDK versions, etc.) and more often the reason for breaking builds were changes of job configurations.

Over the years the plugin got developed and is still under development. New functions were added, that not only view job configurations, but also changes of global and agent configurations. It is also possible to recover old configuration versions. Today the plugin has more than 30,000 installations. For many years JobConfigHistory relieves our daily work — with more than 3,000 Jenkins jobs! Then there was a new type of job: Pipelines.

Pipeline jobs are fundamentally different than classical job types . While classic job types are configured via the Jenkins GUI, Pipeline jobs are configured as code. Every pipeline job indeed gets created via the Jenkins GUI, however that is not necessarily where the pipeline configuration is located. Pipelines can be configured:

With every save operation of the job configuration, JobConfigHistory creates a copy of the actual job configuration if something has changed. That only works for pipeline jobs if the pipeline configuration is inserted in the job configuration page as script. Changes in the Jenkinsfile or the shared libraries are not detected by JobConfigHistory. You have to use the SCM system to view changes of the Jenkinsfile or the shared libraries. It is complex and time intensive to find a correlation between the time of a build and a change to the Jenkinsfile or shared library.

This new problem is much more than JobConfigHistory. A new solution was needed to detect pipeline changes and show these changes in Jenkins. So we developed Pipeline Configuration History.

During every pipeline run the Jenkinsfile and related shared libraries are saved in the builds directory of the job. Pipeline Configuration History saves changes of the pipeline files between the last run and the previous run as history events. Therefore when a pipeline job ceases to build successfully, you can check if something has changed on any used pipeline file. You can also see the build where changes occurred.

Because a pipeline configuration can consist of several files where changes could have occurred, only files with changes between two builds are shown in the diff. That makes the whole thing more compact and effective:

But sometimes you may want to show more than the differences between pipeline files. You may want to see which pipeline files are in use or the content of those files when they were used. So it’s possible to view all files and their content. If required you can download them as well:

We use Pipeline Configuration History successfully in production. It has helped us from the very first day as we solved problems that occurred due to pipeline configuration changes. Pipeline Configuration History won’t replace Job Configuration History. The plugins have different use cases. Many times small changes on job or pipeline configurations also have big impacts. Because of the correlation in time between changes of job or pipeline configurations and different build behavior, it is now possible to substantially reduce the time and effort to analyze build failures. The Job Configuration History and Pipeline Configuration History plugins let us help our users in consulting and in solving issues. We resolve problems much faster through easy access to the configuration history of jobs. These plugins are essential for our daily work.