GSoC Project Intro: Jenkins Remoting over Message Bus/Queue

Published on 2018-06-18 by Pham Vu Tuan

About me

My name is Pham Vu Tuan, I am a final year undergraduate student from Singapore. This is the first time I participate in Google Summer of Code and contribute to an open-source organization. I am very excited to contribute this summer.

Mentors

I have GSoC mentors who help me in this project Oleg Nenashev and Supun Wanniarachchi. Besides that, I also receive great support from developers in remoting project Devin Nusbaum and Jeff Thompson.

Overview

Current versions of Jenkins Remoting are based on the TCP protocol. If it fails, the agent connection and the build fails as well. There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability.

This project aims to develop a plugin in order to add support of a popular message queue/bus technology (Kafka) as a fault-tolerant communication layer in Jenkins.

Why Kafka?

When planning for this project, we want to use traditional message queue system such as ActiveMQ or RabbitMQ. However, after some discussion, we decided to have a try with Kafka with more suitable features with this project:

  • Kafka itself is not a queue like ActiveMQ or RabbitMQ, it is a distributed, replicated commit log. This helps to remove message delivery complexity we have in traditional queue system.

  • We need to support data streaming as a requirement, and Kafka is good at this aspect, which RabbitMQ is lack of.

  • Kafka is said to have a better scalability and good support from the development community.

Current State

The project is reaching the end of the first phase and here are things we have achieved so far:

  • Setup project as a set of Docker Compose components: Kafka cluster, Jenkins master (with plugin) and a custom agent (JAR).

  • Create a PoC with new command transport implementation to support Kafka, which involves of command invocation, RMI, classloading and data streaming.

  • Make neccessary changes in Remoting and Jenkins core to make them extensible for the use of this project.

  • Decide to use Kafka as a suitable final implementation.

We planned to release an alpha version of this plugin by the end of this phase, but decided to move this release to the second phase because we need to wait for remoting and core patches to be released.

Architecture Overview

The project consists of multiple components:

  • Kafka Client Library - new command transport implementation, producer and consumer client logic.

  • Remoting Kafka Plugin - plugin implementation with KafkaGlobalConfiguration and KafkaComputerLauncher.

  • Remoting Kafka Agent - A custom JAR agent with remoting JAR packaged together with a custom Engine implementation to setup a communication channel with Kafka.

  • All the components are packaged together with Docker Compose.

The below diagram is the overview of the current architecture: remoting kafka architecture

With this design, master is not communicating with agent using direct TCP communication anymore, all the communication commands are transfered with Kafka.

Features

1. Kafka Global Configuration

kafka global config

2. Custom agent start up as a JAR

User can start running an agent with the following command: start agent

3. Launch agents with Kafka

launch agent kafka

4. Commands transferred between master and agent over Kafka

kafka commands

Remoting operations are being executed over Kafka. In the log you may see:

  • Command execution (SlaveInstallerFactoryImpl.isWindows())

  • Classloading (Classloader.fetch())

  • Log streaming (Pipe.chunk())

5. Run jobs with remoting Kafka

It is possible to run jobs on Agents connected over Kafka remoting kafka run job

Next Phase Plan

Here are the tasks planned for the next phase:

How to run demo

You can try to run a demo of the plugin by following the instruction.

About the Author
Pham Vu Tuan

Pham Vu Tuan is a developer from Singapore. He starts contributing to Jenkins from Google Summer of Code 2018 for Jenkins Remoting over Message Bus/Queue