Skip to content

Latest commit

 

History

History
46 lines (25 loc) · 2.99 KB

DOCUMENTATION.md

File metadata and controls

46 lines (25 loc) · 2.99 KB
Raw

Technical Documentation

This documentation is to give a technical overview of Remoting Kafka Plugin. To try a demo of the plugin, please visit our demo section.

Overview

Current versions of Jenkins Remoting are based on the TCP protocol. If it fails, the agent connection and the build fails as well. There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability.

This project aims an update of Remoting and Jenkins in order to add support of a popular message queue/bus technology (Kafka) as a fault-tolerant communication layer in Jenkins.

Architecture

The below picture gives an overview architecture of the plugin.

Architecture

All command invocations, RMI, class loading, data streaming between master and agent are now transfered to/from a Kafka broker.

Communication model

The plugin uses manual partition assignment for topic management in Kafka. A master-agent connection will create a topic on the fly with 4 partitions:

  • Partition 0: command from master to agent.

  • Partition 1: command from agent to master.

  • Partition 2: secret message from master to agent.

  • Partition 3: secret message from agent to master.

Communication Model

Shared components (kafka-client-lib)

Shared components to be reused between master and agent, with the core KafkaClassicCommandTransport. This is the core of the plugin, an extension of Jenkins CommandTransport, which supports command transports to/from Kafka.

Plugin componenents

  • GlobalKafkaConfiguration: extension of GlobalConfiguration in Jenkins, where users can input Kafka/Zookeeper connection string. Credentials Plugin is being used for to ensure security input of Kafka secrets.
  • KafkaSecretManager: exchange secret (using HMAC algorithm) between master and agent to ensure a correct connection between master and agent.

Agent components

  • Engine: A rewritten version of remoting original Engine to support Kafka transport.
  • KafkaClientListener: A listener thread to listen to incoming messages from Kafka and produce outgoing response messages. To be used as a secret listener in agent.
  • remoting.jar bundling JENKINS-52871.