Table of contents

CloudBees Jenkins Enterprise 1.x administration guide


Note
CloudBees will no longer be supporting CloudBees Jenkins Enterprise 1.x after July 30, 2020. This end-of-life announcement allows CloudBees to focus on driving new technology and product innovation for CloudBees Core. For information on moving to CloudBees Core, please refer to Migrating from CloudBees Jenkins Enterprise 1.x to CloudBees Core on modern cloud platforms which has been created to help you with the migration process. Existing customers can also contact their CSM to help ensure a smooth transition.

CloudBees Jenkins Enterprise 1.x references

CloudBees Jenkins Enterprise provides a highly elastic infrastructure that supports centralized management of teams, shareable resources across the cluster and automated management of the underlying cluster infrastructure. This section lists a number of CloudBees Jenkins Enterprise reference documents which provide administrators with details and advanced configuration options for further customizing and scaling a CloudBees Jenkins Enterprise environment.

CloudBees Jenkins Enterprise architecture

Once CloudBees Jenkins Enterprise is deployed you’ll have collection of servers, either virtual or physical, which work together to run the Operations Center and to enable dynamic allocation of new Managed Masters and Jenkins build agents

CloudBees Jenkins Enterprise cluster

A CloudBees Jenkins Enterprise cluster uses cluster management software, Apache Mesos, to manage the Operations Center, Managed Masters and Jenkins agent processes that run on the cluster. Mesos is like an operating system for a cluster; it manages CPU and available memory, and it runs applications across the cluster. CloudBees Jenkins Enterprise applications are packaged as Docker containers.

Infrastructure security

Ports used

The cluster needs some ports open in order to be able to work correctly.

Table 1. Ports used

From

To

Port

Description

Elastic Search Worker

Elastic Search Worker

31090,31092

Elastic Search

Master Worker or Build Worker

Elastic Search Worker

31092

Elastic Search

Controller

Master Worker or Build Worker

All ports

 

Controller

Master Worker or Build Worker

31000 - 32000

CloudBees Jenkins Enterprise

Controller

Master Worker or Build Worker

5051

Mesos

Controller

Master Worker or Build Worker

2888,3888

Zookeeper

Internal Load Balancer

Controller

80 - 81

HTTP

Internal Load Balancer

Controller

2222

SSH

Master Worker or Build Worker

Internal Load Balancer

80

HTTP

Internal Load Balancer

Controller

2222

SSH

Master Worker or Build Worker

Internal Load Balancer

443

HTTPS

Load Balancer

Controller

2222

SSH

Load Balancer

Controller

80 - 81

HTTP

Client Workstation

Controller

22

SSH

Client Workstation

Load Balancer

80

HTTP

Client Workstation

Load Balancer

443

HTTPS

CloudBees Jenkins Enterprise Workstation

Controller

22

SSH

CloudBees Jenkins Enterprise Workstation

Controller

80

HTTP

CloudBees Jenkins Enterprise Workstation

Controller

443

HTTPS

CloudBees Jenkins Enterprise Workstation

Master Worker or Build Worker

22

SSH

Master Worker or Build Worker

Master Worker or Build Worker

80, 81

HTTP

Master Worker or Build Worker

Internal Load Balancer

80

HTTP

Master Worker or Build Worker

Internal Load Balancer

443

HTTPS

Master Worker or Build Worker

Controller

8080

Marathon

Master Worker or Build Worker

Controller

5050

Mesos

Master Worker or Build Worker

Controller

2181

Zookeeper

Master Worker or Build Worker

Master Worker or Build Worker

31000 - 32000

CloudBees Jenkins Enterprise

ports

Server roles

There are two kinds of servers in a CloudBees Jenkins Enterprise cluster, Controllers and Workers. The diagram below shows the different applications/processes that run on Controllers and Workers.

cje process architecture

Controller

Within the cluster, controllers are the brains. They decide what tasks need to be executed, and where those tasks should run. They also answer external requests, and route traffic within the cluster.

A controller instance has the following responsibilities:

  • Manages the cluster resources by counting the used and available CPU, memory across the cluster, and offering them to execute tasks. This is accomplished via Apache Mesos, which has an agent running on each worker, and Zookeeper.

  • Schedules tasks, checking the services availability and making sure the amount of services defined on the cluster is actually running. This is done by Marathon, a scheduler that works with the Mesos Master.

  • Routes web traffic, forwarding incoming traffic to the right service depending on the requested service and its current location within the cluster. It also provides the SSL termination if configured. This is done by the Router process running on each controller.

Masters also run processes for monitoring: Logstash for logs and Topbeat for operating system metrics such as CPU, memory usage and disk space. Both of these processes connect to the Elasticsearch index running on the Worker servers.

Note
For a High Availability (HA) cluster setup, you should have at least 3 controllers.

Using multiple controllers enables you to avoid the risks associated with what could otherwise be a single point of failure. On environments supporting it, a load balancer is set up to distribute traffic across the controller instances.

Workers

Workers provide computing capacity within the cluster, which is used to host and run end-user applications including Operations Center, Managed Masters and Agents. You can increase cluster performance and resiliency by adding more workers or configuring workers with more memory and CPU.

Workers run these processes:

  • Operations Center - There will only be one instance of the operations center in a cluster.

  • Managed Masters - Managed Masters can be dynamically created via Operations Center. CloudBees Jenkins Enterprise does health-checks on Masters and can restart them if needed, see the High Availability section below for more on this.

  • Castle - Castle is a volume manager that allocates disk stage for Jenkins data and for Operations Center backups.

  • Palace - Palace is a scheduler works with Mesos to create and manage Jenkins Build Agents.

  • Jenkins Agents - Jenkins Build Agents are created on demand as required by Jenkins jobs run on the Masters.

  • Logstash and Topbeat: which report monitoring information to Elasticsearch.

  • Elasticsearch - used to indexes logs and metrics and makes them available to Operations Center.

Workers can be added and removed over time to adjust cluster capacity for the sake of meeting demand efficiently.

Palace
BinPack algorithm

Palace uses the BinPack algorithm by default to provision single-use agents. In clusters with more than one worker of type build, you may notice unpredictable job run times. This happens because many agents may be packed into one worker and share the worker CPU, so that worker instance is heavily overloaded, while others have little or no load.

This problem usually happens when CloudBees Jenkins Enterprise administrators specify a low CPU value in their templates. Agents should be defined with a higher CPU value. If the CPU value is defined as only 0.1, Mesos will only spread based on other weights, like memory.

Tip
Notice that in Mesos, CPU share is a weight. Setting it to the same value will give the same priority to each container.

Using the LeastLoaded algorithm might work better in cases where there is enough capacity so all build worker nodes are not fully allocated. However, CloudBees Jenkins Enterprise does not provide a permanent way to configure the LeastLoaded algorithm. Each time Palace or Marathon is restarted, the BinPack algorithm will be used. It is the responsibility of the CloudBees Jenkins Enterprise administrator to again switch to the LeastLoaded algorithm.

The change can be made permanent by adding a new environment variable to the env section of the file `$CJE_BINARY/share/setup-templates/core/templates/palace/marathon.json `. After this, it is needed to upgrade the project by applying the change on the template with the command below.

$ cje upgrade-project --apply-template

Changes to that file require a dna stop palace and dna start palace after upgrading the project. However, after a CloudBees Jenkins Enterprise CLI upgrade, this change might be overwritten by the new release.

LeastLoaded algorithm

The LeastLoaded algorithm schedules a task on the least loaded node at the time of scheduling. This spreads the load evenly across build workers.

Instructions for switching to the LeastLoaded algorithm are explained below.

Warning
Changes to the scheduling algorithm are temporary. The BinPack algorithm is used whenever Palace or Marathon restarts. Therefore, CloudBees recommends customizing task weights to control task assignment to workers.
  • Get the URL from Marathon

$ cje run display-outputs
  • Get the required username/password credentials

$ cje run echo-secrets marathon_username
$ cje run echo-secrets marathon_password
  • Open the Marathon UI and log in using the account info above

  • From the Marathon dashboard, select the jce application group

  • Select the Palace application

  • Switch to the Configuration tab below the Scale Application button

cje architecture marathon configuration
  • On the right side of the screen, press the Edit button

  • Select the "Environment Variables" section

  • Press one of the + buttons on the right edge and add an entry: CONSTRAINTS : 8

cje architecture marathon env
  • Press the Change and Deploy Configuration button.

The valid constraint codes are:

  • BinPack: 1

  • LeastLoaded: 8

Types of workers

To give more options for scaling, CloudBees Jenkins Enterprise defines three different types of Workers:

  1. Master Workers run Operations Center and Managed Masters

  2. Build Workers run Jenkins Build Agents

  3. Elasticsearch Workers: (optional) dedicated to running Elasticsearch.

The diagram below shows the four different types of servers we have and the processes that run on each.

cje deployment architecture

Having these different types of servers allows us to scale each type of server independently.

Additional worker capacity enables CloudBees Jenkins Enterprise to run more jobs in parallel, improving the performance of the cluster. Adding workers also increases cluster resiliency, because spare workers are available in the event of a worker outage. This minimizes Managed Master and Agent downtime in the event of a server failure. Adding more Elasticsearch capacity is necessary as the size of the Elasticsearch indexes grow.

Command line interfaces

Many of the CloudBees Jenkins Enterprise cluster and maintenance tasks can be executed through the varied sub-commands provided by the cje command-line tool. These commands are intended for Operators and/or Administrators of the CloudBees Jenkins Enterprise environment, and require a configured project workspace.

For more information, consult the CLI Reference.

Other plugins also provide command line interfaces in CloudBees Jenkins Enterprise, including:

Elasticsearch-based persistence layer

One of the persistence layers underpinning CloudBees Jenkins Enterprise, Elasticsearch, provides storage for various types of data such as raw metrics, job-related information, and logs.

For more details on what types of data, indices, and maintenance procedures, refer to the Elasticsearch Reference.

Configuration files

CloudBees Jenkins Enterprise Cluster configuration is achieved by customizing two text files: cluster-init.config and cluster-init.secrets. These files use the INI file format for overriding default configuration values for CloudBees Jenkins Enterprise.

For a detailed listing of options and tips for customizing CloudBees Jenkins Enterprise configuration files, consult the Configuration Reference.

Integrations

CloudBees Jenkins Enterprise can be integrated with ServiceNow® using a ServiceNow App and a CloudBees Jenkins Enterprise plugin. Refer to the ServiceNow Integration for more information.