Managing Client Masters

This chapter describes how to manage your Client Masters through Operations Center.

Configuring a Client Master

This section describes how to configure a Client Master that has already been connected to your Operations Center instance.

To access a Client Master’s configuration:

  1. Ensure you are logged in to Operations Center as a user with the Client/Managed Master > Configure permission.

  2. From the Operations Center home page/Dashboard, click to the right of your configured Client Master (avoiding its name) and choose Configure from the dropdown menu.
    Client Master dropdown menu

  3. On the resulting Client Master configuration page, you can configure the following properties:

    • Description - Enter an optional description for the Client Master.

    • Health Reporting - When this check box is selected, health-related metrics from this Client Master are collected periodically. The default data collection period is once per minute, when data consumers are present (e.g. Weather columns or CloudBees Jenkins Analytics).

    • Analytics Reporting - When this check box is selected, report events and other metrics from this Client Master for CloudBees Jenkins Analytics are collected.

    • On-master executors - Select Enforce to specify the # of executors, which periodically ensures that the number of executors on the Client Master is the value specified in this # of executors field. Allowing items (i.e. projects or jobs) to execute directly on the Client Master is a security risk since such projects/jobs could potentially access the file system and the build records of all previously run projects/jobs (which may contain sensitive information). Therefore, set this value to 0 to prevent any items from being executing directly on the Client Master.

    • Master Owner - Specify the email address/es (one per line) of the "Owner/s" to be notified whenever this Client Master goes offline or changes state.
      Note: Clicking the Advanced button opens the Delay before notification field, which allows you to specify the number of minutes (a value between 1 and 60) between notifications.

    • Plugin Catalog - Select Specify a plugin catalog for this master to choose a plugin catalog to apply to this Client Master.

Configuring Plugin Catalogs

The Beekeeper Upgrade Assistant feature of your Operations Center’s Manage Jenkins area is the main interface and entry point to the CloudBees Assurance Program.

Beekeeper Upgrade Assistant manages appropriate upgrades (and downgrades) of plugins on your Operations Center instance, in accordance with the CloudBees Assurance Program.

On Client Masters, the Beekeeper Upgrade Assistant feature is also available, and:

  • Is accessible through a Client Master’s Manage Jenkins area

  • Performs exactly the same types of management activities (as this feature does for your Operations Center instance) on their respective Client Master instances.

However, if you also need to install non-CloudBees Assurance Program plugins on your Client Masters, such as plugins which are unknown to CloudBees (perhaps developed internally within your organization) or older versions of plugins, you can configure a plugin catalog. A plugin catalog is used by the Beekeeper Upgrade Assistant to widen the acceptable scope of plugins beyond those defined by the CloudBees Assurance Program. This scope includes the ability to specify plugin dependency rules and acceptable version ranges for these plugins. Your plugin catalog can also specify a Maven 2 artifact repository from which to download plugins.

Once a plugin catalog is defined and is added to your Operations Center instance, it can then be deployed to your connected Client Master instances.

This section contains the following procedures:

Defining a Plugin Catalog

Before you can add a plugin catalog to your Operations Center instance, you must first define it.

A plugin catalog is defined in JSON file format, which in turn defines one or more plugin configurations. A plugin configuration defines the set of rules for one or more plugins (some potentially dependencies) and their versions, which can be installed to your Operations Center instance, along with the CloudBees Assurance Program-defined plugins.

To create this file, using your favorite text editor, copy and paste the unannotated empty template below into a new file and save it with an appropriate file name, similar to that of the name or displayName value.

The following JSON content is an example of the contents of a plugin catalog file, whose annotations are explained underneath. An appropriate name for this plugin catalog file might be
additional-pipeline-plugins-catalog.json. Use this information to help complete your own plugin catalog file.

Annotated plugin catalog file
{
    "type" : "plugin-catalog", (1)
    "version" : "1", (2)
    "name" : "additional-pipeline-plugins", (3)
    "displayName" : "Additional Pipeline plugins", (4)
    "configurations" : [ (5)
        {
            "description" : "Additional Pipeline plugins for masters versions 2.61 and 2.73", (6)
            "prerequisites": { (7)
                "productVersion": "[2.60, 2.73]"
            },
            "includePlugins" : { (8)
                "tier3-plugin": {
                    "groupId": "com.my-company.jenkins.plugins", (9)
                    "version" : "1.8.2"
                },
                "my-corporate-plugin": {
                    "url" : "https://download.my-company.com/jenkins/my-corporate-plugin-3.2.1.hpi"
                },
                "my-other-corporate-plugin": {
                    "sha1" : "SC/TzSN+eOrewaDZJZyzLpIQV7E=", (10)
                    "url" : "https://download.my-company.com/jenkins/my-other-corporate-plugin-1.2.3.hpi" (11)
                }
            }
        },
        { (12)
            "description" : "Additional Pipeline plugins for masters versions 2.89+",
            "prerequisites": {
                "productVersion": "[2.89)"
            },
            "includePlugins" : {
                "tier3-plugin": {
                    "groupId": "com.my-company.jenkins.plugins",
                    "version" : "1.8.3"
                },
                "my-corporate-plugin": {
                    "url" : "https://download.my-company.com/jenkins/my-corporate-plugin-3.4.0.hpi"
                },
                "my-other-corporate-plugin": {
                    "sha1" : "SC/TzSN+eOrewaDZJZyzLpIQV7E=",
                    "url" : "https://download.my-company.com/jenkins/my-other-corporate-plugin-1.3.0.hpi"
                }
            }
        }
    ]
    settings { (13)
        httpCredentialsProvider: { (14)
            credentials: [ (15)
                {
                    "authenticationScope": "ANY", (16)
                    "credentialsId": "download-server-credentials" (17)
                }
            ]
        },
        "repository": { (18)
            "layout": "maven2", (19)
            "url": "https://repo.my-company.com/content/repositories/dev-connect"
        }
    }
}
  1. type (required) - must have the fixed value plugin-catalog, which defines this JSON file as a plugin catalog file.

  2. version (required) - must currently have the value 1, which defines the current metadata format for the plugin catalog.

  3. name (required) - the internal name used to identify the plugin catalog. Avoid using spaces in this name.

  4. displayName (optional) - a human-readable name for the plugin catalog. This is the name of the plugin catalog that appears in the Operations Center interface.

  5. configurations (required) - the plugin catalog can define one or more plugin configurations within this array, where each plugin configuration is represented by an individual element of this array. At least one plugin configuration is required.

  6. description - provides more context regarding the configuration itself.

  7. prerequisites (optional) - an object which allows the definition of the productVersion member, which defines a range of supported Client Master versions (using the Maven Version Range Specification) for the plugin configuration. If the prerequisites object (and hence its productVersion member) is omitted, then any Client Master version is supported.

  8. includePlugins (optional) - an object which allows you to specify the ID values of one or more plugins to include as part of this plugin configuration. For each plugin (specified by its plugin-id), specify the pair made up of the supported groupId and version for the plugin or the URL from where download it.

  9. groupId (optional) - provides the groupId for the plugin when its download will take place from a maven repository. If the groupId is not provided, this field will consider org.jenkins-ci.plugins as default value.

  10. sha1 (optional) - the SHA1 hash of the plugin that will be used to verify the downloaded plugin.

  11. url (optional) - the download URL for the plugin.

  12. (Optional) - an additional plugin configuration that provides support for different plugin versions (or plugins themselves), for different Client Master versions (specified in the prerequisites > productVersion member).

  13. settings (optional) - the plugin catalog defines all parameters used in the download of the plugins.

  14. httpCredentialsProvider (optional) - the plugin catalog defines a set of credentials using the interface CredentialsProvider.

  15. credentials (required) - the array containing the credentials. Currently must contain only one element.

  16. authenticationScope (required) - if the credentials are defined, must currently have the value ANY.

  17. credentialsId (required) - the ID of a credential that must be defined in the Client Master where the plugin catalog will be installed. This credential will be used for plugin downloads from servers which require authentication.

  18. repository (optional) - an object which allows the definition of a Maven 2 repository from which the plugins (defined in the configurations array above) can be downloaded. If a plugin’s url has not been specified within its relevant configurations section, then the Client Master will attempt to download the plugin from this repository‍'s url. Defining a repository is useful when you need to restrict a Client Master’s access to the Internet for security reasons (part of a process known as air gapping), [1] but want the Client Master to install and update plugins from the repository, which can be achieved by setting up a proxy artifact repository.

  19. layout (optional) - must currently have the value maven2. However, if this value is not defined, it is assumed to be maven2 by default.

Copyable plugin catalog file
{
    "type" : "plugin-catalog",
    "version" : "1",
    "name" : "",
    "displayName" : "",
    "configurations" : [
        {
            "description" : "",
            "prerequisites": {
                "productVersion": ""
            },
            "includePlugins" : {
                "replace-with-appropriate-plugin-id": {
                    "groupId": "",
                    "version" : ""
                },
                "replace-with-other-appropriate-plugin-id": {
                    "sha1" : "",
                    "url" : ""
                }
            }
        }
    ]
    settings {
        httpCredentialsProvider: {
            credentials: [
                {
                    "authenticationScope": "ANY",
                    "credentialsId": ""
                }
            ]
        },
        "repository": {
            "layout": "maven2",
            "url": ""
        }
    }
}

Adding a Plugin Catalog to Operations Center

Once you have defined your plugin catalog, you can add it to your Operations Center instance using the Jenkins CLI tool. This section assumes you have configured an alias for the Jenkins CLI tool.

  1. In a terminal/command prompt (window), cd to the directory containing the plugin catalog file you defined above.

  2. Enter the command:

    jenkins-cli plugin-catalog --put < plugin-catalog-file.json

    where plugin-catalog-file.json is the name of your plugin catalog file.
    For example, if you used the plugin catalog example above and saved it to a file named
    additional-pipeline-plugins-catalog.json, then you would enter the command:

    jenkins-cli plugin-catalog --put < additional-pipeline-plugins-catalog.json

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog updated correctly",
      "status": "SUCCESS"
    }

    You can now validate the plugin catalog’s suitability for deployment on a configured Client Master.

Note:

  • If the command was not successful, for example - the JSON format of the plugin file was malformed, the Jenkins CLI tool returns an error like:

    {
      "message": "Unexpected character ('}' ... [Source: ... line: 16, column: 14]",
      "status": "FAILURE"
    }

Updating a Plugin Catalog on Operations Center

If one or more plugins (defined in a plugin catalog) have updates, you may need to specify these plugin updates in the plugin catalog and then update this plugin catalog on Operations Center.

This process is identical to adding the plugin catalog to Operations Center:

  1. Modify the plugin catalog file defined previously (for example, with the updated plugin versions).

  2. Re-add the plugin catalog to Operations Center.

You should then re-validate and then re-deploy the updated plugin catalog to the Client Masters on which the older version of this plugin catalog had been deployed.

If you are unsure of which Client Masters the older version of the plugin catalog had been deployed to, verify the plugin catalog’s deployment on the relevant Client Masters.

Validating a Plugin Catalog’s Suitability for a Client Master

Once you have added a plugin catalog to Operations Center, you can verify this on the Operations Center interface when validating the plugin catalog’s suitability for deployment to a configured Client Master.

This validation process verifies:

  • That the Client Master version and plugin configuration satisfies exactly one plugin catalog definition

  • That plugin dependencies are resolved

  • That credentials referenced in the plugin catalog are defined in the Client Master

  • The credential count

  • The credential scope

To validate your plugin catalog’s suitability for deployment to a Client Master:

  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scoll down to the Plugin Catalog property and select its Specify a plugin catalog for this master check box.

  3. Select the plugin catalog from the Catalog dropdown.
    This name is the displayName specified when the plugin catalog was defined.

  4. Click the Check Validity button.
    If your plugin catalog’s definition is suitable for deployment to the Client Master, the message The catalog is compatible with the master is displayed.
    Plugin Catalog validity check

  5. ( Optional ) Scroll down the page and click the Save or Apply button to deploy the plugin catalog to the Client Master.

Note
To validate your plugin catalog in this manner against all Client Masters that have been connected to your Operations Center instance, repeat this procedure on each Client Master.

Deploying a Plugin Catalog to a Client Master

A plugin catalog which has been added to Operations Center can be deployed to a configured Client Master (after optionally validating its suitability for deployment) using either the Operations Center interface or the Jenkins CLI tool.

Note
Be aware that only one plugin catalog can be deployed to a Client Master at a time.
Deploying a Plugin Catalog Using the Operations Center interface
  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scoll down to the Plugin Catalog property and select its Specify a plugin catalog for this master check box.

  3. Select the plugin catalog from the Catalog dropdown.
    This name is the displayName specified when the plugin catalog was defined.

  4. Scroll down the page and click the Save or Apply button to deploy the plugin catalog to the Client Master.

  5. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Deploying a Plugin Catalog Using the Jenkins CLI Tool

This section assumes you have configured an alias for the Jenkins CLI tool.

  1. In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --push plugin-catalog-name --master client-master

    where plugin-catalog-name is the name of the plugin catalog specified when the plugin catalog was defined and client-master is the name of the Client Master configured on your Operations Center instance.
    For example, if you used the plugin catalog example above and have already added it to your Operations Center instance, then you would enter the command:

    jenkins-cli plugin-catalog --push additional-pipeline-plugins --master client-master

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog applied correctly, it will be propagated in a few seconds",
      "status": "SUCCESS"
    }
    Note
    When using this Jenkins CLI command, the --push option must always be used in conjunction with the --master option. If you only use one of these options, the Jenkins CLI tool returns an error.
  2. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Verifying a Plugin Catalog’s Deployment to a Client Master

To validate that a plugin catalog has been successfully deployed to the Client Master:

  1. Ensure you are logged in to the Client Master as a user with the Administer permission.

  2. From the Client Master home page/Dashboard, click Manage Jenkins on the left.

  3. Click Beekeeper Upgrade Assistant.

  4. Click Plugin Catalog on the left (or the section’s More Info button on the right).
    Plugin Catalog on master
    If you see the message There is a plugin catalog installed, then the plugin catalog was deployed to the Client Master and the remaining details on this page should depict the plugin catalog that had been defined and added to Operations Center, and deployed to this Client Master.

    Note
    If you only just deployed a plugin catalog to the Client Master and you see No Plugin Catalog is installed at the moment, it may take a few minutes before the Client Master registers this deployment. Therefore, wait a few moments and refresh this page, or revisit this procedure at a later point in time.
  5. ( Optional ) For any Included plugins shown on this page, you can now install or update these plugins on the Client Master itself (if these plugins have not yet been installed/updated).

Removing a Plugin Catalog from a Client Master

A plugin catalog that has been deployed to a Client Master can be removed from it using either the Operations Center interface or the Jenkins CLI tool.

Removing a Plugin Catalog Using the Operations Center interface
  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scoll down to the Plugin Catalog property and clear its Specify a plugin catalog for this master check box.

  3. Scroll down the page and click the Save or Apply button to confirm the removal of the plugin catalog from the Client Master.

  4. ( Optional ) Verify the plugin catalog’s removal from the Client Master.

Removing a Plugin Catalog Using the Jenkins CLI Tool

This section assumes you have configured an alias for the Jenkins CLI tool.

  1. In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --remove client-master

    where client-master is the name of the Client Master configured on your Operations Center instance.
    For example, if you have already added a plugin catalog to your Operations Center instance and deployed it to a Client Master whose name is client-master, then you would enter the command:

    jenkins-cli plugin-catalog --remove client-master

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "client-master",
      "message": "Catalog removed",
      "status": "SUCCESS"
    }
  2. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Verifying a Plugin Catalog’s Removal from a Client Master

To validate that a plugin catalog has been successfully removed from a Client Master:

  1. Ensure you are logged in to the Client Master as a user with the Administer permission.

  2. From the Client Master home page/Dashboard, click Manage Jenkins on the left.

  3. Click Beekeeper Upgrade Assistant.

  4. Click Plugin Catalog on the left (or the section’s More Info button on the right).
    If you see the message No Plugin Catalog is installed at the moment, then the plugin catalog was removed from the Client Master.

    Note
    If you only just removed a plugin catalog from the Client Master and you see There is a plugin catalog installed, it may take a few minutes before the Client Master registers this removal. Therefore, wait a few moments and refresh this page, or revisit this procedure at a later point in time.

Removing a Plugin Catalog from Operations Center

A plugin catalog added to your Operations Center instance can be removed from it using the Jenkins CLI tool. This section assumes you have configured an alias for the Jenkins CLI tool.

Note
A plugin catalog can only be removed from your Operations Center instance when no Client Masters are using the plugin catalog.
  • In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --delete plugin-catalog-name

    where plugin-catalog-name is the name of the plugin catalog specified when the plugin catalog was defined.
    For example, if you used the plugin catalog example above, then you would enter the command:

    jenkins-cli plugin-catalog --delete additional-pipeline-plugins

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog [additional-pipeline-plugins] successfully deleted",
      "status": "SUCCESS"
    }

Note:

  • If the Jenkins CLI tool returns an error like:

    {
      "message": "Catalog [additional-pipeline-plugins] is being used by the following master/s: [client-master]. It can not be deleted",
      "status": "FAILURE"
    }

    then the plugin catalog is still being used by at least one Client Master (in this example, a Client Master named client-master). If the plugin catalog is being used by multiple Client Masters, then each Client Master is listed in the error response. To rectify this issue, remove the plugin catalog from each Client Master listed in the error response and then try following this procedure again to remove the plugin catalog from your Operations Center instance.

Installing and updating Client Master plugins

Like the Operations Center, a Client Master makes it easy to install and update plugins that extend its functionality.

Installing a Client Master plugin

Installing a Client Master plugin is performed through the Manage Plugins page.

To install a Client Master plugin:

  1. Access the Manage Jenkins area of the appropriate Client Master.

  2. On the Manage Jenkins page, scroll down and click Manage Plugins.

  3. On the resulting Plugin Manager page, check that the plugin is not already installed by clicking the Installed tab and begin typing the name of the Client Master plugin to install into the top-right Filter field.

    If the page’s list shows the plugin, then it is already installed on Client Master and you need not proceed any further. However, ensure the plugin’s Enabled check box is selected.

    If the plugin does not appear in the list, then it is not installed. Therefore, continue on.

  4. Click the Available tab and begin typing the name of the Client Master plugin to install into the top-right Filter field.

  5. In the page’s list, select the plugin’s Install check box on the left.

  6. Scroll down the page and click either the Install without restart or Download now and install after restart button.

Updating a Client Master plugin

Updating a Client Master plugin is also performed through the Manage Plugins page.

To update a Client Master plugin:

  1. Access the Manage Jenkins area of the appropriate Client Master.

  2. On the Manage Jenkins page, scroll down and click Manage Plugins.

  3. On the resulting Plugin Manager page, check that plugin updates are available by ensuring the Update tab is selected and clicking the Check now button towards the end of the page.

    If the page’s list does not show any plugins, then no plugin updates are available. There is no need (and you cannot) proceed any further.

    If a plugin does appear in the list, then an updated version of the plugin is available. To update the plugin, continue on.

  4. In the page’s list, select the plugin’s Install check box on the left.

  5. Scroll down the page and click the Download now and install after restart button.

Setting up a proxy artifact repository

To maintain the security of Client Masters, as well as your build environment and infrastructure, it is usually a good idea to restrict your Client Masters' access to the Internet as part of a process known as air gapping. [1]

However, Pipelines and other types of build projects configured on your Client Masters and other CloudBees Core masters are likely to require access to artifact repositories on the Internet, since artifacts from such external and/or trusted repositories may need to be integrated into these build projects' builds. Additionally, these external artifact repositories may also house custom plugins, which you can define and configure in a plugin catalog, to install these plugins on your Client Masters.

For instance, the publicly accessible repository https://repo.jenkins-ci.org/releases/ is one such artifact repository to which the Jenkins Community publishes Maven artifacts, as well as hpi/plugin files.

Therefore, to provide your Client Masters with access to content housed by these external artifact repositories, set up a proxy artifact repository (with access to the Internet) internally within your organization’s infrastructure.

To set up a proxy artifact repository to an external artifact repository:

  1. Ensure you have installed an artifact repository system within your organization’s infrastructure, and ensure the artifact repository has access to the Internet.

  2. In your artifact repository system, configure your proxy artifact repository to your external or publicly accessible artifact repository.

    • For Sonatype’s Nexus Repository, configure the Proxy  Remote storage option in your new proxy artifact repository’s settings. Read more about configuring a proxy artifact repository in Nexus Repository using their Repository Management documentation, as well as the Proxy > Remote Storage sub-section of the Managing Repositories and Repository Groups section of this documentation.

    • For JFrog’s Artifactory, read more about configuring a proxy artifact repository in JFrog Artifactory using their Managing Proxies documentation.

Configuring Masters through CLI

Jenkins allows some operations to be invoked through CLI, some of them being useful to configure Client Masters.

But also, when we are talking about Client Masters configuration, applying the same identical configuration to each one is something desirable. Thus we need a way to gather all connected masters from the command line and then perform some operations on each one.

The list-masters CLI command on Operations Center provides information about all connected masters in JSON format, allowing you to use that information to invoke commands on each Client Master:

{
  "version": "1",
  "data": {
    "masters": [
      {
        "fullName": "my master", (1)
        "url": "http://localhost:9090/", (2)
        "status": "ONLINE" (3)
      }
    ]
  }
}
  1. fullName - the name of the Client Master, including folders.

  2. url - URL for the Client Master.

  3. status - the connection status of the Client Master. It can be 'ONLINE' or 'OFFLINE'.

Examples:

Execute a Groovy script and install a plugin

The following bash script will execute a Groovy script and install the 'beer' plugin on all the online Client Masters:

#!/usr/bin/env bash

JENKINS_CLI=jenkins-cli.jar
JENKINS_CJOC_URL=http://localhost:8080/
JENKINS_AUTH=admin:admin

if [ -z "$JENKINS_CJOC_URL" ]; then
    echo "Need to set environment variable JENKINS_CJOC_URL ({CJOC} root URL)."
    exit 1
fi

if [ -z "$JENKINS_AUTH" ]; then
    echo "Need to set environment variable JENKINS_AUTH (format: 'userId:apiToken')."
    exit 1
fi


if [ -f "$JENKINS_CLI" ]
then
	echo "Using $JENKINS_CLI."
else
	wget -O "$JENKINS_CLI" $JENKINS_CJOC_URL/jnlpJars/jenkins-cli.jar
fi

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH list-masters | jq -r '.data.masters[] | select(.status == "ONLINE") | .url' | while read url; do
	java -jar $JENKINS_CLI -s $url -auth $JENKINS_AUTH groovy = < configuration-script.groovy
	java -jar $JENKINS_CLI -s $url -auth $JENKINS_AUTH install-plugin beer
done

Push a Plugin Catalog to all Client Masters

The following bash script will add a Plugin Catalog to Operations Center and push it to all the online Client Masters:

#!/usr/bin/env bash

JENKINS_CLI=jenkins-cli.jar
JENKINS_CJOC_URL=http://localhost:8080/
JENKINS_AUTH=admin:admin

if [ -z "$JENKINS_CJOC_URL" ]; then
    echo "Need to set environment variable JENKINS_CJOC_URL ({CJOC} root URL)."
    exit 1
fi

if [ -z "$JENKINS_AUTH" ]; then
    echo "Need to set environment variable JENKINS_AUTH (format: 'userId:apiToken')."
    exit 1
fi


if [ -f "$JENKINS_CLI" ]
then
	echo "Using $JENKINS_CLI."
else
	wget -O "$JENKINS_CLI" $JENKINS_CJOC_URL/jnlpJars/jenkins-cli.jar
fi

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH  plugin-catalog --put < "java-web-catalog.json" | jq

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH list-masters | jq -r '.data.masters[] | select(.status == "ONLINE") | .fullName' | while read masterName; do
	java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH plugin-catalog --push "java-web" --master "$masterName" | jq
done