Table of contents

CloudBees Jenkins Operations Center administration guide


Move/Copy/Promote

Starting with the Operations Center Context plugin version 1.7.1 CloudBees Jenkins Operations Center (Operations Center) and CloudBees Jenkins Enterprise (CloudBees Jenkins Enterprise) provide enhanced Move and Copy operations for Job, Folders and other items. This functionality enhances and complements the facility to "promote" job configurations from one location to another — between client masters or within the same client master added in Operations Center 1.7.

Note

Each one of the three operations may or may not be supported for a given item type. The following table gives a summary of this support:

Table 1. Supported Operations Per Item Type

Item type

Move

Copy

Promote [1]


1. will be replaced by "copy without build history"

Freestyle Job

Maven Job

Multi Configuration Job (Matrix)

Pipeline Job

Pipeline Multi Branch Job

Folder

GitHub Org

Bitbucket Team / Project

Backup

Long Running Project

Template

Table 2. Supported Operations Per Item Type, Specific to Operations Center

Client Master

Shared Agent

Shared Cloud

Update Center

AWS EC2 Cloud Configuration

Cluster Operation

Jenkins does not have Folders plugin installed by default. Without any folders to move jobs between, it does not make sense to have a move concept. When you install the Folders plugin in Jenkins, this plugin adds a Move operation to all items that can be moved.

legacy action
Figure 1. The move operation provided by the Folders plugin.

Out of the box, Jenkins does not have capabilities for copying jobs between masters. Again, in the absence of folders and when limited to a single master, the major use case for copying a job is to create a new job with the same configuration as an existing job.

That use case is provided for by Jenkins using New Item  Copy an existing Item. When you install the Folders plugin in Jenkins, there are some use cases for copying a job with its entire build history. Typically those use cases are solved by the Jenkins administrator manually copying the job within the JENKINS_HOME directory structure and restarting Jenkins.

With the introduction of Operations Center, however, there now are a lot more use cases for Move, Copy, and Promote operations:

  • Moving jobs between masters in order to distribute the load between multiple masters

  • Copying a job from one master to another to verify that the job and its associated history is transferred correctly before removing the original

  • Copying an example folder with jobs to seed a project initiation

  • Copying a job from a production master to a test master in order to assess the impact of a plugin upgrade.

  • Copying a job from a test master to a production master and skip builds from the copy

Within the above context there is also an additional set of use cases for some customers who have more strict change control processes. Such customers typically will allow users to configure a job in one environment and then move the changes to that job to production. In essence they are configuring the job and then promoting the configuration of that job towards production (typically through a test environment).

Operations Center 1.7 introduced the Promote Configuration operation to support this last use case. Due to the nature of a "promotion" operation, the support for this has to be specifically developed for each job type. At the time of writing configuration promotion support has only been implemented for Free-style jobs.

With Operations Center Context version 1.7.1, the Promote Configuration operation has been subsumed by the new Jenkins  Move/Copy/Promote operation. This new operation also subsumes the Move operation that is provided by the Folders plugin.

new action
Figure 2. The Move/Copy/Promote operation provided by the Operations Center Context plugin, replacing the Move operation provided by the Folders plugin.

Bear in mind the following before you proceed:

  • Upgrade all client masters to at least Operations Center Context 1.7.1 before using the move/copy/promote functionality between client masters.

  • If you attempt to Move/copy/promote to a Jenkins instance that does not have at least version 1.7.0 of the Operations Center Context plugin, the destination will not respond to the validation messages and the originating master may wait for up to one hour before timing out and reporting a validation failure.

  • To be able to move an item from one master to another, you would need to update the client-master security section on Operations Center, setting the authentication mapping to any of the 'trusted with equivalent' security realm options (both restricted master with equivalent security realm and trusted master with equivalent security realm will work). You can choose the one you need in the JENKINS_URL/configureSecurity page.

  • If you attempt to move/copy anything other than a single free-style job to a Jenkins instance that does not have at least version 1.7.1 of the Operations Center Context plugin, the operation will fail on the receiving end.

Moving, Copying, or Promoting Items Using the Web UI

To move/copy/promote a job / folder / item — browse to the item and select the Move/Copy/Promote action from the menu. You will see a screen similar to the following:

initial screen
Figure 3. The initial screen. The default destination reflects the current location of the item to be moved (in this case a folder called "widgets")

Note the following about this screen:

  • The path browser defaults to the current location of the item.

  • The path browser always corresponds to the location containing the item. If there is already an existing item with the same name at the selected destination, then the move operation fails.

  • The three buttons at the bottom of the screen reflect your capabilities for the current item. For example, if support for promotion of the specific type of item has not been written, then the Promote button will be disabled.

Using the path browser you can navigate to the location you want to move/copy/promote the item to.

Note
This functionality is available even on a standalone Jenkins master, although in such cases the operation is limited to that master.

Once the correct destination has been selected, click on the desired operation:

Note
Starting on 2.46.2.x the Copy button has been refactored into a two-options button. The new text is "Copy with builds", and a dropdown menu offers the operation "Copy without builds".
  • The Move button will move the current item with all its build history to the new location. The original item will be removed on confirmation of the item having been successfully received at the destination.

  • The Copy with builds (just Copy in older versions) button will copy the current item with all its build history to the new location. The original item with all its build history will also be retained at its original location.

  • When using "Copy without builds" (in CJE 2.46.2.x or more) the build history is not copied to the new location. This option has better in performance when there are lots of builds and artifacts to copy and there is no interest in copying them.

copy without builds
Figure 4. Copy without builds dropdown split button
  • The Promote button will copy the configuration without any build history of the item to the destination. If an item of the same name already exists at the destination, that item’s configuration will be replaced by the source item’s configuration. The build history of the source item and any pre-existing destination item will remain unchanged by the promotion.

The first thing that will happen is that Jenkins will attempt to validate the operation. There are a list of checks that are performed. These checks depend on the type of operation, the type of item, as well as whether the destination is to a different Jenkins instance. The checks are sent to the destination master for validation.

validation in progress
Figure 5. Initial validation checks in progress
  • If the destination master reports all checks were successful then the operation will be submitted to the Jenkins build queue.

    operation in progress 1
    Figure 6. A move/copy/promote request having been completed validation successfully and accepted into the build queue automatically.
  • If the destination master reports warnings for some checks, then a confirmation screen will be shown:

    validation warnings
    Figure 7. A move/copy/promote request where the validation identified some problems (in this case there are some missing plugins and the destination master is running an older version of Jenkins)

    The user can then decide whether to continue the operation or abort.

    Note
    If the user elects to continue the operation, all warnings will be ignored.
  • If the destination master reports errors for some checks, then a confirmation screen will be shown:

    validation errors
    Figure 8. A move/copy/promote request where the validation indicates that the request will fail (the error handling defaults to the safe option of "Stop on warnings or errors" to prevent accidental selection of the Continue button)

    This screen differs from the case of warnings as the user must explicitly select the error handling behaviour. The default error handling on this screen is to stop for warnings or errors found during the pre-flight check phase.

    Caution

    If validation errors are reported then the operation will almost certainly not succeed.

    In certain cases it may be possible to copy or promote by ignoring warnings or ignoring both errors and warnings.

    Do not attempt a move operation when ignoring errors and warnings unless you have a current backup of the item being moved as when errors are ignored the move operation will always delete the source item.

operation in progress 2
Figure 9. A move/copy/promote operation in the running post-validation checks.

The move/copy/promote operations are injected into the Jenkins build queue as they can be potentially long running operations:

  • Move operations need to ensure that the job is not currently building and also that the job does not try to start building during the move operation.

  • Move operations acting on a folder need to ensure that all jobs contained in the folder are not currently building.

  • Copy and move operations may involve sending significant quantities of data between masters, this can take quite some time if there are a lot of large archived artifacts.

    operation in progress 3
    Figure 10. A move/copy/promote operation waiting for confirmation that the files have been received by the destination.
  • Any of the move/copy/promote operations, when interacting with a remote master, need to allow for the possibility that the destination master may experience a temporary network outage during the operation.

Note
Once the operation has been submitted to the Jenkins build queue, the progress dialog can be closed if the user wants to do other things.

When the operation completes, then progress dialog will report the results.

operation success
Figure 11. A successfully completed move/copy/promote request. Clicking Close will redirect the browser to the destination item.
operation failure
Figure 12. A failed move/copy/promote request. If the destination item exists then clicking Close will redirect the browser to the destination item, otherwise you will be returned to the source item.

If the destination item exists then the user will be taken to the destination item when they close the progress dialog.

Moving, Copying, or Promoting Items Using the Jenkins CLI

The Operations Center Context plugin also adds three commands to the Jenkins CLI:

  • cjp-move-item

  • cjp-copy-item

  • cjp-promote-item

All three of these commands use the same syntax:

java -jar jenkins-cli.jar -s JENKINS_URL cjp-__-item /path/to/source/item uri/of/destination/item

They also support the same command line options:

-q

Submit the request and return immediately.

-w

Wait until the request has started.

-v

Display verbose output of progress.

-e IGNORE_ALL | IGNORE_WARNINGS | IGNORE_NONE

Select the error handling mode.

Except for --skip-builds which is only valid for the cjp-copy-item command. This option will skip the build history from the copy operation (as it does "Copy without builds") in the UI.

The source item is specified by providing the path to the item on the source master. This is a normal path for selecting items using the Jenkins CLI.

The destination folder is specified by providing a URI for the destination. At present there are two supported URI schemes:

  • jenkins://instanceid/path/to/folder.

    Tip
    If you are referencing a path on the same master as the source you can use jenkins://./path/to/folder
  • cjp:///path/from/root/of/cjoc

In general, with all of these paths can be constructed by building the names of each item. Most Jenkins items support using a different display name for the item from the actual name that the item is known as. This can complicate determining the actual path or URI.

The Operations Center Context plugin adds a hidden action to all items: /platform-uri/ that returns the uri of the item.

For example, if you use a web browser and navigate to the destination folder, the browser location will be something like:

https://jenkins.dev.example.com/job/widget/job/blue

By changing the location to:

https://jenkins.dev.example.com/job/widget/job/blue/platform-uri/

We will get back the jenkins:// URI for the "Jenkins » Widgets » Blue" folder.

If we change the location to

https://jenkins.dev.example.com/job/widget/job/blue/platform-uri/?scheme=cjp

We will get back the cjp:/// URI for that folder. On a standalone Jenkins master that would be cjp:///widget/blue but if the master is joined to Operations Center then the path will be relative to the root of Operations Center, for example: cjp://masters/dev/widget/blue.

Note

cjp:// URIs are much easier to work with, but can be broken by moving the client masters within Operations Center.

jenkins:// URIs can be cumbersome to work with but have the advantage that they are not affected by moving the master within Operations Center as the master is looked up using its instance id.

Example: Promoting a Job from one Master to Another

First we need to determine the URI of the destination, in this case we use the curl command:

$ curl http://jenkins.prod.example.com/job/widgets/job/blue/platform-uri/?s\
cheme=cjp
cjp:///masters/production/widgets/blue

$

Now that we have the URI of the destination we can use the CLI command to promote the job configuration:

$ java -jar jenkins-cli.jar -s http://jenkins.dev.example.com/ cjp-promote-\
item /widgets/blue/deploy cjp:///masters/production/widgets/blue
Started masters » dev » widgets » blue » deploy ↑ masters » production » widget
s » blue » deploy
Completed masters » dev » widgets » blue » deploy ↑ masters » production » widg
ets » blue » deploy : SUCCESS

$

We can roll the two steps into one using $(curl …​):

$ java -jar jenkins-cli.jar -s http://jenkins.dev.example.com/ cjp-promote-\
item /widgets/blue/deploy $(curl -L http://jenkins.prod.example.com/job/wid\
gets/job/blue/platform-uri/?scheme=cjp)
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0    0    39    0     0   3296      0 --:--:-- --:--:-- --:--:--  3296
Started masters » dev » widgets » blue » deploy ↑ masters » production » widget
s » blue » deploy
Completed masters » dev » widgets » blue » deploy ↑ masters » production » widg
ets » blue » deploy : SUCCESS

$

We can ask for detailed progress using -v:

$ java -jar jenkins-cli.jar -s http://jenkins.dev.example.com/ cjp-promote-\
item /widgets/blue/deploy cjp:///masters/production/widgets/blue -v
Started masters » dev » widgets » blue » deploy ↑ masters » production » widget
s » blue » deploy
Promotion Starting at: 14/08/15 09:12 triggered by: anonymous
Source: masters » dev » widgets » blue » deploy
Destination: masters » production » widgets » blue » deploy
Error Handling: IGNORE_NONE
        Running pre-flight checks...
        Pre-flight checks: PASS
        Sending files...
        Tray created: DumbWaiterTray{shelf='c033b4a3-1462-4b54-a2f2-10be3c57e30
e', urls=[]}
        Tray is prepared: DumbWaiterTray{shelf='c033b4a3-1462-4b54-a2f2-10be3c5
7e30e', urls=[]}
        Requesting post-validation...
        Post-flight Validation completed.
        Performing Backup
        Backup completed
        Running promotion...
        Promotion completed
        Tidying up...
        Disposing of tray: DumbWaiterTray{shelf='c033b4a3-1462-4b54-a2f2-10be3c
57e30e', urls=[]}

Promotion finished with success at 14/08/15 09:12.
Completed masters » dev » widgets » blue » deploy ↑ masters » production » widg
ets » blue » deploy : SUCCESS

$

We could request that the command returns as soon as the operation starts rather the default of than waiting for the operation to complete using the -w option:

$ java -jar jenkins-cli.jar -s http://jenkins.dev.example.com/ cjp-promote-\
item /widgets/blue/deploy cjp:///masters/production/widgets/blue -w
Started masters » dev » widgets » blue » deploy ↑ masters » production » widget
s » blue » deploy

$

We could ask for the operation to be queued only using the -q option:

$ java -jar jenkins-cli.jar -s http://jenkins.dev.example.com/ cjp-promote-\
item /widgets/blue/deploy cjp:///masters/production/widgets/blue -q

$

We can also control the error handling behaviour. By default the operations will abort if there are any errors or warnings. When transitioning between masters it is not uncommon to find plugin version differences between the masters. Plugin version skew will typically manifest as a WARNING

$ java -jar jenkins-cli.jar -s http://jenkins.dev.example.com/ cjp-promote-\
item /widgets/blue/deploy cjp:///masters/production/widgets/blue -v
Started masters » dev » widgets » blue » deploy ↑ masters » production » widget
s » blue » deploy
Promotion Starting at: 14/08/15 09:29 triggered by: anonymous
Source: masters » dev » widgets » blue » deploy
Destination: masters » production » widgets » blue » deploy
Error Handling: IGNORE_NONE
        Running pre-flight checks...
        Pre-flight checks: WARNING
Aborting
        Tidying up...

Promotion finished with problems at 14/08/15 09:29
Completed masters » production » widgets » blue » deploy ↑ masters » dev » widg
ets » blue » deploy : FAILURE

$

The Jenkins Administrator can inspect the detailed logs of the request using Jenkins  Manage Jenkins  Move/Copy/Promote History on the source Jenkins instance.

detailed history
Figure 13. The detailed logs of a failed promote operation accessed via Jenkins  Manage Jenkins  Move/Copy/Promote History

In this case there are some missing plugins.

Tip

There are some plugins that store side information about a project in a job property.

Typically, these can be identified by examining the job configuration or the config.xml of the job itself.

Note

A job that is deploying things may actually be using the Deployer framework plugin to do the deployment.

While, in general, this specific plugin may inject a mostly unused property into jobs, you need confirmation that this deploy job is not actually using the Deployer framework plugin.

The easiest way to get that confirmation would be to consult with the creator / maintainer of the job. If the creator / maintainer is unavailable or unsure of the answer we can determine the answer by inspecting the configuration of the job.

We take a look at the deploy job’s configuration:

example job config
Figure 14. Example deploy job’s configuration

In this case, the job is using the Copy artifacts from another project and Execute shell build steps. It doesn’t seem likely that this job is actually using the Deployer framework plugin.

We take a look at the deploy job’s raw config.xml just to get final confirmation:

example job raw config
Figure 15. Example deploy job’s config.xml

Looking at the config.xml we can see that the main functionality is actually provided by three steps:

  • The SSH Agent plugin provides a build wrapper that makes the deploy-key credentials available - this is using a credential with an ID that has been explicitly specified so that developement can provide a credential for their server while production can, using the same job configuration, provide the production credentials on the production server.

  • The Copy Artifacts plugin copies the widget from the upstream build job into the workspace.

  • The Execute shell build step uses scp to copy the built widget to the ${TARGET_HOST} - which is presumably an environment variable set on a parent folder so that development can test against a non-production server.

There is a property in the configuration file from the Deployer framework and that is what is causing the warning. The property itself, though, does not seem to contain any significant configuration:

  • The single click deployment functionality has been recorded as disabled.

  • There are no default targets for the the Deploy Now action.

After analysis, we conclude that this job is safe to promote into production even with the reported warnings.

$ java -jar jenkins-cli.jar -s http://jenkins.dev.example.com/ cjp-promote-\
item /widgets/blue/deploy cjp:///masters/production/widgets/blue -v -e IGNO\
RE_WARNINGS
Started masters » dev » widgets » blue » deploy ↑ masters » production » widget
s » blue » deploy
Promotion Starting at: 14/08/15 10:03 triggered by: anonymous
Source: masters » dev » widgets » blue » deploy
Destination: masters » production » widgets » blue » deploy
Error Handling: IGNORE_WARNINGS
        Running pre-flight checks...
        Pre-flight checks: WARNING
        Sending files...
        Tray created: DumbWaiterTray{shelf='ba906833-1e1b-4e85-8f9c-5477f50865d
5', urls=[]}
        Tray is prepared: DumbWaiterTray{shelf='ba906833-1e1b-4e85-8f9c-5477f50
865d5', urls=[]}
        Requesting post-validation...
        Post-flight Validation completed.
        Performing Backup
        Backup completed
        Running promotion...
        Promotion completed
        Tidying up...
        Disposing of tray: DumbWaiterTray{shelf='ba906833-1e1b-4e85-8f9c-5477f5
0865d5', urls=[]}

Promotion finished with success at 14/08/15 10:03.
Completed masters » dev » widgets » blue » deploy ↑ masters » production » widg
ets » blue » deploy : SUCCESS

$

Resolving Common Issues

The following issues may be encountered:

Moving templates

If you move a template from one master to another master, jobs on the source master that were based on the template become disconnected from the template, just as if you had deleted the template on a standalone Jenkins master.

Moving items based on templates

If you move a job based on a template from one master to another, the job on the destination master will become disconnected from the template, just as if you had deleted the template on a standalone Jenkins master.

Plugin version skew

The most common warning is from plugin version skew, i.e. where the destination master has an older version of a plugin used on the source master.

Typically such skew will not be a major issue as the general principle under which plugins are developed is that you should be able to roll back a plugin upgrade without loosing configuration.

If a plugin maintainers has to break that contract, they are expected to set a flag in the plugin’s manifest that triggers a warning in the update center, e.g.

update center warning
Figure 16. Warning about a plugin upgrade where the plugin’s configuration may not be retained if the upgrade is rolled back.

The Deployer framework plugin is not installed on the receiving Jenkins

The Deployer framework plugin can often show up as a missing plugin.

For most jobs, the Deployer framework is storing a cache of whether the job has any archived artifacts that can be auto detected as being supported for deployment with any of the installed deployment engines. The cache is required in order to ensure that the views render quickly as the Deploy Now button should only be rendered for jobs that have at least one artifact that can actually be deployed.

deploy now view
Figure 17. A view with two jobs, the most recent build of the build job has archived artifacts that can be deployed now and as a result the Deploy Now button is visible beside the Build Now button. The deploy job does not have any archived artifacts and therefore there is nothing to for the Deployer framework plugin to deploy from that job.
Note

The best way to know if a job is actually using the Deployer framework plugin is to ask the people using the job.

If all of them do not know, then it is probably safe to conclude that they are not using it.

If you have any remaining doubt, to confirm if a job is actually using the Deployer framework you need to check three things:

  1. There are no Deploy applications build steps

    deploy apps build step
    Figure 18. A Deploy applications build step.
  2. There is no Deploy applications step in the Post-build Actions

    deploy apps publisher
    Figure 19. A Deploy applications post-build action.
  3. The Deploy Now defaults do not contain any significant configuration. The most reliable indicator of significant configuration is if the One-Click Deployment option has been enabled, as that is normally only selected when the defaults have been configured with a valid configuration.

    If the One-Click Deployment option has not been selected, check if the Advanced button is indicating that there is edited configuration within (i.e. the notepad icon to the left of the button)

    deploy now edited
    Figure 20. An Advanced button where the values differ from the default configuration.

    Different deployer engines will inject a sample starting configuration, so often it will be necessary to expand the Advanced button.

    deploy now expanded
    Tip
    If none of the Host service entries contain an Application then the Deploy Now functionality has not been configured.

    The deployer engines may auto-detect applications, for example the AWS deployer engine knows that WAR files are supported for deployment onto Elastic Beanstalk. In order to simplify the use of the Deploy Now action, it will configure an empty application if there is a WAR archived artifact, e.g.

    deploy now aws autodetect
    Figure 21. An auto-detected application entry

    In such cases the lack of customization of the application configuration, example no Application Name or Version specified, the default hint of BucketName/ in the S3 Bucket, etc., all point to this configuration either not being significant or being trivial to reconstruct.

Pipelines

Because a large part of the definition of a Pipeline can be stored in source control, it may not be possible for Jenkins to determine the full list of plugins that are used by the actual Pipeline definition.

This is typically not a serious issue as any of the explicit Jenkins configuration in the Pipeline job (i.e. the parts of the configuration that would be discarded if saved on a Jenkins instance missing a required plugin) will be covered by the validation checks so worst case the Pipeline job can simply be moved to a master that has the plugins required by the Pipeline script.