Upgrading to Jenkins LTS 2.303.x

Jenkins agents communicate with the controller using the remoting library. Individual messages (implementations of the Callable interface) instruct the other side to perform operations and/or provide information. To restrict what agents can instruct controllers to do, Callable implementations are expected to implement a role check, i.e. a declaration on which side of the communication channel they can be executed.

To prevent plugins from (unintentionally) not implementing an effective role check, Jenkins will reject Callables with unimplemented role checks starting with 2.319 and LTS 2.303.3. See the Required Role Check page for more details, including a list of known affected plugins and workarounds.

The developer documentation contains guidelines for the safe and correct implementation of the Callable interface.

SECURITY-2455

Jenkins agents communicate with the controller using the remoting library. For compatibility with existing plugins, agents are allowed some limited access to files on the Jenkins controller file system. Jenkins 2.303.3 and 2.319 fix multiple vulnerabilities that resulted in incomplete access control enforcement.

Any plugins that access files on the Jenkins controller from code running on agents may be affected by these security fixes. Beyond previously allowed operations now being prohibited, the newly introduced path canonicalization means that instances using a custom builds directory (Java system property jenkins.model.Jenkins.buildsDir) or partitioning JENKINS_HOME using symbolic links may fail access control checks to files that are intended to be accessible.

If access to a file is rejected, an exception with the following message is made available to agents; this is typically what also will show up in build logs and similar output:

The Jenkins system log will contain more details.

As of 2021-11-04, no plugins are known to be affected by this security fix.

In case of problems, you can temporarily customize the file access rules to allow specific, intended agent-to-controller file path accesses to pass. Applying custom rules is also the recommended workaround when using a custom builds directory or using symbolic links to distribute JENKINS_HOME across multiple volumes.

Jenkins agents communicate with the controller using the remoting library. For compatibility with existing plugins, agents are allowed some limited access to files on the Jenkins controller file system. The directories agents are allowed to access include the directories storing build-related information, intended to allow agents to store build-related metadata during build execution. As a consequence, this allowed any agent to read and write the contents of any build directory on the Jenkins controller file system with very few restrictions (build.xml and some Pipeline-related metadata).

Since Jenkins 2.319 and LTS 2.303.3, agents are only allowed to access contents of build directories related to builds currently running on the agent attempting to access the directory.

Plugins that implement functionality executing on agents that accesses build directories on the Jenkins controller file system may be impacted by this change. Plugins accessing directories of builds not currently running on the agent accessing the directory will need to be adapted to no longer do that. Jenkins will not associate Pipeline node blocks with the agents they’re running on unless Pipeline: Nodes and Processes has been updated to version 2.40 or newer.

If access to a file by an agent is rejected, an exception with the following message is thrown instead; this is typically what also will show up in build logs and similar output:

The Jenkins system log will contain more detailed information.

Additionally, plugins opening a direct channel between the controller and a process, like Maven Integration Plugin, are completely prohibited from accessing build directories.

If access to a file by a process is rejected, an exception with the following message is thrown instead; this is typically what also will show up in build logs and similar output:

The Jenkins system log will contain more detailed information.

As of 2021-11-08, no plugins are known to be affected by this security fix.

This security fix can be disabled by setting the Java system property jenkins.security.s2m.RunningBuildFilePathFilter.SKIP to true.

The short description of build causes was defined as a short snippet of HTML in Jenkins 2.314 and earlier, LTS 2.303.1 and earlier. It is exposed as the shortDescription field of build causes in the Remote Access API of Jenkins, and rendered on the UI on the overview page of builds, unless a build cause provides a custom UI.

To prevent further security vulnerabilities like SECURITY-2499 from having an impact on Jenkins users, the method has been redefined to return plain text in Jenkins 2.315 and LTS 2.303.2, and its output is no longer rendered as HTML on the UI.

Users of the remote API who are currently rendering this field as HTML on their custom dashboards or similar API clients are advised to no longer do so. With the redefinition, any content that could be interpreted as HTML would be unsafe to render that way, as plugins may no longer escape it.

Advice for maintainers has been added to the developer documentation.

The Jenkins Docker images with tags that do not specify a Java version now use Java 11 instead of Java 8. Jenkins controller images with the following tags will use Java 11 beginning with 2.303.1:

Additional controller images and tags have been added for those users who need to remain on Java 8. The new Java 8 tags include:

Jenkins agents should be run with the same Java version as the controller. With the controller upgrade from Java 8 to Java 11, Jenkins agents need to be updated to run with Java 11. When using Docker images for the agent, that can generally be done by appending the -jdk11 suffix to the agent tag. Cloud agents and static agents will need to be configured to launch with Java 11 instead of Java 8.

Jenkins Docker images for new releases are not published for centos. The centos images use CentOS 8 as a baseline and CentOS 8 is no longer receiving updates from the Red Hat Enterprise Linux upstream.

Jenkins Docker Windows images for new releases are not published for the 1809 images. Microsoft has ended mainstream support for 1809.

Jenkins Docker tags that include a version number are no longer published with the redundant -lts tag. The transition looks like this:

Jenkins Docker images now support multiple architectures, including amd64 and arm64 from selected images. The images with support for multiple architectures include:

Some Jenkins plugins do not support Java 11. Plugins that do not support Java 11 also tend to be unmaintained. Users should remove those plugins before running Jenkins with Java 11.

The ruby runtime plugin is not supported with Java 11. Plugins that depend on the ruby runtime plugin do not load in Jenkins running with Java 11.

Plugins that depend on the ruby runtime include:

Ruby runtime plugins with more than 1000 installations:

Ruby runtime plugins with less than 700 installations:

Ruby runtime plugins with less than 100 installations:

The Jenkins 2.303.1 RPM package has been made more consistent with the Debian Jenkins packaging. It uses the daemonize program to run the Java process of the Jenkins controller as a daemon. The daemonize program is available from the epel-release repository for most RPM based distributions.

Some distributions based on the RPM package manager do not enable the 'epel-release' repository by default. Instructions to enable the epel-release repository are included in the Jenkins installation guide instructions for Red Hat Enterprise Linux / CentOS.

Before updating to Jenkins 2.303.1, run the command:

Docker controller images based on Debian, including jenkins/jenkins:lts, jenkins/jenkins:lts-slim, jenkins/jenkins:2.303.1, and jenkins/jenkins:2.303.1-slim are now based on Debian bullseye, the most recent Debian release. Debian bullseye was released August 14, 2021 after a long period of stabilization and testing.

Some packages that had been included in the Debian buster images may not be installed in these images. If your installation needs one of those packages, you’ll need to install it from inside your custom Dockerfile.

The Apache Commons Digester v2.1 has been removed from Jenkins core. The library was last released in 2010. Plugins that depend on the library need to be updated to remove that dependency.

Plugins that are affected include:

Suspended plugins (no longer distributed by the Jenkins update center) that are affected include:

Certain key Jenkins capabilities were initially created inside the Jenkins core. As Jenkins has developed further, capabilities moved from inside Jenkins core to dedicated Jenkins plugins, like the External Monitor Job plugin, the LDAP plugin, and the PAM authentication plugin. When those plugins were created, the plugins were "bundled" inside the jenkins.war file to retain compatibility for plugins that depended on their functionality to be inside Jenkins core. The External Monitor Job plugin, the LDAP plugin, and the PAM authentication plugin are no longer bundled with Jenkins.

In very rare cases, this could result in problems when attempting to install plugins compatible with Jenkins before 1.467. If you use a plugin that relies on the functionality provided by the External Monitor Job plugin, the LDAP plugin, or the PAM authentication plugin and manage plugins outside the Jenkins plugin manager, you will now need to ensure yourself that a recent release of those plugins are installed. Jenkins will attempt to load such plugins but may fail at any time during startup or afterwards with ClassNotFoundException or similar. In these cases, the best path forward is usually to modify the plugin that has the problem so that it requires a newer version of Jenkins core.

Workarounds that assured older plugin compatibility with the security hardening improvements of JEP-200 have been removed. If you are using any of the following plugins, make sure you use at least the versions specified:

The jna-posix dependency has been removed from Jenkins core. Plugins that use jna-posix functionality, including Maven Repository Scheduled Cleanup, SICCI for Xcode, and java.io.tmpdir cleaner are no longer supported and must be removed.

The JTidy dependency has been removed from Jenkins core. Users of the JDepend plugin must upgrade to the latest version. Other plugins that use JTidy functionality, including NIS notification lamp, must be updated to explicitly declare a dependency on JTidy rather than relying on Jenkins core to provide this library.

Support for plugins that rely on the hudson.model.Queue$Item#id or hudson.model.AbstractProject#triggers fields, including Slave Prerequisites plugin and vertx plugin, has been dropped. Any such plugins must be removed prior to upgrading Jenkins. If you have customized the hudson.ClassicPluginStrategy.noBytecodeTransformer system property, you should remove this customization.