Cloud Native SIG - Pluggable Storage

Below you can find a summary of ongoing activities and their current status:

The list above is not complete. Other storage types may be considered according to the feedback. You can find more information about Pluggable Storage and priorities in this blogpost.

There are many existing plugins allowing to upload and download artifacts from external storage (e.g. S3, Artifactory, Publish over SFTP, etc., etc.), but there are no plugins which can do it transparently without using new steps. In many cases the artifacts also get uploaded through the Jenkins controller, and it increases load on the system. It would be great if there was a layer which would allow storing artifacts externally when using common steps like Archive Artifacts.

Jenkins 2.118+ offers an extended jenkins.util.VirtualFile API which allows implementing external artifact managers using the ArtifactManagerFactory extension point.

Implementation example(s):

Related JEPs:

Logs disk usage causes the same issues previously mentioned for artifacts. Plus a external focused log storage such as AWS CloudWatch Logs allows demanded features as centralized log management, log retention policies, advanced querying, etc. There are already options to externally ship the logs to a backend, or plugins that would do that like the aws-cloudwatch-logs-publisher-plugin, but there is no integrated way to both send and display logs from external log storage. The External log storage work is tracked as issue:38313[].

Reference implementation(s):

Related JEPs:

Although configurations are not big, externalizing them is a critical task for getting highly-available or disposable Jenkins controllers. There are many ways to store configurations in Jenkins, but 95% of cases are covered by the XmlFile layer which serializes objects to disk and reads them using the XStream library. Externalizing these XmlFiles would be a great step forward.

There are several prototypes for externalizing configurations, e.g. in DotCI. There are also other implementations which could be upstreamed to the Jenkins core.

Related JEPs:

In Credentials Plugin 1.15+ there is a CredentialsProvider extension point which allows referencing and resolving external credentials. This engine allows implementing external credentials for plugins implementing Credentials API..

Implementation example(s):

Other credentials API in Jenkins (like jenkinsdoc:hudson.util.Secret) are not supported.

In common CI/CD use-cases a lot of the space is being consumed by test reports. This data is stored within JENKINS_HOME, and the current storage format requires huge overheads when retrieving statistics and, especially, trends. In order to display trends, each report has to be loaded and then processed in-memory.

The main purpose of externalising Test Results is to optimize Jenkins logic by querying the desired data from specialized external storages, e.g. from Document-based databases like Elasticsearch. According to the current plan, JUnit Plugin will be extended in order to support such external storage in its APIs being widely used by test reporting plugins.

Status:

Please try it out, report issues to GitHub and general feedback to GitHub#142.

The fingerprints are stored within JENKINS_HOME inside a local XML-based database. Externalizing fingerprints decreases the dependence of Jenkins on the physical disk storage of the controller, and allows configuring of cloud storages which can be cheaper, and more reliable. Another advantage is that it would allow tracing fingerprints across Jenkins instances and the entire CI/CD flow.

Status:

This page summarizes statuses of the ongoing work only. There are other Pluggable Storage stories we consider in the Cloud Native SIG. See the SIG’s page for more details and links.