Next LTS release

More information about the release process is available on the release guide.

Release Lead

@TheMeinerLP

Prep work

• LTS baseline discussed and selected in the Jenkins developers mailing list.
  If the last release of the preceding LTS line is a security release, consider making the matching weekly release the new LTS baseline.
  For example, 2.462.3 LTS and 2.479 were security releases and it is simpler to use 2.479 as baseline than 2.477.

• Create or update release branch in jenkinsci/jenkins, e.g. stable-2.387, use the init-lts-line script or carry out the equivalent steps therein.

• Create or update release branch in jenkins-infra/release, e.g. stable-2.387. Strike out for initial release.

  • Modify the RELEASE_GIT_BRANCH and JENKINS_VERSION values in the environment file (profile.d/stable) to match the release.
  • Modify the PACKAGING_GIT_BRANCH value in the packaging script (Jenkinsfile.d/core/package) to match the release.
  • For more info, refer to stable.

• Create or update release branch in jenkinsci/packaging, e.g. stable-2.387. Strike out for new point release.

• Create a pull request to update bom to the weekly version that will be the base of the release line (and strike this out for new point release).
  Assure that the bom-weekly version number is already testing the base of the release line or a version newer than the base of the release line.

• Review recent security advisories for fixes in Jenkins weeklies after the LTS baseline, and ensure there are Jira issues for their backport.

• Review Jira and GitHub pull requests for additional LTS candidates, adding the lts-candidate label, and ensure that all tickets are resolved in Jira.

• Send a backporting announcement email to the jenkinsci-dev mailing list, using the default template.
  Remember to exchange the LTS version, release date and Jira URLs.

• Update Jira labels for lts-candidate issues, either add 2.387.2-fixed and remove lts-candidate or add 2.387.2-rejected, and retain lts-candidate.

• Backport changes, run the list-issue-commits script to locate commits via Jira ID, some manual work is required to locate them if the issue ID wasn't present at merge time, backport with git cherry-pick -x $commit.

• Open backporting PR with into-lts label and summary of changes in description from lts-candidate-stats script and:

  • the selected Jira lts-candidates.
  • possible LTS candidates in the release repository.
  • possible LTS candidates in the packaging repository.

• Open a pull request towards the acceptance test harness and plugin compatibility test to confirm the incremental produced by the backporting PR doesn't contain regressions.
  The documentation explains which profiles you have to modify in your PR.

  • 🔴 test(lts): test against rc build 2.504.3 jenkinsci/acceptance-test-harness#2048
  • 🔴 test(lts): test against rc build 2.504.3 jenkinsci/bom#5188

• Prepare LTS changelog based on the style guide using the changelog generator - This is normally done by the docs team, ask in gitter.

• Prepare LTS upgrade guide based on previous upgrade guides - This is normally done by the docs team, ask in gitter.

RC creation

• Merge backporting PR in jenkinci/jenkins using a merge commit (and do not squash).

• Retrieve the URL for the RC from the commit status (Jenkins Incrementals Publisher / Incrementals) of the last build on the stable branch (requires a passing build). Visit the jenkins-war URL and copy the URL of the war file, which would be something like https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/jenkins-war/2.387.1-rc32701.b_06d9cef554c/jenkins-war-2.387.1-rc32701.b_06d9cef554c.war. If the incrementals are broken you can deploy a build from your own machine with mvn -e clean deploy -DskipTests=true.

• Publish a pre-release Github release, e.g. sample currently we don't have a changelog for RCs.

• Confirm the automatic announcement has been sent to the jenkinsci-dev mailing list and community forums. If the automatic announcement is not sent, compose and send the announcement yourself.

• Check with security team that no security update is planned. If a security update is planned, revise the checklist after the public pre-announcement to the jenkinsci-advisories mailing list.

• For a new LTS baseline's ".1" release, if there were recent security advisories for fixes in Jenkins weeklies after the LTS baseline that had to be backported:

  • Update those advisories to mention the new 2.xxx.1 LTS release as an additional fix version (example)
  • Update warnings metadata to exclude the ".1" release (example)
  • Inform the Jenkins security team about the need to update CVE metadata to exclude the new LTS line from affected version ranges.

LTS release

• Publish changelog (one day prior to the release in case of a security update).

• Announce the start of the LTS release process in the #jenkins-release:matrix.org channel.

• Launch job on release.ci.jenkins.io if no security release for Jenkins is planned.

  • Manually review and approve the child release job after carefully checking the "Plan" stage (you can compare with previous stable line).
  • If this is the first release of a new LTS line, the packaging job will fail on its first run. Either run the packaging job once and cancel it before the primary release job is run or accept that the packaging job on the first release of a new LTS line will need to be run a second time after it fails the initial run.
  • ~3 to 4 hours after the beginning of release job, manually review and approve the child packaging job.

• Wait for successful job completion (release: ~3 to 4 hours, packaging ~30 minutes).

• Check LTS changelog is visible on the downloads site.

• Publish GitHub release pointing to LTS changelog, sample.

• Confirm that all Packages are available on the Datadog page.

• Confirm the Debian installer acceptance test is passing.
  For good measures, check the console log to confirm that the correct release package was used (e.g. search for 2.387. If not, launch tests again).

• Confirm the Red Hat installer acceptance test is passing.
  For good measures, check the console log to confirm that the correct release package was used (e.g. search for 2.387. If not, launch tests again).

• Adjust state and Released As of Jira issues fixed in the release (see the changelog for issue links).

• Create pull request to update the lts Maven profile in ATH to the newly released version

• Create pull request to update the jenkins.version in the most recent release profile in plugin BOM to the newly released version.
  Refer to first step before the release and second step after the release for examples

• Create a tag matching the LTS release you create in the docker repository and publish a GitHub release.

• Confirm that the images are available at Docker hub.

• Merge the PR generated by the jenkins-dependency-updater bot in the jenkinsci/helm-charts repository.

• Create a helpdesk ticket to update ci.jenkins.io, trusted.ci, cert.ci and release.ci to the new LTS release, example.

• Send email asking for the next release lead, example, dates for the next one can be found on the Jenkins calendar.