Skip to content

Conversation

Vlatombe
Copy link
Member

@Vlatombe Vlatombe commented Apr 17, 2025

Collect timings for each init milestone
If the startup takes more than a defined initial delay, start collecting thread dumps at regular intervals, until the startup completes.

We consider that an instance taking more than 5 minutes is slow to start and is worth collecting thread dumps for (com.cloudbees.jenkins.support.startup.StartupReport.INITIAL_DELAY_SECONDS).
Then we take one thread dump every 30 seconds (com.cloudbees.jenkins.support.startup.StartupReport.RECURRENCE_PERIOD_SECONDS)

The timings are collected in startup-timings.md with the following format (values for a development instance)


Milestone Time
Started initialization No data
Listed all plugins No data
Prepared all plugins 17 sec
Started all plugins 8.2 sec
Augmented all extensions 0.7 sec
System config loaded 1.8 sec
System config adapted 0.16 sec
Loaded all jobs 1.7 sec
Configuration for all jobs updated 0.44 sec

Total startup time : 30 sec


I will also add a similar logic to collect information when the instance terminates and takes more than the expected time.

CloudBees Internal issue

Testing done

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests - that demonstrates feature works or fixes the issue

Collect timings for each init milestone
If the startup takes more than a defined initial delay, start collecting thread dumps at regular intervals.
@Vlatombe Vlatombe requested a review from a team as a code owner April 17, 2025 14:57
@Vlatombe Vlatombe marked this pull request as draft April 17, 2025 15:00
}
var format = new SimpleDateFormat(DATE_FORMAT);
for (int i = 0; i < iterations; i++) {
File threadDumpFile = fileList.file(format.format(new Date(timestamp)) + ".txt");
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suppressed the prefix -ias the timestamp is always different anyway

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(referring to code moved from SlowRequestThreadDumpsGenerator)

@@ -45,11 +48,41 @@
@Extension(ordinal = -100.0) // run this last as it blocks the channel
public class ThreadDumps extends ObjectComponent<Computer> {

private final Logger logger = Logger.getLogger(ThreadDumps.class.getName());
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long due

@Vlatombe Vlatombe changed the title Collect information about instance startup Collect information about instance startup and shutdown Apr 17, 2025
Copy link
Member

@jglick jglick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far.

}
var format = new SimpleDateFormat(DATE_FORMAT);
for (int i = 0; i < iterations; i++) {
File threadDumpFile = fileList.file(format.format(new Date(timestamp)) + ".txt");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(referring to code moved from SlowRequestThreadDumpsGenerator)


@Override
protected void printTo(PrintWriter out, @NonNull ContentFilter filter) {
var startTime = ManagementFactory.getRuntimeMXBean().getStartTime();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beware that this is JVM startup time, so if any servlet listener runs before Jenkins starts (two such examples come to mind in CloudBees CI), you will get a very skewed measurement.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't have any better from core for now...

@Vlatombe Vlatombe marked this pull request as ready for review April 18, 2025 09:02
@jglick jglick enabled auto-merge April 21, 2025 22:17
@jglick jglick merged commit 397d822 into jenkinsci:master Apr 21, 2025
16 of 17 checks passed
@Vlatombe Vlatombe deleted the startup-report branch April 22, 2025 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants