Add an option to wait for custom startup script to finish before launching the agent by caseyduquettesc · Pull Request #346 · jenkinsci/google-compute-engine-plugin

caseyduquettesc · 2022-07-19T05:54:14Z

Fixes #338

Adds a new option to the instance configuration that enables waiting to launch the agent until google-startup-scripts has finished running.

Agent log - launcher checks startup status and it's already done

Jul 19, 2022 9:48:14 PM null
FINEST: Instance vl843h is running and ready...
Jul 19, 2022 9:48:14 PM null
INFO: Launching instance: vl843h
Jul 19, 2022 9:48:14 PM null
INFO: bootstrap
Jul 19, 2022 9:48:14 PM null
INFO: Getting keypair...
Jul 19, 2022 9:48:14 PM null
INFO: Using autogenerated keypair
Jul 19, 2022 9:48:14 PM null
INFO: Authenticating as jenkins
Jul 19, 2022 9:48:14 PM null
INFO: Connecting to 10.240.0.130 on port 22, with timeout 10000.
Jul 19, 2022 9:48:24 PM null
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
Jul 19, 2022 9:48:24 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 9:48:29 PM null
INFO: Connecting to 10.240.0.130 on port 22, with timeout 10000.
Jul 19, 2022 9:48:37 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.0.130:22
Jul 19, 2022 9:48:37 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 9:48:42 PM null
INFO: Connecting to 10.240.0.130 on port 22, with timeout 10000.
Jul 19, 2022 9:48:42 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.0.130:22
Jul 19, 2022 9:48:42 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 9:48:47 PM null
INFO: Connecting to 10.240.0.130 on port 22, with timeout 10000.
Jul 19, 2022 9:48:47 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.0.130:22
Jul 19, 2022 9:48:47 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 9:48:52 PM null
INFO: Connecting to 10.240.0.130 on port 22, with timeout 10000.
Jul 19, 2022 9:48:52 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.0.130:22
Jul 19, 2022 9:48:52 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 9:48:57 PM null
INFO: Connecting to 10.240.0.130 on port 22, with timeout 10000.
Jul 19, 2022 9:48:57 PM null
INFO: Connected via SSH.
Jul 19, 2022 9:48:57 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 9:48:58 PM null
INFO: Configured startup script is finished.
Jul 19, 2022 9:48:58 PM null
INFO: Verifying: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -fullversion
openjdk full version "11.0.7+10-post-Ubuntu-2ubuntu218.04"
Jul 19, 2022 9:48:58 PM null
INFO: Copying agent.jar to: /var/lib/jenkins/workspace
Jul 19, 2022 9:48:58 PM null
INFO: Launching Jenkins agent via plugin SSH: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -jar /var/lib/jenkins/workspace/agent.jar
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 4.10.1
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

Agent log - launcher checks startup status and it's sleeping for 60s

Jul 19, 2022 10:46:51 PM null
FINEST: Instance 3pi9fj is running and ready...
Jul 19, 2022 10:46:51 PM null
INFO: Launching instance: 3pi9fj
Jul 19, 2022 10:46:51 PM null
INFO: bootstrap
Jul 19, 2022 10:46:51 PM null
INFO: Getting keypair...
Jul 19, 2022 10:46:51 PM null
INFO: Using autogenerated keypair
Jul 19, 2022 10:46:51 PM null
INFO: Authenticating as jenkins
Jul 19, 2022 10:46:51 PM null
INFO: Connecting to 10.240.1.147 on port 22, with timeout 10000.
Jul 19, 2022 10:47:01 PM null
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
Jul 19, 2022 10:47:01 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:47:06 PM null
INFO: Connecting to 10.240.1.147 on port 22, with timeout 10000.
Jul 19, 2022 10:47:07 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.1.147:22
Jul 19, 2022 10:47:07 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:47:12 PM null
INFO: Connecting to 10.240.1.147 on port 22, with timeout 10000.
Jul 19, 2022 10:47:12 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.1.147:22
Jul 19, 2022 10:47:12 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:47:17 PM null
INFO: Connecting to 10.240.1.147 on port 22, with timeout 10000.
Jul 19, 2022 10:47:17 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.1.147:22
Jul 19, 2022 10:47:17 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:47:22 PM null
INFO: Connecting to 10.240.1.147 on port 22, with timeout 10000.
Jul 19, 2022 10:47:22 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.1.147:22
Jul 19, 2022 10:47:22 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:47:27 PM null
INFO: Connecting to 10.240.1.147 on port 22, with timeout 10000.
Jul 19, 2022 10:47:28 PM null
INFO: Connected via SSH.
Jul 19, 2022 10:47:28 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:47:28 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:47:28 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:47:33 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:47:33 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:47:33 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:47:38 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:47:38 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:47:38 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:47:43 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:47:43 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:47:43 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:47:48 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:47:48 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:47:48 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:47:53 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:47:53 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:47:53 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:47:58 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:47:58 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:47:58 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:03 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:03 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:48:03 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:08 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:08 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:48:08 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:13 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:13 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:48:13 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:18 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:18 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:48:18 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:23 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:23 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:48:23 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:28 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:28 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:48:28 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:33 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:33 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:48:33 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:38 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:38 PM null
WARNING: google-startup-scripts has not finished
Jul 19, 2022 10:48:38 PM null
INFO: Waiting for the startup script to finish. Sleeping 5.
Jul 19, 2022 10:48:43 PM null
INFO: Verifying: exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d "=" -f 2)
Jul 19, 2022 10:48:43 PM null
INFO: Configured startup script is finished.
Jul 19, 2022 10:48:43 PM null
INFO: Verifying: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -fullversion
openjdk full version "11.0.7+10-post-Ubuntu-2ubuntu218.04"
Jul 19, 2022 10:48:43 PM null
INFO: Copying agent.jar to: /var/lib/jenkins/workspace
Jul 19, 2022 10:48:43 PM null
INFO: Launching Jenkins agent via plugin SSH: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -jar /var/lib/jenkins/workspace/agent.jar
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 4.10.1
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

Agent log - option disabled and launcher does not check startup status

Jul 19, 2022 10:54:22 PM null
FINEST: Instance o8pom3 is running and ready...
Jul 19, 2022 10:54:22 PM null
INFO: Launching instance: o8pom3
Jul 19, 2022 10:54:22 PM null
INFO: bootstrap
Jul 19, 2022 10:54:22 PM null
INFO: Getting keypair...
Jul 19, 2022 10:54:22 PM null
INFO: Using autogenerated keypair
Jul 19, 2022 10:54:22 PM null
INFO: Authenticating as jenkins
Jul 19, 2022 10:54:22 PM null
INFO: Connecting to 10.240.0.11 on port 22, with timeout 10000.
Jul 19, 2022 10:54:32 PM null
INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
Jul 19, 2022 10:54:32 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:54:37 PM null
INFO: Connecting to 10.240.0.11 on port 22, with timeout 10000.
Jul 19, 2022 10:54:38 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.0.11:22
Jul 19, 2022 10:54:38 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:54:43 PM null
INFO: Connecting to 10.240.0.11 on port 22, with timeout 10000.
Jul 19, 2022 10:54:43 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.0.11:22
Jul 19, 2022 10:54:43 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:54:48 PM null
INFO: Connecting to 10.240.0.11 on port 22, with timeout 10000.
Jul 19, 2022 10:54:48 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.0.11:22
Jul 19, 2022 10:54:48 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:54:53 PM null
INFO: Connecting to 10.240.0.11 on port 22, with timeout 10000.
Jul 19, 2022 10:54:54 PM null
INFO: Failed to connect via ssh: There was a problem while connecting to 10.240.0.11:22
Jul 19, 2022 10:54:54 PM null
INFO: Waiting for SSH to come up. Sleeping 5.
Jul 19, 2022 10:54:59 PM null
INFO: Connecting to 10.240.0.11 on port 22, with timeout 10000.
Jul 19, 2022 10:54:59 PM null
INFO: Connected via SSH.
Jul 19, 2022 10:54:59 PM null
INFO: Verifying: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -fullversion
openjdk full version "11.0.7+10-post-Ubuntu-2ubuntu218.04"
Jul 19, 2022 10:54:59 PM null
INFO: Copying agent.jar to: /var/lib/jenkins/workspace
Jul 19, 2022 10:54:59 PM null
INFO: Launching Jenkins agent via plugin SSH: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -jar /var/lib/jenkins/workspace/agent.jar
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 4.10.1
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests - that demonstrates feature works or fixes the issue

caseyduquettesc · 2022-07-20T05:59:53Z

src/main/java/com/google/jenkins/plugins/computeengine/ComputeEngineLinuxLauncher.java

+          // default value will be 0 until it finishes
+          "exit $(systemctl show google-startup-scripts --property ExecMainExitTimestampMonotonic | cut -d \"=\" -f 2)",
+          // not "initializing" or "starting" - benefit is doesn't require google-startup-scripts
+          // "[ \"$(systemctl is-system-running)\" != \"starting\" ]",


Do we think it's better to check that google-startup-scripts has finished or that systemctl is-system-running does not return "initializing" or "starting"? Those seem to be the two best ways on my ubuntu 18/20 machines, but curious what others think here.

Hi @caseyduquettesc thank you. I think it's just fine to check google-startup-scripts has finished.

When can this be merged? I was hoping I could use it pretty soon.

Will need a project maintainer to review. I'll try asking in the slack channel

caseyduquettesc · 2022-08-21T20:20:29Z

Sorry for the direct ping, but @donmccasland or @alecharp do you know who would be best to review this? Thanks

TheSushiChef · 2023-09-08T01:30:40Z

Is this a feature coming any time soon? We'd really love to have this feature as we currently hit a race condition where the pipeline starts building the agent before the startup script finishes logging in and breaks on first execution.

ogonzalez-sd · 2023-09-14T17:43:30Z

This would be great. Our current workaround is to have the script stop sshd, then start it again at the end.

Thank you!

MTSchnepper · 2024-03-28T01:57:59Z

Any chance we could get codeowners to take a look at this PR? This feature would solve a lot of problems we're facing with setting up Jenkins on GCE

caseyduquettesc commented Jul 20, 2022

View reviewed changes

caseyduquettesc marked this pull request as ready for review July 20, 2022 06:11

caseyduquettesc added 12 commits August 12, 2022 16:15

add testCommand method with configurable successful exit codes

63952f9

add method to linux launcher to check if startup script finished

f51e31a

checkStartupScriptFinished

723b67f

hook up waitForStartupScriptComplete into launcher

3659028

windows launcher stub

5b6e3f0

mvn test reformatted these?

bb81f6b

hook up ui checkbox to instance configuration

1032301

tests

9a89204

fix bug

16344be

fixes

5065dc0

cleanup

5264af8

cleanup

a9137c6

caseyduquettesc force-pushed the casey/wait-for-startup branch from 72f47b5 to a9137c6 Compare August 12, 2022 23:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add an option to wait for custom startup script to finish before launching the agent#346

Add an option to wait for custom startup script to finish before launching the agent#346
caseyduquettesc wants to merge 12 commits intojenkinsci:developfrom
caseyduquettesc:casey/wait-for-startup

caseyduquettesc commented Jul 19, 2022 •

edited

Loading

Uh oh!

caseyduquettesc Jul 20, 2022

Uh oh!

bini-dere Jul 27, 2022

Uh oh!

caseyduquettesc Jul 27, 2022

Uh oh!

caseyduquettesc commented Aug 21, 2022

Uh oh!

TheSushiChef commented Sep 8, 2023

Uh oh!

ogonzalez-sd commented Sep 14, 2023

Uh oh!

MTSchnepper commented Mar 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

caseyduquettesc commented Jul 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

caseyduquettesc Jul 20, 2022

Choose a reason for hiding this comment

Uh oh!

bini-dere Jul 27, 2022

Choose a reason for hiding this comment

Uh oh!

caseyduquettesc Jul 27, 2022

Choose a reason for hiding this comment

Uh oh!

caseyduquettesc commented Aug 21, 2022

Uh oh!

TheSushiChef commented Sep 8, 2023

Uh oh!

ogonzalez-sd commented Sep 14, 2023

Uh oh!

MTSchnepper commented Mar 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

caseyduquettesc commented Jul 19, 2022 •

edited

Loading