support dynamic templates for pipelines

[j3t]

This PR adds the possibility to define job templates dynamically in scripted pipelines. The functionality is similar to the pipeline integration from the Kubernetes plugin mentioned in #81.

Note: For now, only scripted pipelines are supported. See README.md for more details.

[j3t]

With #107 it is now possible to define dynamic templates written in HCL as well. I have tested it and it is working as expected. The only thing I don't have tested yet is when the jobTemplate is provided as file and then referenced via readFile/readTrusted.

The other thing we might have to take care of is support for declarative pipelines, but I'm not sure if we have to support it in the first place.

[j3t]

Jenkins remote agents have two options to establish a connection.

1. JNLP
2. WEBSOCKETS

For both, the JEKNINS_URL is required and for JNLP the JENKINS_TUNNEL is required as well. While the JEKNINS_URL is probably a static value like https://jenkins.service.consul, the JENKINS_TUNNEL is most likely a dynamic value which changes from time to time (e.g. Nomad decides to reschedule Jenkins). Because of that, it makes a lot of sense to introduce a new placeholder %JENKINS_TUNNEL%. This way, pipeline scripts doesn't have to change when the JENKINS_TUNNEL has changed.

What do you think? Maybe it makes sense to introduce %JENKINS_URL% as placeholder as well (just for completion).

[j3t]

the JENKINS_TUNNEL is most likely a dynamic value which changes from time to time (e.g. Nomad decides to reschedule Jenkins)

When this happens then in our case the environment variable NOMAD_HOST_ADDR_jnlp will contain the new value. So in our case it would also be suitable when the jobTemplate evaluates environment variables.

[multani]

Hi @j3t, thanks for the proposition!

I'm not sure if it's a good idea to go that way:

• From a UX perspective, this exposes all the internals of Jenkins, Nomad and how the plugin works to job writers. I really don't see developers writing this in their jobs.
• From a security perspective, this allows anybody who could write a Jenkins pipeline to schedule any kind of job in Nomad using the Jenkins ACL token. As a former Jenkins & Nomad operator, this is really bad: not only the current version can starve Nomad by running unbounded number of jobs, this version would also probably allow to schedule services, system jobs, etc.

I'm not sure exactly what's the actual need here, but AFAIC this looks way too low-level/dangerous to be exposed without a clear benefit.

[j3t]

From a UX perspective, this exposes all the internals of Jenkins, Nomad and how the plugin works to job writers. I really don't see developers writing this in their jobs.

Yes, but on the other hand this what you want. The predefined templates are not suitable.

From a security perspective, this allows anybody who could write a Jenkins pipeline to schedule any kind of job in Nomad using the Jenkins ACL token.

We could add a checkbox allow custom templates to the cloud configuration (default: unchecked) with a hint.

[arsiesys]

Hello there!

There is an example of use-case that we have and seems to match what this PR have to offer:

We have a large farm of build machine.
Each of them build some content that need to be "pre-synchronize" data (AKA, a branch) before being use in "production" continuous build (this step can take up to 8h so we don't want to do all the farm in one shot).
We need to have a flag/information to know when a machine have the synchronized content or not to:
Target machine that have the content ready (flagged using dynamic metadata feature of nomad)
Target machine that do not have the content ready yet to launch a job to synchronize it then set the dynamic metadata "flag"

To avoid the need to manually having to update the jenkins configuration when a new branch need to be synchronized (jenkins as code). The dynamic templates could solve this for us as we would just set a "parameter" for the constraints on the metadata flag to know when a machine is ready or not.

We tried to build the PR with the version 0.10 and sadly, it's not compatible anymore :p (we can build but jenkins reboot loop).

[multani]

@j3t I'm no longer working on this project, feel free to discard my past comments and move this forward if you are still interested! 👍