SSL Handshake failure with mutual TLS auth

[seadub]

Version being used - 1.0.8
Latest version - 1.0.10

The implementation on both the version I am using and latest versions does not seem to provide configuration options for mutual TLS authentication for the plugin and the target server.

A recent dive into attempting communication with internal services led me to complete the following steps:

• Adding certificate presented by service to rundeck's truststore, as defined in rundeck ssl.properties
• Adding a certificate trusted by the service into rundeck's keystore, as defined in rundeck ssl.properties
• Define new job using HTTP Request step, referencing https://<service-url>

The following error was presented in the debug output:

pool-12-thread-1, Exception while waiting for close javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
pool-12-thread-1, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
Error when sending request: Received fatal alert: bad_certificate
Failed executing step plugin [edu.ohio.ais.rundeck.HttpWorkflowStepPlugin]: com.dtolabs.rundeck.core.execution.workflow.steps.StepException: Error when sending request: Received fatal alert: bad_certificate
	at edu.ohio.ais.rundeck.HttpWorkflowStepPlugin.doRequest(HttpWorkflowStepPlugin.java:382)
	at edu.ohio.ais.rundeck.HttpWorkflowStepPlugin.executeStep(HttpWorkflowStepPlugin.java:576)
	at com.dtolabs.rundeck.core.execution.workflow.steps.StepPluginAdapter.executeWorkflowStep(StepPluginAdapter.java:117)
	at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:111)
	at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWFItem(BaseWorkflowExecutor.java:287)
	at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWorkflowStep(BaseWorkflowExecutor.java:683)
	at com.dtolabs.rundeck.core.execution.workflow.engine.StepCallable.apply(StepCallable.java:71)
	at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:73)
	at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:31)
	at com.dtolabs.rundeck.core.rules.WorkflowEngineOperationsProcessor.lambda$processRunnableOperations$1(WorkflowEngineOperationsProcessor.java:222)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
	at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
	at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1779)
	at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:124)
	at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1156)
	at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1266)
	at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1178)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:348)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at edu.ohio.ais.rundeck.HttpWorkflowStepPlugin.doRequest(HttpWorkflowStepPlugin.java:281)
	... 15 more

1: Workflow step finished, result: HTTPFailure: Error when sending request: Received fatal alert: bad_certificate
[workflow] Finish step: 1,edu.ohio.ais.rundeck.HttpWorkflowStepPlugin

With ssl.debug enabled on the JVM, I was able to see the following output during the handshake:

*** ServerHelloDone
Warning: no suitable certificate found - continuing without client authentication
*** Certificate chain
<Empty>
***

While reviewing the code for HttpBuilder in 1.0.10 and HttpWorkflowStepPlugin in 1.0.8, I can see that the capability for setting up a proxy and the disabling of SSL verification are the only options; nothing about defining a truststore or whether to use the org.apache.http.impl.client.HttpClientBuilder#useSystemProperties method within Apache's HttpClientBuilder class to enable possible attempts to load the JVM's configured keystore/truststore.

Expectation: allow plugin to use server's keystore/truststores for mutual TLS authentication, or to have specified which keystores/truststores are to be used.

[kaplanben]

still relevant for us as well