Jersey conflicts

[sverhoeven]

In branch https://github.com/NLeSC/xenon-cli/tree/prep-xenon230 I want to use the Xenon v2.3.0, but the integration tests are failing with

java.lang.NoSuchMethodError: javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;)V
	at org.glassfish.jersey.client.ClientRequest.accept(ClientRequest.java:327)
	at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:229)
	at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:59)
	at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:1249)
	at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:1224)
	at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:1218)
	at com.github.geowarin.junit.DockerRule.<init>(DockerRule.java:60)
	at com.github.geowarin.junit.DockerRuleBuilder.build(DockerRuleBuilder.java:87)
	at nl.esciencecenter.xenon.cli.SftpTest.<clinit>(SftpTest.java:31)
	at sun.misc.Unsafe.ensureClassInitialized(Native Method)
	at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
	at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
	at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
	at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
	at java.lang.reflect.Field.get(Field.java:393)
	at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
	at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
	at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
	at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
	at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:108)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:146)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:128)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:748)

The new dependencies of Xenon are conflicting the docker rule used in the integration tests.

[sverhoeven]

In https://github.com/spotify/docker-client/blob/master/pom.xml it depends on jersey 2.22.2.
In ./gradlew dependencies jersey 1.19 is a dependency of several hadoop libraries.

The MultivaluedMap.addAll was introduced in JAX-RS 2 which jersey2 implements.

[jmaassen]

Any suggestions on fixing this ? Maybe hadoop also works with the 2.22.2 version of jersey ?

[sverhoeven]

I think one of Hadoop dependencies pulls in the JAX-RS 1 version of MultivaluedMap.
I could exclude that dependency, but this will probably break hadoop.

[sverhoeven]

We might be able to shade the javax/ws/rs/* classes in jersey-core-1.9.jar using http://imperceptiblethoughts.com/shadow/#relocating_packages so hadoop uses the shaded javax/ws/rs/* and spotify uses the ones that ship with java.

@jmaassen do you any experience shading?

[jmaassen]

No experience with shading....

What we used to do in an earlier incarnation of Xenon is use separate class loaders for the different adaptors.

Since I had similar issues with the GridFTP and SSH adaptor clashing over a dependency, this may be the time to seriously start looking for a solution

[jmaassen]

I'll compare some potential solutions

[sverhoeven]

The 3.0.0 hadoop client shadows it's Jersey deps.

Adding integrationTestCompile 'org.apache.hadoop:hadoop-client:3.0.0' as dependency and using Xenon 2.4.0 gives no errors.

The ./gradlew dependencies contains org.apache.hadoop:hadoop-common:2.8.1 -> 3.0.0 and com.sun.jersey:jersey-json:1.19 so this probably works because the classpath order is changed.