Exception com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype in Cosmos Client

[KoushikSahu]

We are seeing the following exception while trying to use CosmosDB client in our Synapse Spark 3.4 pools. Same code was working fine in Spark 3.3 pools.

2025-02-07 10:40:45,332 INFO RxDocumentClientImpl [Driver]: Initializing DocumentClient [1] with serviceEndpoint [https://{cosmosdbname}.documents.azure.com:443/], connectionPolicy [ConnectionPolicy{httpNetworkRequestTimeout=PT1M, tcpNetworkRequestTimeout=PT5S, connectionMode=GATEWAY, maxConnectionPoolSize=1000, idleHttpConnectionTimeout=PT1M, idleTcpConnectionTimeout=PT0S, userAgentSuffix='', throttlingRetryOptions=RetryOptions{maxRetryAttemptsOnThrottledRequests=9, maxRetryWaitTime=PT30S}, endpointDiscoveryEnabled=true, preferredRegions=null, multipleWriteRegionsEnabled=true, proxyType=null, inetSocketProxyAddress=null, readRequestsFallbackEnabled=true, connectTimeout=PT5S, idleTcpEndpointTimeout=PT1H, maxConnectionsPerEndpoint=130, maxRequestsPerConnection=30, tcpConnectionEndpointRediscoveryEnabled=true, ioThreadPriority=5, ioThreadCountPerCoreFactor=2, tcpHealthCheckTimeoutDetectionEnabled=true, minConnectionPoolSizePerEndpoint=1, openConnectionsConcurrency=1, aggressiveWarmupConcurrency=16}], consistencyLevel [Session]
2025-02-07 10:40:45,481 INFO RxDocumentClientImpl [cosmos-parallel-1]: Getting database account endpoint from https://{cosmosdbname}.documents.azure.com:443/
2025-02-07 10:40:45,776 ERROR ClientCertificateCredential [cosmos-parallel-1]: Azure Identity => ERROR in getToken() call for scopes [https://{cosmosdbname}.documents.azure.com/.default]: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
2025-02-07 10:40:45,776 ERROR SimpleTokenCache [cosmos-parallel-1]: {"az.sdk.message":"Failed to acquire a new access token"}
2025-02-07 10:40:45,778 ERROR GlobalEndpointManager [cosmos-parallel-1]: Fail to reach global gateway [https://{cosmosdbname}.documents.azure.com:443/], [com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype]
2025-02-07 10:40:45,778 ERROR GlobalEndpointManager [cosmos-parallel-1]: startRefreshLocationTimerAsync() - Unable to refresh database account from any location. Exception: java.util.ServiceConfigurationError: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
java.util.ServiceConfigurationError: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
  at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:589)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1237)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
  at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
  at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
  at com.azure.core.implementation.util.Providers.<init>(Providers.java:61)
  at com.azure.core.implementation.http.HttpClientProviders.<clinit>(HttpClientProviders.java:26)
  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:61)
  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:51)
  at com.azure.identity.implementation.IdentityClientBase.getPipeline(IdentityClientBase.java:893)
  at com.azure.identity.implementation.IdentityClientBase.initializeHttpPipelineAdapter(IdentityClientBase.java:874)
  at com.azure.identity.implementation.IdentityClientBase.getConfidentialClient(IdentityClientBase.java:269)
  at com.azure.identity.implementation.IdentityClient.lambda$getConfidentialClientApplication$4(IdentityClient.java:144)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:45)
  at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:143)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
  at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:102)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
  at reactor.core.publisher.FluxFromMonoOperator.subscribe(FluxFromMonoOperator.java:83)
  at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
  at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
  at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
  at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
  at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
  at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)
2025-02-07 10:40:45,791 ERROR RxDocumentClientImpl [Driver]: Client initialization failed. Check if the endpoint is reachable and if your auth token is valid. More info: https://aka.ms/cosmosdb-tsg-service-unavailable-java. More details: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
java.util.ServiceConfigurationError: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
  at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:589)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1237)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
  at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
  at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
  at com.azure.core.implementation.util.Providers.<init>(Providers.java:61)
  at com.azure.core.implementation.http.HttpClientProviders.<clinit>(HttpClientProviders.java:26)
  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:61)
  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:51)
  at com.azure.identity.implementation.IdentityClientBase.getPipeline(IdentityClientBase.java:893)
  at com.azure.identity.implementation.IdentityClientBase.initializeHttpPipelineAdapter(IdentityClientBase.java:874)
  at com.azure.identity.implementation.IdentityClientBase.getConfidentialClient(IdentityClientBase.java:269)
  at com.azure.identity.implementation.IdentityClient.lambda$getConfidentialClientApplication$4(IdentityClient.java:144)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:45)
  at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:143)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
  at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:102)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
  at reactor.core.publisher.FluxFromMonoOperator.subscribe(FluxFromMonoOperator.java:83)
  at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
  at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
  at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
  at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
  at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
  at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)
2025-02-07 10:40:45,792 ERROR RxDocumentClientImpl [Driver]: unexpected failure in initializing client.

Code snippet we are using is,

this.cosmosClient = new CosmosClientBuilder()
                    .endpoint(endpoint)
                    .credential(new ClientCertificateCredentialBuilder()
                            .tenantId(tenantId)
                            .clientId(clientId)
                            .pfxCertificate(HelperUtils.getCertificate(config))
                            .sendCertificateChain(true)
                            .build())
                    .consistencyLevel(ConsistencyLevel.SESSION)
                    .contentResponseOnWriteEnabled(true)
                    .gatewayMode()
                    .buildClient();

Library versions used:

com.azure.azure-cosmos: 4.61.1
com.azure.azure-identity: 1.13.0
com.azure.core: 1.49.1
com.microsoft.azure.synapse.synapseutils_2.12: 1.4

The exception message that we query about is com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype. We have checked and found that ApacheHttpClientProvider is not coming from our packages. (Our fat jar is not containing this class). We are guessing that ApacheHttpClientProvider is coming from the synapse runtime and getting used by reflection. We wanted to check what version of azure-identity is recommended for Synapse spark 3.4.