[Bug] Broker (or function worker) cannot authenticate against Keycloak (OAuth 2.0)

[naahk37]

Search before reporting

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

Pulsar Version: 4.0.7 (3 broker, 3 bookies, 3 zookeeper)
OS: Ubuntu 24.04
Java: OpenJDK 21.0.8

Broker Client Authentication is enabled, for the brokers as well as for the function workers (running with the brokers)

Issue Description

One of our brokers restarted and wasn't able to authenticate anymore. I'm not sure, but I think it's actually the function worker client that cannot authenticate, because when I start the broker with disabled function workers, it works.
I pasted error code below and uploaded the stacktrace.

We recently deployed a new function, but the cluster was running without any problems since then. I'm not sure if those two things correlate..

Error messages

Message: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments error_message=java.lang.RuntimeException: java.lang.RuntimeException: org.apache.pulsar.broker.PulsarServerException: org.apache.pulsar.client.api.PulsarClientException$AuthenticationException: Unable to retrieve OAuth 2.0 server metadata

Reproducing the issue

I tried to restart the second broker to see if the problem is maybe constrained only to one machine, but it's the same with this broker too..

Additional information

pulsar_oauth_problem.txt

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

"Unable to retrieve OAuth 2.0 server metadata"

This problem occurs when the broker or function worker cannot access the .well-known/openid-configuration endpoint,

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/auth/oauth2/protocol/DefaultMetadataResolver.java

Lines 59 to 73 in 9937d22

	/**
	* Gets a well-known metadata URL for the given OAuth issuer URL.
	*
	* @param issuerUrl The authorization server's issuer identifier
	* @return a URL
	* @see <a href="https://tools.ietf.org/id/draft-ietf-oauth-discovery-08.html#ASConfig">
	* OAuth Discovery: Obtaining Authorization Server Metadata</a>
	*/
	public static URL getWellKnownMetadataUrl(URL issuerUrl) {
	try {
	return URI.create(issuerUrl.toExternalForm() + "/.well-known/openid-configuration").normalize().toURL();
	} catch (MalformedURLException e) {
	throw new IllegalArgumentException(e);
	}
	}

(There's #25052 to make this configurable, but that shouldn't make a difference to Keycloak)

Perhaps there has been a change in Keycloak configuration instead? You might not notice issues in broker-to-broker authentication immediately and that's perhaps the reason why it disabling functions worker seems to fix the issue.

[naahk37]

The Keycloak config hasn't been touched.

How can I test this properly?
I used the following command from the concerning broker to test connectivity and received a proper response:
curl -v https://EXT_KEYCLOAK_URL/realms/MY_REALM/.well-known/openid-configuration

[naahk37]

So, it turns out that I only needed to restart the third broker....
I guess there was some kind of oauth session "alive" that wouldn't let the other brokers with the same credentials authenticate to keycloak...
Everything works again as expected and I guess the takeaway is "if one broker fails, restart all the other ones"..