feat: DH-20578: Groovy remote file sourcing #7451

bmingles · 2025-11-26T22:40:53Z

DH-20578: This PR implements a RemoteFileSource plugin in Deephaven Core that can request file sources from a remote client. Key points of interest

RemoteFileSourceClassLoader.java - registered with the Groovy session to request remote file sources
RemoteFileSourceCommandResolver.java - resolves a flight command to an instance of the plugin service. This allows us to not have to register a magic variable in the session. It exports a PluginMarker object that is associated with a plugin name that the RemoteFileSourcePlugin later recognizes as an object it can suppport
RemoteFileSourcePlugin - the plugin
PluginMarker - named object used to match the flight command with the plugin. Needed since all plugins require an exported object of some sort

Related PRs

deephaven/vscode-deephaven/pull/273 implements the client side code to be used by the VS Code extension.
deephaven-ent/iris/pull/3878 is needed to enable this in Core+

Testing

Install the client side code in VS Code vscode-deephaven-1.0.10-remote.0.vsix

Community

Run Core Groovy server from this branch (I use IntelliJ debug config :web-client-api:gwtCompile server-jetty-app:run -Panonymous -Pgroovy)
Clone this repo https://github.com/bmingles/deephaven-groovy-remote-docs-demo and open the folder in VS Code
In the Deephaven view, expand the Workspace/Groovy/groovy-remote-docs-demo folder under REMOTE IMPORT SOURCES and click the + button to add test folder.
Open docs-sample.groovy file and run against the Deephaven server. Should see testTable panel show up.

Core+

Changes are deployed to bmingles-remote-file-source vm. Can also configure VS Code to connect to https://bmingles-remote-file-source.int.illumon.com:8123/ and run same code

github-actions · 2025-11-26T22:41:04Z

No docs changes detected for `1cdbac4`

bmingles · 2026-01-02T21:50:11Z

server/jetty-app-11/build.gradle

    implementation project(':server-jetty-11')

    implementation project(':extensions-flight-sql')
+    implementation project(':plugin-remotefilesource')


Not sure if these should have been added here since I don't see any other plugins.

bmingles · 2026-01-02T21:50:18Z

server/jetty-app/build.gradle

    implementation project(':server-jetty')

    implementation project(':extensions-flight-sql')
+    implementation project(':plugin-remotefilesource')


Not sure if these should have been added here since I don't see any other plugins.

…H-20578)

…0578)

...otefilesource/src/main/java/io.deephaven.remotefilesource/RemoteFileSourceMessageStream.java

niloc132 · 2026-01-06T15:52:15Z

web/client-api/src/main/java/io/deephaven/web/client/api/JsProtobufUtils.java

+/**
+ * Utility methods for working with protobuf messages in the client-side JavaScript environment.
+ */
+public class JsProtobufUtils {


...how sure are we that we actually need any/all of this? doesnt the generated message types give us this for free?

I thought we needed it in order to support the Any wrapper?

niloc132 · 2026-01-06T15:52:49Z

...pi/src/main/java/io/deephaven/web/client/api/remotefilesource/JsRemoteFileSourceService.java

+@JsType(namespace = "dh.remotefilesource", name = "RemoteFileSourceService")
+public class JsRemoteFileSourceService extends HasEventHandling {
+    /** Event name for generic messages from the server */
+    @JsProperty(namespace = "dh.remotefilesource.RemoteFileSourceService")


this annotation shouldnt be necessary, should get it from the type's namespace+name, ditto below

I got warnings for the @link #EVENT_MESSAGE in the header comment. Claude suggested this fix which worked. Is there a better way?

By adding hte JsType annotation at the class top level, all public static constants are added to the JS automatically. See JsPartitionedTable for some precedent.

@mofojed This produced warnings for the @link #EVENT_MESSAGE syntax in header comments for me

Where are you seeing this warning? I'm not seeing any warning in the header or anywhere else in the file related to this when I remove this annotation (the @JsProperty annotation)

@mofojed
If I remove the annotation and build the JS API types via ./gradlew :web-client-api:types:assemble

I get:

[warning] Failed to resolve link to "dh.remotefilesource.EVENT_REQUEST_SOURCE" in comment for dh.remotefilesource.RemoteFileSourceService.

Although, there are a number of other warnings as well. I just didn't want to keep adding to them:

[warning] Failed to resolve link to "dh.convertArrayToString" in comment for dh.PreviewOptions. [warning] Failed to resolve link to "dh.array" in comment for dh.PreviewOptions. [warning] Failed to resolve link to "dh.transportFactory" in comment for dh.ConnectOptions.useWebsockets. [warning] Failed to resolve link to "dh.useWebsockets" in comment for dh.ConnectOptions.transportFactory. [warning] Failed to resolve link to "Table.uncoalesced" in comment for dh.Column.isPartitionColumn. [warning] Failed to resolve link to "Promise" in comment for dh.IdeConnection.mergeTables. [warning] Failed to resolve link to "io.deephaven.web.shared.fu.JsRunnable" in comment for dh.IdeConnection.onLogMessage. [warning] Failed to resolve link to "io.deephaven.web.shared.fu.JsRunnable" in comment for dh.IdeConnection.onLogMessage.__type. [warning] Failed to resolve link to "dh.includeConstituents" in comment for dh.TreeTable. [warning] Failed to resolve link to "TreeRowImpl.hasChildren" in comment for dh.TreeTable. [warning] Failed to resolve link to "java.lang.UnsupportedOperationException" in comment for dh.TreeTable.applyCustomColumns. [warning] Failed to resolve link to "io.deephaven.engine.table.PartitionedTable.merge" in comment for dh.PartitionedTable.getMergedTable. [warning] Failed to resolve link to "DataOptions.SubscriptionOptions" in comment for dh.Table.subscribe. [warning] Failed to resolve link to "io.deephaven.web.client.api.subscription.AbstractTableSubscription.close" in comment for dh.Table.createSubscription. [warning] Failed to resolve link to "Table.uncoalesced" in comment for dh.Table.size. [warning] Failed to resolve link to "Promise" in comment for dh.IdeSession.getTreeTable. [warning] Failed to resolve link to "Promise" in comment for dh.IdeSession.mergeTables. [warning] Failed to resolve link to "Promise" in comment for dh.IdeSession.emptyTable. [warning] Failed to resolve link to "Promise" in comment for dh.IdeSession.timeTable. [warning] Failed to resolve link to "JsTable#createSnapshot(Object)" in comment for dh.TableViewportSubscription.snapshot. [warning] Failed to resolve link to "dh.remotefilesource.EVENT_REQUEST_SOURCE" in comment for dh.remotefilesource.RemoteFileSourceService.

...pi/src/main/java/io/deephaven/web/client/api/remotefilesource/JsRemoteFileSourceService.java

...otefilesource/src/main/java/io.deephaven.remotefilesource/RemoteFileSourceMessageStream.java

plugin/src/main/java/io/deephaven/plugin/type/PluginMarker.java

niloc132 · 2026-01-06T16:28:45Z

...efilesource/src/main/java/io.deephaven.remotefilesource/RemoteFileSourceCommandResolver.java

+                    "RemoteFileSourcePluginFetchRequest must contain a valid result_id");
+        }
+
+        final String pluginName = request.getPluginName();


it appears that DeephavenRemoteFileSourcePlugin is always the pluginName set by the client, so we end up with exactly one PluginMarker. Do we anticipate more of these, or specific clients who can control this? Nothing seems able to clear old instances, do we want a mechanism (liveness?) for that?

I anticipate that the Python version of the plugin may make use of this at some point.

Clearing instances is probably a good idea. Any suggestions on best way to do that?

mofojed

Functionality seems good overall, nice.

engine/base/src/main/java/io/deephaven/engine/util/RemoteFileSourceClassLoader.java

mofojed · 2026-01-12T16:21:10Z

...otefilesource/src/main/java/io.deephaven.remotefilesource/RemoteFileSourceMessageStream.java

+            throw new IllegalArgumentException("messageStream must not be null");
+        }
+
+        executionContext = new RemoteFileSourceExecutionContext(messageStream, resourcePaths);


Should we throw if the executionContext is already set?

I don't think so. executionContext is static and only 1 MessageStream instance can claim it at a time. This probably doesn't impact Core+, but in Community it could matter if you had multiple connections to the worker.

Added a better block comment to the method.

...otefilesource/src/main/java/io.deephaven.remotefilesource/RemoteFileSourceMessageStream.java

mofojed · 2026-01-12T16:33:10Z

...pi/src/main/java/io/deephaven/web/client/api/remotefilesource/JsRemoteFileSourceService.java

+@JsType(namespace = "dh.remotefilesource", name = "RemoteFileSourceService")
+public class JsRemoteFileSourceService extends HasEventHandling {
+    /** Event name for generic messages from the server */
+    @JsProperty(namespace = "dh.remotefilesource.RemoteFileSourceService")


By adding hte JsType annotation at the class top level, all public static constants are added to the JS automatically. See JsPartitionedTable for some precedent.

mofojed · 2026-01-12T16:34:26Z

...pi/src/main/java/io/deephaven/web/client/api/remotefilesource/JsRemoteFileSourceService.java

+        DomGlobal.setTimeout(ignore -> fireEvent(EVENT_REQUEST_SOURCE,
+                new ResourceRequestEvent(message.getRequestId(), request)), 0);


Comment why this is adding a timeout here instead of just calling it right away.

I'm not actually 100% sure why this is needed except that I saw precedence in JsWidget for dispatching new events in response to message stream events. I assume it's desirable to ensure this happens on the next tick, but I can't speak to a specific problem it may cause if we don't do so.

Looking at the history in JsWidget, looks like it was added with this PR: #6715
In that case, I think the issue was the promise for fetching the widget would resolve after we had already fired off the events, so client didn't have time to add a listener to actually receive the event. So just put emitting the event on the next tick after resolving.
I guess in theory that might be possible here as well... though maybe we should be checking if there's no listeners and throwing in that case, or force a listener to be passed in when retrieving the plugin. When there's a request from the server, it's blocked until there's a response, yeah? So we must have a listener to respond here (or we should default to a no-op response, but then why bother fetching the plugin...), and it doesn't make sense to have multiple listeners registered (since multiple responses for one request is invalid), so really we want exactly one listener... may as well enforce it with creation of the service?
In CoreClient#getRemoteFileSourceService(), we're returning a new instance of the service plugin whenever that method is called - what happens if there's multiple instances? We really only want one instance, right?

I made 2 primary changes to address this:

CoreClient#getRemoteFileSourceService() now creates a single instance of JsRemoteFileSourceService

JsRemoteFileSourceService enforces that only 1 EVENT_REQUEST_SOURCE handler can be registered. It also throws if a request for source is received and there are no registered event handlers

Also removed the setTimeout

...pi/src/main/java/io/deephaven/web/client/api/remotefilesource/JsRemoteFileSourceService.java

mofojed

Changes look good, had some questions about fetching an instance of the service and how we handle multiple instances.

bmingles force-pushed the DH-20578_groovy-remote-file-sourcing branch 3 times, most recently from a48276f to fc106d6 Compare December 24, 2025 16:23

bmingles force-pushed the DH-20578_groovy-remote-file-sourcing branch from 777b0b4 to 4d1b51c Compare January 2, 2026 17:53

bmingles commented Jan 2, 2026

View reviewed changes

bmingles force-pushed the DH-20578_groovy-remote-file-sourcing branch from 3d7a03e to f0ced57 Compare January 5, 2026 21:14

bmingles and others added 22 commits January 5, 2026 15:15

Started building out RemoteFileSource plugin (#DH-20578)

6ec786d

Command resolver now fetches plugin (#DH-20578)

bd985ac

RemoteFileSourceTicketResolverFactoryService (#DH-20578)

51bb7b5

Added stub for JsRemoteFileSourceService (#DH-20578)

c282d40

Generated GWT bindings

1735733

Fetching plugin service working (#DH-20578)

c051a01

Wiring up messagestream (#DH-20578)

ae7d97e

test bidirectional communication (#DH-20578)

ef53eeb

set connection id (#DH-20578)

d3fe8cd

Moved clientSessionId to plugin fetch instead of separate message (#D…

f8e6dc4

…H-20578)

Cleanup (#DH-20578)

e5d8725

set execution context (#DH-20578)

361019a

Simplified execution context (#DH-20578)

2a1525a

Basic file sourcing is working (#DH-20578)

36143fa

Comments and cleanup (#DH-20578)

8246036

Made method private (#DH-20578)

1998da3

Made method static (#DH-20578)

74e193e

onResourceRequest method (#DH-20578)

59afc07

JS API types (#DH-20578)

af3eef0

Refactored to use PluginMarker (#DH-20578)

07f6b8e

Re-using JsWidget for message stream (#DH-20578)

3d7ebc7

Added pluginType field to PluginMarker (#DH-20578)

eea7214

bmingles added 6 commits January 5, 2026 15:15

Cleanup PluginMarker (#DH-20578)

440bd23

Changed back to resourceName convention to match class loaders (#DH-2…

9e74429

…0578)

Moved method (#DH-20578)

140b3ac

Cleanup JsProtobufUtils (#DH-20578)

77ff6b3

Cleanup JsRemoteFileSourceService (#DH-20578)

df05a62

Applying Colin's client proto generation (#DH-20578)

b70f606

bmingles force-pushed the DH-20578_groovy-remote-file-sourcing branch from f0ced57 to b70f606 Compare January 5, 2026 21:15

Changed to runtime dependencies and fixed docs links (#DH-20578)

52feab6

bmingles marked this pull request as ready for review January 5, 2026 23:23

bmingles requested review from cpwright, devinrsmith, niloc132 and rcaudy as code owners January 5, 2026 23:23

bmingles requested a review from mofojed January 5, 2026 23:23

bmingles mentioned this pull request Jan 5, 2026

feat: DH-20578: Groovy remote file source deephaven/vscode-deephaven#273

Draft

niloc132 reviewed Jan 6, 2026

View reviewed changes

bmingles added 4 commits January 6, 2026 12:47

Addressed review comments in RemoteFileSourceClassLoader (#DH-20578)

f46e97d

Addressed review comments in PluginMarker (#DH-20578)

5cc3673

Addressed review comments in RemoteFileSourceMessageStream (#DH-20578)

7ad0a36

Addressed review comments in JsRemoteFileSourceService (#DH-20578)

325f688

bmingles requested a review from niloc132 January 9, 2026 19:50

mofojed requested changes Jan 12, 2026

View reviewed changes

bmingles added 3 commits January 13, 2026 17:13

Changed to getResource (#DH-20578)

1cb84d4

Addressed review comments (#DH-20578)

71c58b8

Addressed review comments (#DH-20578)

1f775cc

bmingles requested a review from mofojed January 14, 2026 00:22

mofojed reviewed Jan 14, 2026

View reviewed changes

Addressed review comments (#DH-20578)

d5ba658

bmingles requested a review from mofojed January 14, 2026 18:25

Removed timeout (#DH-20578)

1cdbac4

		DomGlobal.setTimeout(ignore -> fireEvent(EVENT_REQUEST_SOURCE,
		new ResourceRequestEvent(message.getRequestId(), request)), 0);

feat: DH-20578: Groovy remote file sourcing #7451

Are you sure you want to change the base?

feat: DH-20578: Groovy remote file sourcing #7451

Conversation

bmingles commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related PRs

Testing

Community

Core+

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

No docs changes detected for 1cdbac4

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mofojed left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmingles Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mofojed left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

bmingles commented Nov 26, 2025 •

edited

Loading

github-actions bot commented Nov 26, 2025 •

edited

Loading

No docs changes detected for `1cdbac4`

bmingles Jan 13, 2026 •

edited

Loading