htsget: broken "File->Load from URL" support

[brainstorm]

While investigating igvteam/igv.js#2023 (comment) today on IGV.js, I wondered how IGV desktop was faring... and found the following issue.

I'm using IGV from homebrew on OSX:

When trying to load a simple BAM file from https://htsget-rs.umccr.workers.dev/reads/bam/mt.sorted, the following happens:

Console yields the following message:

(...)
INFO [Aug 13,2025 19:09] [TrackLoader] Loading resource:  https://htsget-rs.umccr.workers.dev/reads/bam/mt.sorted
SEVERE [Aug 13,2025 19:10] [HttpUtils] Range header removed by client or ignored by server for url: https://htsget-rs.umccr.workers.dev/reads/bam/mt.sorted

And after clicking ok on that dialog, IGV gets stuck in "Loading..." state until it times out after a while:

With the following backtrace behind the scenes:

SEVERE [Aug 13,2025 19:13] [TrackLoader] Read timed out
SEVERE [Aug 13,2025 19:13] [TrackLoader] java.net.SocketTimeoutException: Read timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278)
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)
	at java.base/java.net.Socket$SocketInputStream.implRead(Socket.java:1116)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1103)
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)
	at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1462)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1068)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1706)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1615)
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307)
	at org.igv/org.broad.igv.util.HttpUtils.openConnection(HttpUtils.java:789)
	at org.igv/org.broad.igv.util.HttpUtils.openConnection(HttpUtils.java:851)
	at org.igv/org.broad.igv.util.HttpUtils.openConnection(HttpUtils.java:804)
	at org.igv/org.broad.igv.util.HttpUtils.openConnection(HttpUtils.java:665)
	at org.igv/org.broad.igv.util.HttpUtils.openConnection(HttpUtils.java:661)
	at org.igv/org.broad.igv.util.stream.IGVSeekableHTTPStream.openInputStreamForRange(IGVSeekableHTTPStream.java:211)
	at org.igv/org.broad.igv.util.stream.IGVSeekableHTTPStream._read(IGVSeekableHTTPStream.java:125)
	at org.igv/org.broad.igv.util.stream.IGVSeekableHTTPStream.read(IGVSeekableHTTPStream.java:85)
	at htsjdk@4.3.0-module/htsjdk.samtools.seekablestream.SeekableStream.readFully(SeekableStream.java:84)
	at org.igv/org.broad.igv.track.FileFormatUtils.isGzip(FileFormatUtils.java:177)
	at org.igv/org.broad.igv.track.FileFormatUtils.determineFormat(FileFormatUtils.java:43)
	at org.igv/org.broad.igv.track.TrackLoader.load(TrackLoader.java:250)
	at org.igv/org.broad.igv.ui.IGV.load(IGV.java:1277)
	at org.igv/org.broad.igv.ui.IGV.loadResources(IGV.java:1202)
	at org.igv/org.broad.igv.ui.IGV$2.run(IGV.java:383)
	at org.igv/org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:73)
	at org.igv/org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:43)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1575)

SEVERE [Aug 13,2025 19:13] [IGV] Error loading track
SEVERE [Aug 13,2025 19:13] [IGV] org.broad.igv.exceptions.DataLoadException: Read timed out
	at org.igv/org.broad.igv.track.TrackLoader.load(TrackLoader.java:314)
	at org.igv/org.broad.igv.ui.IGV.load(IGV.java:1277)
	at org.igv/org.broad.igv.ui.IGV.loadResources(IGV.java:1202)
	at org.igv/org.broad.igv.ui.IGV$2.run(IGV.java:383)
	at org.igv/org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:73)
	at org.igv/org.broad.igv.util.LongRunningTask.call(LongRunningTask.java:43)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1575)

[jrobinso]

I haven't tested this from the UI in a long time, sometime before the previous GA4GH server stopped working. Looking through the code I don't see how IGV could determine that this was an htsget resource. What version of IGV are you running there?

In some previous version there was an explicit "Load from HTSGET..." menu item. We might need to restore that, I would rather not but its not obvious how this is determined. I'm very curious to know what version you are using there.

[brainstorm]

Looking through the code I don't see how IGV could determine that this was an htsget resource.

You are right, I passed htsget:// scheme instead of https:// to try it out (remember issue samtools/hts-specs#581 ?), but yielded the same result. I do recall that back a few versions ago, this scheme worked and perhaps it was meant to avoid adding that additional "Load from HTSGET" menu item?

What version of IGV are you running there?

As noted in the screenshot: 2.19.5, fresh from Homebrew :)

I'd really like to see htsget:// scheme in the standard, but I guess it very much depends on @jmarshall at this point?

/cc @ohofmann @mmalenic @andrewpatto

[jmarshall]

I think samtools/hts-specs#581 (comment) is still a good summary of my thoughts on the matter. The first paragraph explains why I think a bespoke URL scheme is in general unnecessary. The second paragraph describes why (as I see it) clients might benefit from it a little nonetheless.

However in response to that, samtools/hts-specs#581 (comment) is the IGV author saying that IGV and IGV.js don't need it. Has this changed?

[jrobinso]

I think I know what happened here, and its fixable without an "htsget" scheme.

@brainstorm I was asking what version your users are on, not where you found the bug (I can reproduce it). Its not clear to me when this last could have worked.

[jrobinso]

@jmarshall IGV author here.

igv.js doesn't need it (a htsget:// scheme), because that is an embedded component, and the embedder (developer) presumably knows that the url is to an htsget resource. So you just tell igv.js that it's htsget explicitly.

For the end user applications IGV desktop and IGV web, its just a user-entered URL. The application needs to know if its htsget. An "htsget://" scheme is of course a convenient way to do this, but maybe not required. Looking at the spec I can imagine the following rule -- if the response can be parsed as json and has an "htsget" property assume its an htsget server.

In the worst case we can add explicit menus "load from htsget".

htsget is a topic that seems to popup every few years, and there is a lot of re-education on my part wrt what it is about.

[jrobinso]

Actually this would be better than relying on the parse response, although its not required. For IGV purposes I'm going to work with with Content-Type. Specifically, urls should either (1) use the htsget:// scheme, or (2) specify htsget in the Content-Type header of the response.

JSON responses SHOULD include a Content-Type header describing the htsget protocol version defining the JSON schema used in the response, e.g.,

Content-Type: application/vnd.ga4gh.htsget.v1.0.0+json; charset=utf-8

The question then is what are the allowable values of Content-Type? I don't see any listing for htsget here: https://www.iana.org/assignments/media-types/media-types.xhtml#application. For the time being I'll take any content-type containing the string "vnd.ga4gh.htsget".

[jrobinso]

@brainstorm Back to the problem at hand. For the near term I'm restoring the "htsget://" scheme trick. Just replace https:// with htsget:// for IGV usage.

Determining that a URL is a resource from the response, either the actual response or the content-type, is possible but more complicated.