GDAL Webstart Issues

[wcmatthysen]

There are currently issues in getting GDAL working when deploying WorldWind in a webstart environment. The original discussion is here: #42. This needs to be addressed in some way as this used to work in the upstream repository with the custom version of GDAL.

[wcmatthysen]

@gbburkhardt, I'm assuming you're looking for the GDAL jar files I was describing that must have been put together by the NASA team?

You can download NASA's gdal.jnlp file here.

This is the location from which we downloaded the GDAL jar files years ago. So tonight I downloaded the gdal.jnlp file and then the jar files referenced inside it so that we'd have the originals from NASA. A few of the "nativelib" files referenced in the JNLP file aren't there, but I was able to download the following 7 files (which are the same ones we've been using for the past 6 or 7 years):

gdal.jar
gdaldata.jar
gdal-natives-linux-amd64.jar
gdal-natives-linux-i586.jar
gdal-natives-macosx-universal.jar
gdal-natives-windows-amd64.jar
gdal-natives-windows-i586.jar

I've zipped these 7 jar files and attached them to this post...
gdal.zip

There must be a way to package the new GDAL content in one or more jar files so that we can continue to launch WWJ both from the desktop and via Web Start.

EDIT: I'm assuming this would work just like the jogl-all.jar along with the 5 jogl-all-natives jars and the gluegen-rt.jar along with the 5 gluegen-rt-natives jars.

@gbburkhardt, @robmilton, If you guys look at the ant build script from the upstream repository you'll see that the GDAL native libraries are built by the bundleWebStartLibraries task. The clause in the build.xml file that is of interest is:

    <!-- GDAL library JAR files. -->
    <copy file="${basedir}/gdal.jar" todir="${webstart.unsigned.dir}"/>
    <jar jarfile="${webstart.unsigned.dir}/gdaldata.jar">
        <manifest>
            <attribute name="Permissions" value="all-permissions"/>
        </manifest>
        <fileset dir="${basedir}/lib-external/gdal/data">
            <include name="**"/>
            <type type="file"/>
        </fileset>
    </jar>
    <!-- GDAL native library JAR files. -->
    <jar jarfile="${webstart.unsigned.dir}/gdal-natives-macosx-universal.jar">
        <manifest>
            <attribute name="Permissions" value="all-permissions"/>
        </manifest>
        <fileset dir="${basedir}/lib-external/gdal">
            <include name="GDAL_License.txt"/>
            <include name="LizardTech_DSDK_License.txt"/>
            <include name="LizardTech_3rd_Party.txt"/>
            <include name="LizardTech_Redistributable.txt"/>
            <include name="libgdalalljni.jnilib"/>
            <type type="file"/>
        </fileset>
    </jar>
    <jar jarfile="${webstart.unsigned.dir}/gdal-natives-windows-i586.jar">
        <manifest>
            <attribute name="Permissions" value="all-permissions"/>
        </manifest>
        <fileset dir="${basedir}/lib-external/gdal">
            <include name="GDAL_License.txt"/>
            <include name="LizardTech_DSDK_License.txt"/>
            <include name="LizardTech_3rd_Party.txt"/>
            <include name="LizardTech_Redistributable.txt"/>
            <include name="gdalminjni32.dll"/>
            <include name="gdalalljni32.dll"/>
            <include name="gdalalljni32.dll.manifest"/>
            <type type="file"/>
        </fileset>
    </jar>
    <jar jarfile="${webstart.unsigned.dir}/gdal-natives-windows-amd64.jar">
        <manifest>
            <attribute name="Permissions" value="all-permissions"/>
        </manifest>
        <fileset dir="${basedir}/lib-external/gdal">
            <include name="GDAL_License.txt"/>
            <include name="LizardTech_DSDK_License.txt"/>
            <include name="LizardTech_3rd_Party.txt"/>
            <include name="LizardTech_Redistributable.txt"/>
            <include name="gdalminjni64.dll"/>
            <include name="gdalalljni64.dll"/>
            <include name="gdalalljni64.dll.manifest"/>
            <type type="file"/>
        </fileset>
    </jar>
    <jar jarfile="${webstart.unsigned.dir}/gdal-natives-linux-i586.jar">
        <manifest>
            <attribute name="Permissions" value="all-permissions"/>
        </manifest>
        <fileset dir="${basedir}/lib-external/gdal">
            <include name="GDAL_License.txt"/>
            <include name="LizardTech_DSDK_License.txt"/>
            <include name="LizardTech_3rd_Party.txt"/>
            <include name="LizardTech_Redistributable.txt"/>
            <include name="libgdalalljni32.so"/>
            <type type="file"/>
        </fileset>
    </jar>
    <jar jarfile="${webstart.unsigned.dir}/gdal-natives-linux-amd64.jar">
        <manifest>
            <attribute name="Permissions" value="all-permissions"/>
        </manifest>
        <fileset dir="${basedir}/lib-external/gdal">
            <include name="GDAL_License.txt"/>
            <include name="LizardTech_DSDK_License.txt"/>
            <include name="LizardTech_3rd_Party.txt"/>
            <include name="LizardTech_Redistributable.txt"/>
            <include name="libgdalalljni64.so"/>
            <type type="file"/>
        </fileset>
    </jar>
</target>

There is also a task to build the webstart configuration called compileWebStartConfiguration:

<target name="compileWebStartConfiguration">
    <mkdir dir="${webstart.jnlp.dir}"/>
    <copy todir="${webstart.jnlp.dir}">
        <fileset dir="${webstart.src.dir}">
            <include name="*.jnlp"/>
            <exclude name="JavaWebStartTemplate.jnlp"/>
            <type type="file"/>
        </fileset>
    </copy>
</target>

We need to replicate these tasks in the Gradle build somehow. We also need to ensure that the correct files from the external GDAL installation are copied and assembled into the platform specific jar files.

[gbburkhardt]

Great. It pays to have the source code available. Good find. So @robmilton should be able to adapt this to build his jar files.

One feature of the original NASA build for GDAL that I liked was that they combined all the shared libraries in one file with the JNI interface. When I updated the GDAL build, I made that available as an option in the JNI makefile. The GDAL group was a little wary of doing it that way, so it's not the default.

Collecting all the DLLs from a GISinternals installation should be pretty easy to stick them in a jar file. But for Linux, we'll probably need to use 'ldd -r' or something to figure out what all the dependent shared libraries are.

[robmilton]

Hey guys, I've been working a bunch of late hours the past couple of nights on this issue. I think you're right @gbburkhardt that ultimately the jar files shouldn't be hard to build (especially if I restrict for now the target OS/arch to Win/64).

For now I've set aside the creation of the jar files for use with Web Start, and I'll come back to that later. I'm hoping that a jar structure that mimics the gdal folder structure described below will work.

After looking at the newly modified GDALUtils class in WWJ-CE I structured my project with a "libs" folder that contains a gdal folder with the following structure

  gdal
      data
      gdalplugins

Using the files from the 3 GISInternals MSI installs I populated the "gdal" folder with the gdal.jar and all the DLLs from C:\Program Files\GDAL. I put the contents of C:\Program Files\GDAL\gdal-data and projlib in the "data" folder. And I put the contents of C:\Program Files\GDAL\gdalplugins in the "gdalplugins" folder.

I then made a small change to the GDALUtils class to make the "searchDirs" string array variable a private variable that is global to the class. And I added a static method called "setSearchDirs()" to the class so that you can assign a value to "searchDirs". Then in the static block I check to see if the "searchDirs" variable has been set and if so skip the code that sets the variable in the class.

If I set the "searchDirs" variable to my project's lib\gdal folder AND include my project's lib\gdal folder in my System Path... GDAL 2.4 loads and works fine.

But if I don't add the gdal folder to the System Path no matter what I try I get unsatisfied link errors. Do you have any ideas on how to make this work without having to actually add the gdal folder to the System Path? Or how to accomplish that programmatically in Java?

[gbburkhardt]

If you search, there are a number of hacks suggested on the web to set environment variables like "PATH" programmatically. That has to be done for the JVM process to make it work. There's no "built in" facility in Java to set environment variables for the JVM. I considered doing that for WorldWind, but rejected it because none of techniques looked like they were guaranteed to work if the internals of the JVM changed. This is especially relevant since with Java 11 there are at least three competing JVMs available. There was a method in NASA's release of GDALutils that altered the JVM's "java.library.path" property (GDALutils.alterJavaLibraryPath()), but it either didn't work with Java 11 or used deprecated methods.

There are hooks available so that you don't need to change 'searchDirs'. You can set the environment variable GDAL_DATA to point to the gdal_data directory, and GDAL_DRIVER_PATH to point to the gdalplugins directory, as noted in the GDAL_README.txt file, and the GDAL documentation.

The JVM's "java.library.path" property must include the location of "libgdalall.jni" file, which depends on all the other DLLs. Code in 'gdal.jar' that was generated by SWIG calls "System.loadLibrary("gdalalljni"). The dependent DLLs are found by looking in the directory of the executable, and then through the PATH variable. See DLL Search Order.

Sounds like you are making progress.

[robmilton]

@gbburkhardt, thanks for the response. I'll take a look at the old GDALutils.alterJavaLibraryPath as a possible method to allow us to continue to use WWJ-CE and Web Start until we start making the switch to Java 11+.

When you mention the environment variables GDAL_DATA and GDAL_DRIVER_PATH are you talking about the Windows System environment variables? If so then this is part of the whole challenge with the Web Start application. I've haven't found a reliable "hack" for setting the environment variables from Java... but I'm looking into that. Ultimately it seems I'll have to find a hack that works so that our clients won't have to set these variables.

BTW... I've tried several hacks that I've found, but so far they either flat out don't work or I haven't figured out a way to introduce them early enough in my code. Have you had any success using any hack to set/edit the PATH in Windows?

[robmilton]

We have a separate "login" application that validates credentials and allows the user to select which project/app they want to launch. The selected application is then launched by starting a new process with ProcessBuilder. I am able to append a value to the existing Windows PATH for the new process at this point. However, this is based on my ability to know ahead of time where the GDAL files live on the client machine. Since I don't know of a way to do this using Web Start I've decided to look into continuing to use the version of GDAL that WWJ v2.1.0 was using until we're ready to abandon Web Start for another distribution/update/launch method.

So I copied the method "replaceLibraryLoader" and class "GDALLibraryLoader" from the old GDALUtils. With this method and class in place I can successfully load the old included version of GDAL when my project starts.

However, with this method and class in place every time the project launches I get the following error:

Jun 10, 2019 3:06:20 PM gov.nasa.worldwind.util.SessionCacheRetrievalPostProcessor run
SEVERE: Retrieval failed for https://worldwind26.arc.nasa.gov/elev?EXCEPTIONS=application/vnd.ogc.se_xml&REQUEST=GetCapabilities&SERVICE=WMS&VERSION=1.3.0
Jun 10, 2019 3:06:20 PM gov.nasa.worldwind.util.SessionCacheUtils retrieveSessionData
SEVERE: Exception while retrieving resources for https://worldwind26.arc.nasa.gov/elev?EXCEPTIONS=application/vnd.ogc.se_xml&REQUEST=GetCapabilities&SERVICE=WMS&VERSION=1.3.0
java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at java.net.SocketInputStream.read(Unknown Source)
	at sun.security.ssl.InputRecord.readFully(Unknown Source)
	at sun.security.ssl.InputRecord.read(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
	at sun.net.www.protocol.http.HttpURLConnection.access$200(Unknown Source)
	at sun.net.www.protocol.http.HttpURLConnection$9.run(Unknown Source)
	at sun.net.www.protocol.http.HttpURLConnection$9.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.AccessController.doPrivilegedWithCombiner(Unknown Source)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
	at java.net.HttpURLConnection.getResponseCode(Unknown Source)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
	at gov.nasa.worldwind.retrieve.HTTPRetriever.doRead(HTTPRetriever.java:48)
	at gov.nasa.worldwind.retrieve.URLRetriever.read(URLRetriever.java:368)
	at gov.nasa.worldwind.retrieve.URLRetriever.call(URLRetriever.java:244)
	at gov.nasa.worldwind.retrieve.URLRetriever.call(URLRetriever.java:27)
	at gov.nasa.worldwind.util.SessionCacheUtils.retrieveSessionData(SessionCacheUtils.java:80)
	at gov.nasa.worldwind.util.SessionCacheUtils.getOrRetrieveSessionCapabilities(SessionCacheUtils.java:170)
	at gov.nasa.worldwind.terrain.BasicElevationModel.retrieveResources(BasicElevationModel.java:2028)
	at gov.nasa.worldwind.terrain.BasicElevationModel$3.run(BasicElevationModel.java:2118)
	at java.lang.Thread.run(Unknown Source)

We are not relying on any of the NASA servers in our application, so I don't know where the call to "worldwind26.arc.nasa.gov" is being made from. I'm assuming there's something built into the old gdal.jar that is making this call. Do you have any ideas an how to avoid this?

EDIT
I found that if I set the property "gov.nasa.worldwind.app.config.document" to the location of our WW config file prior to the call to "replaceLibraryLoader" then the attempt to retrieve from "worldwind26.arc.nasa.gov" is never made.

So for now I think we'll continue to use GDAL 1.7.2 until we're ready to move away from Java Web Start.

Thanks for the assistance through this little "adventure".

EDIT 2
If Web Start was a valid option for the future I would continue to work on a resolution. However, since Web Start has been pulled from Java 9 this seems like a reasonable compromise.

[gbburkhardt]

I think that you're better off pursuing the build of the GDAL jar files using the target "bundleWebStartLibraries" in the current "build.xml" file as a starting point.

[robmilton]

I think that you're better off pursuing the build of the GDAL jar files using the target "bundleWebStartLibraries" in the current "build.xml" file as a starting point.

I'm not sure I understand this statement. Do you mean modify the build.xml so that it uses the new GDAL v2.4.0 files? And then use the "replaceLibraryLoader" and "GDALLibraryLoader" from the old GDALUtils to load them instead?

[gbburkhardt]

Yes, modify the build.xml file to use the new GDAL files. The existing "bundleWebStartLibraries" target provides a template that shows how to package the DLLs in a jar file, and how to put the "gdal_data" files into a jar file. I'd poke around in the 'webstart' directory to determine how Webstart accesses the jar files. That said, I'm just guessing. I don't understand at all how Webstart works.

But that 'build.xml' was most likely used to build the native library jar files for Webstart using the GDAL 1.7.2 DLLs, so I'd start looking there.

[robmilton]

Creating the jar files doesn't seem that hard. I've created jar files using the new GDAL content by following the pattern I see in the build.xml. But it seems NASA added some methods to the gdal.jar that aren't present in the new gdal.jar. I'm just not sure it's worth the effort to figure this all out when we will be switching from Web Start (which is technically gone at this point) to another delivery/update/launch method before too long. At that point I'll have control over where the GDAL files are placed and the ability to set environmental variables via a couple of different approaches.

[wcmatthysen]

I have looked a bit at the webstart stuff and it seems as though it is dead. I think it is best if we just drop webstart as it will not be supported in the future.