LGTM.com: Building OpenJDK

[Marcono1234]

How do you get LGTM.com to build the databases for the OpenJDK?

The logs for openjdk/jdk show that the Java database has not been built anymore for multiple days (and was pretty outdated anyways since it performed git checkout jdk-14+36 during extraction).

Updating the analysis configuration to use a newer boot JDK (and not checking out an old Git commit) does not work, see logs:

[2021-06-29 00:32:27] [build-stdout] [2021-06-29 00:32:27] [autobuild] [stderr] [ODASA Javac] Unsupported source Java version 18
[2021-06-29 00:32:28] [build-stdout] [2021-06-29 00:32:27] [autobuild] [stderr] java.util.concurrent.ExecutionException: java.lang.Error: Fatal extractor error detected. Attempting to abort build commands.

But that is probably expected since CodeQL likely does not support the latest Java version yet.

However, trying to build openjdk/jdk16u does not work either, see logs:

[2021-06-29 13:08:32] [build-stdout] [2021-06-29 13:08:32] [autobuild] Creating support/classlist.jar
[2021-06-29 13:08:34] [build-stdout] [2021-06-29 13:08:34] [autobuild] make[3]: *** [/opt/src/build/linux-x86_64-server-release/support/link_opt/classlist] Error 1
[2021-06-29 13:08:34] [build-stdout] [2021-06-29 13:08:34] [autobuild] GenerateLinkOptData.gmk:65: recipe for target '/opt/src/build/linux-x86_64-server-release/support/link_opt/classlist' failed
[2021-06-29 13:08:34] [build-stdout] [2021-06-29 13:08:34] [autobuild] make[2]: *** [generate-link-opt-data] Error 2

However, the C++ build with the same configuration and same build command succeeds, which makes me think it might be related / caused by the CodeQL Java extractor? Running make images locally in a Linux Docker container works as well without issues, but when using CodeQL CLI to run make images it fails with the same error as above.

Do you have any ideas why the build is failing, or what to do to make the build succeed?

Edit: Interestingly the make command make java seems to succeed, see logs (note: for simplicity I omitted the C++ config for that build). Maybe that already suffices for the Java database; it would most likely even cover more than the previous command make java.base.
If you want to build a database for the JDK locally you can try these scripts.

[Marcono1234]

Summarizing the above:

• LGTM.com openjdk/jdk:
  • C++ extraction: Fails because the specified boot JDK is too old
  • Java extraction:
    • Performs manual git checkout jdk-14+36 (should be removed)
    • When checking out latest changes fails because CodeQL Java extractor does not support Java 18 yet
• openjdk/jdk16u
  • Make target make images fails for "Creating support/classlist.jar"; the error information from the JDK build tools is not very helpful
    Possibly caused by a CodeQL Java extractor bug?
  • Make target make java succeeds

Conclusion:

• Newer boot JDK has to be used
• make java should be executed for CodeQL Java extraction
• If the container running the build has limited memory available, --with-memory-size should be used (2 or 3 GB as limit might suffice). The JDK build tools use the total memory as memory limit, not accounting for CodeQL CLI which also needs some memory. During local testing this caused the JDK build to get stuck or fail.

You can use these scripts for building a JDK CodeQL CLI database locally.

[hvitved]

https://lgtm.com/projects/g/AdoptOpenJDK/openjdk-jdk11u works with the following build configuration:

extraction:
  java:
    prepare:
      packages:
      - "libx11-dev"
      - "libxext-dev"
      - "libxrender-dev"
      - "libxrandr-dev"
      - "libxtst-dev"
      - "libxt-dev"
      - "libcups2-dev"
      - "libfontconfig1-dev"
      - "libasound2-dev"
      - "autoconf"
    index:
      build_command: "bash configure --disable-javac-server --disable-warnings-as-errors\
        \ --with-num-cores=4 --with-boot-jdk=/usr/lib/jvm/java-10-openjdk-amd64 --with-build-jdk=/usr/lib/jvm/java-11-openjdk-amd64\
        \ && make images"

I have updated the configuration on the project, so the builds should hopefully succeed in a day or so.

[Marcono1234]

I have updated the configuration on the project, so the builds should hopefully succeed in a day or so.

Thanks! It looks like it did not automatically build the project, so I clicked "Retry build" on https://lgtm.com/projects/g/AdoptOpenJDK/openjdk-jdk11u/logs/languages/lang:java. But now it seems to be stuck at "Not yet scheduled" for a few days already:

[hvitved]

It appears that the analysis is not able to complete within LGTM's 4 hour limit, since some queries are producing too many results for LGTM to process, unfortunately.

[Marcono1234]

Thanks for the update! That is unfortunate indeed, at least you can build the database locally.
I am going to mark this discussion as answered then.