Indexer, memory usage, zombies

[shusterboris]

Hello, colleagues!
My project contains few projects (size of project > 300 Gb). When I started initial indexing or re-indexing with parameters like these:

    -J=-Xmx22G,-J=-Xms22G,-J=-server,
    -j,/opt/java/openjdk/bin/java,
    --jar,/opengrok/lib/opengrok.jar,
    -t,/opengrok/etc/logging.properties,-p,'%PROJ%',-d,'/opengrok/logs/%PROJECT%',
    -P,'%PROJECT%',-U,'http://localhost:8080',
    --,
    --progress,
    -R, /opengrok/etc/readonly_configuration.xml,
    -G,-m,'256',-U,'http://localhost:8080',-H,'%PROJECT%'

In 40% of cases the index process is interrupted and 8 ctags processes became zombie.
Server on VM, 32 Gb memory (usually > 35 Gb is free), 8 Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz
openjdk version "11.0.17" 2022-10-18 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.17.0.8-2.el8_6) (build 11.0.17+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.17.0.8-2.el8_6) (build 11.0.17+8-LTS, mixed mode, sharing)
Opengrok 1.8.1
Linux 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Wed Dec 14 16:00:01 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
Do we have any recommendation to avoid the situation.
Thank you advance

[vladak]

That's strange because normally indexer restarts any exited ctags processes.

Anything interesting in the indexer logs ?

Also, given the size, it might be worth trying raising the heap a bunch - sometimes heap shortages have strange consequences.

[vladak]

Could you post complete stack trace of the exception ?

[vladak]

For the record, the is not reachable message is emitted when the webapp URL is considered as not reachable. This check is done very early in the indexer's main(), right after parsing the command line options:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/Indexer.java

Lines 175 to 179 in 378387f

	if (webappURI != null && !HostUtil.isReachable(webappURI, cfg.getConnectTimeout(),
	cfg.getIndexerAuthenticationToken())) {
	System.err.println(webappURI + " is not reachable and the -U option was specified, exiting.");
	System.exit(1);
	}

which means that if that happens, the ctags processes have not been started yet.

[vladak]

PR #4355 should take care of the extant ctags processes before the sendToConfigHost() is called.

[vladak]

Anyhow, it seems strange to me that the API request to http://localhost:8080 would require certificate chain construction, unless there is some redirect-to-TLS done behind the scenes.

[vladak]

Also, why are the zombies a problem ?