SystemTest: Build/Run time improvement needed

[sreuland]

What problem does your feature solve?

long times for building system-test image during GH workflow action execution, right now the build step of system test(make ... build) takes roughly 45 minutes, as it recompiles all components from source(core, rpc, quickstart, rs-xdr)

What would you like to see?

Shorter runtimes for build of system test image during GH workflow action execution.

What alternatives are there?

• use the system test provided optional config parameters to use an existing docker image of a component rather than compiling from source:
  • CORE_IMAGE - if can use an existing docker image of core, this would reduce build times the most by as much as 20 minutes. - done - allow CORE_IMAGE bin override #95
  • SOROBAN_CLI_IMAGE, SOROBAN_RPC_IMAGE, RS_XDR_IMAGE - if versions of these components were available as pre-build docker images would reduce build times by several minutes.
• smaller subset of test perf improvement suggested on System Test: shouldn't need to recompile the example contracts once for each test #52 to eliminate duplicate contract compilations between tests.

[sreuland]

we started initial discussion on some ideas, paraphrasing here:

System tests take ~an hour to run. Any easy ways we can speed it up through caching.

• Reorg docker file (PR #31) - merged, some marginal effect, overall time is still long
• attempted using docker gha layer caching with buildx during Github action run (PR #27), it's writing out cache layers, but on subsequent builds in gh, seems to be missing finding layers from the cache, not seeing [CACHED] from docker during build, still investigating.
• refer to quickstart gha workflow, it does some effective caching with gh cache
• make quickstart launch faster, since system-test is waiting for that.
• publish soroban enabled core images to dockerhub under some other image like “experimental” because then gh workflow could pull it from there, instead of compiling from source.
• Once decoupled from Horizon, will only have core docker dependency