Leverage projects existing integration tests instead of duplicate

[leighmcculloch]

I think we should make some changes to the system-test so that it is easier to maintain and is not a separate set of tests for folks to maintain.

Today the system-test contains its own set of integration tests that get executed for the stellar-cli and js-stellar-sdk.

However, these projects have, at least the cli does, their own integration tests that are more comprehensive, and so tests are duplicated here and there.

The system-test should be a place that sets up an environment, and then leverages projects existing integration tests to run those tests. We don't need to define new tests here, if each project and application already define their own integration tests.

(Thanks @anupsdf for inspiring this.)

cc @fnando @sreuland

[sreuland]

However, these projects have, at least the cli does, their own integration tests that are more comprehensive, and so tests are duplicated here and there

There is the opposite pathway to consider also. Remove the overlapping integration tests from application component repos and their internal complexity to defer to system-test for that coverage. Similar end result of functional coverage across distributed components but implemented in one place and defined in the context of a product feature use case. Another potential outcome being portability since the tests are compiled as executable images the same versioned tests can be deployed in different CI scenarios like pr scopes and also nightly regression jobs.

Ideally each component repo is only aware of itself and not responsible for knowing how to work with other components. Each application repo can focus on unit tests to hit all lines of code. If a unit test can't reach app code then it likely means the code has some brittle composition that could be broken down further.

I'm not sure if internal integration test results from multiple component repos which interact with each other will result in verifying the same smoothed out external user experience that we think of as product feature or at least it won't be as easy for test orchestrator to determine from multiple integration test outputs and failures which product features are affected. Or worse, all the individual component integration tests still pass, but in the field when they're all used together live, then something fails.

System-test uses a behavioral black box approach for testing external product features which span across multiple components and treats the network stack as opaque, the components have to work together but the test is only concerned with providing inputs and checking outputs from system boundary.

As an example, system-test has a feature use case test of DApp developer compiles, installs, deploys and invokes a contract
, this test uses a combination of cli and js-stellar-sdk in same sequence as a person would use them for deploying and invoking a contract and verifies the end results. Integration tests in separate component repos will likely miss this interaction as they stop short and just want to verify their component role.

[leighmcculloch]

There's tradeoffs. Tests have a greater chance of being maintained if they live close to their application, and can evolve as the application evolves, run when integration tests run on PRs, etc. I don't think we should give up the benefits we get of having integration tests in their project's repo where they can be updated in lock step with changes to the repo.

It's a good point, that the tests we define here can do more. Both of these types can co-exist. The system-test repo only runs 3 tests that it defines itself and we can keep them as they are, while still doing what I propose above and running all application integration tests here in this repo as well, so that integration tests we write in other repos can also run as part of system tests.