current status of CITGM

[mcollina]

I don't think the results are helpful at this point because any issue is tagged as "skip the module" vs "oh maybe there is a problem with this module, who should investigate". So, what if there is actually a bug? I don't think we'd be able to catch breakages... because if we break a module, it would be removed.

On top of that, maintaining a package run as part of CITGM that does any I/O is very hard and frustrating because the failures are seldom reproducible locally. Essentially, we are asking maintainers to add if (process.env.CITGM) (and skip code coverage) to keep them here.

Something that we could improve here is to only run CITGM on a handful of environments that maintainers can easily reproduce.

[RafaelGSS]

I think there are two parts to the "fixing CITGM" initiative.

1. The first objective is already in progress, which involves removing all problematic modules to ensure a reliable CITGM outcome. This may seem like a temporary solution, but it is currently the most practical and feasible approach. However, it is important to keep in mind that this approach only masks the underlying issues.
2. The second approach is to expand the scope of CITGM to cover all crucial modules, which were mostly removed earlier. This can be achieved through a gradual process. Simply initiate a PR check, verify that there are no flaky issues, and then merge the updates.

I'm very confident this is the smoothest approach to achieving a reliable CITGM.

There's an additional step that can and should be moved in parallel to the second part of this initiative which is:

• Why this module is skipping this environment? Does it make sense?
• Also, once we fix most of the flakiness in our ecosystem, we can gradually remove the parallelization -- which very often is the source of machine issues in CITGM -- and this will eventually reduce the failures of I/O tests.

[ljharb]

It would also seem helpful if we could skip not just one platform, but specific node version(s) and platform combos - that way we skip as few tests as possible.

[alfonsograziano]

+1 for @ljharb proposal. That would be very useful.
Something else that may help can be to unpack the lookup file into multiple js files, each one containing the config of a single module.

Currently, in the JSON file we need to use the "comment" attribute to write a comment, but I think this is not the best approach.

Having a file for a single module (in a /lookup folder) will allow for a more complete config.
Also, having one file per module will allow to add useful comments like: why are we skipping this module on this platform for these Node versions? We can refer to GH issues, ongoing discussions and so on.

With a single file per module will also be simpler to track the config changes made over time

[ljharb]

I like that; it would also allow us to use CODEOWNERS to auto-tag a package's maintainers whenever their package's config is changed.

[alfonsograziano]

I didn't know about CODEOWNERS, seems pretty cool!