[FEATURE] Document CI workflows/build processes

[linguini1]

Is your feature request related to a problem? Please describe.

The CI system is maintained by a few individuals who have special knowledge about it, but it puts a lot of strain on them when something goes wrong (resource spikes especially).

Documentation I could find on the system is only:

• https://nuttx.apache.org/docs/latest/testing/citests.html#

Ideally, having these docs will:

• Reduce workload on our excellent CI team so that other contributors can step in to fix problems
• Allow a base understanding of the CI so we can work towards reducing our resource consumption/streamlining our CI processes

Describe the solution you'd like

It would be great to have some documentation about:

• The CI build process that executed on PRs (checkpatch, doc build and specifically system build)
• The custom CI tooling in tools/ci, what they do, options, how they work
• The format for .dat files used in CI builds, what they're specifying, how they're parsed, etc.
• Maintenance of the custom NuttX docker image
• Pitfalls of CI that the experts among us have learned

I know @lupyuen has a wealth of information on his own blogs, so we could maybe draw information from those posts and include them in our own NuttX documentation. This should make it easier for people to patch the CI/reduce Lup and simbit's workload.

Describe alternatives you've considered

No response

Verification

• I have verified before submitting the report.

[lupyuen]

I made a video that explains how I monitor NuttX CI every morning. Hope it's helpful :-)

• "Is Apache #NuttX RTOS really OK?" Daily Routine for NuttX Monitoring

@simbit18 FYI :-)

[linguini1]

Thank you for this Lup!

[lupyuen]

Gemini LLM has some suggestions for automating our Daily Monitoring of NuttX CI. A bit dubious I think 🤔

https://docs.google.com/document/d/1FwkQxy4Fkn8MZZ8-4lpmcE47LxUFxzFI4B0wMuJakt0/edit?usp=drivesdk

[lupyuen]

Maybe the problem is not with the Messy Docs for NuttX CI ... But finding a suitable CI Lead for NuttX? Like @simbit18 or someone else.

(Sorry I can't be CI Lead, I suffer from hypertension)

Our CI Lead needs to be super hands-on in executing operational duties. (Like my Daily Routine for NuttX CI)

Whatever our CI Lead discovers about NuttX CI, that person will add the knowledge to the NuttX Docs.

Somehow I have a hunch: Maintaining a dry set of CI Docs won't be helpful ... Unless our CI Lead actually uses it daily as an Operations Manual 🤔

(Remember: CI is not really a Dev Job, it's more an Operational Job)

[linguini1]

I agree that's a problem too @lupyuen, I had no idea about your daily process until now!

But, I also think documenting the scripts that are used for our CI processes would be helpful so that contributors can safely modify them to help improve the CI. Right now that also falls on the CI lead since it's a delicate beast with a lot of domain knowledge required.

[linguini1]

Wow @lupyuen, I just took the time to watch the video you made about your daily routine. Thank you so much for all your hard work! There is a ton involved!

I also didn't realize that you were paying for the NuttX dashboard out of pocket; we should find a solution for that!

[lupyuen]

I have a new video that explains GitHub Runners and why we're using so many of them...

• Apache NuttX RTOS: GitHub Runners and Mystery Spikes

@simbit18: FYI :-)

[linguini1]

Another very useful video! And I learned something about Buddhist philosophy too :)

[lupyuen]

I have a new video that explains possible reasons for our overuse of GitHub Runners, and potential solutions...

• Apache NuttX RTOS: Too Many GitHub Runners

(0) Too Many PRs? What's the limit?
(1) Build and Test PRs in Our Own NuttX Repo. Possible?
(2) Encourage Devs to build and test PRs in Our Own NuttX Repo
(3) NuttX Apps: Too many GitHub Runners?
(4) Fail to Download Dependencies: ESP32, FDT, ...
(5) Quit Builds Upon Failure: Will it really help?
(6) Self-Hosted GitHub Runners: Need security mgmt
(7) Our Own Distributed Build: Also need Security + Monitoring
(8) CI Upgrade Strategy: CI Lead + Arch Owners + Board Reps

@simbit18 FYI :-)