Transition to Yarn Workspaces

[leeyi45]

Transition to Yarn Workspaces

Yarn has supported Workspaces since Yarn 1, but it has matured over later versions and we can now use it since we have moved to using Yarn 4.

The main motivations for doing this are:

• Reduced install size. Yarn workspaces support focused installs, which means that a single team working on a single bundle or tab won't have to install the entire repository's worth of packages that won't be relevant to them
• Better Compartmentalization. There is a significant amount of code that functioned like a common library for bundles and tabs that didn't really have a proper place to live. Workspaces allow us to separate this code neatly.
• Dependency Management. Previously, if tabs relied on a specific bundle, it was down to the build scripts to determine if the tabs needed rebuilding when the bundle was rebuilt. This is more complicated than expected. Instead, workspaces allow us to rely on Yarn's dependency resolution features to determine what needs to be rebuilt and when.
• Automatic Worktree Detection. If so desired, Yarn automatically detects which bundles/tabs have changed with respect to the main git branch and won't rebuild those that haven't. This responsibility previously fell onto the build scripts, but I think Yarn's implementation is far more robust.
• More Configurability: Each bundle/tab can maintain its own set of dependencies now, separate from other bundles. This allows us to have patches local to a specific bundle (like the @jscad/modeling patch for csg). Also each bundle/tab can customize its own tsconfig etc..

Migration to vitest

This repository was migrated to vitest, which has been designed as a drop in replacement for jest. I find that getting jest to work with the wide mix of ESM and CJS syntaxes has been a major headache, and configuration for jest has always been needlessly complicated.

vitest supports TypeScript and ESM out of the box and has generally provided a far smoother experience, especially when dealing with dependencies that need to be tested (like typedoc).

Playwright Testing

We haven't exactly been able to properly test the interactive functionalities of our tabs and other React components. Now, using playwright, we can do just that.

Further work needs to be done to figure out how to better handle playwright installation.

Why I hate Jest

Basically some scripts relied on __dirname, which is only available in CJS, and so would not work if compiled to ESM. But the scripts also used top level await, so compiling to CJS wasn't an option. Thus, import.meta.dirname would have to be used instead, but that meant that jest couldn't transform these files unless it was run in its experimental ESM mode, which would then cause many of the CJS bundled dependencies to break because they don't use imports with extensions.

The alternatives would have been significantly bulkier jest configurations or major changes to the code, and even after trying both of those I still ended up with errors because jest wouldn't properly transform dependencies.

Also because jest insists on transforming to CJS, we can't use verbatimModuleSyntax: true in tsconfig, instead requiring some weird configuration as a workaround.

Upgrading Module Manifests

This PR is the beginning of the steps required to address #57. Bundles will now be able to specify a requires field in their manifest that specifies the minimum version of Source that the bundle can be used with.

A bundle manifest can now look like:

{
  "tabs": [],
  "requires": 1
}

Each bundle specifies its own manifest, which is then collated by the build tools automatically.

Bundles will also be able to specify versions to help work toward resolving #79. The version field in package.json gets merged with the manifest during compilation.

Vitepress Docs Server

The Modules Repo's wiki has been quite the disorganized mess (oops). While I was thinking about how to go about documenting the changes introduced by this repository I realized that we were probably going to need something more serious than the Github Wiki.

So I've begun porting the Wiki over to this new Vitepress server (found in ./docs) whilst trying to update any outdated documentation.

Better Typing

Many of the modules were written a long time ago, possibly from when before Typescript had typeguards. In any case, a lot of the is_* functions didn't make use of typeguards, so type assertions were still necessary (or even worse, reckless abuse of any)

This PR will fix some of the low hanging fruit in this regard, improving the type safety for modules.

Better (and more consistent) Documentation

I've noticed that for many of the modules' exported functions, their JSDoc documentation doesn't match their signature. There were even cases where the documentation was entirely inaccurate. With new ESLint rules, hopefully these issues get highlighted earlier.

[leeyi45]

There are still many tasks left to be done, this is far from complete, but I thought I'd open this now for feedback.

I'm especially concerned that the structure of the repository looks a lot more complex now, so I'm wondering if there's any need for simplifying things.

[RichDom2185]

A lot of files do not have a trailing final empty newline which is not ideal.

Could you reformat them?

Best to also add in .vscode/settings.json:

{
  // ...other stuff
  "files.insertFinalNewline": true
}

[leeyi45]

RFC:

Effectively the modules repository has three kinds of documentation:

• Bundle Docs for Users (example)
• Bundle Docs for Developers (example)
• Repo Docs for Developers (the various READMEs in ./scripts that are being migrated)

Should we unify all of these under one documentation server? I'm not even sure how we would go about doing that.

I have also been experimenting with using Typedoc to generate Markdown that can be used with Vitepress to document things like our common React components library (which probably will need to become its own fully fledged thing in the future).

[RichDom2185]

RFC:

Effectively the modules repository has three kinds of documentation:

• Bundle Docs for Users (example)
• Bundle Docs for Developers (example)
• Repo Docs for Developers (the various READMEs in ./scripts that are being migrated)

Should we unify all of these under one documentation server? I'm not even sure how we would go about doing that.

I have also been experimenting with using Typedoc to generate Markdown that can be used with Vitepress to document things like our common React components library (which probably will need to become its own fully fledged thing in the future).

@leeyi45 I agree we should move from GitHub Wiki to source code for docs, it's been something in my TODO for a while now. I also agree Vitepress is better, having (unsuccessfully tried Docusaurus in the past.

Though I think since we are now moving it to source code, we should standardize and use something like markdownlint to format our Markdown files.

My original plan for the above-linked repo was to continually pull documentation files from the various individual repositories (added as git submodules), but I am second-guessing it now since it seems like it would be quite messy hmm (though technically doable).

[RichDom2185]

Separately, given the size of codebase, do you have any thoughts on using biome/oxlint over Prettier/ESLint for formatting/linting?

[leeyi45]

Separately, given the size of codebase, do you have any thoughts on using biome/oxlint over Prettier/ESLint for formatting/linting?

At some point I'm thinking about re-incorporating Prettier, but right now I think that would just contribute to the scope creep that this PR is becoming

I have added some configuration to ESLint for linting our markdown and JSON files. Outside of ESLint I have no experience with any other linters so I can't comment on those.

[leeyi45]

Turns out I've made some progress with the matrix workflow, we'll see where this goes...

[leeyi45]

Tasks Left for me:

• Tidy up the different linting rules and package them into configs with the linting plugin
• Fix coverage testing (if possible)
• See if tests can be run without the buildtools for libraries
• Update docs :)
• Tidy up workflow

[leeyi45]

@RichDom2185 Based on the workflow results, any idea what's going on with the ar bundle and the saar package?

[RichDom2185]

@RichDom2185 Based on the workflow results, any idea what's going on with the ar bundle and the saar package?

@leeyi45 sorry for the delay, finally had time to take a look at the PR again. ar bundle uses old React 18 code which doesn't work with React 19.

[leeyi45]

@RichDom2185 Based on the workflow results, any idea what's going on with the ar bundle and the saar package?

@leeyi45 sorry for the delay, finally had time to take a look at the PR again. ar bundle uses old React 18 code which doesn't work with React 19.

But this repository is running React 18?

[RichDom2185]

I've never used Yarn Workspaces before, how do I get the repository set up?

[leeyi45]

I've never used Yarn Workspaces before, how do I get the repository set up?

You can just clone the repo. From there, maybe you can try spinning up the docs server (cd docs && yarn dev) and then see if the info provided there is sufficient to get started

[RichDom2185]

I can't install dependencies due to the above. Is it a Windows thing?

[leeyi45]

Yeah it looks like a Windows thing

[RichDom2185]

Yeah it looks like a Windows thing

Oh are you not on Windows anymore? Is there a fix?

[leeyi45]

Yeah it looks like a Windows thing

Oh are you not on Windows anymore? Is there a fix?

I'm on my Mac. I don't have access to my Windows machine right now.

[leeyi45]

@RichDom2185 I've gone through the tests and code and replaced things with path.join as necessary. If I didn't miss anything, this should work with Windows paths

[RichDom2185]

@leeyi45 Fixed! (I hope, at least installing and building modules works now)

[leeyi45]

I know I've since added like a billion commits, but the main focus was getting the documentation in order. @RichDom2185 If you could, please look through that stuff especially for the modules tutorials to see if anything else needs to be added onto that.

[RichDom2185]

I know I've since added like a billion commits, but the main focus was getting the documentation in order. @RichDom2185 If you could, please look through that stuff especially for the modules tutorials to see if anything else needs to be added onto that.

Sure, I'll take a look again in a bit. Is the PR good to merge otherwise?

[leeyi45]

I know I've since added like a billion commits, but the main focus was getting the documentation in order. @RichDom2185 If you could, please look through that stuff especially for the modules tutorials to see if anything else needs to be added onto that.

Sure, I'll take a look again in a bit. Is the PR good to merge otherwise?

I can't think of anything else that needs to be done within this PR? I've tried to run through the process of making my own bundle/tab and I've fixed all the issues with that process as far as I can tell.

[RichDom2185]

@sentry review

[RichDom2185]

I'm just thinking: wouldn't this upload the cache incorrectly if we focus on a certain package only? We should either use manual strategy or use a custom cache key based on the input parameters to this action

[leeyi45]

I honestly don't know. A lot of the workflows' behaviour remains untested cause we'd actually need to run them against the live codebase