Skip to content

Conversation

@ikusteu
Copy link

@ikusteu ikusteu commented Oct 12, 2025

Some context (motivation)

I work for a team using the wa-sqlite and the Vlcn stack in our projects. There's a lot we love about both projects (rhashimoto/wa-sqlite and this fork, along with the full Vlcn stack), but the two versions of wa-sqlite have diverged quite a bit. In order to be able to use the improvements made in the original repo, along with cr-sqlite extension (and the ecosystem built around it), we've gone through the process of reconciling the two versions (bringing this repo up to speed with the original).

We were going to open this as an issue (to provide the motivation there), but seeing as there's no issue tracker (this being a fork itself), I'll lay out the motivation here.

There's a lot of diff between this (vlcn-io) version of wa-sqlite and the rhashimoto one. Since the divergence, the rhashimoto version had some breaking changes:

  • some restructuring of lib*.(js|h|c) shims
  • the refactor of JS callbacks (e.g. progress hook, update hook, VFS methods etc.)
  • the refactor of the way VFS adapters are defined/used
  • dep updates (emcc, sqlite, etc.)
  • addition of JSPI
  • etc.

We're most interested in OPFS support, as it would provide significant benefits to the way we use cr-sqlite in our (browser) app:

  • FS transparency is a significant benefit
  • we've observed significant performance improvements when syncing
  • as a second order effect, FS transparency makes it even easier for initial sync (if setting up a new node, one can simply download the .sqlite3 file, store it in OPFS and open using the crsqlite-wasm stack)
  • as a second order effect, it's easy to manage import/export (same inherent workflow as above)

Since the divergence, the rhashimoto version added multiple OPFS adapters:

  • opfs-coop-sync-vfs
  • opfs-any-context-vfs
  • opfs-adaptive-vfs
  • opfs-permuted-vfs

We understand those are examples, but we'd really like to use them as beasline and build on top of them if necessary.
However, due to significant divergence those aren't compatible with the current version of this repo.

We've gone through the process of reconciling against the original and have done so within the context of one of our projects to ensure it works with the full Vlcn stack.
We would now like to merge this in so that:

  • the dependency mgmt is cleaner (pointing to this repo's master, instead of our own fork)
  • we believe the rest of the community might benefit from the efforts made in this regard

Some comments

This reconciliation was done without much prior knowledge about the codebase:

  • we've used the final result (crsqlite-wasm + the underlaying stack) prior to this effort and have identified what we want to achieve with this
  • we gained understanding of the codebase as we went along

When solving conflicts (and there were many) we'd inspect history (git blames) and used the following resoning:

  • if the original repo updated/removed something that existed upstream and wasn't touched in vlcn version, we accepted the change
  • if we realised that vlcn version depended on something edited/removed we'd update the depending code (if trivial and could be contained to this repo only)
  • if the update to depending code (from previous point) was non-trivial, we'd keep the upstream patch (for backwards compatilibity)
  • same reasoning (as the previous point) was used if downstream deps (e.g. crsqlite-wasm) depended on something (e.g. str_new and friends)
  • if there was an addition made in the vlcn version, but then the same functionality was tackled (downstream) in the original, the original was kept and this repo updated (e.g. update hooks)

Since we were dealing with a lot of divergence, we didn't merge all at once, instead:

  • we split the diff downstream of divergence point until rhashimoto/master into merge commits and applied one at the time
  • we set up checkpoints throughout:
    • grouping patches that didn't introduce breaking changes
    • using checkpoints as savepoints when we'd either expect breaking changes after the checkpoint, or the checkpoint represented a goal we were going for (e.g. introducing of opfs-any-context-vfs)

Aftger this PR is merged, downstream dependencies should get updated (and we've prepared work to do so) as there are slight changes to the public API (namely the way VFS adapters are initialised).

Breakdown

Below is a full breakdown of updates applied, grouped by checkpoint, in order to help make the review process more iterative and informed.

Checkpoint 1: a939d6b Merge branch 'reconcile/checkpoint-1' into reconcile-rhashimoto-wa-sqlite

Notes:

Checkpoint 1 is a save point, applying minor updates before some (potentially) breaking changes.

  • add JSFILES as deps for make rules for release targets
  • IDBBatchAtomic - support asyncify interprocess communication
  • minor updates to exported functions (add _sqlite3_bind_parameter_index)
  • update libvfs.c - call vfs methods (vfsRead, vfsWrite, vfsTruncate with i64 offset values, instead of pointers to those values -- I would assume some change in 64 bit legalization)
  • update libvfs.js:
    • handle delegalization of i64 params (passed as two i32s)
    • use string names to access properties on Module (Module["method"] instead of Module.method) to prevent issues with minifying of method names
    • allow async handling (by exposing Module["handleAsync"] if asyncify build)
Files changed:

.github/ISSUE_TEMPLATE/bug_report.md
Makefile
README.md
src/examples/IDBBatchAtomicVFS.js
src/exported_functions.json
src/extra_exported_runtime_methods.json
src/libvfs.c
src/libvfs.js
yarn.lock

Checkpoint 2: b4217e9 Merge branch 'reconcile/checkpoint-2' into reconcile-rhashimoto-wa-sqlite

Notes:

Includes a large change in original repo (merge of dev branch -> v1.0)

  • bump emscripten version 3.1.45 -> 3.1.47 (done in CI job, but also applicable to wrapping processes, e.g. crsqlite-wasm)

  • some changes re yarn (too many to break down, shouldn't really affect the cr-sqlite usage)

  • updates to demo (too many to break down, shouldn't really affect the cr-sqlite usage)

  • updates to docs (too many to break down, shouldn't really affect the cr-sqlite usage)

  • updates to tests (too many to break down, shouldn't really affect the cr-sqlite usage)

  • Makefile:

    • add ASYNCIFY_EXPORTS (asyncifying methods exported from SQLite to JS glue)
    • add JSPI support
    • refactor JS libraries flag: single library (libadapters.js), multiple post js files (all the rest)
  • VFS updates:

    • add FacadeVFS - a scaffold for VFS implementations, exposes xMethods and ports them to jMethods (e.g. xRead calls to jRead on JS implemented VFS)
    • add WebLocksMixin - a mixin implementing SQLite VFS locking protocol for FacadeVFS (providing defaults for all VFSes extending FacadeVFS and not implementing locking overrides)
    • replace the xMethod (e.g. xRead) in the JS interface with jMethod (e.g. jRead) -- FacadeVFS handles the porting from one to the other
    • switch from new VFSImplementation() to VFSImplementation.create() as prefered way to initialise an instance of VFS adapter (e.g. IDBBatchAtomic.create(...))
    • remove VFS adapter: IDBMinimalVFS
    • remove VFS adapter: IDBVersionedVFS
    • remove VFS adapter: OriginPrivateFileSystemVFS
    • add VFS adapter: OPFSAdaptiveVFS
    • add VFS adapter: OPFSCoopSyncVFS
    • add VFS adapter: OPFSPermutedVFS
  • major change in the way JS functions get called by WASM code:

    • replace explicitly exposed JS functions (e.g. xRead, callbacks for updateHook, etc.) with function signatures, e.g. ipippj:
      • a function returning i32
      • 1 p (pointer for function pointer)
      • accepting ppij as parameters: (pointer, pointer, i32, i64)
    • all JS functions "exposed" to the WASM code are no exposed directly, but rather registered with the central lookup table (on the JS side) and a key (i32) passed to the WASM code
    • when WASM code calls to a function, it does so by the registered (i32) key and params the function signature expects
    • every signature also exposes the async version (e.g. ipippij_async for ipippij) with async flags passed to the WASM code when registering the function
    • libadapters.js and libadapters.h define both sides of cross environment function calls
    • following this update, I've refactored the way updateHook (defined by Matt) communicates with the WASM code to stick to the new convention (this gets replaced downstream as update hook is defined in the original repo as well)
  • updates to SQLiteAPI (a wrapper around WASM glue code):

    • remove .declare_vtab - seemed unused (and not present in the original wa-sqlite repo)
    • remove .user_data - seemed unused (and not present in the original wa-sqlite repo)
    • move .prepare_v2, .str_new, .str_apendall, .str_finish, .str_value to a backwards compatibility section:
      • all of these functions have been replaced (with .statements method) in the original wa-sqlite code, but kept as wrappers from vlcn.io/js still depend on them being exposed
Files changed:

.github/workflows/ci.yml
.gitignore
.yarn/releases/yarn-3.1.1.cjs
.yarn/releases/yarn-4.0.2.cjs
.yarnrc.yml
Makefile
README.md
demo/ahp-contention.html
demo/ahp-contention.js
demo/ahp-demo.html
demo/ahp-demo.js
demo/ahp-worker.js
demo/benchmarks.js
demo/benchmarks/benchmark1.sql
demo/benchmarks/benchmark10.sql
demo/benchmarks/benchmark11.sql
demo/benchmarks/benchmark12.sql
demo/benchmarks/benchmark13.sql
demo/benchmarks/benchmark14.sql
demo/benchmarks/benchmark15.sql
demo/benchmarks/benchmark16.sql
demo/benchmarks/benchmark2.sql
demo/benchmarks/benchmark3.sql
demo/benchmarks/benchmark4.sql
demo/benchmarks/benchmark5.sql
demo/benchmarks/benchmark6.sql
demo/benchmarks/benchmark7.sql
demo/benchmarks/benchmark8.sql
demo/benchmarks/benchmark9.sql
demo/benchmarks/benchmarks.html
demo/benchmarks/benchmarks.js
demo/benchmarks/index.html
demo/clean-worker.js
demo/contention-sharedworker.js
demo/contention.html
demo/contention.js
demo/contention/contention-worker.js
demo/contention/contention.html
demo/contention/contention.js
demo/contention/index.html
demo/demo-worker.js
demo/demo.html
demo/demo.js
demo/file/index.js
demo/file/service-worker.js
demo/file/verifier.js
demo/hello.html
demo/hello.js
demo/index.html
demo/index.js
demo/retry/RetryVFS.js
demo/retry/retry-worker.js
demo/write-hint/index.html
demo/write-hint/index.js
demo/write-hint/worker.js
y/releases/yarn-4.0.2.cjs
y/sdks/integrations.yml
y/sdks/typescript/bin/tsc
y/sdks/typescript/bin/tsserver
y/sdks/typescript/lib/tsc.js
y/sdks/typescript/lib/tsserver.js
y/sdks/typescript/lib/tsserverlibrary.js
y/sdks/typescript/lib/typescript.js
y/sdks/typescript/package.json
yarn.lock
docs/assets/highlight.css
docs/assets/icons.css
docs/assets/icons.png
docs/assets/icons@2x.png
docs/assets/main.js
docs/assets/navigation.js
docs/assets/search.js
docs/assets/style.css
docs/assets/widgets.png
docs/assets/widgets@2x.png
docs/index.html
docs/interfaces/SQLiteAPI.html
docs/interfaces/SQLiteModule.html
docs/interfaces/SQLiteModuleIndexInfo.html
docs/interfaces/SQLitePrepareOptions.html
docs/interfaces/SQLiteVFS.html
docs/types/SQLiteCompatibleType.html
jsconfig.json
package.json
src/FacadeVFS.js
src/VFS.js
src/WebLocksMixin.js
src/asyncify_exports.json
src/asyncify_imports.json
src/examples/AccessHandlePoolVFS.js
src/examples/ArrayAsyncModule.js
src/examples/ArrayModule.js
src/examples/IDBBatchAtomicVFS.js
src/examples/IDBContext.js
src/examples/IDBMinimalVFS.js
src/examples/IDBVersionedVFS.js
src/examples/MemoryAsyncVFS.js
src/examples/MemoryVFS.js
src/examples/OPFSAdaptiveVFS.js
src/examples/OPFSCoopSyncVFS.js
src/examples/OPFSPermutedVFS.js
src/examples/OriginPrivateFileSystemVFS.js
src/examples/README.md
src/examples/WebLocks.js
src/exported_functions.json
src/libadapters.h
src/libadapters.js
src/libauthorizer.c
src/libauthorizer.js
src/libfunction.c
src/libfunction.js
src/libmodule.c
src/libmodule.js
src/libprogress.c
src/libprogress.js
src/libvfs.c
src/libvfs.js
src/main.c
src/sqlite-api.js
src/sqlite-constants.js
src/types/index.d.ts
test/AccessHandlePoolVFS.test.js
test/IDBBatchAtomicVFS.test.js
test/MemoryAsyncVFS.test.js
test/MemoryVFS.test.js
test/OPFSCoopSyncVFS.test.js
test/OPFSPermutedVFS.test.js
test/OriginPrivateVFS.test.js
test/TestContext.js
test/WebLocksMixin.test.js
test/api.test.js
test/api_exec.js
test/api_misc.js
test/api_statements.js
test/callbacks.test.js
test/obsolete/GOOG.js
test/obsolete/IDBBatchAtomicVFS.test.js
test/obsolete/IDBMinimalVFS.test.js
test/obsolete/MemoryAsyncVFS.test.js
test/obsolete/MemoryVFS.test.js
test/obsolete/OPFSProxy.js
test/obsolete/OPFSWorker.js
test/obsolete/OriginPrivateFileSystemVFS.test.js
test/obsolete/VFS.test.js
test/obsolete/VFSTests.js
test/obsolete/WebLocks.test.js
test/obsolete/api-instances.js
test/obsolete/module.test.js
test/obsolete/sqlite-api.test.js
test/obsolete/tag.test.js
test/sql.test.js
test/sql_0001.js
test/sql_0002.js
test/sql_0003.js
test/sql_0004.js
test/sql_0005.js
test/test-worker.js
test/vfs_xAccess.js
test/vfs_xClose.js
test/vfs_xOpen.js
test/vfs_xRead.js
test/vfs_xWrite.js
typedoc.json
web-test-runner.config.mjs

Checkpoint 3: 3d95bee Merge branch 'reconcile/checkpoint-3' into reconcile-rhashimoto-wa-sqlite

Notes:

This was a milestone for us as we wanted to be able to use OPFSAnyContextVFS, so I've marked a checkpoint here.
The patch adds OPFSAnyContextVFS implementation, along with some updates (testing mostly)

Files changed:

README.md
demo/demo-worker.js
demo/file/index.js
demo/hello.html
demo/hello.js
demo/hello/README.md
demo/hello/hello.html
demo/hello/hello.js
demo/hello/index.html
package.json
src/WebLocksMixin.js
src/examples/OPFSAnyContextVFS.js
src/examples/README.md
test/IDBBatchAtomicVFS.test.js
test/OPFSAdaptiveVFS.test.js
test/OPFSAnyContextVFS.test.js
test/api.test.js
test/data/idbv5.json
test/obsolete/GOOG.js
test/obsolete/IDBBatchAtomicVFS.test.js
test/obsolete/IDBMinimalVFS.test.js
test/obsolete/MemoryAsyncVFS.test.js
test/obsolete/MemoryVFS.test.js
test/obsolete/OPFSProxy.js
test/obsolete/OPFSWorker.js
test/obsolete/VFSTests.js
test/obsolete/WebLocks.test.js
test/sql.test.js
test/test-worker.js
yarn.lock

Final: 0699a79 Merge branch 'master' into reconcile-rhashimoto-wa-sqlite

Notes:

  • add VFS adapter: IDBMirrorVFS
  • bump Emscripten version 3.1.47 -> 3.1.61
  • bump SQLITE_VERSION 3.45.0 -> 3.50.1
  • add libhook.c and libhook.js - replacing .updateHook (previously in libfunction) with .update_hook
  • add .commit_hook method to the API
  • update SQLite amalgamation download link
Files changed:

.github/ISSUE_TEMPLATE/-do-not-post-anything-other-than-a-bug-report.md
.github/workflows/ci.yml
Makefile
README.md
demo/contention/contention-worker.js
demo/demo-worker.js
demo/demo.js
demo/file/index.js
demo/file/service-worker.js
demo/file/verifier.js
demo/hello/hello.js
dist/wa-sqlite-async.mjs
dist/wa-sqlite-async.wasm
dist/wa-sqlite-jspi.mjs
dist/wa-sqlite-jspi.wasm
dist/wa-sqlite.mjs
dist/wa-sqlite.wasm
docs/assets/search.js
docs/interfaces/SQLiteAPI.html
package.json
src/FacadeVFS.js
src/WebLocksMixin.js
src/asyncify_imports.json
src/examples/IDBBatchAtomicVFS.js
src/examples/IDBMirrorVFS.js
src/examples/MemoryVFS.js
src/examples/OPFSAdaptiveVFS.js
src/examples/OPFSCoopSyncVFS.js
src/examples/OPFSPermutedVFS.js
src/examples/README.md
src/jspi_exports.json
src/libadapters.h
src/libadapters.js
src/libhook.c
src/libhook.js
src/sqlite-api.js
src/types/globals.d.ts
src/types/index.d.ts
test/IDBMirrorVFS.test.js
test/TestContext.js
test/api.test.js
test/api_statements.js
test/callbacks.test.js
test/sql.test.js
test/test-worker.js
web-test-runner.config.mjs
yarn.lock

Checklist

  • I grant to recipients of this Project distribution a perpetual,
    non-exclusive, royalty-free, irrevocable copyright license to reproduce, prepare
    derivative works of, publicly display, sublicense, and distribute this
    Contribution and such derivative works.
  • I certify that I am legally entitled to grant this license, and that this
    Contribution contains no content requiring a license from any third party.

simolus3 and others added 27 commits April 30, 2025 17:24
Fix resetting `isHandleRequested` in `OPFSCoopSyncVFS`
…shimoto#272)

* Permit boolean values to be bound to statements, as 0/1

* Add test for boolean binding
…stance

Use a single `TextEncoder` instance
Export HEAP* module members for recent EMSDK changes.
Bumps [tar-fs](https://github.com/mafintosh/tar-fs) from 3.0.8 to 3.0.9.
- [Commits](mafintosh/tar-fs@v3.0.8...v3.0.9)

---
updated-dependencies:
- dependency-name: tar-fs
  dependency-version: 3.0.9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
…yarn/tar-fs-3.0.9

Bump tar-fs from 3.0.8 to 3.0.9
* Replace Proxy with handwritten proxy for jRead/jWrite buffers.

* Replace Proxy with handwritten proxy for VFS return data.

---------

Co-authored-by: Roy Hashimoto <roy@shoestringresearch.com>
* add '-O1' flag - skips Bynarien's 'wasm-opt' -- clashing with '-g' with the currently used version of emcc (3.1.45)
* NOTE: this might not be necessary with emcc '4.x' versions
* this makes it work for me with the latest browser update (it was throwing an error when requesting v5)
* NOTE: this might be squashed downstream
* Use non-CAPTCHA SQLite download URL.

* Use consistent Makefile variable bracing.

---------

Co-authored-by: Roy Hashimoto <roy@shoestringresearch.com>
* Fix WebLocksMixin state initialization.

* Don't fetch state in WebLocksMixin file control unnecessarily.

* Minor fixes.

---------

Co-authored-by: Roy Hashimoto <roy@shoestringresearch.com>
@ikusteu ikusteu force-pushed the reconcile-rhashimoto-wa-sqlite branch from 0699a79 to ba66572 Compare October 17, 2025 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants