forked from rhashimoto/wa-sqlite
-
Notifications
You must be signed in to change notification settings - Fork 3
Bring the repo (vlcn-io/wa-sqlite) up to speed with the latest in rhashimoto/wa-sqlite #4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ikusteu
wants to merge
392
commits into
vlcn-io:master
Choose a base branch
from
codemyriad:reconcile-rhashimoto-wa-sqlite
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Bring the repo (vlcn-io/wa-sqlite) up to speed with the latest in rhashimoto/wa-sqlite #4
ikusteu
wants to merge
392
commits into
vlcn-io:master
from
codemyriad:reconcile-rhashimoto-wa-sqlite
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix resetting `isHandleRequested` in `OPFSCoopSyncVFS`
Bump package version.
Fix hello demo import paths.
…shimoto#272) * Permit boolean values to be bound to statements, as 0/1 * Add test for boolean binding
…stance Use a single `TextEncoder` instance
Export HEAP* module members for recent EMSDK changes.
Bumps [tar-fs](https://github.com/mafintosh/tar-fs) from 3.0.8 to 3.0.9. - [Commits](mafintosh/tar-fs@v3.0.8...v3.0.9) --- updated-dependencies: - dependency-name: tar-fs dependency-version: 3.0.9 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…yarn/tar-fs-3.0.9 Bump tar-fs from 3.0.8 to 3.0.9
Update SQLite to 3.50.1.
Bump package version.
* Replace Proxy with handwritten proxy for jRead/jWrite buffers. * Replace Proxy with handwritten proxy for VFS return data. --------- Co-authored-by: Roy Hashimoto <roy@shoestringresearch.com>
* add '-O1' flag - skips Bynarien's 'wasm-opt' -- clashing with '-g' with the currently used version of emcc (3.1.45) * NOTE: this might not be necessary with emcc '4.x' versions
* this makes it work for me with the latest browser update (it was throwing an error when requesting v5) * NOTE: this might be squashed downstream
* Use non-CAPTCHA SQLite download URL. * Use consistent Makefile variable bracing. --------- Co-authored-by: Roy Hashimoto <roy@shoestringresearch.com>
* Fix WebLocksMixin state initialization. * Don't fetch state in WebLocksMixin file control unnecessarily. * Minor fixes. --------- Co-authored-by: Roy Hashimoto <roy@shoestringresearch.com>
0699a79 to
ba66572
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some context (motivation)
I work for a team using the
wa-sqliteand the Vlcn stack in our projects. There's a lot we love about both projects (rhashimoto/wa-sqlite and this fork, along with the full Vlcn stack), but the two versions ofwa-sqlitehave diverged quite a bit. In order to be able to use the improvements made in the original repo, along with cr-sqlite extension (and the ecosystem built around it), we've gone through the process of reconciling the two versions (bringing this repo up to speed with the original).We were going to open this as an issue (to provide the motivation there), but seeing as there's no issue tracker (this being a fork itself), I'll lay out the motivation here.
There's a lot of diff between this (vlcn-io) version of wa-sqlite and the rhashimoto one. Since the divergence, the rhashimoto version had some breaking changes:
lib*.(js|h|c)shimsemcc,sqlite, etc.)We're most interested in OPFS support, as it would provide significant benefits to the way we use cr-sqlite in our (browser) app:
.sqlite3file, store it in OPFS and open using thecrsqlite-wasmstack)Since the divergence, the rhashimoto version added multiple OPFS adapters:
opfs-coop-sync-vfsopfs-any-context-vfsopfs-adaptive-vfsopfs-permuted-vfsWe understand those are examples, but we'd really like to use them as beasline and build on top of them if necessary.
However, due to significant divergence those aren't compatible with the current version of this repo.
We've gone through the process of reconciling against the original and have done so within the context of one of our projects to ensure it works with the full Vlcn stack.
We would now like to merge this in so that:
Some comments
This reconciliation was done without much prior knowledge about the codebase:
crsqlite-wasm+ the underlaying stack) prior to this effort and have identified what we want to achieve with thisWhen solving conflicts (and there were many) we'd inspect history (git blames) and used the following resoning:
crsqlite-wasm) depended on something (e.g.str_newand friends)Since we were dealing with a lot of divergence, we didn't merge all at once, instead:
rhashimoto/masterinto merge commits and applied one at the timeopfs-any-context-vfs)Aftger this PR is merged, downstream dependencies should get updated (and we've prepared work to do so) as there are slight changes to the public API (namely the way VFS adapters are initialised).
Breakdown
Below is a full breakdown of updates applied, grouped by checkpoint, in order to help make the review process more iterative and informed.
Checkpoint 1: a939d6b Merge branch 'reconcile/checkpoint-1' into reconcile-rhashimoto-wa-sqlite
Notes:
Checkpoint 1 is a save point, applying minor updates before some (potentially) breaking changes.
JSFILESas deps for make rules for release targets_sqlite3_bind_parameter_index)libvfs.c- call vfs methods (vfsRead,vfsWrite,vfsTruncatewithi64offset values, instead of pointers to those values -- I would assume some change in 64 bit legalization)libvfs.js:i64params (passed as twoi32s)Module["method"]instead ofModule.method) to prevent issues with minifying of method namesModule["handleAsync"]if asyncify build)Files changed:
.github/ISSUE_TEMPLATE/bug_report.md
Makefile
README.md
src/examples/IDBBatchAtomicVFS.js
src/exported_functions.json
src/extra_exported_runtime_methods.json
src/libvfs.c
src/libvfs.js
yarn.lock
Checkpoint 2: b4217e9 Merge branch 'reconcile/checkpoint-2' into reconcile-rhashimoto-wa-sqlite
Notes:
Includes a large change in original repo (merge of
devbranch -> v1.0)bump emscripten version
3.1.45->3.1.47(done in CI job, but also applicable to wrapping processes, e.g.crsqlite-wasm)some changes re yarn (too many to break down, shouldn't really affect the cr-sqlite usage)
updates to demo (too many to break down, shouldn't really affect the cr-sqlite usage)
updates to docs (too many to break down, shouldn't really affect the cr-sqlite usage)
updates to tests (too many to break down, shouldn't really affect the cr-sqlite usage)
Makefile:
ASYNCIFY_EXPORTS(asyncifying methods exported from SQLite to JS glue)libadapters.js), multiple post js files (all the rest)VFS updates:
xMethods and ports them tojMethods(e.g.xReadcalls tojReadon JS implemented VFS)xMethod(e.g.xRead) in the JS interface withjMethod(e.g.jRead) --FacadeVFShandles the porting from one to the othernew VFSImplementation()toVFSImplementation.create()as prefered way to initialise an instance of VFS adapter (e.g.IDBBatchAtomic.create(...))major change in the way JS functions get called by WASM code:
xRead, callbacks forupdateHook, etc.) with function signatures, e.g.ipippj:i32p(pointer for function pointer)ppijas parameters: (pointer, pointer, i32, i64)ipippij_asyncforipippij) with async flags passed to the WASM code when registering the functionlibadapters.jsandlibadapters.hdefine both sides of cross environment function callsupdateHook(defined by Matt) communicates with the WASM code to stick to the new convention (this gets replaced downstream as update hook is defined in the original repo as well)updates to SQLiteAPI (a wrapper around WASM glue code):
.declare_vtab- seemed unused (and not present in the originalwa-sqliterepo).user_data- seemed unused (and not present in the originalwa-sqliterepo).prepare_v2,.str_new,.str_apendall,.str_finish,.str_valueto a backwards compatibility section:.statementsmethod) in the originalwa-sqlitecode, but kept as wrappers fromvlcn.io/jsstill depend on them being exposedFiles changed:
.github/workflows/ci.yml
.gitignore
.yarn/releases/yarn-3.1.1.cjs
.yarn/releases/yarn-4.0.2.cjs
.yarnrc.yml
Makefile
README.md
demo/ahp-contention.html
demo/ahp-contention.js
demo/ahp-demo.html
demo/ahp-demo.js
demo/ahp-worker.js
demo/benchmarks.js
demo/benchmarks/benchmark1.sql
demo/benchmarks/benchmark10.sql
demo/benchmarks/benchmark11.sql
demo/benchmarks/benchmark12.sql
demo/benchmarks/benchmark13.sql
demo/benchmarks/benchmark14.sql
demo/benchmarks/benchmark15.sql
demo/benchmarks/benchmark16.sql
demo/benchmarks/benchmark2.sql
demo/benchmarks/benchmark3.sql
demo/benchmarks/benchmark4.sql
demo/benchmarks/benchmark5.sql
demo/benchmarks/benchmark6.sql
demo/benchmarks/benchmark7.sql
demo/benchmarks/benchmark8.sql
demo/benchmarks/benchmark9.sql
demo/benchmarks/benchmarks.html
demo/benchmarks/benchmarks.js
demo/benchmarks/index.html
demo/clean-worker.js
demo/contention-sharedworker.js
demo/contention.html
demo/contention.js
demo/contention/contention-worker.js
demo/contention/contention.html
demo/contention/contention.js
demo/contention/index.html
demo/demo-worker.js
demo/demo.html
demo/demo.js
demo/file/index.js
demo/file/service-worker.js
demo/file/verifier.js
demo/hello.html
demo/hello.js
demo/index.html
demo/index.js
demo/retry/RetryVFS.js
demo/retry/retry-worker.js
demo/write-hint/index.html
demo/write-hint/index.js
demo/write-hint/worker.js
y/releases/yarn-4.0.2.cjs
y/sdks/integrations.yml
y/sdks/typescript/bin/tsc
y/sdks/typescript/bin/tsserver
y/sdks/typescript/lib/tsc.js
y/sdks/typescript/lib/tsserver.js
y/sdks/typescript/lib/tsserverlibrary.js
y/sdks/typescript/lib/typescript.js
y/sdks/typescript/package.json
yarn.lock
docs/assets/highlight.css
docs/assets/icons.css
docs/assets/icons.png
docs/assets/icons@2x.png
docs/assets/main.js
docs/assets/navigation.js
docs/assets/search.js
docs/assets/style.css
docs/assets/widgets.png
docs/assets/widgets@2x.png
docs/index.html
docs/interfaces/SQLiteAPI.html
docs/interfaces/SQLiteModule.html
docs/interfaces/SQLiteModuleIndexInfo.html
docs/interfaces/SQLitePrepareOptions.html
docs/interfaces/SQLiteVFS.html
docs/types/SQLiteCompatibleType.html
jsconfig.json
package.json
src/FacadeVFS.js
src/VFS.js
src/WebLocksMixin.js
src/asyncify_exports.json
src/asyncify_imports.json
src/examples/AccessHandlePoolVFS.js
src/examples/ArrayAsyncModule.js
src/examples/ArrayModule.js
src/examples/IDBBatchAtomicVFS.js
src/examples/IDBContext.js
src/examples/IDBMinimalVFS.js
src/examples/IDBVersionedVFS.js
src/examples/MemoryAsyncVFS.js
src/examples/MemoryVFS.js
src/examples/OPFSAdaptiveVFS.js
src/examples/OPFSCoopSyncVFS.js
src/examples/OPFSPermutedVFS.js
src/examples/OriginPrivateFileSystemVFS.js
src/examples/README.md
src/examples/WebLocks.js
src/exported_functions.json
src/libadapters.h
src/libadapters.js
src/libauthorizer.c
src/libauthorizer.js
src/libfunction.c
src/libfunction.js
src/libmodule.c
src/libmodule.js
src/libprogress.c
src/libprogress.js
src/libvfs.c
src/libvfs.js
src/main.c
src/sqlite-api.js
src/sqlite-constants.js
src/types/index.d.ts
test/AccessHandlePoolVFS.test.js
test/IDBBatchAtomicVFS.test.js
test/MemoryAsyncVFS.test.js
test/MemoryVFS.test.js
test/OPFSCoopSyncVFS.test.js
test/OPFSPermutedVFS.test.js
test/OriginPrivateVFS.test.js
test/TestContext.js
test/WebLocksMixin.test.js
test/api.test.js
test/api_exec.js
test/api_misc.js
test/api_statements.js
test/callbacks.test.js
test/obsolete/GOOG.js
test/obsolete/IDBBatchAtomicVFS.test.js
test/obsolete/IDBMinimalVFS.test.js
test/obsolete/MemoryAsyncVFS.test.js
test/obsolete/MemoryVFS.test.js
test/obsolete/OPFSProxy.js
test/obsolete/OPFSWorker.js
test/obsolete/OriginPrivateFileSystemVFS.test.js
test/obsolete/VFS.test.js
test/obsolete/VFSTests.js
test/obsolete/WebLocks.test.js
test/obsolete/api-instances.js
test/obsolete/module.test.js
test/obsolete/sqlite-api.test.js
test/obsolete/tag.test.js
test/sql.test.js
test/sql_0001.js
test/sql_0002.js
test/sql_0003.js
test/sql_0004.js
test/sql_0005.js
test/test-worker.js
test/vfs_xAccess.js
test/vfs_xClose.js
test/vfs_xOpen.js
test/vfs_xRead.js
test/vfs_xWrite.js
typedoc.json
web-test-runner.config.mjs
Checkpoint 3: 3d95bee Merge branch 'reconcile/checkpoint-3' into reconcile-rhashimoto-wa-sqlite
Notes:
This was a milestone for us as we wanted to be able to use OPFSAnyContextVFS, so I've marked a checkpoint here.
The patch adds OPFSAnyContextVFS implementation, along with some updates (testing mostly)
Files changed:
README.md
demo/demo-worker.js
demo/file/index.js
demo/hello.html
demo/hello.js
demo/hello/README.md
demo/hello/hello.html
demo/hello/hello.js
demo/hello/index.html
package.json
src/WebLocksMixin.js
src/examples/OPFSAnyContextVFS.js
src/examples/README.md
test/IDBBatchAtomicVFS.test.js
test/OPFSAdaptiveVFS.test.js
test/OPFSAnyContextVFS.test.js
test/api.test.js
test/data/idbv5.json
test/obsolete/GOOG.js
test/obsolete/IDBBatchAtomicVFS.test.js
test/obsolete/IDBMinimalVFS.test.js
test/obsolete/MemoryAsyncVFS.test.js
test/obsolete/MemoryVFS.test.js
test/obsolete/OPFSProxy.js
test/obsolete/OPFSWorker.js
test/obsolete/VFSTests.js
test/obsolete/WebLocks.test.js
test/sql.test.js
test/test-worker.js
yarn.lock
Final: 0699a79 Merge branch 'master' into reconcile-rhashimoto-wa-sqlite
Notes:
3.1.47->3.1.613.45.0->3.50.1libhook.candlibhook.js- replacing.updateHook(previously in libfunction) with.update_hook.commit_hookmethod to the APIFiles changed:
.github/ISSUE_TEMPLATE/-do-not-post-anything-other-than-a-bug-report.md
.github/workflows/ci.yml
Makefile
README.md
demo/contention/contention-worker.js
demo/demo-worker.js
demo/demo.js
demo/file/index.js
demo/file/service-worker.js
demo/file/verifier.js
demo/hello/hello.js
dist/wa-sqlite-async.mjs
dist/wa-sqlite-async.wasm
dist/wa-sqlite-jspi.mjs
dist/wa-sqlite-jspi.wasm
dist/wa-sqlite.mjs
dist/wa-sqlite.wasm
docs/assets/search.js
docs/interfaces/SQLiteAPI.html
package.json
src/FacadeVFS.js
src/WebLocksMixin.js
src/asyncify_imports.json
src/examples/IDBBatchAtomicVFS.js
src/examples/IDBMirrorVFS.js
src/examples/MemoryVFS.js
src/examples/OPFSAdaptiveVFS.js
src/examples/OPFSCoopSyncVFS.js
src/examples/OPFSPermutedVFS.js
src/examples/README.md
src/jspi_exports.json
src/libadapters.h
src/libadapters.js
src/libhook.c
src/libhook.js
src/sqlite-api.js
src/types/globals.d.ts
src/types/index.d.ts
test/IDBMirrorVFS.test.js
test/TestContext.js
test/api.test.js
test/api_statements.js
test/callbacks.test.js
test/sql.test.js
test/test-worker.js
web-test-runner.config.mjs
yarn.lock
Checklist
non-exclusive, royalty-free, irrevocable copyright license to reproduce, prepare
derivative works of, publicly display, sublicense, and distribute this
Contribution and such derivative works.
Contribution contains no content requiring a license from any third party.