Fix bug with orderBy on duplicate values

[kevin-dp]

There was a bug in orderBy that could cause duplicate values to be skipped. For example, if we had this data: 1, 2, 3, 3, 3, 4, 5. And we would request LIMIT 3 then we would load 1, 2, 3 and remember that the max value we've loaded is 3. Then on the next load request we would ask for data > 3 which would skip over the duplicate values 3 and immediately return 4, 5 thereby missing the two other occurences of the value 3.

This PR modifies requestLimitedSnapshot such that it loads 2 subsets: all the occurences of maxValue and all data > maxValue. We can't do it in 1 query which is >= maxValue LIMIT N because the duplicate values would count towards the limit but we may already have loaded some of them. e.g. >= 3 LIMIT 3 would give 3, 3, 3 but we had already seen the first occurence of 3, so that's wrong because we actually expect to get 3, 3, 4 (the next two occurences of 3 and the value 4).

TODO:

• I found a bug with the liveQueryCollection which always returns true from setWindow instead of returning the promise of the underlying collection. Need to fix this in a follow up PR and then we can update the unit tests here to use await instead of sleep().

[changeset-bot]

🦋 Changeset detected

Latest commit: ed77bbf

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@tanstack/db	Patch
@tanstack/angular-db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/rxdb-db-collection	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch
todos	Patch
@tanstack/db-example-react-todo	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

More templates

• todos
• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@713

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@713

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@713

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@713

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@713

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@713

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@713

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@713

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@713

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@713

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@713

commit: ed77bbf

[github-actions]

Size Change: +326 B (+0.39%)

Total Size: 84.6 kB

Filename	Size	Change
./packages/db/dist/esm/collection/subscription.js	2.67 kB	+475 B (+21.59%)	🚨
./packages/db/dist/esm/query/compiler/order-by.js	1.42 kB	-8 B (-0.56%)
./packages/db/dist/esm/query/live/collection-subscriber.js	1.97 kB	-141 B (-6.69%)	✅

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/collection/change-events.js	1.63 kB
./packages/db/dist/esm/collection/changes.js	1.01 kB
./packages/db/dist/esm/collection/events.js	413 B
./packages/db/dist/esm/collection/index.js	3.23 kB
./packages/db/dist/esm/collection/indexes.js	1.16 kB
./packages/db/dist/esm/collection/lifecycle.js	1.8 kB
./packages/db/dist/esm/collection/mutations.js	2.52 kB
./packages/db/dist/esm/collection/state.js	3.79 kB
./packages/db/dist/esm/collection/sync.js	2.2 kB
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/errors.js	3.48 kB
./packages/db/dist/esm/event-emitter.js	798 B
./packages/db/dist/esm/index.js	1.62 kB
./packages/db/dist/esm/indexes/auto-index.js	794 B
./packages/db/dist/esm/indexes/base-index.js	835 B
./packages/db/dist/esm/indexes/btree-index.js	2 kB
./packages/db/dist/esm/indexes/lazy-index.js	1.21 kB
./packages/db/dist/esm/indexes/reverse-index.js	577 B
./packages/db/dist/esm/local-only.js	967 B
./packages/db/dist/esm/local-storage.js	2.4 kB
./packages/db/dist/esm/optimistic-action.js	294 B
./packages/db/dist/esm/proxy.js	3.86 kB
./packages/db/dist/esm/query/builder/functions.js	615 B
./packages/db/dist/esm/query/builder/index.js	4.04 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	938 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.55 kB
./packages/db/dist/esm/query/compiler/expressions.js	760 B
./packages/db/dist/esm/query/compiler/group-by.js	2.04 kB
./packages/db/dist/esm/query/compiler/index.js	2.21 kB
./packages/db/dist/esm/query/compiler/joins.js	2.65 kB
./packages/db/dist/esm/query/compiler/select.js	1.28 kB
./packages/db/dist/esm/query/ir.js	785 B
./packages/db/dist/esm/query/live-query-collection.js	404 B
./packages/db/dist/esm/query/live/collection-config-builder.js	5.54 kB
./packages/db/dist/esm/query/live/collection-registry.js	233 B
./packages/db/dist/esm/query/optimizer.js	3.26 kB
./packages/db/dist/esm/scheduler.js	1.29 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/transactions.js	3.05 kB
./packages/db/dist/esm/utils.js	1.01 kB
./packages/db/dist/esm/utils/browser-polyfills.js	365 B
./packages/db/dist/esm/utils/btree.js	6.01 kB
./packages/db/dist/esm/utils/comparison.js	754 B
./packages/db/dist/esm/utils/index-optimization.js	1.73 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 2.89 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	168 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js	1.41 kB
./packages/react-db/dist/esm/useLiveQuery.js	1.31 kB

_{compressed-size-action::react-db-package-size}

[samwillis]

@kevin-dp I think this is the bug we were discussing before I went away and we talked about having a composite custom: [maxValue, lastPK] and always orderbying by the PK after the requested field?

We need to enable using indexes to order by [fieldOne, fieldTwo] anyway, so having it always append the PK ([fieldOne, fieldTwo, PK]) if it is not already used makes sense. The cursor is always the final value of [fieldOne, fieldTwo, ..., PK]then.

This will then be correctly pushed down to the sync layer as a stable cursor for paging through the data on the server.