v2.1.0

[jarohen]

Such a lot to tell you about since our 2.0 release!

As always, see the milestone for the full list of issues closed and PRs merged. Thank you to everyone who's been involved in this release, whether that be by raising issues, helping us repro, helping us benchmark or contributing code - it's massively appreciated 🙏 Particular thanks go out to our clients and Design Partners, who once again have been heavily involved in the direction of XTDB, as well as providing invaluable real-world testing and feedback.

I'd also like to give a special mention to Jacob O'Bryant, who has very kindly contributed an OLTP benchmark based on his Yakread dataset and workload. This has been hugely helpful in guiding our performance work for 2.1 and beyond - working with him, we've already been able to land significant OLTP gains here as a result. Thank you Jacob!

"Multi-DB"

2.1 brings a significant (but still largely backwards-compatible) change to the architecture of XTDB - the introduction of secondary databases!

The database in XT has always been the combination of two core, shared components: a transaction log, and an object-store. This change allows one XTDB node to index and reference multiple tx-logs and object-stores.

Specifically, this decoupling of databases (storage) and clusters (compute) enables a data-mesh architecture - organise your databases around business domains (orders, customers, products), while each application team runs their own XTDB compute cluster. Teams can attach secondary databases to access shared domain data, aligning your data model with your organization's structure while keeping compute independent.

Queries can span multiple databases, enabling powerful cross-domain analytics and insights:

-- attach the secondary databases
ATTACH DATABASE user_preferences WITH $$
  log: !Kafka
    cluster: 'my-kafka'
    topic: 'xtdb.user-preferences'

  storage: !S3
    bucket: 'my-bucket'
    path: 'user-preferences'
$$

-- query across it in a single query - what notifications to send?

FROM orders o 
  JOIN users_preferences.prefs up -- use db_name.table_name
    ON o.user_id = up._id
WHERE o.created_at > CURRENT_DATE - INTERVAL 'P1D'
SELECT o._id, up.notification_settings

We've even made a small scale-factor TPC-H data-set available for you to play with using our 'Play' UI

For more information, and how to get started attaching secondary databases, see 'Databases in XTDB'.

We're really keen to see what you build with this - we think it's a really powerful way to decouple your data and applications.

This has meant a couple of minor breaking configuration changes - see:

• Log configuration changelog
• Kafka configuration changelog
• Storage configuration changelog
• EDN configuration changelog

Additionally, we've made some changes to repeatable queries to support multiple databases:

• WATERMARK -> AWAIT_TOKEN - see 'Transaction consistency'
• We've added SNAPSHOT_TOKEN in addition to SNAPSHOT_TIME - we'd recommend using the former for repeatable queries where possible.

Our current roadmap for this feature is as follows (usual 'subject to change' caveat):

• Multi-partition tx-logs for secondary databases - horizontal write scaling.
• Read-only secondaries - just listening in.
• Removing the requirement to have XT-specific transaction logs - bring your own topics

Client driver support

We've been hard at work improving the support for XTDB through language-native PostgreSQL drivers - we now support ten languages: C/C++, C#, Clojure, Elixir, Go, Java, Kotlin, Node.js, PHP, Python and Ruby.

See 'Language Drivers' for the up-to-date list, and also our 'driver-examples' repository.

OpenID Connect (OIDC) authentication

2.1 adds support for OpenID Connect (OIDC) authentication to XTDB's built-in authentication system - you can now configure XTDB to authenticate users via an OIDC provider, such as Keycloak, Auth0, or Okta. This is likely to become the primary authentication method in XTDB going forward, so that users can leverage existing identity infrastructure rather than attempting to mirror and maintain roles in XTDB.

We'll be adding support for more OIDC authentication methods, as well as OIDC-based authorization/role-mapping in future releases.

See OIDC for more information and how to get started.

Observability

We've heard your feedback regarding observability in XTDB loud and clear, and so 2.1 brings a number of improvements here.

XTDB now supports OpenTelemetry-backed tracing for query introspection and performance analysis. Traces are sent via the OTLP (OpenTelemetry Protocol) HTTP endpoint to your tracing backend (e.g., Grafana Tempo, Jaeger, etc).

tracer:
  # -- required

  # Enable OpenTelemetry tracing.
  enabled: true

  # OTLP HTTP endpoint for sending traces.
  # (Can be set as an !Env value)
  endpoint: "http://localhost:4318/v1/traces"

  # -- optional

  # Service name identifier for traces.
  # (Can be set as an !Env value)
  # serviceName: "xtdb"

Tracing provides detailed introspection into query execution, including:

• Per-query execution times for performance analysis.
• Information on which queries were executed, available through the xtdb.query span attributes.
• Lower-level operation timings, revealing how time is distributed across individual query operations.

See the tracing guide for details on how to get started.

Within the database itself, we've made the EXPLAIN plans much prettier, and added EXPLAIN ANALYZE, which provides detailed timing information for each step of the query plan:

EXPLAIN ANALYZE
SELECT o.o_orderpriority, COUNT(*) AS order_count
FROM orders AS o
WHERE o.o_orderdate >= DATE '1993-07-01'
  AND o.o_orderdate < DATE '1993-07-01' + INTERVAL '3' MONTH
  AND EXISTS (
    FROM lineitem AS l
    WHERE l.l_orderkey = o.o_orderkey
      AND l.l_commitdate < l.l_receiptdate
  )
ORDER BY o.o_orderpriority;

--          depth         |    op     |  total_time   | time_to_first_block | block_count | row_count
--  ----------------------+-----------+---------------+---------------------+-------------+-----------
--   ->                   | project   | "PT0.757754S" | "PT0.757717S"       |           1 |         5
--     ->                 | order-by  | "PT0.757752S" | "PT0.757713S"       |           1 |         5
--       ->               | project   | "PT0.757475S" | "PT0.757423S"       |           1 |         5
--         ->             | group-by  | "PT0.757474S" | "PT0.75742S"        |           1 |         5
--           ->           | project   | "PT0.757327S" | "PT0.670638S"       |         256 |      2539
--             ->         | semi-join | "PT0.757259S" | "PT0.670606S"       |         256 |      2539
--               ->       | project   | "PT0.656491S" | "PT0.000828S"       |        1024 |    189646
--                 ->     | rename    | "PT0.656324S" | "PT0.000816S"       |        1024 |    189646
--                   ->   | select    | "PT0.655886S" | "PT0.000792S"       |        1024 |    189646
--                     -> | scan      | "PT0.65443S"  | "PT0.00054S"        |        1024 |    299814
--               ->       | rename    | "PT0.087767S" | "PT0.000705S"       |         256 |      2765
--                 ->     | scan      | "PT0.087687S" | "PT0.000694S"       |         256 |      2765

Stability/Performance

All that said, the majority of our work in 2.1 has been focused on the stability and performance of XT. As is so often the case with these things, the progress here has been very much incremental - too many small changes to document here (see the milestone for the full list), but with a cumulatively significant effect.

Particularly, we've recently been focusing on a 'deterministic simulation testing' framework, which has already unearthed a handful of otherwise-hard-to-repro concurrency bugs/race conditions. XT already has an advantage here in that the vast majority of critical code is single-threaded, so we don't see the same class of locking/concurrency issues as other databases, but this framework has still been very helpful in improving confidence in XT's correctness under load.

We've also seen some impressive numbers in our internal benchmarks for both OLAP and OLTP queries - look out for future blog posts with more details!

Specifically, we've implemented disk-based joins using a 'grace hash join' algorithm. Previously, XTDB would hold entire join results in memory - fine for most workloads, but not so great when you're joining very large relations.

In 2.1, if a join grows too large to fit in memory, we'll automatically spill partitions to disk and continue processing. This means you can now run those massive analytical queries without worrying about OOM errors.

Other breaking changes

• The XTDB CLI has been split out into multiple top-level commands (à la Git)
• Having been deprecated in the 2.0 release, the HTTP server has now been removed.

If you've any questions or thoughts, please do get in touch - we'd love to hear from you!

• Discord
• hello@xtdb.com

James, Jeremy and the XTDB team

---

This discussion was created from the release v2.1.0.