Embed in non-JVM applications?

[Kinrany]

Have you considered making XTDB embeddable in non-JVM applications?

XTDB looks extremely versatile and a great alternative to monolithic RDBMS. But JVM is a huge limitation for non-JVM apps, it makes it so much harder to use the database than a single static dependency.

[refset]

Hi @Kinrany - great question! It's certainly something we've contemplated and seen others look for previously.

The existing constraints for first-class embedded usage on the JVM essentially relate to serialization and memory transfer costs, as the whole point of being "embedded" is to be as efficient as possible while spreading higher-level query logic over many low-latency lookups (i.e. individual Datalog query / entity calls).

Currently the only supported approach for non-JVM languages is to connect to XT via the HTTP/2 server, which can still happen locally to be reasonably fast (e.g. inside the same Docker container), but it's also the case that HTTP introduces bigger overheads than something like gRPC (which we don't support). This is partly why we are currently working on a new engine for XT that uses Apache Arrow, to remove any JVM-based restrictions and allow for ~ideal polyglot interop: https://github.com/xtdb/core2

See previous discussion relating to gRPC here https://juxt-oss.zulipchat.com/#narrow/stream/194466-xtdb-users/topic/HTTP.20API.20question/near/274173030

You may also be curious to see an example of XT being used from Python via HTTP:
https://github.com/minvws/nl-kat-octopoes - this overall system/project was presented here https://www.youtube.com/watch?v=cIHiSeHvA-A (the presentation is very light on detail about any XT specifics, but they're a friendly team though if you wanted to reach out to get their perspectives on limitations/performance)

[Kinrany]

Thank you!

[willmruzek]

I'd also be interested in embedding XTDB in a non-JVM application. For my use case, I'd like to use XTDB in a suite of React / React Native apps: desktop (Electron/Node.js, Tauri/Rust, or React Native), iOS (React Native and/or Swift) and Android (React Native and/or Kotlin). A web app might also be involved but that could use the typical client/server architecture.

Ideally most of the code would be in Clojure(Script).

I wondered if one could create a GraalVM shared native library that would expose native methods to work as one would in Java, or if there's some other way of embedding XTDB in these scenarios that I'm not aware of.

Excited to see what Core2 has in store. Hoping full-featured EDN Datalog can be included despite the focus on SQL (but could deal with a full-featured SQL if not).

[refset]

Hey @mruzekw thanks for chiming in! Building a Graal Native Image isn't beyond the realms of possibility in the future 🤔 We've done some experimentation in the past, see https://github.com/xtdb/xtdb/search?q=graal (but that was more to sanity check the performance impact, rather than attempting to ship mobile-friendly artifacts)