Building (for WASM!) inside a flake.nix

[c4lliope]

Hello all,

I am eager to begin using gitoxide embedded in a .wasm module, for a client-side app similar to https://radicle.xyz

As phase one, I need to compile gitoxide somehow on NixOS; this means adding a flake.nix to the code and declaring all dependencies. Here is flake.nix:

{ inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = { nixpkgs, ... }: let
  arch = "x86_64-linux";
  pkgs = import nixpkgs { system = arch; };
in with pkgs; {
  packages.${arch} = { default = deno; };
  devShells.${arch}.default = mkShell {
    packages = [ cargo openssl.dev pkg-config gcc sqlite.dev ];
}; }; }

When I build, the error I see is based on libsqlite3-sys

error: failed to run custom build command for `libsqlite3-sys v0.35.0`

Caused by:
  process didn't exit successfully: `/home/calliope/disc/nvme0n1p6/code/gitoxidelabs/gitoxide/target/debug/build/libsqlite3-sys-3e189fef835d58fc/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=LIBSQLITE3_SYS_USE_PKG_CONFIG
  cargo:include=/home/calliope/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/libsqlite3-sys-0.35.0/sqlite3
  cargo:rerun-if-changed=sqlite3/sqlite3.c
  cargo:rerun-if-changed=sqlite3/wasm32-wasi-vfs.c
  cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER
  cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH
  cargo:rerun-if-env-changed=SQLITE_MAX_COLUMN
  cargo:rerun-if-env-changed=LIBSQLITE3_FLAGS
  OUT_DIR = Some(/home/calliope/disc/nvme0n1p6/code/gitoxidelabs/gitoxide/target/debug/build/libsqlite3-sys-073edc65d80d3faa/out)
  OPT_LEVEL = Some(0)
  TARGET = Some(x86_64-unknown-linux-gnu)
  HOST = Some(x86_64-unknown-linux-gnu)
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
  CC_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
  CC_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CC
  HOST_CC = None
  cargo:rerun-if-env-changed=CC
  CC = Some(gcc)
  cargo:rerun-if-env-changed=CC_KNOWN_WRAPPER_CUSTOM
  CC_KNOWN_WRAPPER_CUSTOM = None
  RUSTC_WRAPPER = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(true)
  CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
  CFLAGS_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
  CFLAGS_x86_64-unknown-linux-gnu = None
  CARGO_ENCODED_RUSTFLAGS = Some()

Nothing I search for has any promising leads; I seem to be missing some environment variable or dependency package. As this is my first production rust / wasm build, I am lost for direction and hope someone can lend their experience. Much obliged, as usual. I hope I can lend a hand on issues soon.

[c4lliope]

Seems like I found a guide that addresses this precisely in their onboarding docs, using openssl and libsqlite3-sys as the "nontrivial dependency" examples.

https://github.com/rustshop/flakebox/blob/master/docs/building-new-project.md#extra-dependencies

I may open a PR if I get flakebox running for gitoxide; this could help make builds more reproducible, and cross-compiling rather simple.

[c4lliope]

Nope. rustshop/flakebox#209

I'll read up on Crane though, as they recommend. Seems like a more basic building block.

[Byron]

It's great that you could figure something out as I have no idea of how anything would work on nix.
However, as an important pointer: I'd start compiling only the gix crate, not gitoxide, and configure its features which may pull in additional dependencies. OpenSSL is only pulled in if cloning from HTTPS is required, and there are ways to go without it as well while supporting HTTPS.

Also, sqlite isn't anything that gix (the crate) needs.

[c4lliope]

@Byron - perhaps you could help me imagine what gix or gitoxide may look like in client-side WASM. Here is my inspiration:
https://wasm-git.petersalomonsen.com

There's a branch of my codebase where I did compile and use the libgit2.wasm he made, but all in all it seemed clunky, and I ended up blocked by CORS errors. I hope to use this on my blogging engine, https://operand.online - where I currently call to libgit2 from the server-side Elixir bindings.

Possible gitoxide approaches:

• Using gitoxide on the server, package necessary files into a stream for client consumption. Easy, could do this by the end of the week. This may be good for a single-author blog, where the most updated copy is displayed. This is less good for an offline wiki where people are skimming through history.

• Using nix to manage the compiler target and flags, compile gitoxide.wasm and drop it onto the client page.

  • Harness CORS in the git server (https://github.com/charmbracelet/soft-serve)
  • directly clone the codebase into some wasm-based filesystem emulator
  • Move the markdown rendering to the client, add client-side changelogs, diffs, and so on
  • apply for sponsorship from https://radicle.xyz/work, and add codemirror and tree-sitter layers
  • apply for sponsorship from https://inkandswitch.com and bring in https://automerge.org
  • apply for sponsorship from http://archive.org and begin scraping https://wikiapiary.com/wiki/Special:AllPages, using https://wikiapiary.com/wiki/Special:Export
  • buy more machines
  • bring large chunks of the internet along on my first oceanic sailing expedition

(a) is where we are now, and (b) is compelling enough to make me learn Rust. Lemme go speak to Radicle really quick while you chime in.

See also: #2212

[Byron]

Thus far the WASM story has been spotty at best, but this issue has all the details and history for it.

I don't think the relevant code in gix-* crates is ready for WASM, and clones would need some work to compile to WASM at all. Object access is a similar story.

Needs work - I let you catchup on the issue and maybe from there you get an idea of what's needed to compile it.