Adding GRPC or node mini-protocol support to `cardano-wasm` in a JS independent way

[palas]

Adding GRPC or node mini-protocol support to cardano-wasm in a JS independent way #999

Problem description

The JavaScript library already facilitates obtaining information such as protocol parameters and UTxO data from the node via web‑gRPC (when using the browser) or gRPC (when using Node.js).

There is demand for a .wasm module that is JavaScript‑independent, so it can be used from other languages. This is being addressed by the PR #995.

However, supporting a gRPC (or any other) connection to the node is problematic because the wasm32-wasi compiler we are currently using only targets wasi1-preview, which does not support the creation of sockets.

Goal summary

In summary, our solution would ideally be:

• Portable
  • Compiling our code to WASM achieves this
• Simple to use
  • Not requiring the user to manually allocate memory helps
  • Incorporating node querying in the API would help
• Feasible to develop realistically (in terms of required effort)
  • Not having to support many languages helps with this

Ideas explored

WAGI (or a CGI-style wrapper for WASM)

See wagi. The idea is treating the WASM module as a CGI program (HTTP requests are turned into environment variables and stdin, and the module returns a response via stdout). This makes the WASM library a portable Web Service.

Advantages

• No need for the client to manage memory manually. Information is piped, so no need to allocate space for the parameters of functions which
• HTTP and CGI are pretty standard and simple

Limitations

• It doesn't really solve the issue with outbound connections. There is a solution: wasi-experimental-http. But:
  • The repo is archived.
  • It is tagged as experimental and temporary.
  • It is not covered by CGI, so it is not standard.
  • It is my understanding that it depends on the runtime supporting this feature, which reduces portability.

WASI1 (wasi_snapshot_preview1)

See wasi1.

Advantages

• We are already doing this and it works
• It is fully supported by the wasm32-wasi-ghc compiler

Limitations

• It doesn't support sockets (we cannot connect to node, nor grpc, nor web-grpc, nor Blockfrost...)
• It doesn't support passing arbitrary strings as parameters (AFAIK), so it requires us to make the API so that the user needs to manually allocate space, set the memory, and free afterwards.

WASIX

See wasix. An extended version of WASI1 that includes socket support and other things.

Advantages

• Socket support would theoretically allow us to connect to node directly, through grpc, through web-grpc, and to Blockfrost... We could even make the wasm module be a web server directly and combine it with the WAGI idea.
• It may be quite straightforward to get it to work with the current set-up, because it is backwards compatible with wasi1-preview (I think), so it may just be a matter of writing some ccall imports.

Limitations

• It doesn't support passing arbitrary strings as parameters (AFAIK), so it requires us to make the API so that the user needs to manually allocate space, set the memory, and free afterwards.
• It is probably less supported than wasi1-preview.
• It may be replaced by wasi2 eventually.

WASI2

See wasi2-preview. The evolution of WASI2, which also includes socket support and other things. It is foreseeable that this will eventually become the standard.

As of now, because gRPC can generate client stubs for many languages, it would be theoretically possible to build the gRPC client in a language that can compile to WASI2, then link that client with the .wasm generated by Haskell after converting it to WASI2. There exist tools for converting WASI1 to WASI2, but they tend to fail when the module has unresolved imports.

Also as of now, I've tried and I couldn't get this to work. The most convenient way would be to have ghc generate WASI2. I would hope this eventually happens, but at the moment it doesn't, and getting to that stage would be a considerable amount of work.

Advantages

• It supports passing strings directly (which wouldn't require manual memory allocation)
• It supports sockets
• It may likely become the standard

Disadvantage

• It is not currently supported by ghc

Expose cardano-api as a HTTP server

We can just make a Haskell executable that is an HTTP server that exposes the cardano-api as a REST API, for example. Since HTTP and JSON are very standard that would be very portable.

What wouldn't be so portable would be the actual server, which would require being compiled to the target architecture, and doing this with Haskell is not trivial. Also it makes the architecture of

We could provide binaries to make this easy or even incorportate it as part of the node. At this point the portability would be the same as that of the node.

Advantages

• Moderate ease of use (just REST API)
• Compatibility with virtually every language
• Ease of development (could be done purely in Haskell)

Disadvantage

• Lower portability of the server
• Lower coupling between the api and the host application
• Potential worse latency for queries (HTTP has higher latency than RPC)
• More complicated deployment of the host application + library

Write library in each host language that uses WASM module under-the-hood

Alternatively, we may use existing libraries that are already in many languages and platforms, or decide to write code in many languages and platforms ourselves, but this is much more work to maintain, so it would be a direct trade off between portability and effort, but it would make the libraries potentially really easy to use.

Advantages

• Potential very high ease of use

Disadvantages

• Either high effort or low portability

Conclusion

The added value of this effort would be to give users a convenient way to connect to a node and retrieve the data required by the cardano-wasm API as well as facilitate the use of cardano-wasm API itself. That said, the same data gathering functionality is already covered by the gRPC client that can be generated for most programming languages, so that feature could be considered not essential.

We can revisit this possibility in the future. It may become trivial if GHC adds support for WASI2 or WASIX, or if new tooling emerges. We may want to try WASIX straight-away if it is not too difficult to get it to work.