Support for /proc/self/exe

[vadimkantorov]

I'm developing an alternative file package system. For ELF and Linux, one can just append a ZIP file to the ELF binary and then open a file descriptor of the process's file on the file system and then unzip library can parse it.

@sbc100 can I do the same in Emscripten? Could I append a ZIP file to the contents of the .wasm file (to allow for easy post-hoc adding some ZIPped data by end-user without the wasm toolchain)? Will it then be parsed okay in the browser or break? Otherwise will #embed work? Does it have any file size limitations? Another thing for embedding a blob in Linux land is using ld -r -b some_file -o some_file.o and then linking it in. I wonder if wasm-ld supports this? Is special file /proc/self/exe or argv[0] supported in terms of opening it for reading?

[sbc100]

#embed should work if/when its supported by clang I think yes. I don't think that requires any special support from emscripten.

For the file_packager we use an assembly file to kind of do what #embed does. See

emscripten/tools/file_packager.py

Lines 285 to 295 in a19f7df

	.section .rodata.{f.c_symbol_name},"",@
	# The name of file
	{f.c_symbol_name}_name:
	.asciz "{dstpath}"
	.size {f.c_symbol_name}_name, {len(dstpath)+1}
	# The size of the file followed by the content itself
	{f.c_symbol_name}:
	.incbin "{srcpath}"
	.size {f.c_symbol_name}, {size}

Does does ld -b do? According the man page -b is for specifying an input format so I'm a little confused. I can imagine add a objcopy or wasm-ld feature that could embed files as data segments, but I'm not sure its necessary.

[vadimkantorov]

Somehow ld does a similar thing to this asm: https://tratt.net/laurie/blog/2022/whats_the_most_portable_way_to_include_binary_blobs_in_an_executable.html

For in-browser context, is it possible somehow to have the whole wasm-image mapped in memory? I'm wondering then for this post-hoc appending to a .wasm file: will it break the WASM parser and how to mmap that last thing in memory (or the whole image "file")

[vadimkantorov]

For the file_packager we use an assembly file

Ah interesting, it feels like file_packager.py got updated recently? Last time I checked it produced a js and a data file... Some my old thoughts for improving file_packager.py: #14385

[sbc100]

Yes, file_packager with --embed-file not always just puts the data directly into the wasm memory. This is very efficient with zero copying of the file data.

[sbc100]

We could probably improve llvm-objcopy to support the flags used in https://tratt.net/laurie/blog/2022/whats_the_most_portable_way_to_include_binary_blobs_in_an_executable.html. Seems like the end result would be the basically the same as what file_packager does with asm today.

[vadimkantorov]

What about wasm-ld? I guess it doesn't support this mode as lld probably does not? (there need to be -b binary though)

[sbc100]

/proc/self/exe is not supported by emscripten.

[sbc100]

Yes, it should be fairly easy to create a file in the virual file system that corresponds to the wasm bytes. You would do something like FS.createDataFile("/proc/self", "exe", <data>).

However, you would then need to write a file format parser in your C/C++ code. Seem less is efficient is many ways that just embedding the data.

Another alternative would be to put the data in Wasm custom sections and then you can use the JS API (WebAssembly.Module.customSections()) to access them. I think llvm-objcopy can already add such custom sections.

[vadimkantorov]

However, you would then need to write a file format parser in your C/C++ code. Seem less is efficient is many ways that just embedding the data.

What format parser do you mean? For embedding ZIP the fun part is that existing unzip libraries can work work as is (they seek to the file end and detect there some ZIP special section and then seek back via discovered negative offset), as long as the browser can accept WASM image with some extra undefined bytes appended. ELF and classical loaders do not mind. Do you know if it works with WASM? Can we take a .wasm file and just cat it with some .zip without breaking WebAszembly.initializeModule (or similar)?

[sbc100]

You would need to attach the zip file as custom section at the end of the wasm file. I think that would make the zip bytes appear at the very end of the file (i.e. there are not end markers to EOF markers that follow the last section).

You probably can't just use cat though (i'm not totally sure, it probably depends on the engine). Safer to use a custom section which means adding a small header (I think llvm-objcopy can do this).

[vadimkantorov]

For adding a section, does it need to have any "trailing" footer after the binary file? Or must its header specify the size in advance? E.g. if we can have the section stub pre-appended to the wasm file and then just append the binary or size + binary should make it super easy for the end-user to append their binary via some simple linux commands without a wasm toolchain...

[sbc100]

There is no trailing footer. But the section header encodes its size as LEB128. I would recommend using llvm-objcopy for this. Trying to do it with cat is likely not going to work. You could also write a small script to do it.. but I would recommend objcopy.