Improve support for "random-access" assets (e.g., ZIP files)

[andriyDev]

Currently, Bevy assumes that an asset loader loads a whole asset. This means something like a ZIP asset loader would load literally all assets in the ZIP, even if only a single asset in the ZIP is needed. Ideally, it should be possible to load singular assets from something like a ZIP file, rather than the whole ZIP file. I call these "random-access" assets, as the common workflow is probably not to load the whole thing and instead to load the individual pieces that are needed.

[andriyDev]

One possible solution is to instead make random-access assets be asset sources instead of asset loaders. This makes sense as asset sources are already good at loading individual assets, and are more designed for loading arbitrary assets. This at least works for assets like ZIP files which already store "file-like" things.

An existing blocker is #10162, which discusses the possibility of dynamic asset sources. That way you can register an asset source at runtime (like a ZIP) and do that for any other ZIP files.

This could also work with asset processing, since the file is an asset source, so its assets could be processed (though in some situations this is probably undesirable which we should make configurable anyway due to #12483).

[andriyDev]

Another alternative is to directly support random-access assets through a new kind of loader. This loader could have 2 methods: init_index(reader) -> Index, and load_subasset(reader, index, subasset_name). init_index would do something like read the index of a ZIP file, allowing it to cache the index to avoid repeatedly reading the entire ZIP file. Then load_subasset could use the index to directly read the one asset it needs.

I'm not super excited about this idea though. First, this adds a bunch of new complexity to the asset system (which is already brimming with undue complexity). Second, on the web, we probably don't want to keep reading and rereading the ZIP file, since that incurs a download of the whole file for each read. So we would prefer to "cache" the reader in some way even across multiple subasset loads (maybe we can do this by storing the reader with the index? I'm not sure). This also won't play well with asset processing, since the "unit" of asset processing is a whole file, not subassets.

One thing this does address though is non-file-like assets. ZIP files are the trivial case, where every "subasset" is a file that can be loaded normally - this can be served easily with an asset source. On the other hand are situations where the subassets are not generic files. For example GLTF subassets are not files in their own right. So trying to make this into an asset source would be difficult. You'd need to somehow encode any metadata from the GLTF JSON index into the bytes returned by the asset reader, and you'd need custom asset loaders to parse that data into the actual subassets. While the GLTF example is a little silly, here's a user who wanted to do something like this for Scratch project files https://discord.com/channels/691052431525675048/749332104487108618/1426231519332012266.

Practically, these situations are currently already bad, due to the fact that loaders unconditionally load the whole asset, but at least after loading is done, all unused assets are dropped.

[hukasu]

On the definition for GRF, which is an asset that I use on my hobby project they use the term virtual filesystem, which I think represents better what they are than random access asset

https://ragnarokresearchlab.github.io/file-formats/grf/