WebSeed specifications

[Jorropo]

This is a meta issue discussing required features to enable webseed like features for the IPFS network.

A webseed is qualified as a server:

• Server that implements a version of the HTTP protocol. Or protocols with equivalent GET semantics. (like FTP)
• Is able to serve static files.
• Supports RFC 7233 bytes HTTP Range request on such files. Or protocols with equivalent ranging semantics. (like FTP)
• Does not run IPFS aware pieces of code.
• And finally can be used by IPFS clients to download content without compromising on the IPFS features (extensible content addressed incremental verification).

The endgame here would be to upload some file to such server which would contain both the merkle dag cryptographic proofs and an index which would allow clients to implement O(log n) seeking and random read behaviors.
Here for unixfs we have 2½ usecases which are useful to separate:

• 1. IPFS proofs and index packed together with the leaves node. This mode would be blind to content of the blocks and could work with any IPFS application.
  Potential applications target IPFS native applications which want to leverage extensive HTTP compatible storage services which do not understand IPFS.
• 2. IPFS proofs and index stored separately from the de-serialized leaves of the tree. This is usefull for applications which also want to serve their content over non IPFS means, this allows to reuse existing non IPFS architectures and have IPFS clients download the same files as non ipfs clients for the raw-leaves of the file.
• 2½. Similar as point 2 however it allows for using non webseeds (servers that run IPFS aware code) for downloading the proofs.
  This also include usecases where the proofs are packed into an archive and shared OOB (see .torrent files for an example).

I can think of at least one other possible implementation (one plain file per block) but I am not including it here due to dubious value, could be added if this interest peoples, this could also be implemented without updating the spec using IPIP402 and very minimal header rewriting.

Milestones

2½

• IPIP-0445: Option to Skip Raw Blocks in Gateway Responses #445
• Figure out signaling and expected semantics of webseeds over content routers.

2

• TODO

1

• TODO

[lidel]

If we need a test data for proper webseeds (2), Wikipedia snapshots at https://download.kiwix.org/zim/wikipedia/ provide both good use case and variety (biggest archive is 96GiB).

The web-based ZIM reader described in ipfs/distributed-wikipedia-mirror#140 could even utilize the MVP from (2½) – fetch proof from one gateway, and then range-requests for deserialized raw bytes from others.