[mnet-devel] FEC parameters, backwards compatibility

[Zooko]

Someone asked on IRC [1] if the GF(2^16) mode of Rizzo's FEC was a good choice.

First of all, I question whether a large number of erasure code shares actually 
makes any difference.  It probably makes no difference to file robustness.

I.e., you could do 400 out of 1600, or you could just do 4 out of 16 and make 
100 copies of each share.  It certainly makes no difference in the simple model 
where you have a certain probability of recovering each copy, and it eases 
repair.  That is: if you 1600 different shares with one copy each, and you lose 
a copy, now you have 1599 different shares, any 400 of which are necessary and 
sufficient to reconstruct.  To replace the lost share you would have to perform 
a whole reconstruction.  If you had 100 copies each of 16 different shares, and 
you lost a copy, you could make a new replacement copy from one of the other 
99 copies of that share.

Nonetheless, I intend to do large numbers of shares in my mythical 
"decentralized filestore format v2".  The reason is that it makes a difference 
to block size -- we can keep block sizes small (like 64 KB or 256 KB) even if 
the file is huge (like 16 GB).  The alternative is to allow blocks to be big 
(like 1 GB), which means you have to have another layer of design to deal with 
big blocks (streaming them on download, for one thing).

So I intend to do *both* GF(2^8) and GF(2^16), where we use the fastest one of 
the two that is big enough to handle the current file.  My envisioned parameters 
make it so that "Zooko's decentralized filestore format" can handle 20 GB files 
before running out of FEC GF() space.  When this limit begins to chafe, I might 
add support for GF(2^32).

Another topic I saw on IRC was backwards compatibility.  I'm pretty sure we can 
achieve smooth backwards compatibility by denoting the version somewhere early 
in the metadata.  I've worked out concrete formats that include this feature.

I'm not sure I want to actually go to the effort to support the old-style code 
though!  If someone else who cares more about backwards compatibility than I do 
wants to manage the "roll-over" process then we can surely do it.

Regards,

Zooko

[1] server: irc.freenode.net, channel #mnet



-------------------------------------------------------
This SF.NET email is sponsored by:
SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
http://www.vasoftware.com
_______________________________________________
mnet-devel mailing list
mnet-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mnet-devel