[Mnet-devel] erasure code optimization project

[Bryce Wilcox-O'Hearn]

I forgot to mention one major drawback of LDPCs (already discussed in 
ancient mnet-devel archives): they are not as attack-resistant.  In the 
example you ran, you generated 255 shares, and when trying random 
shares it took 110 of them to regenerate the file.  If instead of 
trying random shares you allowed an enemy to delete some shares of his 
choosing then you might have had to try even more than that.  In fact, 
I'm not even sure that recovery of the file would always be possible.


On 2004, Aug 05, , at 17:39, Lenny G Arbage wrote:
>
>   I'm somewhat agnostic on this point, to tell you the
> truth.  I'd considered the same tradeoff in the design
> of a system similar to Mnet (done for a course I took)
> -- using RSE for smaller files and LDGM for larger. 
> But simplicity is often better than cleverness, even
> when cleverness actually pays off.

I agree.

>   The other curves that neither of these papers
> discuss are the relative performance gains of CPUs vs.
> Networks, and what those curves might imply for
> systems that want to be "built to last."

It is hard to predict, especially if what happens in the future isn't 
"bigger faster computer" but instead "smaller slower computer".  For 
example, there are already cell phones that could (at least in theory) 
run Mnet v0.7, using various cellular transports.  Also maybe rural 
Africa will never get the Internet but instead will get something 
resembling FIDOnet or an ad-hoc routing scheme...

>   I was just browsing through mnetlib/filesystem.  It
> looks like the implementation allows plugging in a new
> encoder/decoder in a fairly clean way (good for you!)
> -- is that assessment accurate?

Yes.

I would be more than happy to just use a Reed-Solomon code for now and 
worry about other stuff later, *except* that the current ones are 
already too slow to be usable.  I think that this is because of the way 
they process the file when encoding (they walk through the entire file 
to generate the first check share, then they walk through the entire 
file again to generate the second check share, etc.), instead of 
because of the computational limitations as discussed in the various 
papers.

So my planned optimization is primarily just to make it so that they 
walk through the file once and generate all of the check shares in 
parallel.

Regards,

Zooko