[mnet-devel] Chunks... what where they for?

[Jim McCoy]

On Thursday, March 6, 2003, at 09:48 AM, icepick at icepick.info wrote:

> I note one of the key things that isn't in the new filesystem spec is 
> the
> lack of Chunks.  What where their purpose, and what's changed so that 
> we
> don't need them anymore.

One of the original goals was to be able to stream data directly from 
the mn datastore to a media player.  This was in the heyday of Napster 
and content distribution for large media files was one of the things we 
were pitching.  The information dispersal algorithm we were using was 
not systematic, so we had to put in a short hack to allow a user to 
start streaming the first chunk while later chunks were being 
downloaded.

Okay, that is not exactly true.  The reason was that if you run the 
Rabin IDA on a very big file you end up with very large shares, by 
chunking the data you could control the size of the resulting shares.  
At the time we did not know about FEC and the advantages that a 
systematic FEC offered.

Okay, that is not true either.  The real reason was that originally MN 
was designed by me to serve as the public shared database for an 
anonymous email system.  This system was similar to penet-style 
remailers rather than the cypherpunk/mixmaster remailers in that you 
had a persistent ID, etc.  One of the design requirements was that I 
needed to be able to do random access on the stored data files.  Chunks 
were the mechanism for allowing this sort of random access.

The real truth was originally answer #3, then answer #2 as the original 
design was first implemented and we were trying to decide whether or 
not to keep the original chunking method, and finally it was a 
combination of answer #2 and answer#1 by the time any of this was ever 
made public.

Jim



-------------------------------------------------------
This SF.net email is sponsored by: Etnus, makers of TotalView, The debugger 
for complex code. Debugging C/C++ programs can leave you feeling lost and 
disoriented. TotalView can help you find your way. Available on major UNIX 
and Linux platforms. Try it free. www.etnus.com
_______________________________________________
mnet-devel mailing list
mnet-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mnet-devel