[mnet-devel] FEC, blocksizes, chunking

[Hauke Johannknecht]

Hi.

this i my try to sum up yesterdays irc conversation about the
FEC/blocksize/chunking issue.

Problem: FEC eats up CPU when using too many blocks.

so, currently the number of blocks is limited to 256 (by
switching FEC to 8bit mode) and growing the blocks to fit this
restriction.

that means, at the current expansion rate of 4, a file can be up
to 16MB with 256kb blocks. that should cover the majority of
files currently on mnet.

with bigger files, we get bigger blocks. with blocks at 16MB, we
get max 1GB filesize.

zooko: could you please extend that hardcoded 4M msg-size limit you
       found to be 20M or so?

i asserted that we will start to see problems with interrupted
blocktransfers at such big blocksizes. if this problem arises, it
can be fixed by adding partial blocktransfers (blocklets). the
blocklet table could be exchanged between downloader and
blockserver when trying to get-block, triggered by a user-supplied
sizelimit.

since we have plenty of space in or inode blocks with max 256 blocks
anyways, we could also fill up the inode blocks with
blocklet-tables if we publish bigger files.

but thats all "when we see problems with big blocks" extensions.

the only change to the current situation i want to make is,
choose the size of the inode block in the same way as choosing
blocksize for the datablocks (round up to next power of two and
pad). (at the moment the inode blocks are same-size as
datablocks, which makes a 1MB inode block about 99.6% padding.)

and i will send this now, since else i have to repeat it four
times on irc. ;)

Gruss,
  Hauke

-- 
Hauke Johannknecht        Berlin / Germany        HJ422-RIPE
Use PGP ! -> lynx -dump http://www.ash.de/ash.asc | pgp -kaf




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
mnet-devel mailing list
mnet-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mnet-devel