[mnet-devel] more fast dumpers

[Zooko]

"fastdumper" writes the serialized dict directly to the file, and writes it in 
minimal format.

"fastdumpermencode" writes the serialized dict directly to the file, and 
writes it in mencode format, so that we can use mdecode for the undumper.

Each of these has a C and a Python implementation.

"{c,py}stringdictdump{s,sz}" returns a string containing the serialized dict 
and then the caller writes the string to a file.  I don't know why this is 
faster.

It uses libzstr, so you can't run it without first installing libzstr and 
setting a "LIBZSTR_DIR" env var.  I think I'll rewrite it so it doesn't use 
libzstr, which it hardly needs.  Then I'll make "cstringdictdumpsmencode()" 
which should be nearly as fast as cstringdictdumpsz(), but it won't use 
libzstr and it will write in mencode format.

--Z

HACK pion:~/playground/mnet/mnet_new$ time /usr/local/stow/python2.3-O2-athlon-xp/bin/python -OO ~/playground/mnet/mnet_new/mnetlib/scripts/benchpickle.py
NOTE: c_mencode found, using accelerated C version of mencode
NOTE: c_fastdumper found, using accelerated C version of fastdumper
mencode_to_file
N:   16384, time: ave:    0.29,   5th-best:    0.29,   5th-worst:    0.31 (of 128), ave rate:    55658
N:   65536, time: ave:    1.20,   4th-best:    1.18,   4th-worst:    1.21 (of  50), ave rate:    54790
N:  262144, time: ave:    4.86,   3th-best:    4.84,   3th-worst:    4.88 (of  13), ave rate:    53944
N: 1048576, time: ave:   19.83,   2th-best:   19.83,   2th-worst:   19.83 (of   3), ave rate:    52881
cfastdumpermencode
N:   16384, time: ave:    0.06,   5th-best:    0.06,   5th-worst:    0.06 (of 128), ave rate:   291535
N:   65536, time: ave:    0.23,   5th-best:    0.22,   5th-worst:    0.25 (of 128), ave rate:   288122
N:  262144, time: ave:    0.91,   5th-best:    0.90,   5th-worst:    0.93 (of  66), ave rate:   287865
N: 1048576, time: ave:    3.73,   3th-best:    3.69,   3th-worst:    3.73 (of  17), ave rate:   280825
cfastdumper
N:   16384, time: ave:    0.03,   5th-best:    0.03,   5th-worst:    0.03 (of 128), ave rate:   482567
N:   65536, time: ave:    0.14,   5th-best:    0.13,   5th-worst:    0.15 (of 128), ave rate:   480906
N:  262144, time: ave:    0.55,   5th-best:    0.54,   5th-worst:    0.57 (of 110), ave rate:   476558
N: 1048576, time: ave:    2.25,   4th-best:    2.24,   4th-worst:    2.26 (of  27), ave rate:   465265
pyfastdumpermencode
N:   16384, time: ave:    0.16,   5th-best:    0.16,   5th-worst:    0.18 (of 128), ave rate:   100421
N:   65536, time: ave:    0.66,   5th-best:    0.65,   5th-worst:    0.68 (of  92), ave rate:    99944
N:  262144, time: ave:    2.62,   4th-best:    2.61,   4th-worst:    2.63 (of  23), ave rate:    99981
N: 1048576, time: ave:   10.54,   2th-best:   10.51,   2th-worst:   10.56 (of   6), ave rate:    99479
pyfastdumper
N:   16384, time: ave:    0.09,   5th-best:    0.09,   5th-worst:    0.10 (of 128), ave rate:   188895
N:   65536, time: ave:    0.35,   5th-best:    0.34,   5th-worst:    0.37 (of 128), ave rate:   188122
N:  262144, time: ave:    1.40,   4th-best:    1.38,   4th-worst:    1.42 (of  43), ave rate:   187564
N: 1048576, time: ave:    5.64,   3th-best:    5.62,   3th-worst:    5.65 (of  11), ave rate:   186025
cstringdictdumpsz
N:   16384, time: ave:    0.01,   5th-best:    0.01,   5th-worst:    0.01 (of 128), ave rate:  1307136
N:   65536, time: ave:    0.05,   5th-best:    0.05,   5th-worst:    0.06 (of 128), ave rate:  1252902
N:  262144, time: ave:    0.21,   5th-best:    0.21,   5th-worst:    0.23 (of 128), ave rate:  1224031
N: 1048576, time: ave:    0.90,   5th-best:    0.89,   5th-worst:    0.92 (of  67), ave rate:  1162015
pystringdictdumps
N:   16384, time: ave:    0.08,   5th-best:    0.08,   5th-worst:    0.09 (of 128), ave rate:   208687
N:   65536, time: ave:    0.31,   5th-best:    0.30,   5th-worst:    0.33 (of 128), ave rate:   208837
N:  262144, time: ave:    1.25,   4th-best:    1.23,   4th-worst:    1.27 (of  49), ave rate:   209912
N: 1048576, time: ave:    5.02,   3th-best:    5.00,   3th-worst:    5.04 (of  12), ave rate:   208722
1183.281



-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
mnet-devel mailing list
mnet-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mnet-devel