[mnet-devel] Ickle me, pickle me, tickle me too...

[Jim McCoy]

On Friday, August 8, 2003, at 05:23 AM, Zooko wrote:

>
>> [...] Maybe it's better
>> to write the checkpoint to a new file, fsync it, and then rename it
>> over the old one.
>
> That would be guarantee that either the old or the new version is in 
> place
> (assuming certain guarantees from the underlying filesystem), right?  
> That
> *is* better.
>
> But we still have the possibility that it is the old version, which 
> can cause
> problems.

I guess the real question is whether or not having no checkpoint file 
worse than having an old checkpoint file. If you have an old checkpoint 
file you will need to update it, but this is going to happen eventually 
at some point when a re-index is requested just check the current index 
(this should happen infrequently, but needs to happen in case things 
have gone horribly wrong somewhere.) If you have an old checkpoint file 
they the worst that can happen is that the localblockstore has some 
blocks in the storefile that it can't see, right?  It will not cause 
the system to overwrite existing data with new insertions; the freelist 
and block writing happens in the storefile object, the localblockstore 
just says "store this and tell me where it is located", the 
localblockstore does not tell the storefile where to put the new data.

This leads me to believe that un-locatable (to the localblockstore) 
data is the worst problem we would see if the localblockstore recovered 
from an old checkpoint. An eventual reindex will find these blocks, so 
this is not a long-term problem. OTOH, being forced to rebuild the 
index at start-up is a problem that should be avoided if possible.  It 
is a performance hit at the worst possible time. One other thing that 
will probably need to happen is that the reindexer will need to be told 
what to do if it finds a duplicate block.  Right now it assumes that 
since it short-circuits the write of duplicate blocks this event will 
not occur, right? Just a quick "remove this specific block" where the 
block is selected according the the store maintainer policies will 
probably be sufficient.

I think that the dump to new file, fsync, and then relink the file over 
the old pickle is probably a better solution. If we were truly paranoid 
we would also re-read the pickle and confirm that it rebuilds the data 
we dumped before doing the relink.

Jim



-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
mnet-devel mailing list
mnet-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mnet-devel