Data corruption

[Brian K. White]

----- Original Message ----- 
From: "Don Coleman" <dcoleman at dgcreact.com>
To: "'Brian K. White'" <brian at aljex.com>; <filepro-list at lists.celestial.com>
Sent: Thursday, December 13, 2007 11:59 AM
Subject: RE: Data corruption


>> -----Original Message-----
>> From: filepro-list-bounces+dcoleman=dgcreact.com at lists.celestial.com
>> [mailto:filepro-list-bounces+dcoleman=dgcreact.com at lists.celestial.com] On
>> Behalf Of Brian K. White
>> Sent: Thursday, December 13, 2007 10:30 AM
>> To: filepro-list at lists.celestial.com
>> Subject: Re: Data corruption
>> 
>> 
>> ----- Original Message -----
>> From: "Don Coleman" <dcoleman at dgcreact.com>
>> To: "'Kenneth Brody'" <kenbrody at bestweb.net>
>> Cc: "'Filepro-List at Lists. Celestial. Com'" <filepro-
>> list at lists.celestial.com>
>> Sent: Thursday, December 13, 2007 9:57 AM
>> Subject: RE: Data corruption
>> 
>> 
>> >
>> >> -----Original Message-----
>> >> From: Kenneth Brody [mailto:kenbrody at bestweb.net]
>> >> Sent: Wednesday, December 12, 2007 5:55 PM
>> >> To: Don Coleman
>> >> Cc: 'Filepro-List at Lists. Celestial. Com'
>> >> Subject: RE: Data corruption
>> >>
>> >> Quoting Don Coleman (Wed, 12 Dec 2007 14:49:00 -0500):
>> >>
>> >> [...]
>> >> >> >  So I'm thinking I can write a
>> >> >> > clerk process to update, then save, then delete these records.
>> >> >>
>> >> >> Just remember that, being deleted records, you can't select them via
>> >> the
>> >> >> normal methods.
>> >> [...]
>> >> >> > a graphic dash.  Yet if I run a report with the debug on and I
>> >> display
>> >> >> the
>> >> >> > value contained in this field I get 00000211.  Why does the
>> debugger
>> >> >> display
>> >> >> > something different than the screen?  This wouldn't be so bad
>> except
>> >> >> this
>> >> >> > file contains 1.6M records.  One at a time may take a while.
>> >> >>
>> >> >> How did you get to the record in a report?  If the header marks the
>> >> record
>> >> >> as deleted, no selection set (not even the "-a" flag) will select
>> it.
>> >> >
>> >> > I ran the report (just a simple listing) with a selection set of @RN
>> eq
>> >> 13
>> >> > (one of the corrupted records).  I then displayed the field value in
>> the
>> >> > debugger.  However, the report didn't actually print.
>> >>
>> >> If record 13 is marked deleted because it's one of the corrupted
>> records,
>> >> then it won't be selected.  What processing were you in when in the
>> >> debugger?  Were you, perchance, in @DONE?  Check @RN and @TS.
>> >>
>> >> --
>> >> KenBrody at BestWeb dot net        spamtrap:
>> <g8ymh8uf001 at sneakemail.com>
>> >> http://www.hvcomputer.com
>> >> http://www.fileProPlus.com
>> >
>> >
>> > Ken:
>> >
>> > Yesterday I thought I had just run a simple listing from dreport with
>> the
>> > debugger on.  However, when I tried to replicate this today it doesn't
>> > select record #13 from a selection set nor does it select it when put
>> into a
>> > select table.  So you're your premise is correct.  Is my only option
>> looking
>> > for bad data on a manual view screen?
>> 
>> 
>> How many times does someone have to say "map a non-filepro file over the
>> key" ?
>> 
>> Brian K. White    brian at aljex.com    http://www.myspace.com/KEYofR
>> +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
>> filePro  BBx    Linux  SCO  FreeBSD    #callahans  Satriani  Filk!
>> 
>> _______________________________________________
>> Filepro-list mailing list
>> Filepro-list at lists.celestial.com
>> http://mailman.celestial.com/mailman/listinfo/filepro-list
> 
> Can anyone tell me if, when defining the non-filepro file (overlay) should I
> use the existing corrupted file field edits or use "*" for all field edits?

Up to you.

Lets digest that true and complete answer into something possibly more helpful...
"How do you decide if you want any edits?"

1) will you be doing a lot of manual/interactive on-screen fixing of data in clerk?
in that case you may want to specify edits on the fields to gain the usual data-entry sanity checking.

2) will you be using indexes to find records quickly in the alien file?
In that case you may want to specify edits so that the indexes sort correctly.

Lets digest that a little further into "what would I do?"...

I would probably not bother specifying any edits anywhere in the alien file since you are probably just viewing the data or analysing it almost byte for byte with custom code anyways, so the edits mostly won't help much, and may get in the way in fact.

Just to confuse things a little, using an edit _may_ actually be the simplest way to check for data sanity in some particular case. It depends on the nature of the corruption, the nature of your data, the nature of your app. It may be that you can say something like "if this field ever doesn't pass a MDYY/ edit, then that record is automatically junk"
No one can say anything remotely like that about your app but you or someone who has worked on your code enough to know the whole system inside-out.

I'd probably not define any edits in the alien file, and if I wanted to use an edit to sanity-check the data in a field in processing, I'd use the edit in a dummy or use doedit() and compare the result with "".

if: 3 ne "" and doedit(3,"mdyy/") eq ""
then: msgbox "bad record"

Mostly I wouldn't trust such a simple test myself. I'd maybe use something that simple only to flag such records for me to look at manually or to do something else with. The best way to identify and then deal with corrupted data is always a very case-by-case kind of thing.

Brian K. White    brian at aljex.com    http://www.myspace.com/KEYofR
+++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
filePro  BBx    Linux  SCO  FreeBSD    #callahans  Satriani  Filk!