Indexing a file with 3,000,000 records FAST ?

[John Esak]

I don't believe anyone is/can give you much more hope with PF variables.
Could be, setting PFIXBUILD=2 might help... but I would, of course, test
this. Also, I'm not sure what is meant by the "old 4.1 sorting". Doe it mean
you can't have multi-segment indexes?  Don't know. Ken, can illucidate.

Otherwise,

I think you need to look to an overall hardware/software upgrade to get the
results you want. For now, an overnight of index building seems the only
fate you have.

You did not mention the hard disks and type that you are using. I blieve
these would be of paramount importance with questions of index-building
speed. Well, it's true that the CPu does the intensive work of indexing, but
you have to get the records "to" the CPU and that is disk i/o and memory
caching. Assuming the filePro routines let you do the best you can with
caching, the onus is on the disk i/o to speed things to the fastest they can
be at this bottle neck... and this is the sloooowwest point in the chain for
indexing. So, here is a scenario that just happened to me. I cahnged a
single disk 9gig Baracuda running on a SCSI-2 adpator... (and a Pentium
100Mhz machine to a 36Gb ultra 310 dirve, running an LVD adaptor on a 2.8Ghz
Celeron. The indexing of one large file used to take 4.5 hours. On the new
system, it took just under 3 minutes. Needless to say the customer was
pretty happy. Also, here daily which took about 5 to 8 minutes to run
happened so fast that she called me and said it was broken. Becuase it
happened in the blink of an eye she thought it hadn't run. :-)

Okay, the addition of the 2.8Gb Celeron over the Pentium 100Mhz is a 28
times increase in processor speed... but I believe it was the fantastic
difference in the drive chain. If you can afford 15K or 20K drives and put
these on your same 2.8 it will be great. If you put them on a Pentium Extrem
3.73Ghz CPU... I think your hour per index will drop substantially... maybe
down to single digit minutes even. Judging (guessing) by the 5.0.9 software
you are running, it is probable that you are not running the newest SCSI
drives... more probably SCSI 2.  So, you can expect huge speed-ups with new
hardware.

So, changing to this new stuff will make a world of difference... and we
haven't even mentioned optical drives yet. :-)

John Esak


  -----Original Message-----
  From: filepro-list-bounces at lists.celestial.com
[mailto:filepro-list-bounces at lists.celestial.com]On Behalf Of Mike Fedkiw
  Sent: Friday, March 10, 2006 10:40 AM
  To: filepro-list at lists.celestial.com
  Subject: Indexing a file with 3,000,000 records FAST ?


  system
  filepro 5.0.09
  windows 2000 advances server
  P4 2.4 GHz
  2 gig of ram

  BTW, I don't want to upgrade to 5.0.14. I am waiting for next ver. Ray
said
  that the bug slowing browse window scrolling and flickering when using
lots
  of show commands would be fixed in ver 5.0.15 and above.

  I have a file with about three million records. Recently I deleted some
  records out of this file. Now I am getting node errors on a particular
  Index. I rebuilt one index this morning and it took over an hour. now add
up
  (a-p times 1 hour). That's 16 hours. There has to be a faster/better way.
I
  did try setting PFNUMIXBUF=15000 and that helped a little. What about
  PFNUMIXBUILD.

  Any help will be greatly appreciated. I am not looking forward to fifteen
  hours of indexing for one file.

  Thanks in Advance, Mike Fedko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.celestial.com/pipermail/filepro-list/attachments/20060310/18ad2ca1/attachment.html