Computer arithmetic

Jean-Pierre A. Radley appl at jpr.com
Thu Jun 13 13:18:39 PDT 2013


I'm generating a CSV file for Excel comprising about 6500 invoices.

Each invoice (row) has charges allocated to one or the other of six
category columns (e.g., goods, tax, freight...), and then there's a
total column.  (I'm using filePro's '$' edit for the CSV entries.)

At the bottom, the columns are totaled, but the sum of the first six
columns does not equal the total in the seventh column; the error is
slightly different if I let filePro do all the totalling, or if I 
use Excel SUM formulas.

The error is typically a few thousand bucks out of a grand total over
six million dollars.

Is this an inherent result of representing decimal numbers in a computer's
binary language?

--
JP


More information about the Filepro-list mailing list