Computer arithmetic
Jean-Pierre A. Radley
appl at jpr.com
Thu Jun 13 13:18:39 PDT 2013
I'm generating a CSV file for Excel comprising about 6500 invoices.
Each invoice (row) has charges allocated to one or the other of six
category columns (e.g., goods, tax, freight...), and then there's a
total column. (I'm using filePro's '$' edit for the CSV entries.)
At the bottom, the columns are totaled, but the sum of the first six
columns does not equal the total in the seventh column; the error is
slightly different if I let filePro do all the totalling, or if I
use Excel SUM formulas.
The error is typically a few thousand bucks out of a grand total over
six million dollars.
Is this an inherent result of representing decimal numbers in a computer's
binary language?
--
JP
More information about the Filepro-list
mailing list