Float data type precision
Mike Dunn
mike.dunn at amd.com
Tue Jun 26 04:55:05 EST 2001
I'm running ACEDB Version 4_7l and am getting rounding errors with
large Float values. Here is an example class, the data I enter in the
database, and what the db returns to me
Class model:
?Float_class Title UNIQUE Text
Value Float
Input:
Float_class : "Example"
Value 0.1234567
Value 1234567
Value 1234567.1234567
Output:
Float_class : "Example"
Value 0.123457
Value 1234567.000000
Value 1234567.125000
Is any of this rounding error reduced in later versions of acedb? Is
there anyway to get more precision? I'm using aceperl, so one
workaround is to store the large floats as text, then let perl sort it
out, but I'd rather find a way to keep the data in the db as Float.
Regards,
Mike Dunn
------------------------------------------------------------------
Mike Dunn CMD / CPG
Advanced Micro Devices email: Mike.Dunn at amd.com
5204 E. Ben White Blvd. phone: (512) 602-5231
Austin, TX 78741 fax: (512) 602-5155
(Mail Stop 521) pager: (512) 622-6413
------------------------------------------------------------------
More information about the Acedb
mailing list