I'm developing a Scientific Calculator using Adobe AIR. I'm aware Flash is not a great platform for math, however that is decided. I expect Flash's double-precision floating-point format should be adequate for a scientific calculator comparable to a hardware one (e.g. Casio FX series). I've become aware about the many pitfalls of using binary floating-point formats for computations with (decimal) numbers, and will surely ask more questions about that lately.
I'm developing the calculation results display, for that I need to format a numeric value into one of several display modes. Depending on the mode, I need to either round half away from zero, or round towards zero (truncate extra digits). I've written my own formatting function which converts the Number to a string using Number.toExponential, breaks down the string using RegExp, rounds to the desired number of digits with the desired rounding mode and returns the string in the desired format. The reason I used Number.toExponential because it's the only function that outputs any number from very small to very large in the same format, so that is easy to parse and work with.
However I found out that toExponential have some strange behaviour when it comes to rounding. It appears to me that it truncates the extra digits instead of rounding half away from zero (or half to even or whatever). AS3 docs don't mention any details about rounding.
const number1:Number = 0.000000123455555555544; trace(number1.toString()); // 1.23455555555544e-7 trace(number1.toExponential(14)); // 1.23455555555543e-7 trace(number1.toExponential(20)); // 1.23455555555543993039e-7 const number2:Number = 0.000000123455555555567; trace(number2.toString()); // 1.23455555555567e-7 trace(number2.toExponential(14)); // 1.23455555555566e-7 trace(number2.toExponential(20)); // 1.23455555555566995278e-7
It appears that Number.toString() performs correct rounding and returns the same number as the literal values used in the code to initialize the constants. But toExponential, set to print the same number of digits as toString returns for these particular numbers, appears to always round to zero.
Some more examples.
const DOUBLE_MIN:Number = 4.94065645841246544176e-324; trace(DOUBLE_MIN.toString()); // 4.9406564584124654e-324 trace(DOUBLE_MIN.toExponential(16)); // 4.9406564584124654e-324 trace(DOUBLE_MIN.toExponential(20)); // 4.94065645841246544176e-324 const DOUBLE_MIN_NORMAL:Number = 2.22507385850720138309e-308; trace(DOUBLE_MIN_NORMAL.toString()); // 2.2250738585072014e-308 trace(DOUBLE_MIN_NORMAL.toExponential(16)); // 2.2250738585072013e-308 trace(DOUBLE_MIN_NORMAL.toExponential(20)); // 2.22507385850720138309e-308 const DOUBLE_MAX:Number = 1.79769313486231570814e+308; trace(DOUBLE_MAX.toString()); // 1.79769313486231e+308 trace(DOUBLE_MAX.toExponential(14)); // 1.79769313486231e+308 trace(DOUBLE_MAX.toExponential(20)); // 1.79769313486231570814e+308
Is this the expected behavior of toExponential? If so, I'm forced to use toString and have two different RegExps (or one more complicated RegExp), depending on if toString prints the number in normal format or exponential notation.
I was also considering using toExponential, then rounding it to 16 (or 15?) digits, and then finally rounding to my desired number of digits (according to the display mode). However such double rounding generally can create problems (e.g. assuming half away from zero rounding, 9.49 rounded to 1 decimal digit gives 9.5, then rounded second time to whole number gives 10; but 9.49 rounded straight to whole number gives 9). I wonder if in this situation there's some reason double rounding could be used?
Any advice is welcome, as well as explanations (or refuting examples) on why my ideas are wrong.
Apologies for such verbose writing, when I'm writing I can never figure out what I should cut, cut, cut. And apologies if I made mistakes with my language, English is not my native language.