Results 1 to 3 of 3
  1.    #1  
    On my Treo 600, if I subtract 225.3 from 226.2, instead of 0.9 I get

    0.89999999999998 (that's 12 9's).

    The calculator is in advanced mode and the settings are Math, Float, Degrees.

    A coworker's Treo 600 does the same thing. Mine is Sprint and his is T-Mobile.

    In the simple mode I do get an answer of 0.9.

    Does anyone know why this happens? I've read the other long dicussions about the % key but this seems to be some kind of defect rather than an ambiguity in the operation.
  2. McTreo's Avatar
    238 Posts
    Global Posts
    239 Global Posts
    This is really funny behavior. Mine did the same thing on your calculation, so I tried some others: 1.2-.3=.9, 26.2-25.3=.9, but get this... 126.2-125.3=.90000000000001.

    I guess the reason this is still funny to me is, I use Abacus for HP-12C functionality. Wait a minute, I better try Abacus....

    Good luck,

    "I Sync, therefore I AM!"

    BellSouth became Cingular, left for ATT, who became Cingular, who became ATT. Is that right?
    Visor Pro > Vx > Treo600 > Treo650 > Treo680 > Tilt!
  3. #3  
    It has to do with integer vs floating-point type claculations and rounding problems.The error you are getting is 0.000000000002 [did I put enough 0's?]


Posting Permissions