Results 1 to 3 of 3
  1.    #1  
    On my Treo 600, if I subtract 225.3 from 226.2, instead of 0.9 I get

    0.89999999999998 (that's 12 9's).

    The calculator is in advanced mode and the settings are Math, Float, Degrees.

    A coworker's Treo 600 does the same thing. Mine is Sprint and his is T-Mobile.

    In the simple mode I do get an answer of 0.9.

    Does anyone know why this happens? I've read the other long dicussions about the % key but this seems to be some kind of defect rather than an ambiguity in the operation.
  2. McTreo's Avatar
    Posts
    238 Posts
    Global Posts
    239 Global Posts
    #2  
    This is really funny behavior. Mine did the same thing on your calculation, so I tried some others: 1.2-.3=.9, 26.2-25.3=.9, but get this... 126.2-125.3=.90000000000001.

    I guess the reason this is still funny to me is, I use Abacus for HP-12C functionality. Wait a minute, I better try Abacus....

    Good luck,

    Mac
    "I Sync, therefore I AM!"

    BellSouth became Cingular, left for ATT, who became Cingular, who became ATT. Is that right?
    Visor Pro > Vx > Treo600 > Treo650 > Treo680 > Tilt!
  3. #3  
    It has to do with integer vs floating-point type claculations and rounding problems.The error you are getting is 0.000000000002 [did I put enough 0's?]

    Salim

Posting Permissions