Page 1 of 2 12 LastLast
Results 1 to 20 of 35
  1.    #1  
    Figured I would post some 800w vs. 700wx benchmark results (using SPB Benchmark 1.6):



    I ran the benchmark 3 times and it's very consistent (percentage-wise), plus each test iterates multiple times and then averages to keep the times consistent, so one run should be all that's needed. It takes roughly 30 minutes to run.

    To make the results as accurate as possible, I did the following:
    1) Hard-reset
    2) Turned the the phone radio off
    3) Ran on AC power, not USB
    4) Dismissed any notifications
    5) Disabled the backlight and power timeouts
    6) Disabled the Pocket Word benchmark (which never finishes)

    The benchmark is free for personal use, as well as journalist/reviewer use. It helps to eliminate the apples-to-oranges factor when comparing different processor types running at different speeds with custom tweakage by OEM's such as Palm.

    If anyone wants to try it, you can download it here.

    Note: there are some odd numbers in the graphics/filesystem results, especially when compared to their reference model. It could be that the tests use deprecated interfaces, no hardware acceleration or sub-optimal blocksizes, so take the results with a grain of salt. Anyways, the results provide a better perspective than, my device feels faster than yours.

    Any Treo Pro or Touch Pro owners want to give it a shot?
  2. #2  
    Surprising how slow the 800w is compared to the 700wx in a bunch of the tests.
  3. #3  
    How many times did you repeat your tests?
  4. #4  
    Quote Originally Posted by nsxprime View Post
    Surprising how slow the 800w is compared to the 700wx in a bunch of the tests.
    People who are experienced with these phones will tell you that WM2003 devices run faster than WM5 which run faster than WM6.

    Throw in the higher pixel count with a 320x320 it's not hard to see how there would be certain differences, all else being equal.

    Assuming the results are consistent, they tell us that both devices are quite comparable in overall performance, which is a good thing actually (no one complained about the 700wx being a slouch). The 800w "wins" 15 out of 25 categories.

    Going further, when it comes to heavy lifting (file processing, reading, listing) the 800w excels--that is on everyday tasks and running core apps. (as far I can tell).

    Where it lags, evidently, is in multimedia performance which is exactly where a business messenger should have a weakness if there has to be one. Then again, in terms of real-world performance, I'm not aware of anyone complaining about slow graphics, stuttering video, etc.

    btw, thanks zbop for doing this/posting. Interesting numbers.

    The Touch Pro should trounce the 800w in certain areas (it has 200mhz more just for raw processing), so that should be a no brainer. I'd be more curious though about the Treo Pro, especially the CDMA version.
    Last edited by Malatesta; 12/21/2008 at 01:30 PM.

    WMExperts: News, Reviews & Podcasts + Twitter
  5. #5  
    wow some of those results I can't believe. I would have thought the 800 would be twice as fast as the 700. Just goes to show you never know. The proof is on the pudding. Don't judge a book by it's cover. Looks are decieving.
  6. #6  
    It should also be noted: SPB benchmark is OLD

    It has not been updated since 2005 and is technically not "WM6" optimized. What that means for actual testing is unclear--it might not matter at all.

    WMExperts: News, Reviews & Podcasts + Twitter
  7. #7  
    Quote Originally Posted by Malatesta View Post
    It should also be noted: SPB benchmark is OLD

    It has not been updated since 2005 and is technically not "WM6" optimized. What that means for actual testing is unclear--it might not matter at all.
    It's clear that it's not optimized for WM6 (or even 6.1) from the graphics test. The 800w has acceleration that the 700wx lacks, and the tests are obviously not using that.

    It's still rather interesting.
  8.    #8  
    Quote Originally Posted by Ebag333 View Post
    How many times did you repeat your tests?
    As I stated, I ran the entire benchmark 3 times, even though it wasn't necessary. All of the tests repeat (I think at least 10 times). The results are pretty consistent.
  9.    #9  
    If you've spent any time working with benchmarks (and I have), you know that each individual benchmark is only as meaningful as your ability to interpret the results.

    Cherry-picking results is what's known as marketing.

    I am more interested in the results of a Treo Pro or a Touch Pro, but I posted the 700wx data so that there was something to compare the results to. Except for the graphics performance, overall the numbers favor the 800w. To make a better comparison, I would have to load WM6.1 on the 700wx to eliminate any WM deltas.

    In general, you can't argue the validity of the numbers unless the test was improperly run or the benchmark is coded improperly, you can only argue their relevance. I can't vouch for the code, but I don't really doubt the numbers. You can argue that the code is old, however the same code was run on both systems.

    CPU benchmark results - I think they are valid, these are industry standard and pretty hard to screw up.

    FS benchmark results - I think they are valid, but I believe that WM6.1 is causing some of the numbers to be worse on the 800w (versus WM5 on the 700wx).

    Application benchmarks - I think they are valid, wish I could have completed the Pocket Word test.

    Graphics benchmark results - There is something funny going on, but without knowing more about the benchmark I can't quite explain. In particular the DIB Bitblt test is much worse on the 800w but this test is supposed to be independent of any driver acceleration (that's the purpose of the test). However I do agree that the results are inconsistent with what I've observed on the 800w. I suspect that the results are related to using the deprecated GAPI interfaces rather than DirectDraw interfaces. This performance hit should only be present you if you are using very old apps. WM5 may have provided better support for GAPI than WM6.

    Like I said, take it with a grain of salt. If we can get someone else to run this on their WM6.1 device, then we'll have some interesting data.
  10. #10  
    Quote Originally Posted by zbop View Post
    I ran the benchmark 3 times and it's very consistent (percentage-wise), plus each test iterates multiple times and then averages to keep the times consistent, so one run should be all that's needed. It takes roughly 30 minutes to run.

    To make the results as accurate as possible, I did the following:
    1) Hard-reset
    2) Turned the the phone radio off
    3) Ran on AC power, not USB
    4) Dismissed any notifications
    5) Disabled the backlight and power timeouts
    6) Disabled the Pocket Word benchmark (which never finishes)

    The benchmark is free for personal use, as well as journalist/reviewer use. It helps to eliminate the apples-to-oranges factor when comparing different processor types running at different speeds with custom tweakage by OEM's such as Palm.

    If anyone wants to try it, you can download it here.

    Note: there are some odd numbers in the graphics/filesystem results, especially when compared to their reference model. It could be that the tests use deprecated interfaces, no hardware acceleration or sub-optimal blocksizes, so take the results with a grain of salt. Anyways, the results provide a better perspective than, my device feels faster than yours.

    Any Treo Pro or Touch Pro owners want to give it a shot?
    This is good, because we can see stock device test numbers if everyone does the same test the way Zbop suggested.

    However, given we know the Treo Pro and Touch Pro default OS configurations are different from the 800w, it would be best for Treo Pro owners to try this test once as described above and again with ClearType turned off. Given the 800w has ClearType off by default. That way, we can see results with near similar configurations as well as the device's "stock default" configuration.

    Touch Pro owners should try as Zbop mentioned above and also try with Touch Flo off and ClearType off. And if you have time, Touch Flo on with ClearType off and vice versa.

    If we get enough participants with various devices, this will REALLY help consumers in knowing more about the various devices capabilities.

    I'm really used to the Tom's Hardware and AnandTech benchmark test reports and they ALWAYS make sure the software is configured as near similar as possible. Because that gives results for how well the raw hardware performs. Users can change the software to their desired preferences on their own. So I'm looking at this with that bit of bias. I know with handhelds it's more difficult because of differing ROMs, but I think it's good to try and keep them as similar in testing as possible. It would be great if someone made a site doing for handhelds what Tom's Hardware and AnandTech have done. It is very much needed.
  11.    #11  
    Quote Originally Posted by darnell View Post
    This is good, because we can see stock device test numbers if everyone does the same test the way Zbop suggested.

    However, given we know the Treo Pro and Touch Pro default OS configurations are different from the 800w, it would be best for Treo Pro owners to try this test once as described above and again with ClearType turned off. Given the 800w has ClearType off by default. That way, we can see results with near similar configurations as well as the device's "stock default" configuration.

    Touch Pro owners should try as Zbop mentioned above and also try with Touch Flo off and ClearType off. And if you have time, Touch Flo on with ClearType off and vice versa.

    If we get enough participants with various devices, this will REALLY help consumers in knowing more about the various devices capabilities.

    I'm really used to the Tom's Hardware and AnandTech benchmark test reports and they ALWAYS make sure the software is configured as near similar as possible. Because that gives results for how well the raw hardware performs. Users can change the software to their desired preferences on their own. So I'm looking at this with that bit of bias. I know with handhelds it's more difficult because of differing ROMs, but I think it's good to try and keep them as similar in testing as possible. It would be great if someone made a site doing for handhelds what Tom's Hardware and AnandTech have done. It is very much needed.
    To minimize variables, sure it would be better to turn off ClearType. But I think the only tests that might be affected by ClearType are the PIE tests.
  12. #12  
    Quote Originally Posted by zbop View Post
    To minimize variables, sure it would be better to turn off ClearType. But I think the only tests that might be affected by ClearType are the PIE tests.
    I'm suspecting more than that, but we'll see once someone runs the test .

    Can you do a run on the 800w with ClearType on and if it is an option on the 700wx it would be interesting to see results on that device also. Although I'm more interested in the Treo Pro and Touch Pro in this regard, given the reports from users.
  13.    #13  
    Quote Originally Posted by darnell View Post
    I'm suspecting more than that, but we'll see once someone runs the test .

    Can you do a run on the 800w with ClearType on and if it is an option on the 700wx it would be interesting to see results on that device also. Although I'm more interested in the Treo Pro and Touch Pro in this regard, given the reports from users.
    Heheh, want to bet me a new laptop?

    ClearType only comes into play if you are drawing text on windows.

    The CPU benchmarks measure integer and floating point operations, the filesystem benchmarks measure file I/O, and the graphics benchmarks measure the ability to blit pixels. If the benchmarks are coded correctly, there shouldn't be a difference except in the Explorer and Word tests. If I get a chance, I'll rerun it. I'm currently debugging other stuff and would need to hard reset.
  14. #14  
    Quote Originally Posted by zbop View Post
    Heheh, want to bet me a new laptop?
    Nice try, but I'm not a betting kind of man . And even if I was, NO WAY I'd put my new laptop on the line .


    ClearType only comes into play if you are drawing text on windows.
    Well that's what it is supposed to improve and it does. However I'll await your test results to see if it's just sitting dormant apart from the PIE test. Being an anti-aliasing algorithm, I would not be surprised if it's running anytime anything graphics related is accessed. I just want to know for certain . I figure it's better if you run the test given you've provided the recognized "baseline" values already.

    The CPU benchmarks measure integer and floating point operations, the filesystem benchmarks measure file I/O, and the graphics benchmarks measure the ability to blit pixels. If the benchmarks are coded correctly, there shouldn't be a difference except in the Explorer and Word tests. If I get a chance, I'll rerun it. I'm currently debugging other stuff and would need to hard reset.
    I'm absolutely not saying you're wrong about how the test will turn out. You may be 100% correct in saying that only the PIE test will show a difference. I'm just saying I have suspicions. That's why I keep bugging you to run it . So we can see how and where it impacts your baseline. It's really hard to find this sort of benchmarking on handhelds, you're doing some good stuff.

    I only wish the test were updated. But I'm sort of glad it's a little known test. Because if it were a well known test, device makers would tweak their ROMs to "beat" the test although they might not perform well in regular usage. (Something I'm sure you're aware has been done in other areas of computing.)
  15.    #15  
    Quote Originally Posted by darnell View Post
    Nice try, but I'm not a betting kind of man . And even if I was, NO WAY I'd put my new laptop on the line .




    Well that's what it is supposed to improve and it does. However I'll await your test results to see if it's just sitting dormant apart from the PIE test. Being an anti-aliasing algorithm, I would not be surprised if it's running anytime anything graphics related is accessed. I just want to know for certain . I figure it's better if you run the test given you've provided the recognized "baseline" values already.



    I'm absolutely not saying you're wrong about how the test will turn out. You may be 100% correct in saying that only the PIE test will show a difference. I'm just saying I have suspicions. That's why I keep bugging you to run it . So we can see how and where it impacts your baseline. It's really hard to find this sort of benchmarking on handhelds, you're doing some good stuff.

    I only wish the test were updated. But I'm sort of glad it's a little known test. Because if it were a well known test, device makers would tweak their ROMs to "beat" the test although they might not perform well in regular usage. (Something I'm sure you're aware has been done in other areas of computing.)
    Well if you had called my bluff, I wouldn't have taken the bet because I've seen too many unexpected results in my day. Simply too many variables.

    Benchmarks are a marketing tool for sure, but they do serve a purpose. In reality, users care more about how responsive their device feels, rather than raw performance numbers.

    BTW, some websites do use this benchmark when reviewing smartphones. The full price on this software is a staggering $495.
  16.    #16  
    Finally got a chance to run this. There is some variance between the two runs, but most of the deltas are "in the noise" and can't be attributed to ClearType (right column is with ClearType, left is without).

    Note, this doesn't mean there is NO perfomance penalty with ClearType on the 800w, it just means that if there is a hit, it was either insignificant or outside the scope of this benchmark.
  17. #17  
    Quote Originally Posted by zbop View Post
    Note, this doesn't mean there is NO perfomance penalty with ClearType on the 800w, it just means that if there is a hit, it was either insignificant or outside the scope of this benchmark.
    Yeah, these look basically the same to me.

    Now, I've not noticed a hit at all when enabling/disabling Clear Type, unlike what folks have seen with the Touch/Treo Pro.

    Perhaps there's something different about the way the 800w handles Clear Type.
  18. #18  
    Quote Originally Posted by Ebag333 View Post
    Now, I've not noticed a hit at all when enabling/disabling Clear Type, unlike what folks have seen with the Touch/Treo Pro.

    I've not noticed a difference either. I have noticed how much slower my friends Treo Pro is compared to the 800w running the same apps. I tried turning off his clear type and it made no noticeable difference.
    Pilot 1000 -> Pilot 5000 ->Palm Pilot Professional -> HP 620LX -> TRG Pro -> Palm V -> Palm Vx -> Palm M505 -> Palm i705 -> Palm Tungsten|T -> Samsung i500 -> Treo 600->Treo 650 -> Treo 600-> Treo 700p ->Centro ->Treo 800w + Redfly C8n -> Palm Pre -> HP Touchpad
    R.I.P Palm 1996-2011
  19. #19  
    Interesting.

    Yeah, we gotta see what the Treo Pro numbers are in this regard--perhaps it's just placebo. It's very odd that there would be such a dramatic change with CT on any device...it shouldn't have one at all.

    WMExperts: News, Reviews & Podcasts + Twitter
  20. #20  
    Quote Originally Posted by rc46 View Post
    I've not noticed a difference either. I have noticed how much slower my friends Treo Pro is compared to the 800w running the same apps. I tried turning off his clear type and it made no noticeable difference.
    I don't have a Pro so I can't attest to it.

    I just know that there's been quite a few posts about turning that off for better results. *shrug*
Page 1 of 2 12 LastLast

Posting Permissions