Results 1 to 6 of 6
  1.    #1  
    I got the idea for this thread while reading a post about bugs on the treo 680 platform. (see http://discussion.treocentral.com/sh....php?t=132408) I do not own a treo 680, but bug reports interest me because I'm a software tester and seeing what "gets by" other testers makes me a better tester.

    First of all (and as a summary) I think Palm is getting better at testing but still not good enough. After giving the problem some thought, I'd like to propose a possible solution for Palm to consider.

    Palm should consider revamping their testing model to include a beta test period with real users on at least one or two carriers before announing general availability of a new device. The beta tests could be performed by "early adopter" types that frequent this forum, and Palm could offer discounts, and prizes for bugs found. Palm could put the beta testers under NDA and have a website similar to the Palm Plugged-In website and a forum to discuss bugs. This sort of thing would significantly improve initial quality for probably less money than Palm is spending now.

    I'm not sure what the problems would be for Palm to take such an approach, but from my vantage point, the benefits of a well-managed program would probably outweigh the costs. I suspect the first hurdle would be getting carriers to buy into such an idea. I think it could be justified by faster test turnaround time, more certainty of customer acceptance when new models are introduced, and dramatically reduced customer support costs. (Customer support would be lower because initial problems will have been identified, so carriers can write scripts for tech-support to be trained on in advance of product release.)

    I'd be interested in hearing other TC'ers opinions and/or ideas on how this idea could work. Hopefully peope with telco and marketing experience will contribute their ideas about what might and might not work. I'm sure Palm reads this forum and will follow this thread, so if you want to encourage such an approach, I'm sure your ideas can lead to higher quality treo releases in the future! Based on history of past device releases, plus Palm's stated goals of releasing more varied devices at both the high and low ends, I do not believe Palm's current testing strategy is scalable. No matter how much testing they do, embarassing bugs are still getting through! Both ends of the market segment they want to expand into will be less tolarant of the current level of quality. I think carriers would like increased certainty of the level of quality before final-signoff with Palm and the "go" decision for the device release is made. Again, even if you think Palm would not go for this idea, I'd like to learn by hearing your opinions about what might and might not work. Lastly, I hope Palm listens, or at least brainstorms solutions based on these ideas. Your future as a company depends on this.

    thank you,
    --
    Bob Clancy
    Last edited by sudoer; 12/28/2006 at 10:18 PM.
    I'm both super! ... and a doer!
  2. #2  
    Quote Originally Posted by bclancy View Post
    I got the idea for this thread while reading a post about bugs on the treo 680 platform. (see http://discussion.treocentral.com/sh....php?t=132408) I do not own a treo 680, but bug reports interest me because I'm a software tester and seeing what "gets by" other testers makes me a better tester.

    First of all (and as a summary) I think Palm is getting better at testing but still not good enough. After giving the problem some thought, I'd like to propose a possible solution for Palm to consider.

    Palm should consider revamping their testing model to include a beta test period with real users on at least one or two carriers before announing general availability of a new device. The beta tests could be performed by "early adopter" types that frequent this forum, and Palm could offer discounts, and prizes for bugs found. Palm could put the beta testers under NDA and have a website similar to the Palm Plugged-In website and a forum to discuss bugs. This sort of thing would significantly improve initial quality for probably less money than Palm is spending now.

    I'm not sure what the problems would be for Palm to take such an approach, but from my vantage point, the benefits of a well-managed program would probably outweigh the costs. I suspect the first hurdle would be getting carriers to buy into such an idea. I think it could be justified by faster test turnaround time, more certainty of customer acceptance when new models are introduced, and dramatically reduced customer support costs. (Customer support would be lower because initial problems will have been identified, so carriers can write scripts for tech-support to be trained on in advance of product release.)

    I'd be interested in hearing other TC'ers opinions and/or ideas on how this idea could work. Hopefully peope with telco and marketing experience will contribute their ideas about what might and might not work. I'm sure Palm reads this forum and will follow this thread, so if you want to encourage such an approach, I'm sure your ideas can lead to higher quality treo releases in the future! Based on history of past device releases, plus Palm's stated goals of releasing more varied devices at both the high and low ends, I do not believe Palm's current testing strategy is scalable. No matter how much testing they do, embarassing bugs are still getting through! Both ends of the market segment they want to expand into will be less tolarant of the current level of quality. I think carriers would like increased certainty of the level of quality before final-signoff with Palm and the "go" decision for the device release is made. Again, even if you think Palm would not go for this idea, I'd like to learn by hearing your opinions about what might and might not work. Lastly, I hope Palm listens, or at least brainstorms solutions based on these ideas. Your future as a company depends on this.

    thank you,
    --
    Bob Clancy
    Kudos Bob. Palm can improve quality significantly with such a program. I think Palm would even be able to charge for membership in such a program, and would provide discounts to members who provide a certain level of feedback.

    The problem I see is that of inevitable leaks. Who do you trust? You can assess a penalty, but unless it's ridiculously high, some web site will reimburse leakers just to get leaks and therefore hits. Also, how would they address hardware updates? Say they send out 1000 and now have to recall them to make some small hardware changes. Who pays for shipping? What happens when only 900 come back?
    A new Avatar to commemorate Silly Season.
  3.    #3  
    Quote Originally Posted by skfny View Post
    ... The problem I see is that of inevitable leaks. Who do you trust? You can assess a penalty, but unless it's ridiculously high, some web site will reimburse leakers just to get leaks and therefore hits. ...
    <rant value="on">
    My first reaction is: "That's Palm's problem to solve.", but aside from being a consumer who is tired of beta testing with no regcognition, and leaving Palm for a better alternative, my first reaction is not very helpful. Personally, with lack of a beta test program and improved initial quality, I'll refrain from buying any more Palm products that prove to have unacceptable initial quality, and if more people did this, Palm would have to adapt in order to survive.
    </rant>

    I think they should look to what motivates people and build a program that recognizes individually while collectively discouraging cheating.

    * First of all, the program would be "invite only".

    * Secondly, maybe a "startup mentaility" would be a better way to do something like this rather than having Palm run it. The company would succeed or fail and this would discourage cheating. Holy crap, is there anyone here with business smarts and/or Palm/Telco connections that would like to explore this idea to determine feasability. I could provide testing guidance and we could do this in addition to our day jobs.! Regardless of who would run the company, or if Palm would do it internally, there could be stock options based on performance and quality metrics such as: 1) initial quality / feedback from customer initial satisfaction surveys, 2) success of secrecy of issues found during beta, 3) success/sales of device, 4) success of Palm as a whole, and perhaps other similar ideas.

    * Thirdly, we'd have to capitalize on some social networking aspects in order to keep people honest: ie: we could have a rule where we can "ban" a member by something like a 2/3 membership vote or a majority vote of the board of directors. The banning could be done without cause, just a simple vote. Membership would be a privelage. Kind of like being an open-source developer! (Borrow the parts of that model that are transferrable.)

    * Get Palm-Plugged-in developers to agree to give a discount to beta test members. Palm could do this as a condition of continued membership in the plugged-in program.


    I guess those are my ideas so far to your comment about how to keep things honest. Anyone else with ideas please chime in!

    -- Bob
    I'm both super! ... and a doer!
  4. #4  
    Palm has a beta test program with all its carriers.

    This users are largely selected through a third party company.
  5.    #5  
    Quote Originally Posted by dutchtrumpet View Post
    Palm has a beta test program with all its carriers.

    This users are largely selected through a third party company.
    Good, already done.

    This means either: a) bugs are known and releases go ahead anyway or b) the beta program needs to be improved.

    If a), there needs to be faster followup work with developers and better transparency as per Bob-C's thread.

    If b), they need to take a serious look at how to make the program better.


    -- Bob
    I'm both super! ... and a doer!
  6. #6  
    Quote Originally Posted by bclancy View Post
    Good, already done.

    This means either: a) bugs are known and releases go ahead anyway or b) the beta program needs to be improved.

    If a), there needs to be faster followup work with developers and better transparency as per Bob-C's thread.

    If b), they need to take a serious look at how to make the program better.
    Palm builds a model, all carriers take a look at it and make recommendations as to what they want changed. Palm makes changes. Device is beta tested. Comes back to Palm. Device is fixed, tested again. Milestones are reached where the device is deemed ready to release. Device goes back to carrier for testing. Updates to software. Updates certified and tested. Updates released.

    This is why a beta program that you speak of is in place, because of all of these steps. However, development and testing is the loophole, not the ability of the developers (usually). If you want things to move faster, you need to change the testing methodology of those who purchase the devices build to order - aka the carriers.
    MMM | AntoineRJWright.com | BH | Jaiku

    Moved on to Symbian, but still will visit from time to time.

Posting Permissions