webOS Nation Forums >  webOS Developers >  webOS Development > Autoreplace Upgrade
Autoreplace Upgrade
  Reply
 
Thread Tools Display Modes
Old 06/21/2009, 04:34 PM   #1 (permalink)
Member
 
Posts: 54
EDIT: There are a few autoreplace list links in this post, but this is the one you want: http://drop.io/dimfeld/asset/autoreplace-common-zip

I spent the last few hours working on an upgrade to the autoreplace system. I found that when typing on the Pre, most of my typing mistakes would come from pressing the key either to the left or the right of the correct key. Personally, I never have a problem with hitting the wrong row, so that makes things somewhat simpler.

I wrote a Python script to read in a word list file (I used a standard Mac OSX word list with contractions added) and generate all the possible single-letter mistakes from that file that could result from pressing the button to the left or right of the intended button. This script also reads in the Pre's autoreplace file, and it gives precedence to the corrections in there. To keep the output file to a reasonable size, I didn't do corrections for words more than 7 letters long. With this restriction, the file is 9MB, and with a max length of 9 letters, it's 27MB.

I've uploaded the Python script, my modified word list, and the resulting output file at [removed because it's outdated]
I haven't included the Pre's autoreplace file here to avoid any IP concerns, but if you can get the replacement file in to the Pre, I'm sure you can get the original out

EDIT: The original file here causes Webkit to use a lot of memory and load slowly. I've uploaded a new file with words only up to 5 characters long at http://drop.io/dimfeld/asset/autoreplace5-zip which seems to work much better.

EDIT2: And here's another file with a word list generated from a list of the 15000 most common English words. I recommend using this one as it also has other improvements. See posts #21 through #24 for more details. http://drop.io/dimfeld/asset/autoreplace-common-zip

To run the script:
python generate_autoreplace.py original-autoreplace wordlist new-autoreplace

Upload the output file to
/etc/palm/autoreplace/en_us/text-edit-autoreplace

Then reboot your Pre and you're all set.

I haven't done extensive testing on this, so do this at your own risk, but it's working well for me so far.

Last edited by dimfeld; 06/25/2009 at 01:39 PM. Reason: Remove link to old huge list.
dimfeld is offline   Reply With Quote
Old 06/21/2009, 05:25 PM   #2 (permalink)
Member

 
Posts: 6,869
From my understanding, it seems to be the latter file that webOS uses.
jhoff80 is offline   Reply With Quote
Old 06/21/2009, 05:33 PM   #3 (permalink)
Member
 
FuManChuu's Avatar
 
Posts: 477
sweet mother of lord this is exactly what I was planning on doing, but had no idea in hell how to do it

Edit: what an odd collection of words :P However, a great improvement none the less
__________________
Just waiting for the day my Pre has contacts grouping, and a "speed dial" for text messaging

Last edited by FuManChuu; 06/21/2009 at 06:10 PM.
FuManChuu is offline   Reply With Quote
Old 06/21/2009, 07:03 PM   #4 (permalink)
Member
 
Posts: 1,395
Holly (can't swear, but imagine the 10 worse words you have ever heard).


Have you considered sending this to palm? Just make sure they make it optional...

Can't wait for the pre to come out in canada. one of the first things i will try.
nimer55 is offline   Reply With Quote
Old 06/21/2009, 07:12 PM   #5 (permalink)
Member
 
Posts: 7
I imagine it's more likely that the file in /etc is either placed there by the language selection app, or it's a symlink/hardlink to the other file.
posguy99 is offline   Reply With Quote
Old 06/21/2009, 07:20 PM   #6 (permalink)
Member
 
Posts: 211
crap u beat me to it lol... nonetheless great job!!!
knightflores is offline   Reply With Quote
Old 06/21/2009, 08:27 PM   #7 (permalink)
Member
 
Posts: 27
Nice job. Works great. Also I was really hoping for someone to make the autoreplace better. Thanks
jclusso is offline   Reply With Quote
Old 06/21/2009, 08:38 PM   #8 (permalink)
Member
 
Posts: 1,395
Could anyone add spell check, and grammar like the one microsoft word 2007 (the old words spell check is worse then nothing).

Or even firefox's spell would be great, dosen't do grammar, but who cares, spell check is good enough.
nimer55 is offline   Reply With Quote
Old 06/22/2009, 01:29 AM   #9 (permalink)
Member
 
Posts: 54
Glad everyone likes it so far. Unfortunately a large autoreplace list like this seems to cause Webkit to both use a lot of memory and take forever to start up. So email and the web browser are not too useful with this list. It may work better with a small max word length like 5 or 6, so I'll try that and report back.
dimfeld is offline   Reply With Quote
Old 06/22/2009, 01:58 AM   #10 (permalink)
Member
 
Posts: 54
Quote:
Originally Posted by posguy99 View Post
I imagine it's more likely that the file in /etc is either placed there by the language selection app, or it's a symlink/hardlink to the other file.
Well, that made me curious so I did a couple of tests. It's not a hard or symbolic link, and switching languages doesn't seem it update /etc/palm/text-edit-autoreplace either.

If that file and the other one are different, it uses /etc/palm/autoreplace/en_us/text-edit-autoreplace, which is in line with what everyone was saying. So it looks like /etc/palm/text-edit-autoreplace is simply unused.

On another note, performance with a file that has a max word length of 5 seems to be much better. The file is about 1.5 MB instead of 9 MB which I'm sure helps a lot. I've uploaded this to drop.io/dimfeld/asset/autoreplace5-zip
dimfeld is offline   Reply With Quote
Old 06/22/2009, 01:47 PM   #11 (permalink)
Member
 
Posts: 4
Quote:
Originally Posted by dimfeld View Post
Then just upload the output file to
/etc/palm/text-edit-autoreplace
and
/etc/palm/autoreplace/en-us/text-edit-autoreplace
you have a typo, it should be an underscore
/etc/palm/autoreplace/en_us/text-edit-autoreplace
sur4die is offline   Reply With Quote
Thanked By: Dogg006
Old 06/22/2009, 01:49 PM   #12 (permalink)
Member
 
Posts: 54
Fixed. Thanks for letting me know.
dimfeld is offline   Reply With Quote
Old 06/22/2009, 02:36 PM   #13 (permalink)
Member
 
sacherjj#AC's Avatar
 
Posts: 1,639
Need to get a good sample of emails, SMSs, etc and do a word frequency count. Should be able to reduce the base list quite a bit without reducing functionality much. i.e. Better performance with nearly the same functionality.
sacherjj#AC is offline   Reply With Quote
Old 06/22/2009, 02:36 PM   #14 (permalink)
Member
 
Posts: 46
Instead of shortening the word length to improve performance, is there a way to shortened the list to words that are commonly used? I don't think I will ever use "Zyzomys" or "xysti" (from the original 7 letter Autoreplace file).

I'm thinking of looking up something like "most commonly used words" or something like that. But thanks for getting the ball rolling on this. This is a really useful mod!
jngai is offline   Reply With Quote
Old 06/22/2009, 02:39 PM   #15 (permalink)
Member
 
Posts: 54
Yeah, a max word length was quick to implement, so I did that first. Changing the list to include only commonly-used words would definitely be more useful though.

You can use whatever word list you want with the python script I included in the ZIP file, so if you get a list of commonly used words, you can just feed that and the original autoreplace file into the script and you'll get a much better autoreplace file.
dimfeld is offline   Reply With Quote
Old 06/22/2009, 02:52 PM   #16 (permalink)
Member
 
sacherjj#AC's Avatar
 
Posts: 1,639
The hardest part is getting the list. It needs to not reformat common "slang" terms used often in texting and posting. It is hard to take just a wild feed from somewhere without filtering for all the misspellings people have done in the feed.
sacherjj#AC is offline   Reply With Quote
Old 06/22/2009, 03:55 PM   #17 (permalink)
Member
 
Posts: 1,395
Quote:
Originally Posted by sacherjj View Post
The hardest part is getting the list. It needs to not reformat common "slang" terms used often in texting and posting. It is hard to take just a wild feed from somewhere without filtering for all the misspellings people have done in the feed.
Yea, it's hard..
nimer55 is offline   Reply With Quote
Old 06/22/2009, 04:01 PM   #18 (permalink)
Member
 
FuManChuu's Avatar
 
Posts: 477
hrm, i'm all for this idea, so ill dig around and see if i can come up with a word list... I'm a tard though, so no promises <3
__________________
Just waiting for the day my Pre has contacts grouping, and a "speed dial" for text messaging
FuManChuu is offline   Reply With Quote
Old 06/22/2009, 04:04 PM   #19 (permalink)
Member
 
Posts: 54
There's a list of 15000 most commonly used English words at Sustainable English

It's based on a project from England though, so some British->American conversion will be needed before it's useful for most of us. Also need to add contractions and I'm not sure if it included plurals or not. Might be a good starting point though. I'm at work right now so I can't do more than post quick replies, but I'll be looking into this more later tonight.
dimfeld is offline   Reply With Quote
Old 06/22/2009, 09:44 PM   #20 (permalink)
Member
 
FuManChuu's Avatar
 
Posts: 477
I found this while googling...

Using Python to detect the most frequent words in a file | Zen and the Art of Programming

Its a python script, and many variants, for collecting commonly used words from a text file. It also has links to perl and command line alternatives.

Either way, what I'm looking for now, and it might be helpful if someone else has theirs and could try one of these scripts, are AIM logs / text message logs. Unfortunately I hard reset my phone all the time, so my treo logs are worthless, but I've got a buddy with about 6 months of iphone logs, and I'm sure I've got my AIM logs somewhere... I think those would be a pretty good starting point to run one of these scripts on.

Anyway, I'll post when I start filtering out muh logs <3
__________________
Just waiting for the day my Pre has contacts grouping, and a "speed dial" for text messaging
FuManChuu is offline   Reply With Quote
Reply

 

Tags
autoreplace

Thread Tools
Display Modes



 


Content Relevant URLs by vBSEO 3.6.0