PDA

View Full Version : clearing the swap memory


Pages : 1 [2] 3 4 5

kbrabble
05/16/2011, 10:44 PM
Just wanted to report, I have been running the latest methods (post 74 and the softswap install script) with very good results. I have been running it for two days now, and while I still needed a reboot once to keep things running smooth, there is definitely a lot less lag, and none of the minute + freeze ups I was experiencing before.

-- Sent from my Palm Pre using Forums (http://developer.palm.com/appredirect/?packageid=com.newnessdevelopments.forums)

theXfactor2011
05/17/2011, 12:33 AM
I would say to put things back to normal. restore any files you changed related to compcache. reboot, turn off compcache in govnah, run the softswap install script or copy the script posted in post 234 to a file in /var/palm/event.d and reboot. Once you reboot you should be able to run free -m from the terminal and see 159mb of free swap.

so have you noticed improvement with this over the change to lunasysmgr and adding the other sysctl file or are you using them all together?[/QUOTE]

I have removed the overcommit value in lunasysmanager, I have my sysctl.conf configured from a post back around page 5 or 6 and I'm running the install script I made a few days ago. Overall I would say performance has improved. Is it perfect, nope, can it get better, maybe. We are limited to palm's / hp's coding and 256mb of ram and there's only so much that can happen. But I am not rebooting every night and lag has dropped. If you haven't tried it but know what to do then I say go for it.

CptOx
05/17/2011, 08:16 AM
After disabling all methods for a week and undergo several freeze ups, I have begun to use both methods theXfactor2011 mentioned in previous post. In some days I hope I will report some improvement as it seems initially.

As some users are claiming what methods are being tested, I'm going to do the same now. I am using both methods at once:

1) Swappiness (modifying /etc/event.d/LunaSysMgr and creating /etc/sysctl.conf). Post #74 of Xanadu73 in 05/02/2011, 10:53 PM (http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-4.html#post2951712) (using the step-by-step instructions and files from Post #110 of NickVTPre in 05/06/2011, 05:56 AM (http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-6.html#post2954838)).

2) Softswap (install script creating /var/palm/event.d/SoftSwapEvent and enabling softswap). Post #234 of theXfactor2011 in 05/14/2011, 02:34 AM (http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-12.html#post2961656).

graffix31
05/17/2011, 01:47 PM
ok so i just ran the softswap script. am i not supposed to be able to reenable compcache after doing that? does this replace that more or less? if i remove it then everything is fine so i am assuming this is as intended.
thanks

theXfactor2011
05/17/2011, 03:28 PM
ok so i just ran the softswap script. am i not supposed to be able to reenable compcache after doing that? does this replace that more or less? if i remove it then everything is fine so i am assuming this is as intended.
thanks

So the script doesn't replace or really do anything to compcache at all. It all comes down to the modules that are loaded. xvmalloc and ramzswap.

when configuring ramzswap compcache uses different parameters then softswap. So softswap unloads the modules xvmalloc.ko and ramzswap.ko and reconfigures them. while ramzswap is active if you turn on compcache in govnah you will get an error that xvmalloc.ko and ramzswap.ko already exist.

The install script creates an event that is in /var/palm/event.d that runs after compcache is setup and just turns it off and reconfigures the modules and starts it back up again. compcache and softswap are excatly the same except the following:


#compcache
modprobe xvmalloc
modprobe ramzswap memlimit_kb=32768 backing_swap=/dev/store/swap



#softswap
insmod /lib/modules/`uname -r`/extra/xvmalloc.ko
insmod /lib/modules/`uname -r`/extra/ramzswap.ko disksize_kb=163840


side note: insmod and modprobe do exactly the same thing. Just modprobe has more options if need and is used by hp/palm. insmod pretty much loads the module and doesnt care about diagnostic feedback and stuff. more basic tool.

Its just a change in configuration. When softswap is loaded govnah doesnt know what to do since the modules are already active.

The uninstall command of the script turns off softswap, and then deletes the script. when softswap is off the modules are not loaded and govnah runs as normal.

If you run the script and then go into govnah and turn on compcache you should get an error. If compcache starts then softswap is not running. you can get it running by typing the following in the terminal.

initctl start /var/palm/event.d/SoftSwapEvent
or
sh /var/palm/event.d/SoftSwapEvent

graffix31
05/19/2011, 08:30 AM
i think this actually works pretty well. I notice a bit of a trade off. the huge minute or two hangs are gone but you get more slight hangs it seems. but the swap stays much lower and all and all as long as you arent trying to load a bunch of things at once everything runs nice. but with 256k of memory that is to be expected. thanks everyone for all of your work. this is using both methods btw.

rksand
05/19/2011, 01:23 PM
My experience with my Sprint Pre-, metadr 2.1, and F104A

With mods from post 74 only, swap slowly grew until I needed to restart my phone. It would become very laggy at about 70+ MB swap.

With mods from post 234 only, there appeared to be no swap at all based on running a top command in terminal (is that right?). I also was unable to run NFS as it froze my phone (which makes sense if I had no swap available).

Running both mods from posts 74 and 234 concurrently seems to be working well for the past 2 hours. I have not noticed any significant lag and have been able to load quit and load another memory intensive app without trouble. I have for the first time noticed I can run at 70+ MB swap without lag, and used swap has actually decreased too.

I'll post more as time passes.

Xanadu73
05/19/2011, 03:51 PM
Hey!

I've been keeping an eye on all this, I just have been running my brain ragged the past couple weeks and haven't had really been able to post much.

I have made a slight change to my /etc/sysctl.conf "tweaks". My current sysctl.conf looks like:

vm.swappiness = 0
vm.vfs_cache_pressure = 200
vm.page-cluster = 1

vm.dirty_expire_centisecs = 180000
vm.dirty_writeback_centisecs = 60000

A few big changes there. The times for the dirty_* stuff is the same. What I changed was "vm.swappiness", "vm.vfs_cache_pressure", and have added "vm.page-cluster". I'll explain:

I dropped the swappiness to zero because the kernel will only shove stuff out to swap when it's in DIRE need to free up RAM, *BUT*, putting the vfs_cache_pressure at 200 will make cached file/directory inodes VERY expendable and totally dump that first to free up RAM. The page-cluster is a bit more complex. First, from the kernel documentation:

page-cluster controls the number of pages which are written to swap in
a single attempt. The swap I/O size.

It is a logarithmic value - setting it to zero means "1 page", setting
it to 1 means "2 pages", setting it to 2 means "4 pages", etc.

The default value is three (eight pages at a time). There may be some
small benefits in tuning this to a different value if your workload is
swap-intensive.

Got that?!? :) Basically, reducing this value won't have any direct impact on what gets swapped and what doesn't, *BUT* what it will do is take smaller chunks at a time when it does. It'll still swap out what ever it's there to swap, but it should be FAR less I/O intensive of a operation hopefully reducing the HUGE lags we see when it is doing a lot of swapping.

Cool.

Bottom line:

Further reduced the kernels tendency to swap by making chached inodes VERY expendable (vfs_cache_pressure) and the kernel will do everything it can not to swap (swappiness). When it does swap, we shouldn't see the huge I/O lags that we currently do (page-cluster).

Good stuff!

Also, if you want to "watch" all this, check out "vmstat". Fire up SDL Terminal, and type "vmstat", you'll see:

root@My Pre:/# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 520 24140 4352 66640 0 0 110 24 414 548 15 4 72 9

That's FAR too wide for our little screens, so, I did this:

#!/bin/sh

vmstat -S K 5 | awk '{print $3, $4, $5, $6}'

Which looks like:

---swap-- -----io---- -system-- ----cpu----
swpd free buff cache
520 22692 4484 66768
520 22548 4484 66768

That fits really nice on our screens. It'll output a new line every 5 seconds so you can watch your system's VM at work. To make life easy on yourself, you may want to consider doing what I do to my Pre. Make a directory /media/internal/Scripts. Add that to your path in /etc/profile:

At the top you'll see:

PATH="/usr/local/bin:/usr/bin:/bin"

Change it to:

PATH="/usr/local/bin:/usr/bin:/bin:/media/internal/Scripts"


Drop all your scripts in /media/internal/Scripts and because that is now a part of your path, you don't have to type the full path to the script you want to run, just type it:

root@My Pre:/# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 520 23120 4572 66792 0 0 94 21 379 491 13 4 75 8
root@My Pre:/# ls -l /media/internal/Scripts/prevmstat
-rwxrwxrwx 1 57 May 19 11:42 /media/internal/Scripts/prevmstat
root@My Pre:/# prevmstat
---swap-- -----io---- -system-- ----cpu----
swpd free buff cache
520 23096 4572 66792
520 22952 4572 66792


Enjoy!

#----------
# EDIT:

I did change the way ramzswap is being loaded also. The relevant line in /etc/event.d/compcache looks like:

modprobe ramzswap disksize_kb=131072

That's 128*1024, of course. I decided to try using a nice "round" powers-of-two number that computers like. :)
#----------


M.

graffix31
05/19/2011, 04:09 PM
wow that was a lot and all makes sence. but if we are already using the old way what is the method to change everything to what you are saying? step by step would be much appreciated :-) a little much to wrap my head around. but when i can walk through it i can understand it much better.

Xanadu73
05/19/2011, 04:18 PM
wow that was a lot and all makes sence. but if we are already using the old way what is the method to change everything to what you are saying? step by step would be much appreciated :-) a little much to wrap my head around. but when i can walk through it i can understand it much better.


Really it's just changing a couple existing values in /etc/sysctl.conf and adding one more to it.

"Old":

vm.swappiness = 20
vm.vfs_cache_pressure = 180

vm.dirty_expire_centisecs = 180000
vm.dirty_writeback_centisecs = 60000(proposed) "New":

vm.swappiness = 0
vm.vfs_cache_pressure = 200
vm.page-cluster = 1

vm.dirty_expire_centisecs = 180000
vm.dirty_writeback_centisecs = 60000Two values and one line. I just had to drone on and on explaining the changes. :)

The bit at the end about making life easy on yourself by adding a Scripts dir to your path was more for those that don't really use Linux and/or those who didn't have it occur to them. Just a tip to pass on, ya know.


M.

grappler
05/19/2011, 04:31 PM
Really it's just changing a couple existing values in /etc/sysctl.conf and adding one more to it.

"Old":

vm.swappiness = 20
vm.vfs_cache_pressure = 180

vm.dirty_expire_centisecs = 180000
vm.dirty_writeback_centisecs = 60000(proposed) "New":

vm.swappiness = 0
vm.vfs_cache_pressure = 200
vm.page-cluster = 1

vm.dirty_expire_centisecs = 180000
vm.dirty_writeback_centisecs = 60000Two values and one line. I just had to drone on and on explaining the changes. :)

The bit at the end about making life easy on yourself by adding a Scripts dir to your path was more for those that don't really use Linux and/or those who didn't have it occur to them. Just a tip to pass on, ya know.


M.

What's your current thinking on the so-called "dangerous" mods you proposed earlier in the thread. I still have those in place, along with the old swappiness values. Also, are you running this alone, or with the softswap suggestions?

Xanadu73
05/19/2011, 04:42 PM
What's your current thinking on the so-called "dangerous" mods you proposed earlier in the thread. I still have those in place, along with the old swappiness values. Also, are you running this alone, or with the softswap suggestions?

Yeah, I did stick an edit in at the end of my huge rant:


I did change the way ramzswap is being loaded also. The relevant line in /etc/event.d/compcache looks like:

modprobe ramzswap disksize_kb=131072That's 128*1024, of course. I decided to try using a nice "round" powers-of-two number that computers like. :)

~~~~~~~

I don't think commenting out the two lines I had found are really all that "dangerous" any more. I had only called it that since I had/have zero idea what it does. Honestly, I don't know that it's done much of anything. I suppose the test would to put them back the way they were and see what happens. ;)


M.

rksand
05/19/2011, 05:10 PM
# EDIT:

I did change the way ramzswap is being loaded also. The relevant line in /etc/event.d/compcache looks like:

modprobe ramzswap disksize_kb=131072

That's 128*1024, of course. I decided to try using a nice "round" powers-of-two number that computers like. :)


I don't see that line in my /etc/event.d/compcache. I see the line:

modprobe ramzswap memlimit_kb=32768 backing_swap=/dev/store/swap

Was that something you added?

Xanadu73
05/19/2011, 05:24 PM
I don't see that line in my /etc/event.d/compcache. I see the line:

modprobe ramzswap memlimit_kb=32768 backing_swap=/dev/store/swapWas that something you added?


That's the line I changed. Essentially, that's all the whole script/upstart/etc. script is doing - just changing that one line. I (personally - and mean ZERO offense) don't see the purpose in the whole creating yet another thing to start at boot time to remove modules and putting them back and all that. All that has to happen is changing that one line (commenting out the original of course).


M.

graffix31
05/19/2011, 05:46 PM
so can i just go into the sysctl in internalz using the linux formatting of course and change and add those lines and restart
?

graffix31
05/19/2011, 05:48 PM
That's the line I changed. Essentially, that's all the whole script/upstart/etc. script is doing - just changing that one line. I (personally - and mean ZERO offense) don't see the purpose in the whole creating yet another thing to start at boot time to remove modules and putting them back and all that. All that has to happen is changing that one line (commenting out the original of course).


M.

so what changes are you currently using? are you saying you dont understand that new softswap script? sorry if i sound like a total noob with all of this. i am eating it all up though. love learning about it.

rksand
05/19/2011, 06:55 PM
so can i just go into the sysctl in internalz using the linux formatting of course and change and add those lines and restart
?

Be careful with internalz. See this link:

http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-5.html#post2953390

Xanadu73
05/19/2011, 07:03 PM
so can i just go into the sysctl in internalz using the linux formatting of course and change and add those lines and restart
?

You'll have to make / writable, of course as well. Yes, you can use Internalz just make sure you change the EOL setting or your phone may not boot!


M.

graffix31
05/19/2011, 07:26 PM
oh boy that doesnt sound good. so how would you replace the file? i have a copy of the original so i can make the changes. can i just follow the steps from the first time? will it overwrite it if it is the same file name? or is there a way i have to delete it?

Xanadu73
05/19/2011, 07:35 PM
Are you saying you don't understand that new softswap script?

I understand what the script is doing, and that's why I don't understand its existence. I mean, it looks like it should do exactly what it's advertised to do, but just changing the one line in /etc/event.d/compcache does the exact same thing. I guess it helps people who don't really understand what it is they are playing around with from totally hosing their phones, but, at the cost of another point of failure (the script not starting for whatever reason) and the user of a Pre- is (potentially!) left with no swap at all seems to be a fairly high price to pay.

I like the idea, and it *looks* like it should work fine, and I'm sure it helps those that seem to be afraid of text on a black screen, so I say if it works for people go for it. Novaterm (or SDL Terminal when on-the-go) and Nano is much simpler, though.


M.

Xanadu73
05/19/2011, 07:45 PM
oh boy that doesn't sound good. so how would you replace the file? i have a copy of the original so i can make the changes. can i just follow the steps from the first time? will it overwrite it if it is the same file name? or is there a way i have to delete it?

I'd have to dig though here to see what "the first time" was, but if it worked then, there's no reason it wouldn't now.

I'm just saying that with Internalz (as Loopy found out for us...), the "File Type" needs to be changed to Linux. For a reason I'll probably never come to grips with, a text editor on a Linux OS, able to modify Linux files, should default to a Linux file type. I'm sure Jason had his reasons for setting the default to the Windows EOL, but, I don't see it.

Then again, I only support Windows for money, I don't actually *use* the P.O.S., so 95%+ of my files are Linux EOLs. The ones that aren't are .vbs scripts I've cobbled together over the years for Group Policy pushing, SCCM package installs, Machine setup/imaging, registry hacking scripts, auto-printer installs and migrations (I rather liked that one - took almost a week to get 100%, but it works great), etc., so they need to stay Windows EOL. *BUT* that's what Unix2Dos (http://www.xs4all.nl/%7Ewaterlan/dos2unix.html) is for... :)


M.

graffix31
05/19/2011, 07:47 PM
ok i can dig it, ltm. i know how to revert the soft swap script. so here is my question. what is the best method to one, change the settings in the sysctl and then two to change the line in compcache.event.d?

thanks for your help and clarrification.

edit, just saw your second post. so if i repeat the method used to install sysctl the first time using the edited file will it overwrite the original? and how did you modify the other event.d file. again sorry for all the questions but thanks for all the help.

Xanadu73
05/19/2011, 08:02 PM
ok i can dig it, ltm. i know how to revert the soft swap script. so here is my question. what is the best method to one, change the settings in the sysctl and then two to change the line in compcache.event.d?

thanks for your help and clarrification.

edit, just saw your second post. so if i repeat the method used to install sysctl the first time using the edited file will it overwrite the original? and how did you modify the other event.d file. again sorry for all the questions but thanks for all the help.


I don't know that there is a "best way". It's just two different approaches to same solution. Use which ever you feel most comfortable with (not "you" as *you*, but "you" as in whomever is reading this).

~~~~~~~

If you're referring to NickVTPre's post (#110) with the step by step thing, then yes, assuming you've remember to make the file system writable, it'll over-write the existing file. You can always verify afterward (and you probably should just to be on the safe side - it never hurts to be on the safe side of things).


M.

graffix31
05/19/2011, 08:26 PM
ok so if i also edit the compcache file i can then send it the same way as sysctl? i have changed my test editor to linux formatting in internalz. i made a back up copy of compache.d so can i just edit that in there or should i use the method i just said?

graffix31
05/19/2011, 08:43 PM
NickVTPre where are you for the write up, ltm ltm ltm

rmausser
05/19/2011, 10:36 PM
hey so I was wondering, when you set compcache to 131 072 kilobytes (128mb) it shows up in Govnah as "36796". Why is this??

Xanadu73
05/19/2011, 10:40 PM
hey so I was wondering, when you set compcache to 131 072 kilobytes (128mb) it shows up in Govnah as "36796". Why is this??

I have no idea since we're disabling compcache in Govnah so it didn't interfere with the things we've been playing with...


M.

rksand
05/19/2011, 10:51 PM
If it shows up at all in Govnah, then softswap isn't running.

Xanadu73
05/19/2011, 10:52 PM
ok so if i also edit the compcache file i can then send it the same way as sysctl? i have changed my test editor to linux formatting in internalz. i made a back up copy of compache.d so can i just edit that in there or should i use the method i just said?

Oops, didn't see this...

You can edit it any way you feel comfortable editing very low-level system files. As Loopy's experience taught everyone reading these words, one wrong move and the thing won't boot.

Though I do have to add again, if the "SoftSwap" script thing has been working fine for you, I'm not all that sure it'd be a good idea to back out of it now. This is super pre-alpha grade tests here. It's rarely a good idea to change a lot at once.

It does baffle me though: If people are so willing to potentially make their phone not boot again, why aren't people taking the time to learn to do these things the right way? Novaterm to the thing, or use SDL Terminal, edit the file with nano. Reboot. Boom. Done. It's never a good idea to only know GUI ways to "work on computers".

What happens when the GUI can't load?


M.

rksand
05/19/2011, 11:13 PM
Personally, I love openssh. Wifi loads early in the boot process and I can putty or filezilla in before luna loads - no cables. With a shared key, its very secure. With 3 basic commands, vi is very easy to learn and use: load, insert, save. Its All in the wiki:

Application:OpenSSH (http://en.m.webos-internals.org/wiki/Application:OpenSSH)

rksand
05/19/2011, 11:33 PM
Additionally, when you edit a file, comment out notes to yourself with a # at the start of the line. I can't always remember what mods I do months later, it makes it easy to see what I did, when, why, and I can reference the source.

theXfactor2011
05/20/2011, 12:20 AM
Looks like this thread is moving again. Wooooo! So I'm also guessing people upgraded to the new F105-51 kernel?

Just to join the convo, im a ubuntu kind of guy. The terminal is a powerful tool and if you havent used it you should give it a try. I normally do my editing in vi but every now and then I have to use emacs :D

rmausser
05/20/2011, 01:23 AM
the F105-51 kernel is giving me problems.

It wont screenstate from 500mhz to 1.05ghz.

It just sticks at 500mhz. Reinstalled govnah, same problem.

Any suggestions?

Xanadu73
05/20/2011, 07:26 AM
the F105-51 kernel is giving me problems.

It wont screenstate from 500mhz to 1.05ghz.

It just sticks at 500mhz. Reinstalled govnah, same problem.

Any suggestions?


*I* only use Harrier (I personally have found webOS 2.1 to like a constant freq better - 1.4.5 liked to scale), but I believe 'Psycho changed the charger poll rate to FULL seconds now and Govnah doesn't know how to handle that yet. So I think what you're sing is not that is *doesn't* scale, it's just taking FAR longer than you think it should. If you have it set at "100", let's say, that's 100 FULL seconds now. I do believe Rod/oil are aware of the change and a new rev of Govnah should be out soon.


M.

graffix31
05/20/2011, 08:30 AM
well yeah the soft swap is actually working but if there is another way to get the same result without the chance of having it not start up and run out of memory. i dont want to have to check it every time i restart. if your way of changing the line in compache essentially does the same then i would like to try it.

NickVTPre
05/20/2011, 08:36 AM
well yeah the soft swap is actually working but if there is another way to get the same result without the chance of having it not start up and run out of memory. i dont want to have to check it every time i restart. if your way of changing the line in compache essentially does the same then i would like to try it.

It would be great to have upstart run this on boot, i agree.
The other option i'm advocating for, and i'm repeating myself, is to request that commands be executable by Mode Switcher. Sconix has this planned for a future release, and would probably up-prioritize it if a bunch of folks post in that thread: http://forums.precentral.net/enlightened-linux-solutions/279166-app-mode-switcher-2-x.html

rmausser
05/20/2011, 09:41 AM
*I* only use Harrier (I personally have found webOS 2.1 to like a constant freq better - 1.4.5 liked to scale), but I believe 'Psycho changed the charger poll rate to FULL seconds now and Govnah doesn't know how to handle that yet. So I think what you're sing is not that is *doesn't* scale, it's just taking FAR longer than you think it should. If you have it set at "100", let's say, that's 100 FULL seconds now. I do believe Rod/oil are aware of the change and a new rev of Govnah should be out soon.


M.

How do you find battery life to be on 1 ghz constantly? battery life is very important to me.

graffix31
05/20/2011, 10:14 AM
Oops, didn't see this...

You can edit it any way you feel comfortable editing very low-level system files. As Loopy's experience taught everyone reading these words, one wrong move and the thing won't boot.

Though I do have to add again, if the "SoftSwap" script thing has been working fine for you, I'm not all that sure it'd be a good idea to back out of it now. This is super pre-alpha grade tests here. It's rarely a good idea to change a lot at once.

It does baffle me though: If people are so willing to potentially make their phone not boot again, why aren't people taking the time to learn to do these things the right way? Novaterm to the thing, or use SDL Terminal, edit the file with nano. Reboot. Boom. Done. It's never a good idea to only know GUI ways to "work on computers".

What happens when the GUI can't load?


M.


can you recommend something i can read to better understand how to do things through terminal, i have been using it more but would like to understand it more so i can make those changes as you said. i want to edit those two files and remove the soft swap script.

rksand
05/20/2011, 10:41 AM
can you recommend something i can read to better understand how to do things through terminal, i have been using it more but would like to understand it more so i can make those changes as you said. i want to edit those two files and remove the soft swap script.

Start here:

Basic Linux Use - WebOS Internals (http://www.webos-internals.org/wiki/Basic_Linux_Use)

theXfactor2011
05/20/2011, 10:49 AM
the F105-51 kernel is giving me problems.

It wont screenstate from 500mhz to 1.05ghz.

It just sticks at 500mhz. Reinstalled govnah, same problem.

Any suggestions?

I noticed that. I deleted my profiles, pulled the battery recreated a profile to my liking. seems to work fine now.

Xanadu73
05/20/2011, 11:46 AM
Just found this while doing some research. It's quite informative, actually.

Help! Linux ate my RAM! (http://www.linuxatemyram.com/)


M.

graffix31
05/20/2011, 02:10 PM
NickVTPre, can you make another sysctl with the new settings? i dont have a linux system here at work i can do it on. that is how you have to do it correct? then you can just send the file? or can i edit the file through internalz using the linux formatting and then send it?

xanadu can you post exactly what your compache file looks like? i dont want to take anything out that shouldnt be based on your abbreviated posts.

thanks

Xanadu73
05/20/2011, 02:30 PM
Xanadu can you post exactly what your compache file looks like? i dont want to take anything out that shouldnt be based on your abbreviated posts.


I wasn't aware they were "abbreviated". I've pasted exactly what /etc/sysctl.conf should look like (http://forums.precentral.net/2967404-post260.html) and exactly the line that needs to change in /etc/event.d/compcache (http://forums.precentral.net/2967462-post262.html) (even going as far as saying that the original should be commented out so it's easier to go back - which really shouldn't have to be said).

Make the edits, save, reboot. I'm not sure I understand where the confusion is coming from.

You may want to wait, though. I'm starting to think, after watching the VM system, that I don't know that a swappiness of zero is a good idea. I do still think that the page-cluster should be lower than default to hopefully reduce I/O hit. I'm trying that at zero now with a couple other ideas also.


M.

graffix31
05/20/2011, 02:34 PM
ok i will wait a bit. i am sorry for all of the questions. i just want to leave as little to chance as possible. maybe i think the line in compache is more than is actually there because of how it looked in internalz. i felt like there was a lot more to the line at the end then what you showed. i dont mean to be a pain and apologize if i am. just trying to make sure i fully understand and can see everything cleanly before i start playing.

graffix31
05/21/2011, 03:08 PM
i just edited sysctl in ubuntu and then replaced it on the pre. no when i go to check it, the file opens in word not in internalz. does this mean the formatting is wrong and the file is doing nothing?

rksand
05/21/2011, 03:19 PM
...I'm starting to think, after watching the VM system, that I don't know that a swappiness of zero is a good idea. ...
M.

I just changed from sysctl.conf settings from post 258 back to those of post 74. I had found that my swap grew to over 100MB and I had to reboot. This happened several times. I think the settings from post 74 worked better for me.

I don't think it makes a difference, but I did change from F104 to the just released F105 kernel. Love the new boot screen!

graffix31
05/21/2011, 03:25 PM
ok well i just made the change to sysctl, just adding the new line not changing the swappiness value and added the new line in compcache. everything is booting properly, will report using soon. i to am also thinking about changing from f-105 from f-104.

Xanadu73
05/21/2011, 04:11 PM
I just changed from sysctl.conf settings from post 258 back to those of post 74. I had found that my swap grew to over 100MB and I had to reboot. This happened several times. I think the settings from post 74 worked better for me.


Yes, that was a bad idea. I was just sitting and watching swap counting up for no reason. New plan of attack test.


vm.swappiness = 30
vm.vfs_cache_pressure = 200
vm.page-cluster = 8


M.

graffix31
05/21/2011, 04:26 PM
Looking forward to the results.


-- Sent from my Palm Pre using Forums (http://developer.palm.com/appredirect/?packageid=com.newnessdevelopments.forums)

loopytee
05/21/2011, 06:53 PM
Yes, that was a bad idea. I was just sitting and watching swap counting up for no reason. New plan of attack test.


vm.swappiness = 30
vm.vfs_cache_pressure = 200
vm.page-cluster = 8


M.

just to be sure, if testing this, i will replace the code to what is written above only, i should not include the other bit of code from the last test (vm.dirty.....)

graffix31
05/22/2011, 10:57 AM
Well so far so good. things have been running pretty smooth. no real issues. this is changing the line in compcache and addind the page cluster line to sysctl. everything else was the same. the swap memory does get high but i dont see the huge lag when it does. continues to run smooth with a few momentary lags here and there.

graffix31
05/22/2011, 05:34 PM
OK well after using it for a while swap hit about 85 and it had a good freeze. came back after about a minute. anyone try the new settings for sysctl? wondering how that is.

snpalavan
05/22/2011, 05:48 PM
I of course appreciate all of the time and effort people are putting into this, however, noting the title, there has yet to be any reduction of the memory haha. I have tested all of these various methods, but they each attempt to get the phone to manage memory better, but there hasn't been a solid solution on how to get the phone to lose unncessary/unused memory.

With that said, i want to reurge the fact that I appreciate all the hard work and also that I am not even sure it is possible to tell Linux to throw away memory.

rmausser
05/22/2011, 07:59 PM
I definitely think there has been progress, my phone is much better.

However, I feel that the options in post 74 really help, as well as the very latest sysctrl, BUT I honestly think that softswap is crap.

Its good until a lot of tasks happen at once, and then your phone just bogs down like crazy, freezes, and you gotta battery pull.

Honestly, so far I have found that keeping the compcache at 10mb has proved the best, with the settings in post 74.

Why? Because that is how 2.1 was written with memory management in mind by Palm.

Changing compcache changes the swap size and the RAM size, and thus it really messes up how the kernel and OS was set to manage memory.

That being said, this OS was really written to use 256mb of ram, not 512, but compcache isnt going to solve that.

What we need is to find out where to possibly change the ram size to 256mb, if there are settings like that. (I hope)

Also I would like to note that the RAID buffer settings in bootmisc.sh to 0 does have a detrimental effect: playing music over bluetooth stutters and skips. I think thats what the buffer is for.

graffix31
05/22/2011, 10:22 PM
when you say latest systctl did you just change the 3 values and leave the other 2 alone or does it only consist of the three values noted?

graffix31
05/23/2011, 09:48 AM
I to just took off the softswap. everything seems to be working well, still wondering what settings you are using when you say new sysctl settings.
thanks

UncleVanya
05/23/2011, 09:51 AM
That being said, this OS was really written to use 256mb of ram, not 512, but compcache isnt going to solve that.

What we need is to find out where to possibly change the ram size to 256mb, if there are settings like that. (I hope)


'eh? :confused:

Why on Earth would you reduce your RAM from 512 to 256?

graffix31
05/23/2011, 10:14 AM
no he is talking about in the software everything is set for 512 because 2.1 was designed for a pre+ or pre2 not a pre minus with 256mb therefor the memory allocation is off.

theXfactor2011
05/23/2011, 10:37 AM
Sorry to be missing in action for a few days but I'm back. So I decided to mix and mangle some of the recent tweaks. So here is what I got

1. Softswap with the 128mb swap size
2. my sysctl.conf (/etc/sysctl.conf)


vm.swappiness = 20
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 120000
vm.dirty_writeback_centisecs = 60000
vm.page-cluster = 5
vm.dirty_background_ratio = 60
vm.dirty_ratio = 95


vm.dirty_ratio - The highest % of your memory that can be used to hold dirty data. If you set this to a low value, the kernel will flush small writes to the disk more often. Higher values allow the small writes to stack up in memory. They'll go to the disk in bigger chunks.

vm.dirty_background_ratio - The lowest % of your memory where pdflush is told to stop when it is writing dirty data. You'll want to keep this set as low as possible.

In short, when your memory begins filling with little pieces of data that needs to be written to the disk, it will keep filling until it reaches the dirty_ratio. At that point, pdflush will start up, and it will write data until it reduces the dirty data to the value set by dirty_background_ratio.


So why did I choose a high value for dirty_ratio: Its a phone with very little memory and our swap shares that memory. Also most of the things we do on our phone only increments the memory by a little bit. We dont want to be constantly firing up disk I/O every time we open a program or close it. So using a high number we will up the memory and then once we hit the 95% marker it all purges.

So why not set dirty_background_ratio to some thing lower: Once the flush begins its going to cause the phone to slow up. So by making it a 60% dump I believe we can clear out what we need fast enough to avoid too much lag.

Both of these values are experimental. Feel free to try them and let us know if you see any improvement.

ps.
the page-cluster = 8 seems like it might be a little high. I may mess with that a little more.

graffix31
05/23/2011, 10:44 AM
interesting settings. look forward to some results. i dont have a linux system to change sysctl with so i cant try it while i am at work. i have the original sysctl settings but i added page-cluster = 1. running that with with no softswap.

theXfactor2011
05/23/2011, 11:22 AM
My thoughts on the page-cluster

I came across this link (http://www.redhat.com/magazine/001nov04/features/vm/). The key part I found was this:

page-cluster

The kernel attempts to read multiple pages from disk on a page fault to avoid excessive seeks on the hard drive. This parameter defines the number of pages the kernel tries to read from memory during each page fault. The value is interpreted as 2page-cluster pages for each page fault. A page fault is encountered every time a virtual memory address is accessed for which there is not yet a corresponding physical page assigned or for which the corresponding physical page has been swapped to disk. If the memory address has been requested in a valid way (for example, the application contains the address in its virtual memory map), then the kernel associates a page of RAM with the address or retrieves the page from disk and places it back in RAM. Then the kernel restarts the application from where it left off. By increasing the page-cluster value, pages subsequent to the requested page are also retrieved, meaning that if the workload of a particular system accesses data in RAM in a linear fashion, increasing this parameter can provide significant performance gains (much like the file readahead parameters described earlier). Of course if your workload accesses data discreetly in many separate areas of memory, then this can just as easily cause performance degradation.

So would we say the palm pre is more linear memory loading or discreetly accessing separate areas. with a page-cluster = 8 we could load a max loading 256mb of memory worth of cache every time we swap out. Since we dont use 256mb of cache normally we basically touch every thing. possibly never making the cache dirty hence it will never clear.

I'm trying a new setting in sysctrl.conf
page-cluster = 5

Reasoning is that this will pull in 32mb of cache around the program we have loaded. Most likely in our case of phone use and phone apps should be plenty. That should also leave the unused stuff to be wiped out. On that note a page-cluster = 4 would be 16mb and would be a good test also.

graffix31
05/23/2011, 01:04 PM
other than using terminal to edit thie sysctl file is there another app you can use? it opens up in quick office and not internalz so i worry about the line formatting not being in linux.
thanks

trying to use the vi command but fumbling a bit. thank god i figured out how to exit without saving. just having a hard time with all of the cursor position and text editing commands

loopytee
05/23/2011, 01:47 PM
open up launcher, tap upper left for drop down, select default applications. All the extensions are listed, tap on the one you want (.conf) and select internalz pro. I don't know if this holds through a reset ( I think it does) now, as long as you have chosen text editing, linux, in internalz preferences, the file should open through Internalz.

graffix31
05/23/2011, 02:05 PM
Awesome Thanks :-)
i knew you could change the defaults like that. didnt even think to look there for something like this.

ryancell
05/23/2011, 02:25 PM
This swap getting filled up makes me think its why the Pre minus gets slow w/ 2.1, a memory/swap bug, not the hardware. I redoctored a spare pre to wifi only 2.1 and it was snappy again. Its just after time the OS slows itself down for some reason and gets more bugs that cause it to slow down. If we didnt have these kind of bugs, I almost feel 2.1 would be more stable/less lag, but the 256mb ram isnt helping. I may read more into this swap discussion because mines getting unbearable at times, but not always.

Xanadu73
05/23/2011, 03:29 PM
So would we say the palm pre is more linear memory loading or discreetly accessing separate areas. with a page-cluster = 8 we could load a max loading 256mb of memory worth of cache every time we swap out. Since we dont use 256mb of cache normally we basically touch every thing. possibly never making the cache dirty hence it will never clear.

I'm trying a new setting in sysctrl.conf
page-cluster = 5

Reasoning is that this will pull in 32mb of cache around the program we have loaded. Most likely in our case of phone use and phone apps should be plenty. That should also leave the unused stuff to be wiped out. On that note a page-cluster = 4 would be 16mb and would be a good test also.


I agree 100% with the logic as this is where my mind was going with it all: QUICKLY dump/grab "memory" for current allocation needs - HUGE I/O hit, yes, but it'd be over with "faster" because it's one/two LARGE reads/writes rather than a bunch of small ones (in theory).

However, I don't understand your math because the Linux page size is (usually) only 4K. So, (8*2)*4096 == 65,536K == 64M. Agreed that's kind of large, I've dropped it down as well to 5: (5*2)*4096 == 40,960K == 40M.

The "times two" is from the Kernel Documentation:



page-cluster controls the number of pages which are written to swap in
a single attempt. The swap I/O size.

It is a logarithmic value - setting it to zero means "1 page", setting
it to 1 means "2 pages", setting it to 2 means "4 pages", etc.

(I'm not a math major... That pattern looks like "times two" to me! :) )


I'll see how this goes!




M.

theXfactor2011
05/23/2011, 04:38 PM
I agree 100% with the logic as this is where my mind was going with it all: QUICKLY dump/grab "memory" for current allocation needs - HUGE I/O hit, yes, but it'd be over with "faster" because it's one/two LARGE reads/writes rather than a bunch of small ones (in theory).

However, I don't understand your math because the Linux page size is (usually) only 4K. So, (8*2)*4096 == 65,536K == 64M. Agreed that's kind of large, I've dropped it down as well to 5: (5*2)*4096 == 40,960K == 40M.

The "times two" is from the Kernel Documentation:



(I'm not a math major... That pattern looks like "times two" to me! :) )


I'll see how this goes!




M.

I'll admit I pulled my math out of no where lol. It was monday morning and I completely forgot that we were talking about page sizes. I also over looked the proper formula. I was thinking logs. 2^8 in mb. But at least we aimed for the same goal.

UncleVanya
05/23/2011, 05:20 PM
The "times two" is from the Kernel Documentation:

(I'm not a math major... That pattern looks like "times two" to me! :) )


From another source (http://www.kernel.org/doc/Documentation/sysctl/vm.txt):

page-cluster

page-cluster controls the number of pages which are written to swap in
a single attempt. The swap I/O size.

It is a logarithmic value - setting it to zero means "1 page", setting
it to 1 means "2 pages", setting it to 2 means "4 pages", etc.

The default value is three (eight pages at a time). There may be some
small benefits in tuning this to a different value if your workload is
swap-intensive.


2x would mean 3 = 6 but it clearly = 8 per the source I found. While it clearly says LOG it doesn't say what base but after looking it's clear this is LOG2 not LOG10. So to figure out the impact of a given setting simple raise 2 to that value. Thus 0=1;1=2; 2=4; 3=8; 4=16; 5=32; 6=64; 7=128; 8=256.

NOTE: This indicates the number of pages not KB or MB of RAM as pointed out earlier in the thread.

UncleVanya
05/23/2011, 05:22 PM
no he is talking about in the software everything is set for 512 because 2.1 was designed for a pre+ or pre2 not a pre minus with 256mb therefor the memory allocation is off.

Thank you - I read that backwards from what he was meaning - english (while my first and only really proficient language) is a loosely interpreted thing.

UncleVanya
05/23/2011, 05:37 PM
My thoughts on the page-cluster

So would we say the palm pre is more linear memory loading or discreetly accessing separate areas. with a page-cluster = 8 we could load a max loading 256mb of memory worth of cache every time we swap out. Since we dont use 256mb of cache normally we basically touch every thing. possibly never making the cache dirty hence it will never clear.

I'm trying a new setting in sysctrl.conf
page-cluster = 5

Reasoning is that this will pull in 32mb of cache around the program we have loaded. Most likely in our case of phone use and phone apps should be plenty. That should also leave the unused stuff to be wiped out. On that note a page-cluster = 4 would be 16mb and would be a good test also.

Is page size 1MB? Page-Cluster setting of 5 = 32 pages - each page would have to be 1MB for your concerns to be true. Page-Cluster set to 8 = 256 pages which at 4KB per page is 1MB which doesn't seem like too much memory to be dirty unless I'm confusing myself. In fact due to the way Flash is prone to wearing out due to writes I would think you might want to raise the limit not lower it - all of this is based on a 4KB page size default.

I was unable to verify that webOS uses a page-size of 4KB due to the fact that the getconf command suggested by several sources doesn't seem to be included in the default shell and I didn't feel like digging around for another means of verification.

Xanadu73
05/23/2011, 06:18 PM
I was unable to verify that webOS uses a page-size of 4KB due to the fact that the getconf command suggested by several sources doesn't seem to be included in the default shell and I didn't feel like digging around for another means of verification.


I ran into the same thing. I even went through 'Psycho's .config and grep'ed the arch/arm tree of the kernel source. I don't see an actual "default" being set (though I'm no developer...). I do see one of the options in the source being set to 4k, and I found a post on LKML from a higher up kernel hacker (who's name escapes me at the moment) that arm is locked at 4k. That was from late 2010. Linux 2.6.24 is quite a lot older than that. Still, I think it's fairly safe to assume a default of 4k page size.


M.

rmausser
05/23/2011, 07:38 PM
These latest sysctrl settings seem too aggressive.

My swap is constantly at 0. I think this is too much, as its relying too heavily on RAM, which is not plentiful on the Pre-

I'm going to try the settings with compcache disabled though, just to see how that works, considering its putting nothing in it anyways.

graffix31
05/23/2011, 07:51 PM
What newest settings are you talking about. There are a lot being thrown around now.

-- Sent from my Palm Pre using Forums (http://developer.palm.com/appredirect/?packageid=com.newnessdevelopments.forums)

Xanadu73
05/23/2011, 07:54 PM
These latest sysctrl settings seem too aggressive. Ĥ

My swap is constantly at 0. I think this is too much, as its relying too heavily on RAM, which is not plentiful on the Pre-

I'm going to try the settings with compcache disabled though, just to see how that works, considering its putting nothing in it anyways.


I've never seen that. Verify your swap even got initialized. "free -tm"

M.

rmausser
05/23/2011, 07:57 PM
hmm maybe it didnt. Its hanging around 30mb now. (after reboot) With no compcache my phone is flying. Lets see if this lasts a day.

rmausser
05/23/2011, 08:07 PM
no compcache is no go. PDK games hang and stutter like crazy.

Hmm...is there any way to give RAM priority to the UI? I dont mind if things are slow, I just hate when the whole OS is unresponsive.

I'd like to be able to give priority to the UI so that even if an app locks up, you can still swipe it away or whatever.

theXfactor2011
05/24/2011, 02:20 PM
So last night my pre- finally filled up all its swap and ram. :shake: Lets just say things were horrible. Nothing worked and battery pull was required. We have done an amazing job at avoiding swap as much as possible. However once stuff dumps into swap it just sits there. It seems like the vm stuff helps with memory but nothing really apply's to swap. Yeah it will drop 3 -4 mbs every now and then but when we dump 20mb into it when we open a a game or a ton of apps its not good enough. The only true way to ensure a clear swap is to turn it off and on again. however doing that manually sucks and doing by reading the total remaining or in use and just turning off and on could really suck if we are doing something. So I propose we try something. I'm trying to work on script that looks at the swap usage every x mins and then if its at a high point aka 50 mb out of 128 then look at the cpu / system load. if the load is blow a threshold we echo 3 > /proc/sys/vm/drop_caches and then swapoff /dev/ramzswap and swapon /dev/ramzswap. what does that do. The 3 drops everything not in use either to disk or swap. then we wipe out the swap. I figure setting the check value low ensures that it doesnt take long to clear and if the cpu load is high then we still have a good safe buffer of swap available until the phone is good again.

I found this script to check for cpu load in percentage but I cant get it to work on the pre- i get errors that Im not sure how to solve. still learning. But if you know what your doing then feel free to drop it on /media/internal and run it. all it does is loop through every second and display the cpu utilization in %. I figure we can use some of this code and make the magic happen.


#!/bin/bash
# by Paul Colby (http://colby.id.au), no rights reserved ;)

PREV_TOTAL=0
PREV_IDLE=0

while true; do
CPU=('cat /proc/stat | grep '^cpu '') # Get the total CPU statistics.
unset CPU[0] # Discard the "cpu" prefix.
IDLE=${CPU[4]} # Get the idle CPU time.

# Calculate the total CPU time.
TOTAL=0
for VALUE in "${CPU[@]}"; do
let "TOTAL=$TOTAL + $VALUE"
done

# Calculate the CPU usage since we last checked.
let "DIFF_IDLE=$IDLE-$PREV_IDLE"
let "DIFF_TOTAL=$TOTAL-$PREV_TOTAL"
let "DIFF_USAGE=(1000*($DIFF_TOTAL-$DIFF_IDLE)/$DIFF_TOTAL+5)/10"
echo -en "\rCPU: $DIFF_USAGE% \b\b"

# Remember the total and idle CPU times for the next check.
PREV_TOTAL="$TOTAL"
PREV_IDLE="$IDLE"

# Wait before checking again.
sleep 1
done


to run type bash what_ever_you_saved_as

the errors I get are:

cpu_usage: line 15: let: TOTAL=0 + : syntax error: operand expected (error token is " ")
cpu_usage: line 21: let: DIFF_USAGE=(1000*(0-0)/0+5)/10: division by 0 (error token is "+5)/10")


I figure 21 is because nothing happened on line 15.

Xanadu73
05/24/2011, 02:23 PM
#!/bin/bash
# by Paul Colby (http://colby.id.au), no rights reserved ;)

PREV_TOTAL=0
PREV_IDLE=0

while true; do
CPU=('cat /proc/stat | grep '^cpu '') # Get the total CPU statistics.
unset CPU[0] # Discard the "cpu" prefix.
IDLE=${CPU[4]} # Get the idle CPU time.

# Calculate the total CPU time.
TOTAL=0
for VALUE in "${CPU[@]}"; do
let "TOTAL=$TOTAL + $VALUE"
done

# Calculate the CPU usage since we last checked.
let "DIFF_IDLE=$IDLE-$PREV_IDLE"
let "DIFF_TOTAL=$TOTAL-$PREV_TOTAL"
let "DIFF_USAGE=(1000*($DIFF_TOTAL-$DIFF_IDLE)/$DIFF_TOTAL+5)/10"
echo -en "\rCPU: $DIFF_USAGE% \b\b"

# Remember the total and idle CPU times for the next check.
PREV_TOTAL="$TOTAL"
PREV_IDLE="$IDLE"

# Wait before checking again.
sleep 1
done
to run type bash what_ever_you_saved_as

the errors I get are:

cpu_usage: line 15: let: TOTAL=0 + : syntax error: operand expected (error token is " ")
cpu_usage: line 21: let: DIFF_USAGE=(1000*(0-0)/0+5)/10: division by 0 (error token is "+5)/10")


I figure 21 is because nothing happened on line 15.


Fist things first, the webOS's shell is not bash, it's ash. That first line should be changed to:

#!/bin/sh

I'm playing with the rest of it.

M.

CptOx
05/24/2011, 02:59 PM
A week ago from this post:
After disabling all methods for a week and undergo several freeze ups, I have begun to use both methods theXfactor2011 mentioned in previous post. In some days I hope I will report some improvement as it seems initially.

As some users are claiming what methods are being tested, I'm going to do the same now. I am using both methods at once:

1) Swappiness (modifying /etc/event.d/LunaSysMgr and creating /etc/sysctl.conf). Post #74 of Xanadu73 in 05/02/2011, 10:53 PM (http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-4.html#post2951712) (using the step-by-step instructions and files from Post #110 of NickVTPre in 05/06/2011, 05:56 AM (http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-6.html#post2954838)).

2) Softswap (install script creating /var/palm/event.d/SoftSwapEvent and enabling softswap). Post #234 of theXfactor2011 in 05/14/2011, 02:34 AM (http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-12.html#post2961656).
Slowly, after three days all swap full and Pre- stuck. After that, disabling Method 2) Softswap but only with Method 1) Swappiness but stoppages appears. Also disabled Method 1).
And as Sconix won't add support for executing commands in MS service (http://forums.precentral.net/enlightened-linux-solutions/279166-app-mode-switcher-2-x-23.html#post2970322) the simple solution of run a set of commands to clearing swap memory with Mode Switcher is gone. I am officially lost now :(.

theXfactor2011
05/24/2011, 03:07 PM
A week ago from this post:

Slowly, after three days all swap full and Pre- stuck. After that, disabling Method 2) Softswap but only with Method 1) Swappiness but stoppages appears. Also disabled Method 1).
And as Sconix won't add support for executing commands in MS service (http://forums.precentral.net/enlightened-linux-solutions/279166-app-mode-switcher-2-x-23.html#post2970322) the simple solution of run a set of commands to clearing swap memory with Mode Switcher is gone. I am officially lost now :(.

Since we are still in very early testing to see what works its all scripts and commands. Im sure after we figure out what works and what doesnt we can find a way to make it very easy for everyone.

theXfactor2011
05/24/2011, 03:28 PM
Fist things first, the webOS's shell is not bash, it's ash. That first line should be changed to:

#!/bin/sh

I'm playing with the rest of it.

M.

I got the script working if I run it with bash. I think i installed it from the opt feed for the hell of it once before. I am now trying to get it to work with sh. Here are my updates for bash.


#!/bin/sh
# by Paul Colby (http://colby.id.au), no rights reserved

PREV_TOTAL=0
PREV_IDLE=0

while true; do
CPU=(`cat /proc/stat | grep '^cpu '`) # Get the total CPU statistics.
unset CPU[0] # Discard the "cpu" prefix.
IDLE=${CPU[4]} # Get the idle CPU time.

# Calculate the total CPU time.
TOTAL=0
for VALUE in "${CPU[@]}"; do
let "TOTAL=$TOTAL+$VALUE"
done

# Calculate the CPU usage since we last checked.
let "DIFF_IDLE=$IDLE-$PREV_IDLE"
let "DIFF_TOTAL=$TOTAL-$PREV_TOTAL"
let "DIFF_USAGE=(1000*($DIFF_TOTAL-$DIFF_IDLE)/$DIFF_TOTAL+5)/10"
echo -en "\rCPU: $DIFF_USAGE% \b\b"

# Remember the total and idle CPU times for the next check.
PREV_TOTAL="$TOTAL"
PREV_IDLE="$IDLE"

# Wait before checking again.
sleep 1
done


of course if I run this with sh I get the following error:

cpu_usage: line 8: syntax error: unexpected "(" (expecting "done")

I believe that is because line 8 isnt wrapped in "" but if I do that then I get
cpu_usage: line 10: syntax error: bad substitution

most likely because the CPU array isnt correct

graffix31
05/24/2011, 09:01 PM
This looks sweet. Let me know if you need to test something.


-- Sent from my Palm Pre using Forums (http://developer.palm.com/appredirect/?packageid=com.newnessdevelopments.forums)

rmausser
05/25/2011, 12:47 AM
I am personally having good success right now with these settings:

(Lunasysmanager hack on post 74 is applied)

vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 120000
vm.dirty_writeback_centisecs = 60000
vm.page-cluster = 5

Compcache= 131270 (128MB) WITH disk swap.

Why this setting?

Well, I thought about it, and I am assuming that compcache is looked at as swap by the phone. But, its actually ram, thats compressed.

So, I thought, what way can I increase the amount of ram? Well, why not treat compcache as RAM, by increasing swappiness, and so that we have a lot more, but slower RAM, and a separate swap on the disk for background tasks. Rather than limit how much swap is used, increase the swap size to a huge amount with compcache and use that as RAM by telling the kernel to use swap a lot.

This way, my hope is the RAM is organised like this.

Speed

Fast [ Very few uncompressed RAM MB's for most important app (one being used) ]

Slower [ 128MB of compressed RAM for less important tasks (other open apps) ]

Slowest [ 100MB of disk swap for background tasks and services ]

So that is my hope at least. Its working well so far.

*EDIT*

Now I am trying it with 96MB of compcache instead. Seems to work even better... hmm.

Xanadu73
05/25/2011, 07:59 AM
vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 120000
vm.dirty_writeback_centisecs = 60000
vm.page-cluster = 5

Compcache= 131270 (128MB) WITH disk swap.


Just a note, if you didn't already know:



swappiness

This control is used to define how aggressive the kernel will swap memory pages. Higher values will increase aggressiveness, lower values decrease the amount of swap.

The default value is 60.


So putting that on 90 is telling the kernel to prefer to swap much more than it normally would.


M.

Xanadu73
05/25/2011, 08:08 AM
I got the script working if I run it with bash. Of course if I run this with sh I get the following error:

cpu_usage: line 8: syntax error: unexpected "(" (expecting "done")

I believe that is because line 8 isn't wrapped in "" but if I do that then I get
cpu_usage: line 10: syntax error: bad substitution

most likely because the CPU array isn't correct


Do keep in mind also that bash and ash are two very different beasts. I have a couple scripts that I use on webOS that had to go through a few major changes to run in ash after having been written in bash. Mostly ash didn't support sine fancier things that bash does. There's the good chance that ash doesn't support arrays and that it would require some major awk'ing to work around that.

Also, just as a side-note, the "#!/bin/sh" is NOT ash specific. /bin/sh is a symlink to your shell's executable. It's just a "good practice" thing to use that instead of specifying the shell directly (#!/bin/bash, #!/bin/ash, #!/bin/zsh, etc.). It's a bit more portable that way.


M.

rmausser
05/25/2011, 10:13 AM
Just a note, if you didn't already know:




So putting that on 90 is telling the kernel to prefer to swap much more than it normally would.


M.

yes I know, and by doing so its keeping the RAM much cleaner and thus there is more RAM to run tasks with and the phone is a lot faster.

Since I have 196mb of swap (100 on disk and 96mb from compcache) the swap never gets full to the point of locking the phone.

Xanadu73
05/25/2011, 10:35 AM
yes I know, and by doing so its keeping the RAM much cleaner and thus there is more RAM to run tasks with and the phone is a lot faster.

Since I have 196mb of swap (100 on disk and 96mb from compcache) the swap never gets full to the point of locking the phone.


I can follow the logic. The really good test to run is to call your voice mail (you can see with JSTop that the Phone app, once used, grabs a LOT of memory and hangs onto it indefinitely even after close), then open and close a couple large games (all the free versions of all the Angry Birds works fine), and browse the web for a bit. I can usually kill the phone after just a couple games. I'd recomend to also use Ampache Mobile for a while (caches songs to RAM), but that requires a lot more back-end work then you may be able to setup somewhere.


M.

graffix31
05/25/2011, 11:04 AM
let us know the result. hoping we are getting somewhere

Xanadu73
05/25/2011, 11:18 AM
Also for those that are playing along, you are able to WATCH what the VM System is doing which is a huge help, and this is why I cobbled this together:

#!/bin/sh

vmstat -S K 5 | awk '{print $3,"\t",$4,"\t",$5,"\t",$6,"\t",$7,"\t",$8}'

That's it. It's just WAY too long of a command line to type all the time. :)

It looks like this:

root@My Pre:/# prevmstat
---swap-- -----io---- -system-- ----cpu----
swpd free buff cache si so
7932 5904 1472 54024 1 1
7932 5904 1480 54024 0 0
7932 5904 1488 54024 0 0
7932 5904 1488 54024 0 0

You'll have to ignore that first line with all the dashes. I've tried various ways to cut it out, but, nothing works yet. The rest of the output fits perfectly on our screens for use with "SDL Terminal". The output updates itself every 5 seconds.

The first four columns are pretty self-explanatory, but 5 and 6 require a bit of typing:

si = pages swapped back INTO RAM from Swap.
so = pages swapped OUT TO Swap.

The memory numbers are in K. Good stuff to watch while stress testing! (and having another PDK app open at the same time of testing is extra stress as well!)


M.

graffix31
05/25/2011, 06:56 PM
I am personally having good success right now with these settings:

(Lunasysmanager hack on post 74 is applied)

vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 120000
vm.dirty_writeback_centisecs = 60000
vm.page-cluster = 5

Compcache= 131270 (128MB) WITH disk swap.

Why this setting?

Well, I thought about it, and I am assuming that compcache is looked at as swap by the phone. But, its actually ram, thats compressed.

So, I thought, what way can I increase the amount of ram? Well, why not treat compcache as RAM, by increasing swappiness, and so that we have a lot more, but slower RAM, and a separate swap on the disk for background tasks. Rather than limit how much swap is used, increase the swap size to a huge amount with compcache and use that as RAM by telling the kernel to use swap a lot.

This way, my hope is the RAM is organised like this.

Speed

Fast [ Very few uncompressed RAM MB's for most important app (one being used) ]

Slower [ 128MB of compressed RAM for less important tasks (other open apps) ]

Slowest [ 100MB of disk swap for background tasks and services ]

So that is my hope at least. Its working well so far.

*EDIT*

Now I am trying it with 96MB of compcache instead. Seems to work even better... hmm.

Are you still having success with these settings?


-- Sent from my Palm Pre using Forums (http://developer.palm.com/appredirect/?packageid=com.newnessdevelopments.forums)

rmausser
05/25/2011, 11:33 PM
Are you still having success with these settings?


-- Sent from my Palm Pre using Forums (http://developer.palm.com/appredirect/?packageid=com.newnessdevelopments.forums)

I am still having success but haven't had time to really drive the phone hard, just day to day tasks today.

But, i removed "vm.page-cluster = 5", so its just:

vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 120000
vm.dirty_writeback_centisecs = 60000

I have found all the newest other addons did nothing but lock my phone, especially the cluster setting.

carrel
05/26/2011, 01:11 AM
I'm a bit confused on how the size of swap is set. To get 96 meg of compcache, I presume you are editing the stock /etc/event.d/compcache and only changing the one number (I think it originally was 10 meg) to 96 meg. But where is the swap size set to 100 meg?

Thanks.

graffix31
05/26/2011, 07:47 AM
I am still having success but haven't had time to really drive the phone hard, just day to day tasks today.

But, i removed "vm.page-cluster = 5", so its just:

vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 120000
vm.dirty_writeback_centisecs = 60000

I have found all the newest other addons did nothing but lock my phone, especially the cluster setting.

are you still running your compcache at 96mb? are you just editing it through govnah or did you perm. change it?

rmausser
05/26/2011, 02:29 PM
I changed it to 96mb in etc/event.d/compcache

changed it to 98304 (kilobytes)

The SSD swap is always set at 100mb by Palm. I'd love to change it to 200mb but I dont know how.

Xanadu73
05/26/2011, 02:38 PM
I changed it to 96mb in etc/event.d/compcache

changed it to 98304 (kilobytes)

The SSD swap is always set at 100mb by Palm. I'd love to change it to 200mb but I dont know how.

/etc/init.d/swaphack.sh

Be vewy vewy caafuu. You're hunting wabbits.


M.

carrel
05/26/2011, 07:17 PM
Are you sure swaphack.sh is being run? If it was, then /dev/store/swap should have been added as a swap area. I think it can't be running for two reasons. 1) swapon -s doesn't list /dev/store/swap and 2) compcache is using that area for backing store and it can't be both.

I'm thinking that backing store is a problem. The compcache developers admit that backing store is often less efficient than just having compcache and a separate physical swap. I want to try compcache without backing store and a discreet swap with a lower priority. I think there is no swap on our devices other than compcache. At least not on mine. When people are talking about compcache and swap, they really mean compcache and backing store. Again, at least that's what I am seeing on my Pre-. But I think compcache and swap would be a good combination.

CompilingAndUsingNew - compcache - Compiling and Using ramzswap (0.6 or newer) - Compressed Caching for Linux - Google Project Hosting (http://code.google.com/p/compcache/wiki/CompilingAndUsingNew)
and go down to March 19th.

carrel
05/26/2011, 07:19 PM
/etc/init.d/swaphack.sh

Be vewy vewy caafuu. You're hunting wabbits.


M.

Wabbit Stuuu

rmausser
05/26/2011, 07:20 PM
Well lets get swap running then and disable the backing!

dave75
05/26/2011, 07:32 PM
I've been following this thread for a while and I love the fact that you guys are experimenting. What I have been hoping for though, is an app that when launched would run the script to clear the swap and then close itself. I am not a developer so I'm not sure if this is even possible. If it is, we could use Mode Switcher to launch it at any frequency we want. Thoughts from people with knowledge?

rmausser
05/26/2011, 07:56 PM
I've been following this thread for a while and I love the fact that you guys are experimenting. What I have been hoping for though, is an app that when launched would run the script to clear the swap and then close itself. I am not a developer so I'm not sure if this is even possible. If it is, we could use Mode Switcher to launch it at any frequency we want. Thoughts from people with knowledge?

I feel like this is a band-aid solution.

Everytime you run this your phone is going to be unresponsive for about 30 seconds while it clears out the swap.

Im hoping we get things to the point where you dont have to clear out the swap.

loopytee
05/26/2011, 08:28 PM
yes that would be great, but in the meantime we could have mode switcher running this app everytime the screen shuts off. Any chance that someone can create this even as a band aid solution?

carrel
05/27/2011, 01:18 AM
OK. so now I have SWAP. I mean real swap!!

root@ - Palm Pre:/# swapon -s
Filename Type Size Used Priority
/dev/ramzswap0 partition 98296 29700 100
/dev/mapper/store-swap partition 106488 0 -2

I just started this experiment. So I have no real data yet, but I'll report back.

The compcache has no backing store. The VM system will definitely favor the compcache over the physical swap because of the priorities. I also have some tweaks in /etc/sysctl.conf taken from some of the recent postings here. Most notably the high swappiness. If this is stable, then I'll try playing with tuning it. But for now I just want to leave it alone and see how it handles a few days of hard use.

Xanadu73
05/27/2011, 06:18 AM
OK. so now I have SWAP. I mean real swap!!

root@ - Palm Pre:/# swapon -s
Filename Type Size Used Priority
/dev/ramzswap0 partition 98296 29700 100
/dev/mapper/store-swap partition 106488 0 -2I just started this experiment. So I have no real data yet, but I'll report back.


What was it you played with to get that working? Comment out a couple things in swaphack.sh?


M.

carrel
05/27/2011, 10:13 AM
swaphack.sh isn't getting run by default. It's just sitting there to tempt us. I thought of running it, but it does a little more than I wanted.

I simply changed /etc/event.d/compcache to create a 96M compcache with no backing store. I changed the line to:
modprobe ramzswap disksize_kb=98304
Then I did the next step by hand. (This is my phone, after all. I don't want to get stuck if this all fails miserably.) I didn't want this to auto-start if the thing crashed. (Eventually I'll put this in /etc/event.d/compcache) I ran:
mkswap /dev/store/swap
swapon /dev/store/swap
And here is my sysctl.conf
vm.swappiness = 90
vm.vfs_cache_pressure = 200
#vm.page-cluster = 5
vm.dirty_expire_centisecs = 120000
vm.dirty_writeback_centisecs = 60000

I know it is way to early to really know, but so far I am really happy. I ran 8 apps all night. This morning the phone is responsive. That's far from conclusive, but it is promissing. I am also seeing that swap is actually being used:
carrel@ - Palm Pre:~$ /sbin/swapon -s
Filename Type Size Used Priority
/dev/ramzswap0 partition 98296 86340 100
/dev/mapper/store-swap partition 106488 5200 -2

More later...

UncleVanya
05/27/2011, 12:28 PM
Has anyone looked at this page:
Tips for running Linux on a flash device (http://www.cyrius.com/debian/nslu2/linux-on-flash.html)

A lot of similar ideas are expressed there.

carrel
05/27/2011, 12:54 PM
Hmmm, I am beating on this puppy hard and she keeps recovering. But I do notice swapping delays sometimes. Well, of course some of that is just inevitable. But I was looking at tuning and I went back to post #74 and found some math there is wrong. In particular, the centiseconds values are off by a factor of 10. Instead of 3 minutes, that values gives you 30 minutes. Without rebooting I have changed my values to
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500

I closed a bunch of apps to see if swap was reclaimed. It was. But interestingly about 20M of compcache was free-ed up and almost 16M remains in the physical swap. I was hoping to see the vm system migrate that memory to the higher priority compcache swap partition. But it did not. Maybe it wont bother with it until I access it. But the good news is that I have seen the physical swap partition's usage actually decrease over time. Not much, but it has. Now to keep watching...

carrel
05/27/2011, 01:02 PM
Has anyone looked at this page:
Tips for running Linux on a flash device (http://www.cyrius.com/debian/nslu2/linux-on-flash.html)

A lot of similar ideas are expressed there.

Yep, good ideas. It seems though that we'd like to apply much of what they say to the true swap flash storage, but not to the compcache swap storage in ram. Unfortunately the kernel doesn't give us that much flexibility.

UncleVanya
05/27/2011, 01:13 PM
Yep, good ideas. It seems though that we'd like to apply much of what they say to the true swap flash storage, but not to the compcache swap storage in ram. Unfortunately the kernel doesn't give us that much flexibility.


+1

Is there any easy way to disable the swap partition and use swap files? I would think due to flash wear we should be using files not partitions and creating a second one when we want to disable the old one and move to new flash areas. This assumes the flash in the phone is not automatically implmenting some type of wear leveling.

Xanadu73
05/27/2011, 01:17 PM
+1

Is there any easy way to disable the swap partition and use swap files? I would think due to flash wear we should be using files not partitions and creating a second one when we want to disable the old one and move to new flash areas. This assumes the flash in the phone is not automatically implmenting some type of wear leveling.

Yes, it's quite simple (in theory). Move upstart's compcache out of there. Create a swap *file* with dd. Hand type that file in /etc/fstab as swap. It should be that simple. Performance might suffer some because swap *files* aren't as good as swap *partitions*, but we really don't have the choice here.


M.

UncleVanya
05/27/2011, 01:21 PM
Yes, it's quite simple (in theory). Move upstart's compcache out of there. Create a swap *file* with dd. Hand type that file in /etc/fstab as swap. T should be that simple. Performance might suffer some because swap *files* aren't as good as swap *partitions*, but w really don't have the choice here.


I know that files once upon a time were not as fast as partitions - but my reading of info on current kernels seems to suggest there isn't much if any penalty for doing that now.


IGNORE BELOW. I talked to a buddy who reminded me that swap might not be in /etc/fstab so I went looking with his help and found it in /dev/store/swap and got it back.


On a related note - if I disable swap using:

swapon -a

and then try to restart it with:

swapon -a

I get a free -m output showing 0 swap. I'm not sure what I'm doing wrong here... The only reason I did this was to clear out swap to see what impact some new settings would have. After that failed I rebooted and tried without any changes and it persists. I have swap when I restart - but as soon as I use swapoff - I can't seem to get it back with swapon.

FWIW - swapon -s shows no swap devices/files.

I guess I can reboot and see what swapon -s shows when first restarted...

rksand
05/27/2011, 02:55 PM
Hmmm, I am beating on this puppy hard and she keeps recovering. But I do notice swapping delays sometimes. Well, of course some of that is just inevitable. But I was looking at tuning and I went back to post #74 and found some math there is wrong. In particular, the centiseconds values are off by a factor of 10. Instead of 3 minutes, that values gives you 30 minutes. Without rebooting I have changed my values to
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500

I closed a bunch of apps to see if swap was reclaimed. It was. But interestingly about 20M of compcache was free-ed up and almost 16M remains in the physical swap. I was hoping to see the vm system migrate that memory to the higher priority compcache swap partition. But it did not. Maybe it wont bother with it until I access it. But the good news is that I have seen the physical swap partition's usage actually decrease over time. Not much, but it has. Now to keep watching...

This seems to be the best implementation I've tried thus far. I can open and close muntiple apps, including hogs like Classic, and the phone recovers well and is generally more responsive, but for a few swap pauses that i think are tollerable.

Xanadu73
05/27/2011, 03:59 PM
But I was looking at tuning and I went back to post #74 and found some math there is wrong. In particular, the centiseconds values are off by a factor of 10. Instead of 3 minutes, that values gives you 30 minutes.

Yumpin' Yimminy! I can't believe I messed that up! You are 100% correct. I'll go back an edit that post for future generations.

:shake:


M.

grappler
05/27/2011, 04:33 PM
mkswap /dev/store/swap
swapon /dev/store/swap


I get

/dev/store/swap: Device or resource busy


with the first command. Help?

rmausser
05/27/2011, 04:36 PM
This is awesome work you guys:

I have also added these values and am having good results:

vm.swappiness = 100
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500
vm.dirty_ratio = 40
vm.dirty_background_ratio= 0

This is a bit extreme so maybe you want to try:

vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500
vm.dirty_ratio = 40
vm.dirty_background_ratio= 10

first.

Lets get the swap to 200mb. I can afford the space on my memory.

rksand
05/27/2011, 04:57 PM
I agree, this is awesome. My swap is running high (like 102) after running many intensive apps, but the phone remains zippy. I think its doing a better job of keeping ram available. I've been able to go from Classic, to NFS to web browsing and back without a notable change in responsiveness.

carrel
05/27/2011, 06:09 PM
I get

/dev/store/swap: Device or resource busy


with the first command. Help?

By default /etc/event.d/compcache is configured to use /dev/store/swap as compcache's backing store. It can not be both at the same time. Did you disable that and set compcache to have no backing store??

carrel
05/27/2011, 06:20 PM
+1

Is there any easy way to disable the swap partition and use swap files? I would think due to flash wear we should be using files not partitions and creating a second one when we want to disable the old one and move to new flash areas. This assumes the flash in the phone is not automatically implementing some type of wear leveling.

I don't think I'd worry about flash wear with modern flash. /dev/store/swap is this 100M partition available to us. (Actually it's a link to /dev/mapper/store-swap.) I haven't figured out exactly what/where it is. Maybe that's my next investigation. What is block device 254,5??? Does anyone know?

Off to go sailing now...

rmausser
05/27/2011, 07:05 PM
This is awesome work you guys:

I have also added these values and am having good results:

vm.swappiness = 100
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500
vm.dirty_ratio = 40
vm.dirty_background_ratio= 0

This is a bit extreme so maybe you want to try:

vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500
vm.dirty_ratio = 40
vm.dirty_background_ratio= 10

first.

Lets get the swap to 200mb. I can afford the space on my memory.

Update, changed it to


vm.swappiness = 100
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500
#vm.dirty_ratio = 40
#vm.dirty_background_ratio= 0

Dont add those two last lines yet. Im not happy with the results.

swappiness at 100 is fine tho.

rmausser
05/27/2011, 07:11 PM
swaphack.sh isn't getting run by default. It's just sitting there to tempt us. I thought of running it, but it does a little more than I wanted.

I simply changed /etc/event.d/compcache to create a 96M compcache with no backing store. I changed the line to:
modprobe ramzswap disksize_kb=98304



May I suggest using Memlimit rather than Disksize? As in

modprobe ramzswap memlimit_kb=98304

memlimit is resizeable on demand, and disksize is a static size.

With memlimit, if there isnt any swap, the kernel can use that space as actual ram instead, and dynamically make the size of the ramzswap smaller or larger depending on demand.

Why use up ram that you arent using?

grappler
05/27/2011, 07:18 PM
By default /etc/event.d/compcache is configured to use /dev/store/swap as compcache's backing store. It can not be both at the same time. Did you disable that and set compcache to have no backing store??

Yes, I made your other suggested change to kill the backing store and allocate 96 MB to compcache. Or at least I think I did. :)

rmausser
05/27/2011, 08:29 PM
THESE NEW SETTINGS ARE AMAZING!!

Just had to say. Good work everyone. My phone is usable again. THANK GOD.

To recap so far: (use Internalz Pro on the phone if ur not good with terminal to change these files BUT FIRST MAKE SURE that Internalz Pro is set to LINUX mode. To switch to linux mode open Internalz Pro and "left drop down menu>Preferences>Text Editor>Newline format: LINUX") Also scroll to the bottom of preferences and enable Master mode. BE CAREFUL, you now have the ability to modify system files with Internalz. Turn this off after you hack your phone so you dont delete something by mistake.

Step 1: Change /etc/event.d/LunaSysMgr from:

echo "1" > /proc/sys/vm/overcommit_memory".

to

#echo "1" > /proc/sys/vm/overcommit_memory".

Step 2: Make a file in /etc called "sysctl.conf". Open the file with a text editor and write:


vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 1800
vm.dirty_writeback_centisecs = 6000


Step 3: Open /etc/event.d/compcache.

edit line 10 to say:

modprobe ramzswap disksize_kb=98304

change line 10 to say ONLY that. Remove the part that says: dev/store/swap after it.

then, after the line 12 in Compcache (swapon /dev/ramzswap0 -p 100, add:


mkswap /dev/store/swap
swapon /dev/store/swap


Reboot the phone. Enjoy. :)

**I would highly recommend using the thundercheif F105 kernel if you can, as it overclocks the RAM to 200mhz, which will help with the slow speed of Compcache.**

Explanation:

What we are doing, is making essentially a pyramid of memory.

On the top, we have real RAM, but less of it available. This is reserved only for applications that are currently being open or run and used, so they are responsive to the users input.

In the middle of the pyramid we have Compresses Cache, or RAM that is being used as a compressed swap file. It is slower, but since it compresses the pages, there is more available for the phone to use than the Pre came with. It has the second priority, and is used for applications in the background and things that are not at your immediate attention.

At the bottom we have disk swap. This is the slowest memory, but since we essentially have 8GB of memory on the phone, we could eventually turn this up to be even larger in size. Dirty pages and memory that is currently not being used at all, but the kernel thinks might be available on short notice, like background processes, lives here.

We have organized the kernel to as quickly as possible dump any pages that arent being used to compcache, so that the faster ram is available for the app or device the user is using, hopefully eliminating lag. The advantage is that compcache increases the available amount of memory by compressing the pages, and therefore the kernel doesnt have to clean out the swap and ram as often, which is what is causing those long and painful lags we keep experiencing.

**EDIT**

changed memlimit to disksize.

memlimit isnt working right and is stuck on 60mb according to /sbin/swapon -s. Dont know why.

kbrabble
05/27/2011, 08:36 PM
THESE NEW SETTINGS ARE AMAZING!!

Just had to say. Good work everyone. My phone is usable again. THANK GOD.



I agree, my initial impression, after a few hours of heavy use, are very good!

-- Sent from my Palm Pre using Forums (http://developer.palm.com/appredirect/?packageid=com.newnessdevelopments.forums)

loopytee
05/28/2011, 12:07 AM
Wow, nice work yall! These new settings are making my Pre- exciting again! Whoop Whoop!!!

dave75
05/28/2011, 12:19 AM
Is "compressed swap" supposed to enabled in Govnah?

cyberprashant
05/28/2011, 01:41 AM
cool = can this be a patch in preware to apply (to make it simpler and reversible if needed)

grappler
05/28/2011, 09:12 AM
THESE NEW SETTINGS ARE AMAZING!!

Just had to say. Good work everyone. My phone is usable again. THANK GOD.

To recap so far: (use Internalz Pro on the phone if ur not good with terminal to change these files BUT FIRST MAKE SURE that Internalz Pro is set to LINUX mode. To switch to linux mode open Internalz Pro and "left drop down menu>Preferences>Text Editor>Newline format: LINUX")

Step 1: Change /etc/event.d/LunaSysMgr from:

echo "1" > /proc/sys/vm/overcommit_memory".

to

#echo "1" > /proc/sys/vm/overcommit_memory".

Step 2: Make a file in /etc called "sysctl.conf". Open the file with a text editor and write:


vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500


Step 3: Open /etc/event.d/compcache.

edit line 10 to say:

modprobe ramzswap disksize_kb=98304

change line 10 to say ONLY that. Remove the part that says: dev/store/swap after it.

then, after the line 12 in Compcache (swapon /dev/ramzswap0 -p 100, add:


mkswap /dev/store/swap
swapon /dev/store/swap


Reboot the phone. Enjoy. :)

**I would highly recommend using the thundercheif F105 kernel if you can, as it overclocks the RAM to 200mhz, which will help with the slow speed of Compcache.**

Explanation:

What we are doing, is making essentially a pyramid of memory.

On the top, we have real RAM, but less of it available. This is reserved only for applications that are currently being open or run and used, so they are responsive to the users input.

In the middle of the pyramid we have Compresses Cache, or RAM that is being used as a compressed swap file. It is slower, but since it compresses the pages, there is more available for the phone to use than the Pre came with. It has the second priority, and is used for applications in the background and things that are not at your immediate attention.

At the bottom we have disk swap. This is the slowest memory, but since we essentially have 8GB of memory on the phone, we could eventually turn this up to be even larger in size. Dirty pages and memory that is currently not being used at all, but the kernel thinks might be available on short notice, like background processes, lives here.

We have organized the kernel to as quickly as possible dump any pages that arent being used to compcache, so that the faster ram is available for the app or device the user is using, hopefully eliminating lag. The advantage is that compcache increases the available amount of memory by compressing the pages, and therefore the kernel doesnt have to clean out the swap and ram as often, which is what is causing those long and painful lags we keep experiencing.

**EDIT**

changed memlimit to disksize.

memlimit isnt working right and is stuck on 60mb according to /sbin/swapon -s. Dont know why.

Ah, I get it now . . . When the earlier poster said he did mkswap, etc., "by hand," I thought he meant they were to be entered at the command-line. Duh.

I have the same question as the poster above: are Govnah swap settings overruled by all this, or do they still need to be set?

rksand
05/28/2011, 09:26 AM
THESE NEW SETTINGS ARE AMAZING!!

Just had to say. Good work everyone. My phone is usable again. THANK GOD.

To recap so far: (use Internalz Pro on the phone if ur not good with terminal to change these files BUT FIRST MAKE SURE that Internalz Pro is set to LINUX mode. To switch to linux mode open Internalz Pro and "left drop down menu>Preferences>Text Editor>Newline format: LINUX")

Step 1: Change /etc/event.d/LunaSysMgr from:

echo "1" > /proc/sys/vm/overcommit_memory".

to

#echo "1" > /proc/sys/vm/overcommit_memory".

Step 2: Make a file in /etc called "sysctl.conf". Open the file with a text editor and write:


vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500


Step 3: Open /etc/event.d/compcache.

edit line 10 to say:

modprobe ramzswap disksize_kb=98304

change line 10 to say ONLY that. Remove the part that says: dev/store/swap after it.

then, after the line 12 in Compcache (swapon /dev/ramzswap0 -p 100, add:


mkswap /dev/store/swap
swapon /dev/store/swap


Reboot the phone. Enjoy. :)

**I would highly recommend using the thundercheif F105 kernel if you can, as it overclocks the RAM to 200mhz, which will help with the slow speed of Compcache.**

Explanation:

What we are doing, is making essentially a pyramid of memory.

On the top, we have real RAM, but less of it available. This is reserved only for applications that are currently being open or run and used, so they are responsive to the users input.

In the middle of the pyramid we have Compresses Cache, or RAM that is being used as a compressed swap file. It is slower, but since it compresses the pages, there is more available for the phone to use than the Pre came with. It has the second priority, and is used for applications in the background and things that are not at your immediate attention.

At the bottom we have disk swap. This is the slowest memory, but since we essentially have 8GB of memory on the phone, we could eventually turn this up to be even larger in size. Dirty pages and memory that is currently not being used at all, but the kernel thinks might be available on short notice, like background processes, lives here.

We have organized the kernel to as quickly as possible dump any pages that arent being used to compcache, so that the faster ram is available for the app or device the user is using, hopefully eliminating lag. The advantage is that compcache increases the available amount of memory by compressing the pages, and therefore the kernel doesnt have to clean out the swap and ram as often, which is what is causing those long and painful lags we keep experiencing.

**EDIT**

changed memlimit to disksize.

memlimit isnt working right and is stuck on 60mb according to /sbin/swapon -s. Dont know why.

To clarify, is this done with softswap, or compcache via Govnah?

PreGP
05/28/2011, 11:00 AM
THESE NEW SETTINGS ARE AMAZING!!

Just had to say. Good work everyone. My phone is usable again. THANK GOD.



Thanks for these relatively easy to follow instructions using Internals Pro (master mode needs to be turned on in preferences as well as changing to Linux for the Newline format to edit the files). Seems to have made a huge difference to performance on my Pre minus running 2.1.0 particularly as I've also changed from UK to F105 Thunderchief. If the F105 kernel gets updated, will I need to repeat the hack or will it survive an update?

rmausser
05/28/2011, 12:01 PM
Is "compressed swap" supposed to enabled in Govnah?

No. I think because we modified the Compcache file, govnah is confused by it and shows compcache as disabled.

If you go to Terminal and type:

/sbin/swapon -s

You can see both compcache and the new swap file and how its working.

To clarify, is this done with softswap, or compcache via Govnah?

This is done with neither. If you read the instructions, its done by modifying the "/etc/event.d/compcache file."

Updated instructions for master mode.

rmausser
05/28/2011, 12:04 PM
Ah, I get it now . . . When the earlier poster said he did mkswap, etc., "by hand," I thought he meant they were to be entered at the command-line. Duh.

I have the same question as the poster above: are Govnah swap settings overruled by all this, or do they still need to be set?

actually he did enter it by hand at terminal so it wouldn't stick on reboot in case it bricked his phone.

But we now know its relatively safe.

rmausser
05/28/2011, 12:06 PM
If the F105 kernel gets updated, will I need to repeat the hack or will it survive an update?

Updating a kernel would never modify any system files like Compcache or the sysctl.conf file we made. Quite the opposite, these files actually modify the kernel when it boots up.

carrel
05/28/2011, 12:19 PM
Just a quick update... I am at 34 hours running with this way and all seems pretty good. There are occasional stutters, which are almost certainly swapping. But they are not long and not too frequent. I have really used my phone hard during this time. She definitely would not have survived this long before this change. The screen comes right on and apps start fast. Swap usage is holding steady with most of compcache used and a small amount of the swap partition used.

I even put off the temptation to update Mode Switcher (an update is available) because the last update required a phone reboot and I want to see how long this will last.

Thank you rmausser for writing up how to do this with InternalzPro. (I prefer ssh and a full size screen/keyboard.) My sysctl settings are slightly different, but that is just fine tuning.

carrel
05/28/2011, 12:27 PM
Ah, I get it now . . . When the earlier poster said he did mkswap, etc., "by hand," I thought he meant they were to be entered at the command-line. Duh.

Actually I did mean that I entered it at the command line. Anything that you do in a shell script, you can do on the command line (shell). I chose to do this at the command line because then it wouldn't automatically get re-done if the phone reboots. I was just being cautious because I was trying something significant on my primary phone. If it crashed or failed miserably, I wanted to increase the chance that a reboot would be able to start fresh, without this. Now that I have run this for 35 hours, I feel safe putting it in a startup script.

As to why the mkswap didn't work for you: Did you reboot after changing the line in /etc/event.d/compcache ?

loopytee
05/28/2011, 01:24 PM
just an update, pre- 2.1 f104 (my pre- is unhappy with 105) Govnah currently show memory/swap at 235/103 and my pre- is SNAPPY! I am amazed and loving it! Thanks again



oh and do you think I should edit the first post to redirect from 74 to this most current test?

Xanadu73
05/28/2011, 01:52 PM
What we are doing, is making essentially a pyramid of memory.



On the top, we have real RAM, but less of it available.
In the middle of the pyramid we have Compresses Cache.
At the bottom we have disk swap.




I can follow all this and I am sacrificing my Pre to the test as well!


M.

PreGP
05/28/2011, 02:19 PM
just an update, pre- 2.1 f104 (my pre- is unhappy with 105) Govnah currently show memory/swap at 235/103 and my pre- is SNAPPY! I am amazed and loving it! Thanks again



oh and do you think I should edit the first post to redirect from 74 to this most current test?

I think you should direct people to rmausser's post #372 and someone should make this thread sticky! :)

carrel
05/28/2011, 02:29 PM
Explanation:

What we are doing, is making essentially a pyramid of memory.



Which, by the way, is exactly what Palm tried to do. They did a similar pyramid with RAM, Compcache and Compcache backing store. However it seems there is a problem with the Compcache backing store. My guess is that the backing store has no knowledge of flash behavior. If it doesn't know about flash pages, it could easily use up all the backing store by leaving just a little bit of needed memory in a page when the rest is stale. Then the page can't be cleared and it moves on to a new page. And so on and so on... This fits the behavior we were all seeing.

The VM system does know about flash and implements swap on flash much more intelligently. So I figured, why not try letting it do the job. Besides, the system gives me better tools to prioritize the two types of swap and then to observe how they are getting used. So far, the theory seems solid.

rmausser
05/28/2011, 02:47 PM
I think you should direct people to rmausser's post #372 and someone should make this thread sticky! :)

Lets not jump ahead of ourselves yet people, lets keep testing it for a bit and I want to test some other ideas I have in my head first before we make this sticky.

Im going to try tweaking some settings to this method, and even try a very different method as well soon so stay tuned.

Maybe we can eradicate even the tiny lags completely.

What I know for sure is

a) Lunasysmgr commented out to overcommit definitely makes things better regardless of any setting.

What I want to try is

a) Taking off sysctl.conf and just try Palms memory managment with 96mb compcache and the new disk swap

b) Trying the older sysctl.conf settings with only 10mb of compcache (palm default) and the new disk swap

c) Just 10mb compcache and new disk swap, no sysctl.conf

I want to isolate what of the three changes is mostly responsible for making the phone better, the compcache size, the new disk swap, or sys.ctl.
Then from there see what combination works the best, and from there just tiny tweaking of the settings till we get it as good as possible.

Its hard to really know what is making things better without isolating each part. Then on top of that, one setting might make it better or worse, but isolation is still the best bet.

If you guys want to help I would like to set up three testing settings.

Test 1:

The settings we are currently using

Test 2:

96MB compcache with new disk swap. No sysctl.conf settings (rename or move the file)

Test 3:

10MB compcache with new disk swap. No sysctl.conf. (rename or move the file)

Test 4:

10MB compcache with new disk swap. Set sysctl.conf to:


vm.swappiness = 20
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000


For baseline make sure that you do at least these things.

1) Open PDK game
2) Check voicemail
3) Text message some people
4)Go on web browser, access some full page sites
5) Open camera
6)Watch youtube video
7) Open a bunch of mojo apps at once

Since we all have different phones you will have to record the changes based on compared to how your phone operates without any of these settings. (Stock, but keep the kernel and CPU settings the same, so if you overclock, dont modify that)

Then state which test out of 4 you ran, and the results after a day of usage.

I am going to do Test number 3 personally.

Thanks.

*****EDIT*****
Ok so 10mb compcache officially sucks. So thats out of the question.

Im trying 96mb with disk swap and no sysctl.conf. So far working great. no lag.

carrel
05/28/2011, 05:52 PM
Hmmmm, at 39 hours, my phone locked up. I had to pull the battery. :-(

I can't say why it locked up. It was responsive right up to the end. It could be the something other than the VM system, but it could have been these changes too. But it was a good 39 hours, so I'm going to keep testing my config the way it was.

theXfactor2011
05/28/2011, 06:55 PM
Hmmmm, at 39 hours, my phone locked up. I had to pull the battery. :-(

I can't say why it locked up. It was responsive right up to the end. It could be the something other than the VM system, but it could have been these changes too. But it was a good 39 hours, so I'm going to keep testing my config the way it was.

Most likely at 39 hours you ran out of swap. Once swap runs out and memory fills up its game over. The lag to push stuff from swap to disk just to make room to put things in memory is so great that nothing seems to function.

rmausser
05/28/2011, 07:16 PM
This is why I am trying 96mb compcache with new disk swap and no sysctl.conf settings.

I have an inkling those sysctl.conf settings are too aggressive.

Xanadu73
05/28/2011, 07:21 PM
This is why I am trying 96mb compcache with new disk swap and no sysctl.con settings.

I have an inkling those sysctl.conf settings are too aggressive.

I came up with those numbers because the defaults are even more aggressive. I was trying to slow down pdflush allowing time for the garbage collector to do it's thing trying to avoid swapping *known* garbage from clogging up the works.


M.

carrel
05/28/2011, 07:53 PM
Most likely at 39 hours you ran out of swap. Once swap runs out and memory fills up its game over. The lag to push stuff from swap to disk just to make room to put things in memory is so great that nothing seems to function.

Possible, but I don't think so. I have been checking swap usage regularly with 'swapon -s'. Since I first booted, swap filled continuously until all 96 megs of compcache filled and then 20 of the 100 megs of the swap partition filled. That was the peak and happened within 12 hours. After that, compcache stayed mostly full and swap fluctuated between 20 and 12 megs of usage. I last checked it about 2 hours before it hung and only 12 meg of swap was in use.

rmausser
05/28/2011, 08:11 PM
I came up with those numbers because the defaults are even more aggressive. I was trying to slow down pdflush allowing time for the garbage collector to do it's thing trying to avoid swapping *known* garbage from clogging up the works.


M.

I was referring to the sysctl.conf settings that I made.

carrel
05/28/2011, 08:23 PM
I came up with those numbers because the defaults are even more aggressive. I was trying to slow down pdflush allowing time for the garbage collector to do it's thing trying to avoid swapping *known* garbage from clogging up the works.


M.

I think the direction you took was really good at alleviating the symptoms we were all seeing when using compcache backing store. However, with that gone and a proper swap partition, I'm not sure they are beneficial any more. I too am now running with no sysctl.conf changes. I think the linux defaults are likely very close to ideal. Hopefully we are now tweaking these values to minimize stuttering instead of stopping the entire VM system from clogging up.

On the other hand I think vm.overcommit_memory may be more interesting to play with. Palm set it to 1 because they had to know that their system allocates more memory than it needs or uses. 1 allows unlimited overcommits, but 1 is dangerous. 0 does not restrict it to not over commit memory, but does reduce it somewhat. And why did Palm think they needed to be able to overcommit so much?? Maybe some rationed over committing is useful. I am running with it set to zero right now just to feel out the swap changes without too many variables. I might try setting vm.overcommit_memory to 2 and playing with the ratio. But not just yet...

For more info, try this The Linux kernel: Memory (http://www.win.tue.nl/~aeb/linux/lk/lk-9.html)

theXfactor2011
05/28/2011, 08:54 PM
I think the direction you took was really good at alleviating the symptoms we were all seeing when using compcache backing store. However, with that gone and a proper swap partition, I'm not sure they are beneficial any more. I too am now running with no sysctl.conf changes. I think the linux defaults are likely very close to ideal. Hopefully we are now tweaking these values to minimize stuttering instead of stopping the entire VM system from clogging up.

On the other hand I think vm.overcommit_memory may be more interesting to play with. Palm set it to 1 because they had to know that their system allocates more memory than it needs or uses. 1 allows unlimited overcommits, but 1 is dangerous. 0 does not restrict it to not over commit memory, but does reduce it somewhat. And why did Palm think they needed to be able to overcommit so much?? Maybe some rationed over committing is useful. I am running with it set to zero right now just to feel out the swap changes without too many variables. I might try setting vm.overcommit_memory to 2 and playing with the ratio. But not just yet...

For more info, try this The Linux kernel: Memory (http://www.win.tue.nl/~aeb/linux/lk/lk-9.html)

I played with the overcommit a while back. I set it to 2 and tried with overcommit ratios of 25, 50,75, 80 and the os failed to boot. when I used a ratio of 100 it was able to boot. But that says you can overcommit as long as you request less then the total available memory. seemed kind of pointless. I bailed on the idea for some more interesting tests ;)

carrel
05/28/2011, 09:29 PM
I played with the overcommit a while back. I set it to 2 and tried with overcommit ratios of 25, 50,75, 80 and the os failed to boot. when I used a ratio of 100 it was able to boot. But that says you can overcommit as long as you request less then the total available memory. seemed kind of pointless. I bailed on the idea for some more interesting tests ;)

Did you have swap configured when you did that?? I'm guessing that the VM system knows nothing about compcache backing store. With a setting of 2 and no swap, you would only have "ratio" percentage of ram to allocate from. I could see that failing badly.

theXfactor2011
05/28/2011, 09:36 PM
Did you have swap configured when you did that?? I'm guessing that the VM system knows nothing about compcache backing store. With a setting of 2 and no swap, you would only have "ratio" percentage of ram to allocate from. I could see that failing badly.

Good point. I don't think I had dedicated swap at the time. Just the compache and backing.

carrel
05/28/2011, 10:00 PM
Good point. I don't think I had dedicated swap at the time. Just the compache and backing.

In fact... Now that you mention it, that would explain what Palm did. Because they were using backing store, they had no way for the VM system to accurately estimate the amount of available memory. So they simply allowed unrestrained allocation. Using a value of 2 would fail, and even a 0 would likely underestimate. So they threw in the towel and set it to 1.

Xanadu73
05/28/2011, 10:13 PM
In fact... Now that you mention it, that would explain what Palm did. Because they were using backing store, they had no way for the VM system to accurately estimate the amount of available memory. So they simply allowed unrestrained allocation. Using a value of 2 would fail, and even a 0 would likely underestimate. So they threw in the towel and set it to 1.

That would explain it. I've been thinking about that off and on for a while now and couldn't figure out why they would do that (unless it was self-admitting that their code is buggy as hell, of course... :) ).

Good show!


M.

theXfactor2011
05/28/2011, 11:13 PM
After about 10 hours of using ramzswap with a separate official linux swap I havent noticed the tiered effects. I see ram getting used it fills and clears and sometimes spills into compache. Compache grows over time and shrinks every now and then. once I filled up compache the swap on /dev/store started to be used. I predict in the long run compache will stay full and swap will grow. once swap is full then the phone will freeze and crash.

is anyone seeing the ram use and swap to compcache and every now and then compcache dumps to swap, keeping compcache with available space?

carrel
05/29/2011, 02:33 AM
I'm confused... You say you haven't seen a tiered effect, but then you describe a perfectly operating tiered operation of filling ram, then compcache and then physical swap. The only negative is your prediction. The rest all sounds perfect. WebOS does use more and more memory over time. But eventually it seems to top out. From there, usage goes up and down.

And yes, I saw the compcache fill and then empty some even when swap was partially used. That is normal. No need to move every possible bit of unused memory from lower priority swap to higher priority swap.

Xanadu73
05/29/2011, 12:31 PM
After just using the newer idea(s) since yesterday afternoon, I have decided that high swappiness is a bad idea. Swap just fills and fills until the kernel reaches a point that it decides it has to start juggling things around and that's when our Pre- falls over dead.

I'm sticking with this multi-swap idea but have dropped the swappiness down to 30 and put back in the higher dirty times (my corrected ones - not the lower carrel's - I still am convinced that Palm's defaults are WAY too fast, though).

I'm thinking to use the 2nd swap as "emergency" swap. With the lower swappiness, the kernel should only decide to use that when it ABSOLUTELY HAS TO to keep itself alive. This is what swap is for, not for filling up just to fill up (which is what the high swappiness will/does do).

Will report back.


M.

rmausser
05/29/2011, 02:40 PM
After just using the newer idea(s) since yesterday afternoon, I have decided that high swappiness is a bad idea. Swap just fills and fills until the kernel reaches a point that it decides it has to start juggling things around and that's when our Pre- falls over dead.

I'm sticking with this multi-swap idea but have dropped the swappiness down to 30 and put back in the higher dirty times (my corrected ones - not the lower carrel's - I still am convinced that Palm's defaults are WAY too fast, though).

I'm thinking to use the 2nd swap as "emergency" swap. With the lower swappiness, the kernel should only decide to use that when it ABSOLUTELY HAS TO to keep itself alive. This is what swap is for, not for filling up just to fill up (which is what the high swappiness will/does do).

Will report back.


M.

I agree completely.

The Palm settings are better than high swapiness, but there are still some issues.

Trying with a setting of 30 too.

Im also gonna try it with both 96mb compcache, and 32mb compcache. Will report the differences.

Xanadu73
05/29/2011, 02:52 PM
I'm also gonna try it with both 96mb compcache, and 32mb compcache. Will report the differences.

My head was going there too. I'm currently using 96M. I think that with the second swap area, 96M might actually be too much. I haven't changed it yet because I don't want to reboot at this time. Not that we HAVE to reboot to change this, but, I am trying not to taint the experience too much yet by clearing out various swaps mid-run and all that.


M.

rmausser
05/29/2011, 04:27 PM
I am trying this now:


vm.swappiness = 30
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000


ramzswap disksize= 32768. (memlimit is messed up, will always show at 60mb in sbin/swapon -s)

Will report back. If 32768 proves to not be enough I am going to switch to 96mb and just keep the same sysctl settings.

rmausser
05/29/2011, 04:58 PM
Just a recap for those following at home:

Things we know for sure:

1) LunaSysMgr is better without over committing memory

2)A dedicated Linux Swap Disk is much better with Compcache than the default Backing Store.

3) Setting Compcache to Memlimit rather than Disksize does not seem to work right with a Linux Swap Disk

4) 10MB of Compcache is not enough under any circumstance on the Pre-

What we arent sure of yet:

1) What SysCtl.conf settings are ideal

2) If 32mb or 96mb of compcache is ideal

carrel
05/29/2011, 06:06 PM
Just a recap for those following at home:

Things we know for sure:

1) LunaSysMgr is better without over committing memory

2)A dedicated Linux Swap Disk is much better with Compcache than the default Backing Store.

3) Setting Compcache to Memlimit rather than Disksize does not seem to work right with a Linux Swap Disk

4) 10MB of Compcache is not enough under any circumstance on the Pre-

What we arent sure of yet:

1) What SysCtl.conf settings are ideal

2) If 32mb or 96mb of compcache is ideal

I think I would disagree with your #1 "things we know". From looking at what the overcommit does, I do not think it is really doing that much. I think it did help back with backing store. The setting of zero would actually cause some malloc()s to fail and that would slightly slow down the backing store getting totally clogged up and becoming useless. With backing store disabled and a swap partition enabled, I have never yet run into a situation where VM is full, so basically a setting of 0 or 1 would behave the same. (Now using a value of 2 might be helpful if we were experiencing the OOM killer, but I have no reason to believe that my usage is getting me there. Sure if I open enough cards, I will, but I don't.) This value is not effecting performance, just how the system will behave when it truly runs out of VM. This, and the overcommit ratio, should be the last parameters we tune, after all the others.

I look at this "experiment" slightly differently. There are two things going on.

First, there definitely seems to be a bug in compcache backing store, and using compache along with a true swap partition seems to fix this.

Second, with a working VM system, we should be able to tune it to perform best on our devices. All the kernel variable changes (sysctl.conf and LunaSysMgr) and compcache sizing are working on that.

The two parts are completely separable, but I do think we should re-think/reevaluate any tuning efforts that were done previously with backing store. The experiments were hugely helpful to me to better understand the VM system, but the answers are no longer relevant. We will almost certainly want different tuning parameters for compache and a swap partition.

carrel
05/29/2011, 06:46 PM
After just using the newer idea(s) since yesterday afternoon, I have decided that high swappiness is a bad idea. Swap just fills and fills until the kernel reaches a point that it decides it has to start juggling things around and that's when our Pre- falls over dead.

I'm sticking with this multi-swap idea but have dropped the swappiness down to 30 and put back in the higher dirty times (my corrected ones - not the lower carrel's - I still am convinced that Palm's defaults are WAY too fast, though).

I'm thinking to use the 2nd swap as "emergency" swap. With the lower swappiness, the kernel should only decide to use that when it ABSOLUTELY HAS TO to keep itself alive. This is what swap is for, not for filling up just to fill up (which is what the high swappiness will/does do).

Will report back.


M.

I'm very curious to hear how this works. One word of caution about the "emergency" swap idea. If you delay swapping until you absolutely need it, then you will get larger delays when you do need more memory. It's a delicate balance. You don't want to swap too early, but also not too late. I want my system using swap before it absolutely has to. That should give me smoother operation.

Also note, that I don't think you can set VM configuration parameters that apply to the swap partition that are different than the compcache. Both are considered swap to the VM system. The compcache has a higher priority so it will get used first. But the swapiness (for example) that you set will apply to both the swap partition and the compcache swap.

FWIW: I am not any longer using the lower values that I mentioned a while ago. I am using all Palm defaults. I'm not claiming that these are better. I'm just trying to get a baseline that I understand before I go tweaking. The palm values are
vm.swappiness = 60
vm.vfs_cache_pressure = 100
vm.page-cluster = 3
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 500
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5

Here's something else to consider. Take a look at swaphack.sh and at the end, Palm is setting some tuning parameters that they thought would be useful when using a swap partition. The one that interested me was the I/O scheduler. Palm is setting it to cfq, but most of us running a non-stock kernel probably have a different setting (noop, anticipatory or deadline). This one can be set with Govanah.

The hard part with all this is doing truly objective testing. Changing more than one variable at a time makes it hard to learn how much effect each had. But, on the other hand, the variables are inter-related and each does effect the other. So you can't tune one variable, claim it is ideal and move on to the next. Sigh ... it's never easy. :)

rmausser
05/29/2011, 08:19 PM
I have to admit im having VERY good results with:


vm.swappiness = 70
vm.vfs_cache_pressure = 180
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000


Compcache: 32768kb (and Linux Swap Disk)

dave75
05/29/2011, 10:33 PM
THESE NEW SETTINGS ARE AMAZING!!

Just had to say. Good work everyone. My phone is usable again. THANK GOD.

To recap so far: (use Internalz Pro on the phone if ur not good with terminal to change these files BUT FIRST MAKE SURE that Internalz Pro is set to LINUX mode. To switch to linux mode open Internalz Pro and "left drop down menu>Preferences>Text Editor>Newline format: LINUX") Also scroll to the bottom of preferences and enable Master mode. BE CAREFUL, you now have the ability to modify system files with Internalz. Turn this off after you hack your phone so you dont delete something by mistake.

Step 1: Change /etc/event.d/LunaSysMgr from:

echo "1" > /proc/sys/vm/overcommit_memory".

to

#echo "1" > /proc/sys/vm/overcommit_memory".

Step 2: Make a file in /etc called "sysctl.conf". Open the file with a text editor and write:


vm.swappiness = 90
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 1500


Step 3: Open /etc/event.d/compcache.

edit line 10 to say:

modprobe ramzswap disksize_kb=98304

change line 10 to say ONLY that. Remove the part that says: dev/store/swap after it.

then, after the line 12 in Compcache (swapon /dev/ramzswap0 -p 100, add:


mkswap /dev/store/swap
swapon /dev/store/swap


Reboot the phone. Enjoy. :)

**I would highly recommend using the thundercheif F105 kernel if you can, as it overclocks the RAM to 200mhz, which will help with the slow speed of Compcache.**

Explanation:

What we are doing, is making essentially a pyramid of memory.

On the top, we have real RAM, but less of it available. This is reserved only for applications that are currently being open or run and used, so they are responsive to the users input.

In the middle of the pyramid we have Compresses Cache, or RAM that is being used as a compressed swap file. It is slower, but since it compresses the pages, there is more available for the phone to use than the Pre came with. It has the second priority, and is used for applications in the background and things that are not at your immediate attention.

At the bottom we have disk swap. This is the slowest memory, but since we essentially have 8GB of memory on the phone, we could eventually turn this up to be even larger in size. Dirty pages and memory that is currently not being used at all, but the kernel thinks might be available on short notice, like background processes, lives here.

We have organized the kernel to as quickly as possible dump any pages that arent being used to compcache, so that the faster ram is available for the app or device the user is using, hopefully eliminating lag. The advantage is that compcache increases the available amount of memory by compressing the pages, and therefore the kernel doesnt have to clean out the swap and ram as often, which is what is causing those long and painful lags we keep experiencing.

**EDIT**

changed memlimit to disksize.

memlimit isnt working right and is stuck on 60mb according to /sbin/swapon -s. Dont know why.


Are the changes in this post still valid with the exception of what is in your last post?

carrel
05/29/2011, 10:52 PM
Are the changes in this post still valid with the exception of what is in your last post?

Yes, those instructions will work.

I think step #1 will actually have no effect, but it doesn't hurt. Step #2 is fine tuning. Several folks are playing with different numbers. Its still to be seen what the best settings are. You can do this or leave this step out. Step #3 will have the most impact. Those instructions use 96Megs of compcache. That's a fine starting value. Some folks are trying out smaller values too. Hopefully we can come up with an optimal value, but my guess is that "optimal" will end up being rather subjective and different people will prefer different values based on how they use their Pre.

If you are just starting with this, I'd suggest just doing step #3. Try it a bit. Then try step #2 or a variation of #2 or try changing the compcache size in #3.

rmausser
05/30/2011, 12:10 AM
Those instructions use 96Megs of compcache. That's a fine starting value. Some folks are trying out smaller values too. Hopefully we can come up with an optimal value, but my guess is that "optimal" will end up being rather subjective and different people will prefer different values based on how they use their Pre.

I completely agree with this. I have found that with 32MB of compcache the phone is overall mroe responsive, but with 96MB I can open more apps at once, but the whole UI overall is slightly laggy.

rmausser
05/30/2011, 12:36 AM
Personally though I am finding 96mb with linux disk swap to be the best overall.

Cards sometimes take a little bit longer to load than with 32mb compcache, but overall things stay more responsive and snappy, and overall less huge "lockups"

so I have to say that 96mb is appearing to be the magic number for compcache, at least with me.

rmausser
05/30/2011, 02:41 AM
I have personally found this to be the best so far.


vm.swappiness = 70
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000


With compcache at 96MB and Linux Disk Swap

I am sorry, but I do not like a low Swapiness setting personally. It causes constant hangs.

snpalavan
05/30/2011, 01:34 PM
So far so good, but webOS still does not like getting texts, calls and doing things at the same time. Definitely have lots of hang ups doing that.

rmausser
05/30/2011, 04:27 PM
vm.swappiness = 60
vm.vfs_cache_pressure = 100
vm.page-cluster = 3
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 500
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5



These defaults were essentially configured for the Pre Plus however. Which has 512 mb of ram. Palm never configured 2.1 to work on a Pre-, or at least the doctor we are using is essentially for a Pre Plus.

where did you find this information? Would there be any way to find these values for 1.4.5 on a Pre Minus?

Then we could use those values as a baseline because the values for 1.4.5 on a Pre Minus were infact designed for a Pre Minus.

carrel
05/30/2011, 06:03 PM
where did you find this information? Would there be any way to find these values for 1.4.5 on a Pre Minus?

Then we could use those values as a baseline because the values for 1.4.5 on a Pre Minus were infact designed for a Pre Minus.

Just run "sysctl -a | grep vm" and you will see all the VM settings. Some of these are at the kernel defaults and some have been configured in startup scripts. The command will show the settings as they are so your changes in sysctl.conf and LunaSysMgr will be reflected in the output.

You should definitely be able to run this on a 1.4.5 Pre, though I don't have one I can try. However, I would bet that the 1.4.5 values are much less relevant than 2.1.0 Pre+ values. The reasoning is that 1.4.5 didn't have compcache. Nonetheless, it would be interesting to see what they are. My guess is that Palm has done very little tuning and is mostly using stock linux defaults. Hmmm, did 1.4.5 even have swap??

Also keep in mind that most of us are NOT running stock kernels. So any settings that were not overridden in startup files may have been changed by psycho or whoever built your kernel. But it does seem that Palm is using startup files to make kernel parameter changes, so it's likely that what you see with sysctl is pretty close to what Palm did, even with a non-stock kernel.

carrel
05/30/2011, 06:33 PM
Yay! I just broke 2 full days running with my config. I have run many programs including Angry Birds and much more. And my Pre- is pretty darn happy. I have not changed the config at all during this time just so I could test stability. The VM system is behaving properly, allocating swap and releasing it appropriately. And it is favoring compcache over physical swap. Quick summary: I am running 96 Meg of compcache, a separate physical swap and NO custom settings of the VM system.

For anyone starting out I highly recommend that you start with just this one change: Edit the file /etc/event.d/compcache so that it looks like this:
# -*- mode: shell-script; -*-
start on stopped swaphack
stop on runlevel [!2]
console none
script
if [ -e /etc/palm/preferences/compcache_enabled ]; then
swapoff -a
sleep 1
modprobe xvmalloc
# modprobe ramzswap memlimit_kb=98304 backing_swap=/dev/store/swap
modprobe ramzswap disksize_kb=98304
sleep 3
swapon /dev/ramzswap0 -p 100
mkswap /dev/store/swap
swapon /dev/store/swap
fi
end script


I highlighted the lines to add in green and the line in red was there before and has to go (or you can add the # at the beginning to comment it out like I did).

Do just that and try it for a while. In my next post I'll start a discussion about compcache size.

WARNING: If you edit the file with Internalz make sure you set the Newline Format to "Linux" and NOT Windows!!!

rmausser
05/30/2011, 06:52 PM
Yay! I just broke 2 full days running with my config. I have run many programs including Angry Birds and much more. And my Pre- is pretty darn happy. I have not changed the config at all during this time just so I could test stability. The VM system is behaving properly, allocating swap and releasing it appropriately. And it is favoring compcache over physical swap. Quick summary: I am running 96 Meg of compcache, a separate physical swap and NO custom settings of the VM system.

For anyone starting out I highly recommend that you start with just this one change: Edit the file /etc/event.d/compcache so that it looks like this:
# -*- mode: shell-script; -*-
start on stopped swaphack
stop on runlevel [!2]
console none
script
if [ -e /etc/palm/preferences/compcache_enabled ]; then
swapoff -a
sleep 1
modprobe xvmalloc
# modprobe ramzswap memlimit_kb=98304 backing_swap=/dev/store/swap
modprobe ramzswap disksize_kb=98304
sleep 3
swapon /dev/ramzswap0 -p 100
mkswap /dev/store/swap
swapon /dev/store/swap
fi
end script


I highlighted the lines to add in green and the line in red was there before and has to go (or you can add the # at the beginning to comment it out like I did).

Do just that and try it for a while. In my next post I'll start a discussion about compcache size.

Awesome work Carrel!

I agree with you that 96mb compcache seems to be the best, and what people should just be doing right now.

At least, anything below 96mb is no good, Im going to try 128mb and 160mb just to be adventurous,

But, I have to admit, that today I am finding this setting in sysctl.conf very good.


vm.swappiness = 50
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000


I am constantly sitting at about 16mb of swap cache at idle, and the phone is staying snappy. When I do things the swap cache goes up, but then it clears itself out pretty quickly.

Its just a tad swappier than palms.

(I am using 96mb compcache)

Does anyone have a Pre Minus with the stock kernel running 1.4.5?

Can you please run this command and give us the results:

"sysctl -a | grep vm"

Thanks

snpalavan
05/30/2011, 07:08 PM
I have one...but not on me (girlfriend's haha) if noone else reports I will do so tonight when I get it!

carrel
05/30/2011, 07:23 PM
Ok, so now that I am comfortable that VM is stable, I'm starting to look at optimizing it. My 2 cents worth is that the biggest bang for the buck is setting the compcache size. I chose to initially set compcache to 96 meg in a VERY unscientific way. Others have tried running with smaller values with some success, but there seems to be some agreement that results are very subjective. So how can we be more scientific and more objective??

What do we know? ... RAM is fast, compcache is slower and physical swap is the slowest. We also know that increasing the size of compcache reduces the size of RAM but increases the total size of RAM + compcache. Also the size of physical swap is fixed. (Even if we could increase physical swap, who cares, it's slow.) Ideally we barely use physical swap, but we do use it a little. We want to adjust the size of RAM + compcache to be as small as possible (because that makes RAM bigger) but large enough that we only use physical swap a little bit.

Using "swapon -s" we can observe how we are doing. Initially usage always starts low, so don't pay much attention to the values for a while after a reboot (or Luna restart). Luna usage grows over time as you run different apps and the memory isn't all released when that app closes. This isn't a leak, its just Luna doing its thing. It does level off. Now once we are stable, let's run "swapon -s":
carrel@ - Palm Pre:~$ /sbin/swapon -s
Filename Type Size Used Priority
/dev/ramzswap0 partition 98296 96836 100
/dev/mapper/store-swap partition 106488 5728 -2

Looking at that, compcache is basically full and under 6 Megs of physical swap are in use. That's pretty ideal, I think. Now the thing to do is check this at many points in time. It's fine to use more swap if you are temporarily using more apps than you normally do. You'll get some delays, but remember, we are optimizing for our normal usage.

You probably will never see physical swap usage go to zero. Even when compcache usage is less than full. Some memory gets allocated and seems to be rarely, if ever, used so it remains in swap and never brought up to higher priority compcache or RAM. (Why should the VM system bother using resources to move it if you aren't using it.)

If you find that compcache is ususally not close to full, then you probably can decrease the size of your compcache. If you see that a significant amount of swap is usually being used, then you probably should increase the size of compcache. The test of your settings is how balanced you are with your usage.

OK, so that's my analysis. I included no numbers here because I'm hoping to discuss the concepts and analysis methodology, rather than just people's personal observations. What do people think? Make sense? Any other analysis tools that people are using?

snpalavan
05/30/2011, 08:44 PM
Ok, I have a few questions, i type in swapon -s and get this:

blahblah:$ /var/home/root/swapon -s
Filename Type Size Used Priority
/dev/mapper/store-swap partition 106488 0 -2
/dev/ramzswap0 partition 131064 27464 1


So basically same idea, but the questions I have are
1) Why (if it matters) is my store listed first?

2) Why is the priority of my ramzswap0 only 1?

3) Why is my ramzswap0 size so damn big?

EDIT: Keep in mind I'm using the same settings, and the sysctl.conf that rm recently posted.

carrel
05/30/2011, 09:21 PM
Ok, I have a few questions, i type in swapon -s and get this:

blahblah:$ /var/home/root/swapon -s
Filename Type Size Used Priority
/dev/mapper/store-swap partition 106488 0 -2
/dev/ramzswap0 partition 131064 27464 1


So basically same idea, but the questions I have are
1) Why (if it matters) is my store listed first?

2) Why is the priority of my ramzswap0 only 1?

3) Why is my ramzswap0 size so damn big?

EDIT: Keep in mind I'm using the same settings, and the sysctl.conf that rm recently posted.

Hmmm, that is different.

I would guess, 1) because the physical swap was created before the compcache, 2) & 3) I would guess something got mistyped.

Did you do any of this by hand?

Did you ever do any of the things posted much earlier in this thread where there was a script to occasionally remove swap and then re-create it? (Could that still be around?) This is my number #1 guess as to what is happening!!!

How long ago did you last reboot? How does it look right after a reboot?

Can you post your entire /etc/event.d/compcache file.

Dave

carrel
05/30/2011, 09:25 PM
Ok, I have a few questions, i type in swapon -s and get this:

blahblah:$ /var/home/root/swapon -s
Filename Type Size Used Priority
/dev/mapper/store-swap partition 106488 0 -2
/dev/ramzswap0 partition 131064 27464 1



Uh... why is swapon in /var/home/root/ ?? You should be using /sbin/swapon

rmausser
05/31/2011, 12:02 AM
I have to admit, been using the Sysctl settings I made in post 420 (hehehe) for half the day today, and they have been stellar.

On idle my swap whittles itself down to a nice 65mb, not too small but not too big.

Really enjoying this setting.

snpalavan
05/31/2011, 12:20 AM
That's directly from copy/paste out of linux terminal from novaterm. But I did restart and now its ramz first at 100 priority (98296 mb) then swap at -2 priority (106488 mb). However, it runs from wherever doesn't show /var/home/root unless i'm in that folder.

EDIT: The problem was leftover code from previous testing :) totally forgot to check thanks! Now it works fine haha.

snpalavan
05/31/2011, 12:33 AM
And for the base Palm kernel on 1.4.5 (I have to type on phone excuse me):


error: permission denied on key 'kernel.sched_nr_migration'
error: permission denied on key 'net.ipv4.route.flush'
vm.overcommit_memory = 1
vm.panic_on_oom = 0
vm.oom_kill_allocating_task = 0
vm.oom_late_helper = /usr/sbin/oom_late_helper
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 6000
vm.nr_pdflush_threads = 2
vm.swappiness = 60
vm.lowmem_reserve_ratio = 25632
vm.drop_caches = 0
vm.min_free_kbytes = 1999
vm.percpu_pagelist_fraction = 0
vm.max_map_count = 65536
vm.laptop_mode = 0
vm.block_dump = 0
vm.vfs_cache_pressure = 100


that's all of the output I get :)

carrel
05/31/2011, 12:38 AM
And for the base Palm kernel on 1.4.5 (I have to type on phone excuse me):


error: permission denied on key 'kernel.sched_nr_migration'
error: permission denied on key 'net.ipv4.route.flush'
vm.overcommit_memory = 1
vm.panic_on_oom = 0
vm.oom_kill_allocating_task = 0
vm.oom_late_helper = /usr/sbin/oom_late_helper
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 6000
vm.nr_pdflush_threads = 2
vm.swappiness = 60
vm.lowmem_reserve_ratio = 25632
vm.drop_caches = 0
vm.min_free_kbytes = 1999
vm.percpu_pagelist_fraction = 0
vm.max_map_count = 65536
vm.laptop_mode = 0
vm.block_dump = 0
vm.vfs_cache_pressure = 100


that's all of the output I get :)

Thanks. That's identical to 2.1.0. I think that shows that Palm wasn't really tuning this and more certainly wasn't tuning it for the Pre- vs. the Pre+.

snpalavan
05/31/2011, 12:45 AM
no problemo! Sucked...had to type that ALL on the phone haha!

snpalavan
05/31/2011, 10:49 AM
Ok I had an issue with the settings using the video recording. After 15 seconds or so the phone froze and reset. This was no ordinary reboot, however, when the phone turned back on I had lost 'settings'. (Govnah, Preware, etc. acted as if they had never been run)

First, has any one else experienced this, or is it due to some other tinkering I have done because so far everything else has worked flawlessly. And second, what would have happened to cause the setting loss...perhaps memory related??

somline
05/31/2011, 11:15 AM
Its well known, that if you don't make a clean shutdown you may loose seting.
I think it's filesystem related.
and it looks like you had a hard crash.

UncleVanya
05/31/2011, 12:05 PM
Its well known, that if you don't make a clean shutdown you may loose seting.
I think it's filesystem related.
and it looks like you had a hard crash.

Save Restore might be useful in these situations.

snpalavan
05/31/2011, 01:10 PM
Save Restore might be useful in these situations.

Oh yes, I forgot to shamelessly ;) plug Save/Restore, i had used that two days ago and it did indeed save my booty. No major settings, just things like the initial page on apps, and stuff like that.

Xanadu73
05/31/2011, 01:19 PM
Its well known, that if you don't make a clean shutdown you may loose seting.
I think it's filesystem related.
and it looks like you had a hard crash.


This is true, and there's a way to "fix" that also.

Some dolt at Palm thought it was a good idea to change /var's mount option to: "data=writeback" in /etc/fstab. GET RID OF THAT so the kernel falls back to the default for ext3 of "data=ordered". First from the man page:



writeback

Data ordering is not preserved - data may be written into the main filesystem after its metadata has been committed to the journal. This is rumoured to be the highest-throughput option. It guarantees internal filesystem integrity, however it can allow old data to appear in files after a crash and journal recovery.



ordered

This is the default mode. All data is forced directly out to the main file system prior to its metadata being committed to the journal.


In human terms, in the event of a dirty shutdown, there's NO GUARANTEE that the journal was updated with what is current in the filesystem (the kernel devs even say "it gets to it when it gets to it; it could be seconds, minutes, or even hours later."), so when the journal replays at next boot, the file system will get over written with old/bad/incomplete/etc. data which, in our case, often wipes out all our personal settings.

I made this change about a year ago and with all the dirty shutdowns from bad kernel settings, battery pulls, etc., I have NOT ONCE lost all my settings again.

Stupid, stupid, Palm.


M.

UncleVanya
05/31/2011, 01:41 PM
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.160.2456&rep=rep1&type=pdf

It appears someone outside of this group is looking at optimizing flash storage on Linux and they propose a new way of doing that.

rksand
05/31/2011, 02:38 PM
This is true, and there's a way to "fix" that also.

Some dolt at Palm thought it was a good idea to change /var's mount option to: "data=writeback" in /etc/fstab. GET RID OF THAT so the kernel falls back to the default for ext3 of "data=ordered".

Could you enlighten those of us who are not so linux savvy on how to do that? I'm a little tired of that issue myself.

Ron

rmausser
05/31/2011, 03:12 PM
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.160.2456&rep=rep1&type=pdf

It appears someone outside of this group is looking at optimizing flash storage on Linux and they propose a new way of doing that.

It would be awesome if we could optimize our flash memory to the point that compcache would not even be necessary.

Could you imagine? 256mb of flash disk swap at the same speed as compcache with all 256mb of ram available as actual ram..... :D

UncleVanya
05/31/2011, 03:15 PM
It would be awesome if we could optimize our flash memory to the point that compcache would not even be necessary.

Could you imagine? 256mb of flash disk swap at the same speed as compcache with all 256mb of ram available as actual ram..... :D

That would be nice but the article only promised about 20% improvement if I recall - it also reduced unnecessary write cycles on the flash.

snpalavan
05/31/2011, 04:22 PM
This is true, and there's a way to "fix" that also.

...

Stupid, stupid, Palm.


M.

Totally wish i had known about this a long time ago, has happened a few times haha thanks!

On a separate note, I was running fine with the new settings, but out of nowhere my phone no longer works. What i mean is:
I click on name to call and phone app shows up but no number in the area at the top.
Pressing the call button after typing a number does nothing, it sits there.
Texting works fine both in and out of wifi and in and out of roaming.
Phone is not 'offline' or in 'airplane mode'.
I get the alert for missed calls, but never have any indication that a call is incoming until then.

The reason i ask here is because this is the most recent fiddling I have done with my phone. I assume it has something to do with altering the memory, but it also might not. I'm going to doctor it to see if that helps (it's about time for one of those anyway).

carrel
05/31/2011, 04:33 PM
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.160.2456&rep=rep1&type=pdf

It appears someone outside of this group is looking at optimizing flash storage on Linux and they propose a new way of doing that.

That's a cool idea. Nice find.

I did a quick look and didn't find any source code anywhere. But I have been told that I have bad "Google-**". Implementing from scratch is a big project.

carrel
05/31/2011, 04:36 PM
Hey, the auto-censor-bot edited my post above. I wrote "Google-Fu" as in Kung-Fu and somehow, that is too racy for this crowd. :-)

rksand
05/31/2011, 07:12 PM
So I'm running 96mb compcache and the following in sysctl.conf:

vm.swappiness = 50
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000

Things had been chugging along nicely all day, but then I was running Classic and there was a drastic change in the swap utilization followed by a Luna restart.

swpd free buff cache si so
91904 2520 1464 50840 0 0
91904 3896 1504 49136 37 0
91904 3896 1512 49140 0 0
91904 8272 1528 49148 0 0
91904 8812 1544 49176 8 0
91904 8752 1560 49180 0 0
91376 12780 1664 43220 14 0
32500 108604 2012 46400 491 0
32472 67968 9800 59532 74 0
32472 48392 12840 70764 0 0
32472 36960 13024 82784 1 0
32468 10644 13624 88524 153 0

Any thoughts?

snpalavan
05/31/2011, 08:00 PM
So I'm running 96mb compcache and the following in sysctl.conf:

...

Things had been chugging along nicely all day, but then I was running Classic and there was a drastic change in the swap utilization followed by a Luna restart.

...

Any thoughts?

Same issue with me but when using the camcorder, seems that any high use of memory just overloads the phone...

rmausser
05/31/2011, 08:19 PM
I am not sure but it cant be the sysctl settings, they are not that far off from the palm defaults.

Try the palm sysctl defaults and see if the issue still arises.

If the issue still is there, then change the compcache size to 32mb, then if that makes things worse and not better try changing the compcache size to 128mb.

Basically just seeing if first the sysctl is the problem, then ruling that out and moving onto a "too large" or "too small" compcache.

carrel
05/31/2011, 09:12 PM
So I'm running 96mb compcache and the following in sysctl.conf:

vm.swappiness = 50
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000

Things had been chugging along nicely all day, but then I was running Classic and there was a drastic change in the swap utilization followed by a Luna restart.

swpd free buff cache si so
91904 2520 1464 50840 0 0
91904 3896 1504 49136 37 0
91904 3896 1512 49140 0 0
91904 8272 1528 49148 0 0
91904 8812 1544 49176 8 0
91904 8752 1560 49180 0 0
91376 12780 1664 43220 14 0
32500 108604 2012 46400 491 0
32472 67968 9800 59532 74 0
32472 48392 12840 70764 0 0
32472 36960 13024 82784 1 0
32468 10644 13624 88524 153 0

Any thoughts?

Well, I know its kind of obvious, but you are running out of memory. Even if we completely nail the VM problems and optimize it perfectly, this is still just a Pre- and it really needs more memory. Now that doesn't mean that we have nailed all the problems or optimized perfectly, but I'm just saying... There are limits. In your case, I can't tell if Classic went on a memory feeding frenzy or if the VM system went wonky.

But, let's be optimistic and see what we can figure out... A few questions:

What else were you running? Did you have a lot of apps open? What were you doing in Classic?

Above you say that you are running a 96 Meg compcache, but you don't mention physical swap. Did you enable the physical swap? Can you send the output of '/sbin/swapon -s' ? If you are not running the physical swap with compcache, then running out of memory is completely unsurprising.


One area we really haven't explored is increasing the size of the physical swap. Assuming that you are using a physical swap, then you need more memory and swap is going to be your only option. Yes swap is slow, but I personally will take slow over crashed anyday. Now we have a 100M swap partition. I'd love to be able to increase it to 256 or 384. Maybe we can create and add a swap file, but it just seems wrong to me to create a swap file on a flash file system. Might work, but something about it scares me. I'd prefer to understand what that device is that /dev/mapper/store-swap calls out.

rmausser
05/31/2011, 11:04 PM
Maybe we can create and add a swap file, but it just seems wrong to me to create a swap file on a flash file system. Might work, but something about it scares me.

I am pretty sure that the swap partition we are currently using is on the same flash media (actually its an SSD) that the rest of the OS resides on.

So I dont really understand the difference, its not like there is a separate media device in the phone for the physical swap. That would be utterly pointless and better spent on more RAM instead.

So I dont really see what the issue is with moving the swap from one part of the SSD to another part of the SSD.

carrel
06/01/2011, 03:28 AM
OK, if its an SSD, that make some sense. Let's find the disk and have a look...

I rebooted so I could look at the top of dmesg and see the root device. Then let's see what's on it.

root@ - Palm Pre:/var/home/root# /usr/sbin/parted /dev/mmcblk0
GNU Parted 1.8.9
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
print
Model: Unknown (unknown)
Disk /dev/mmcblk0: 8221MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 524kB 4719kB 4194kB primary palo
2 4719kB 38.3MB 33.6MB primary ext3
3 38.3MB 8221MB 8183MB primary lvm

Partition #3 is an LVM with most of the space. Let's look at it.

root@ - Palm Pre:/var/home/root# /usr/sbin/vgdisplay
--- Volume group ---
VG Name store
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 14
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 8
Open LV 7
Max PV 0
Cur PV 1
Act PV 1
VG Size 7.62 GB
PE Size 8.00 MB
Total PE 975
Alloc PE / Size 972 / 7.59 GB
Free PE / Size 3 / 24.00 MB
VG UUID MiFyGF-ov2x-hX1e-PA40-k1jU-ge1d-M0kpNK

and
root@ - Palm Pre:/var/home/root# /usr/sbin/lvdisplay
--- Logical volume ---
LV Name /dev/store/root
VG Name store
LV UUID I3Nc7e-z4kT-zMzg-nJ9A-L418-03eS-C6s8uD
LV Write Access read/write
LV Status available
# open 1
LV Size 456.00 MB
Current LE 57
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Persistent major 254
Persistent minor 0
Block device 254:0

--- Logical volume ---
LV Name /dev/store/var
VG Name store
LV UUID RA9oHF-HSaT-lvHN-AaxK-wfod-pxS2-535TyB
LV Write Access read/write
LV Status available
# open 1
LV Size 64.00 MB
Current LE 8
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Persistent major 254
Persistent minor 1
Block device 254:1

--- Logical volume ---
LV Name /dev/store/update
VG Name store
LV UUID W1axj4-I67Q-WrsH-l57s-YWcM-NGNK-EDqrMY
LV Write Access read/write
LV Status available
# open 0
LV Size 16.00 MB
Current LE 2
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Persistent major 254
Persistent minor 2
Block device 254:2

--- Logical volume ---
LV Name /dev/store/log
VG Name store
LV UUID 079euH-Myjc-H6US-IKl9-P7yy-QP2N-6roUM3
LV Write Access read/write
LV Status available
# open 1
LV Size 24.00 MB
Current LE 3
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Persistent major 254
Persistent minor 3
Block device 254:3

--- Logical volume ---
LV Name /dev/store/media
VG Name store
LV UUID KK2qLk-x96W-eCgi-TTH8-CVyV-zQTk-kp591L
LV Write Access read/write
LV Status available
# open 1
LV Size 6.67 GB
Current LE 854
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Persistent major 254
Persistent minor 4
Block device 254:4

--- Logical volume ---
LV Name /dev/store/swap
VG Name store
LV UUID 8dRgu4-UfRo-bwYU-hjyK-PwPq-7yTd-8yBIlz
LV Write Access read/write
LV Status available
# open 1
LV Size 104.00 MB
Current LE 13
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Persistent major 254
Persistent minor 5
Block device 254:5

--- Logical volume ---
LV Name /dev/store/mojodb
VG Name store
LV UUID j9AsLj-iDoF-eniK-2K2M-VeIF-xvPb-udrP02
LV Write Access read/write
LV Status available
# open 1
LV Size 144.00 MB
Current LE 18
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Persistent major 254
Persistent minor 6
Block device 254:6

--- Logical volume ---
LV Name /dev/store/filecache
VG Name store
LV UUID S2JJWS-nmtT-nX7D-jys6-7fpK-XSaQ-DTZD5q
LV Write Access read/write
LV Status available
# open 1
LV Size 136.00 MB
Current LE 17
Segments 4
Allocation inherit
Read ahead sectors auto
- currently set to 256
Persistent major 254
Persistent minor 7
Block device 254:7



So there it is. The lvm named "store" has all the Linux partitions in it. We can see that swap is 104M and media is 6.67G. Now what we want to do is decrease the size of media and use that to increase the size of swap. I think we should be able to do that with an LVM, but it's late and I'm going to sleep now. But that was a fun look. More later...

carrel
06/01/2011, 04:11 AM
And the good news is that Palm provided us with resize2fs and lvreduce so we have all the tools necessary to shrink /media/internal and the tool lvextend so we can then grow swap.

Basic outline:

reboot to ensure swap is not in use.
swapoff /dev/store/swap
umount /media/internal/ (this may be hard)
e2fsck -f /dev/mapper/store-media
run resize2fs for /dev/mapper/store-media with the new size (about 256 M smaller)
run lvreduce for /dev/mapper/store-media with the new size
run lvextend for /dev/mapper/store-swap with the new size
reboot


Just gotta work on the sizes... But really... I'm going to sleep now...

UncleVanya
06/01/2011, 10:38 AM
And the good news is that Palm provided us with resize2fs and lvreduce so we have all the tools necessary to shrink /media/internal and the tool lvextend so we can then grow swap.

Basic outline:

reboot to ensure swap is not in use.
swapoff /dev/store/swap
umount /media/internal/ (this may be hard)
e2fsck -f /dev/mapper/store-media
run resize2fs for /dev/mapper/store-media with the new size (about 256 M smaller)
run lvreduce for /dev/mapper/store-media with the new size
run lvextend for /dev/mapper/store-swap with the new size
reboot


Just gotta work on the sizes... But really... I'm going to sleep now...

Before you go down that path. Most sources say that swap files are just as effective as swap partitions with current kernels. At this stage - rather than resize the partition, why not just add a file to the swap devices and see what happens?

graffix31
06/01/2011, 12:01 PM
Man my phone died on thursday and my new one is ready to pick up today. so much has happened in the past few days with this. looks like some great progress though. are there any settings that people are having success with right now?

theXfactor2011
06/01/2011, 03:31 PM
Before you go down that path. Most sources say that swap files are just as effective as swap partitions with current kernels. At this stage - rather than resize the partition, why not just add a file to the swap devices and see what happens?

A while ago I added a swap file to my media internal part of the file system. Yeah I had more swap but when I tried to get into usb mode it for fail because the phone couldn't lock the file system. If you dont mind not having usb mode then you can give it a shot. Nobody is saying you cant do it. I was able to get Usb mode back by unmounting the swap. in the end I didnt see much benefit with a swap file.

UncleVanya
06/01/2011, 03:35 PM
A while ago I added a swap file to my media internal part of the file system. Yeah I had more swap but when I tried to get into usb mode it for fail because the phone couldn't lock the file system. If you dont mind not having usb mode then you can give it a shot. Nobody is saying you cant do it. I was able to get Usb mode back by unmounting the swap. in the end I didnt see much benefit with a swap file.

By USB Mode do you mean to share files with your PC or USB Passthru? In either case - I guess you could still validate that extra swap would help before resizing partitions on the phone which is the goal of this.

rksand
06/01/2011, 04:13 PM
Or setup sftp and don't use the usb cable anymore. Its my preferred method, but transferring movies is a little slow.

loopytee
06/01/2011, 07:08 PM
Graffix31- read post #411 and continue on from there, I think this is a good place to re-engage

http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-21.html

loopytee
06/01/2011, 07:11 PM
That being said , Should I update the first post on this thread? point them to start reading from post #411. I refer to #411 because it quotes the most recent directions and then the next post begins the most current discussion and tweaking of this information. what do you all think?

the first post currently directs people to post #74 I think.

rmausser
06/01/2011, 07:53 PM
not yet, I want to first finalise some things, I think we have found some better settings since that time.

I will update it when i know for sure.

rmausser
06/02/2011, 12:30 AM
Latest settings. Your gonna laugh. :D

/etc/sysctl.conf


vm.swappiness = 30
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000



/etc/event.d/compcache

make sure to add these lines if you havent already!!


mkswap /dev/store/swap
swapon /dev/store/swap


comment out line "modprobe ramzswap disksize_kb="

#modprobe ramzswap disksize_kb=

comment out line "swapon /dev/ramzswap0 -p 100"

#swapon /dev/ramzswap0 -p 100

Yup thats right.

Completely disabling compcache.

With the low swapiness and the new disk swap, I am having excellent results.

Maybe we can increase the disk swap size to 200mb for this set up, as there isnt the compcache as well.

graffix31
06/02/2011, 09:10 AM
looks good, ill give it a try.

theXfactor2011
06/02/2011, 11:02 AM
By USB Mode do you mean to share files with your PC or USB Passthru? In either case - I guess you could still validate that extra swap would help before resizing partitions on the phone which is the goal of this.

PC. I haven't tried anything with passthru

Jake_M
06/02/2011, 11:27 AM
rmausser should I do what you said in http://forums.precentral.net/palm-pre/281268-clearing-swap-memory-21.html#post2976158 or your latest post 458?

rksand
06/02/2011, 11:42 AM
Latest settings. Your gonna laugh. :D

/etc/sysctl.conf


vm.swappiness = 30
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000



/etc/event.d/compcache

make sure to add these lines if you havent already!!


mkswap /dev/store/swap
swapon /dev/store/swap


comment out line "modprobe ramzswap disksize_kb="

#modprobe ramzswap disksize_kb=

comment out line "swapon /dev/ramzswap0 -p 100"

#swapon /dev/ramzswap0 -p 100

Yup thats right.

Completely disabling compcache.

With the low swapiness and the new disk swap, I am having excellent results.

Maybe we can increase the disk swap size to 200mb for this set up, as there isnt the compcache as well.

And the default backing swap in /etc/event.d/compcache is still commented out too, right?

modprobe ramzswap memlimit_kb=32768 backing_swap=/dev/store/swap

rmausser
06/02/2011, 01:06 PM
yes. comment that out too.

I am finding that this setup (without compcache) is good if you only use a couple apps at a time.

I tend to be a heavier app user and have multiple apps open at once.

So if you are an app power mutlitasker, stick with 96mb compcache.

Your phone will be slightly slower overall, but wont lock up.

graffix31
06/02/2011, 01:08 PM
when you say heavier app user with apps running at the same time can you ellaborate? of course for different people heavy user means diff things. just curious.
thanks

dave75
06/02/2011, 01:21 PM
Disabled compcache. My standard way of using my Pre- is to have Carbon, Facebook, Calendar, Messaging, Email, Weatherdashboard, and Morestocks running. It would work for a while but eventually, even with all the things you guys have come up with, it would lock up. That may still happen but this is running great for the first 30 minutes. Since going to 2.1, I was never able to open the camera with those things open. I just did, awesome! And I am getting no missed letters as I type right now. Hopefully this holds up, but regardless this is the best my Pre has ever run, thanks.

graffix31
06/02/2011, 01:25 PM
im a bit confused by your post, did you disable compcache just now and with everything else your now working great and when you had all the other tweaks but still had compcache enabled it would lock up?

dave75
06/02/2011, 01:30 PM
im a bit confused by your post, did you disable compcache just now and with everything else your now working great and when you had all the other tweaks but still had compcache enabled it would lock up?

I have made almost all of the changes in this thread. The only thing that really has made a difference is the latest post from rmausser that disables compcache.

carrel
06/02/2011, 02:22 PM
yes. comment that out too.

I am finding that this setup (without compcache) is good if you only use a couple apps at a time.

I tend to be a heavier app user and have multiple apps open at once.

So if you are an app power mutlitasker, stick with 96mb compcache.

Your phone will be slightly slower overall, but wont lock up.

Did you observe/record any swap usage stats while running this test?

What you are saying here makes sense. Without compcache, you get more RAM, but once you use up the RAM, then you go to swap which is slow. So this might be good for people running just a few apps, hence using less memory and needing less/no swap. For a user using more apps, then having compcache increases the amount of VM before you need to go to physical swap. You get less RAM, but RAM+compcache is larger than RAM without compcache and compcache is slower than RAM but faster than swap. So this is exactly what I would expect.

One thing though... Luna seems to grow over time, though it does top out. So even a user running fewer apps may find that after a day or so, things get quite slow with a zero compcache setup. I would think that almost everyone will want some amount of compcache. (But I get in trouble when I think too much.) :-)

Jake_M
06/02/2011, 02:29 PM
Still a little confused between these two things.

Xanadu73
06/02/2011, 03:03 PM
Latest settings. Your gonna laugh. :D

/etc/sysctl.conf


vm.swappiness = 30
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000



Those numbers look familiar. :)

I'll give it a shot completely disabling compcahe and see what happens here.


M.

carrel
06/02/2011, 03:32 PM
Can someone that is running with zero compcache post the output of "free -m" ?

Thanks

Xanadu73
06/02/2011, 03:34 PM
Can someone that is running with zero compcache post the output of "free -m" ?

Thanks

root@My Pre:/# free -m
' total used free shared buffers cached
Mem: 239 219 20 0 3 69
-/+ buffers/cache: 146 93
Swap: 103 0 103I just rebooted about a half hour ago, though.

M.

dave75
06/02/2011, 03:48 PM
Can someone that is running with zero compcache post the output of "free -m" ?

Thanks

I have only been using Internalz Pro to change these files. How do I do that?

rksand
06/02/2011, 03:50 PM
free -m
total used free shared buffers cached
Mem: 239 231 7 0 2 46
-/+ buffers/cache: 182 57
Swap: 103 83 20

4 hours uptime and at 82 MB swap

carrel
06/02/2011, 03:59 PM
Thanks. But unfortunately it is not telling me the information that I was hoping to find. My output (with 96M compcache is)
carrel@ - Palm Pre:~$ free -m
total used free shared buffers cached
Mem: 239 223 15 0 1 43
-/+ buffers/cache: 178 60
Swap: 199 76 123


I was trying to see how much RAM compcache was using. Mine is configured to provide a 96 M "swap" partition. But it should be using about 1/4 that much RAM to do it. Palm did not include the rzstat tool. I was hoping that the RAM that compcache used would be deducted from the total that "free -m" is reporting. But 239 is the same for both of us, so that is how much memory the kernel sees regardless. Anyone know how to see what compcache is using other than porting the compcache rz* tools?

carrel
06/02/2011, 04:11 PM
Ah ha! I found it.
carrel@ - Palm Pre:~$ /sbin/swapon -s
Filename Type Size Used Priority
/dev/ramzswap0 partition 98296 96780 100
/dev/mapper/store-swap partition 106488 0 -2
carrel@ - Palm Pre:~$ cat /proc/ramzswap
DiskSize: 98304 kB
NumReads: 26978
NumWrites: 26079
FailedReads: 0
FailedWrites: 0
InvalidIO: 0
NotifyFree: 3390
ZeroPages: 534
GoodCompress: 72 %
NoCompress: 6 %
PagesStored: 17968
PagesUsed: 7111
OrigDataSize: 71872 kB
ComprDataSize: 27828 kB
MemUsedTotal: 28444 kB


At the bottom two lines, you can see that I am using about 27M for a 96M compcache partition. That would be a 4:1 ratio. Not sure why OrigDataSize is 71M. But swapon says that I am using basically all of the compcache.

Here's an interesting paper. Eeepc701 - compcache - Application benchmark on Eeepc 701. - Compressed Caching for Linux - Google Project Hosting (http://code.google.com/p/compcache/wiki/Eeepc701)

carrel
06/02/2011, 04:14 PM
Oh yeah... One more thought for people testing this. If you have an overclocked kernel that drops down to a lower speed when you are on the charger, then be careful that you are comparing apples to apples. Compcache is awful when I am charging, but not so bad when I am clocked up at 1.1GHz.

I'm sure you all know this, but it's easy to forget.

dave75
06/02/2011, 05:08 PM
Swap finally filled up on me and everything is laggy again. Is there a way to make swap clear out faster?

rmausser
06/02/2011, 07:07 PM
If you are using the no compcache method I would highly suggest using Preset Reset to restart your phone every night.

This phone eats swap for breakfast.

Ironically I still think 96mb compcache with a high swapiness is best, which was the first option I tried out almost a week ago.

Go figure.

My last test before I give up and settle on 96mb is going to be a crazy 256mb compcache (which will use up 64 mb of actual ram) and 200mb swap (provided we can resize the swap) with swapiness set to 100.

Ramp it up until it breaks, I say.

graffix31
06/02/2011, 07:14 PM
looking forward to the results. there is only so much one can do with not enough memory. as long as we can find the best setting to optimize what we do have, i think restarting daily is good practice anyway. no harm in it thats for sure. i dont mind having to reset if the phone boggs way down from heavy use either as long as it doesnt get so bad that it requires a battery pull.

dave75
06/02/2011, 07:27 PM
I do use Preset Reset, I was just wondering if there was a setting to free up swap faster. I'm not really sure how all this works. No compcache is still the best settings I have used by far. I have not had to restart yet. It seems like things bog down for a bit but it recovers better.

Jake_M
06/02/2011, 08:37 PM
I do use Preset Reset, I was just wondering if there was a setting to free up swap faster. I'm not really sure how all this works. No compcache is still the best settings I have used by far. I have not had to restart yet. It seems like things bog down for a bit but it recovers better.

Is the bogging down worse than before? I mean 2.1 on a Pre- gets pretty rough. It's bearable for like 10 seconds but then it starts locking up, and the best option is to run Garbage Collector. I'd prefer bogging down vs locking up honestly. At least it's still kind of moving :P

dave75
06/02/2011, 10:40 PM
I think it's better. I'm probably not turning compcache back on. I did have to reboot, but I've been rebooting every day anyway. The performance with 8-10 apps running is much better.

loopytee
06/02/2011, 11:12 PM
so how do we make the swap 200?

rmausser
06/02/2011, 11:18 PM
"modprobe store-swap disksize_kb=262144" ?

Dont try this, I have no idea what I am talking about.

Just a guess.

Xanadu73
06/02/2011, 11:29 PM
so how do we make the swap 200?


It requires resizing the partitions. Luckily, Palm gave us the tools to do it right on the device. It's just a matter of getting it 100% correct the first time to avoid mandatory Doctoring in the event of a "OOPS!". :-)


M.

loopytee
06/02/2011, 11:38 PM
sort of like the game "operation" make a slip and BBZZZZZZZZZZZZZZZZZ

Xanadu73
06/02/2011, 11:39 PM
One thing though... Luna seems to grow over time, though it does top out. So even a user running fewer apps may find that after a day or so, things get quite slow with a zero compcache setup. I would think that almost everyone will want some amount of compcache. (But I get in trouble when I think too much.) :-)


This is why I looked into disabling overcommit. LunaSysMgr and especially WebAppMgr keep eating up "memory". I figured that they were doing something they probably shouldn't be (judging on the other things in the OS that Palm has done...).

I also do agree that some amount of compcache is good. I am currently testing the zero compcache idea, but, I think that just a thimble size 10M - 12M might be good for this. Big enough to keep actual real stuff worth compressing and keeping, and PLENTY small enough not to catch to much binary junk that'll bog down the system trying to compress the un-compressible.

P.S.

memlovers:
#!/bin/sh
ps -A --sort -rss -o comm,pmem,vsize | head -n 11Nice pretty display of the top 10 Memory Hoggers.

root@My Pre:/# memlovers
COMMAND %MEM VSZ
WebAppMgr 37.2 236596
LunaSysMgr 16.8 83072
sdlterminal 8.4 29584
BrowserServer 4.8 56580
mojodb-luna 4.1 52412
mojomail-imap 2.4 22348
PmWsfDaemon 2.2 46452
LunaDownloadMgr 1.9 17284
mediaserver 1.9 27604
pubsubservice 1.7 18844
M.

rmausser
06/02/2011, 11:46 PM
I am currently trying out 256mb of compcache.

The phone actually booted, and so far hasnt blown up into a million pieces.

Also made swapiness 90 to make use of all the swap.

first impressions: everything is slower, but in a very responsive and understanding way.

Rather than this peppy race of a phone that suddenly trips over its own feet and then crashes into a long hang, it seems to smooth out all the bumps so its slower overall, but in a consistent way.

We will see what my opinion is after a day of use.

carrel
06/03/2011, 12:20 AM
"modprobe store-swap disksize_kb=262144" ?

Dont try this, I have no idea what I am talking about.

Just a guess.

Do Not do this!!!

I can't imagine that this would actually do anything because the commands are garbage, but just avoid this. It is bogus.

Posts #449 and #448 look at the actual partition tables and explain what will be needed to change the partition size. I haven't worked out the exact numbers, but I think I will over the weekend. I have been reading up on VM systems and many Linux distros get wonky when they are running low on VM. With our Pre-'s we are getting too close to running out of VM, even with compcache and 100M of swap. Adding another 256 M of VM would put us on par with the newer Pre's and while it won't be as fast as they are, it should be as stable.

carrel
06/03/2011, 12:33 AM
This is why I looked into disabling overcommit. LunaSysMgr and especially WebAppMgr keep eating up "memory". I figured that they were doing something they probably shouldn't be (judging on the other things in the OS that Palm has done...).


What overcommit does may be a bit different than you think. Setting overcommit to 1 says that the kernel will never fail any attempt at malloc() no matter how much memory is asked for and how much VM is free. But once those processes actually start to use the memory, then the OOM killer process will kill off processes to free memory as needed. Setting overcommit to 0 causes the kernel to guess at how much malloc()ing to allow and it still allows more than the actual size of VM. But it isn't infinite. The OOM killer still runs.

Neither will keep Luna from growing. But in the "1" case, some random processes will get killed off when Luna uses too much memory. In the "0" case, there is a slight chance that Luna might actually get an error when calling malloc() and then Luna will have to abort itself.

carrel
06/03/2011, 12:36 AM
memlovers:
#!/bin/sh
ps -A --sort -rss -o comm,pmem,vsize | head -n 11Nice pretty display of the top 10 Memory Hoggers.

root@My Pre:/# memlovers
COMMAND %MEM VSZ
WebAppMgr 37.2 236596
LunaSysMgr 16.8 83072
sdlterminal 8.4 29584
BrowserServer 4.8 56580
mojodb-luna 4.1 52412
mojomail-imap 2.4 22348
PmWsfDaemon 2.2 46452
LunaDownloadMgr 1.9 17284
mediaserver 1.9 27604
pubsubservice 1.7 18844
M.

Wow, that is interesting. I wonder what WebAppMgr is. What a pig! :-)

Xanadu73
06/03/2011, 08:07 AM
Wow, that is interesting. I wonder what WebAppMgr is. What a pig! :-)


I've been trying to hunt that one down myself (on and off). It's not an actual "app" like LunaSysMgr and friends are. I don't know what it is, where it lives, how it's called, or how the mem usage is controlled.

me@aragorn ~/webOS/ROOTS/a $ find -name '*WebAppMgr*'
me@aragorn ~/webOS/ROOTS/a $Nothing. OK, let's try this:

me@aragorn ~/webOS/ROOTS/a $ grep -rn WebAppMgr *
etc/palm/db/permissions/com.palm.contextupload:16: "caller": "com.palm.eventreporter.WebAppMgr",
Binary file usr/bin/LunaSysMgr matches
usr/share/dbus-1/system-services/com.palm.luna.service:2:Name=com.palm.appinstaller;com.palm.applicationManager;com.palm.lunastats;co m.palm.nativealertmanager;com.palm.systemmanager;com.palm.vibrate;com.palm.ambientLightSensor;com.pa lm.display;com.palm.keys;com.palm.appDataBackup;com.palm.eventreporter.LunaSysMgr,com.palm.eventrepo rter.WebAppMgr;com.palm.webos.eventreporter
usr/share/ls2/roles/prv/com.palm.luna.json:5: "allowedNames": ["", "com.palm.appinstaller", "com.palm.applicationManager", "com.palm.lunastats", "com.palm.nativealertmanager", "com.palm.systemmanager", "com.palm.vibrate", "com.palm.ambientLightSensor", "com.palm.display", "com.palm.keys", "com.palm.appDataBackup", "com.palm.eventreporter.LunaSysMgr", "com.palm.eventreporter.WebAppMgr", "com.palm.webos.eventreporter", "com.palm.luna-*"]
usr/share/ls2/roles/prv/com.palm.luna.json:64: "service":"com.palm.eventreporter.WebAppMgr",
Not terribly helpful...

And this is even less so:

root@My Pre:/# ipkg list_installed | grep -i web
com.quickoffice.webos - 1.0-696 -
luna-webkit - 1.1-183.25 -
node-webos - 1.0-3 -
webview - 1.0-23.1 -


M.

somline
06/03/2011, 08:13 AM
From top:
1561 root 19 -1 259m 123m 29m S 2.7 25.0 10:28.79 WebAppMgr

Then ps -afe | grep 1561
root 1561 1512 1 03:02 ? 00:10:28 /usr/bin/LunaSysMgr -s
and finaly:
root@ DrBattery:/var/home/root# ps -afe | grep Luna
root 1135 1 0 02:59 ? 00:00:03 /usr/bin/LunaDownloadMgr
root 1188 1 0 02:59 ? 00:00:00 /usr/bin/LunaUniversalSearchMgr
root 1189 1 0 02:59 ? 00:00:01 /usr/bin/LunaSysService
root 1512 1 0 03:02 ? 00:04:07 /usr/bin/LunaSysMgr -s
root 1561 1512 1 03:02 ? 00:10:29 /usr/bin/LunaSysMgr -s
root 30606 30520 0 14:12 pts/0 00:00:00 grep Luna


So its a child of the LunaSysMgr

rksand
06/03/2011, 12:48 PM
...the best option is to run Garbage Collector...

I'm not yet sure if this is real or not, but since I started JStops Auto GC, I'm finding my swap fills much more slowly.

Xanadu73
06/03/2011, 12:57 PM
So its a child of the LunaSysMgr

It seems like maybe it's some kind of "keep alive" / "watch dog" type daemon.

me@aragorn ~/webOS/ROOTS/a/usr/bin $ strings LunaSysMgr | grep -i webapp
void WebAppBase::blurActivity()
void WebAppBase::focusActivity()
void WebAppBase::destroyActivity()
void WebAppBase::createActivity()
virtual int WindowedWebApp::resizeEvent(int, int, bool)
virtual void WindowedWebApp::onTouchEvent(const SysMgrTouchEvent&)
virtual void WindowedWebApp::paint(bool)
bool WindowedWebApp::showWindowTimeout()
virtual void WindowedWebApp::keyEvent(QKeyEvent*)
virtual void WindowedWebApp::stagePreparing()
virtual void WindowedWebApp::stageReady()
virtual void WindowedWebApp::attach(WebPage*)
virtual void WindowedWebApp::loadFinished()
void AlertWebApp::stopPowerdActivity()
void AlertWebApp::startPowerdActivity()
void CardWebApp::cancelCrossAppScene()
void CardWebApp::removeChildCardWebApp(CardWebApp*)
void CardWebApp::addChildCardWebApp(CardWebApp*)
virtual void CardWebApp::paint(bool)
void CardWebApp::prepareSceneTransition()
void CardWebApp::cancelSceneTransition()
void CardWebApp::runCrossAppTransition(bool)
void CardWebApp::crossAppSceneActive()
void CardWebApp::runSceneTransition(const char*, bool)
virtual WebAppBase* WebAppFactoryLuna::createWebApp(Window::Type, WebPage*, PIpcChannel*)
virtual void WebAppManager::windowedAppRemoved(WindowedWebApp*)
virtual void WebAppManager::windowedAppAdded(WindowedWebApp*)
void WebAppManager::mimeHandoffUrl(const char*, const char*, const char*)
void WebAppManager::markUniversalSearchReady()
void WebAppManager::onDeleteHTML5Database(const std::string&)
virtual void WebAppManager::onDisconnected()
virtual void WebAppManager::serverDisconnected()
void WebAppManager::onProcMgrLaunch(const std::string&, const std::string&, const std::string&, const std::string&)
void WebAppManager::onProcMgrLaunchBootTimApp(const std::string&)
virtual void WebAppManager::onMessageReceived(const PIpcMessage&)
static gboolean WebAppManager::BootupIdleCallback(void*)
static gboolean WebAppManager::BootupTimeoutCallback(void*)
void WebAppManager::stopGcPowerdActivity()
bool WebAppManager::gcPowerdActivtyTimerCallback()
void WebAppManager::startGcPowerdActivity()
static bool WebAppManager::displayManagerCallback(LSHandle*, LSMessage*, void*)
void WebAppManager::launchLowMemoryWarningDashboard()
void WebAppManager::launchLowMemoryWarningAlert()
virtual void WebAppManager::windowedAppKeyChanged(WindowedWebApp*, int)
bool WebAppManager::headlessAppWatchCallback()
WebAppBase* WebAppManager::launchUrlInternal(const std::string&, Window::Type, const std::string&, const std::string&, const std::string&, const std::string&, const std::string&, bool)
WebAppBase* WebAppManager::launchWithPageInternal(WebPage*, Window::Type, ApplicationDescription*)
WebAppManager
WebAppMgr
void WebAppMgrProxy::onLowMemoryActionsRequested(bool)
virtual void WebAppMgrProxy::onMessageReceived(const PIpcMessage&)
void WebAppMgrProxy::webKitDied(GPid, gint)
virtual void WebAppMgrProxy::onDisconnected()
void WebAppMgrProxy::setExtendedSelection(Window*, bool)
void WebAppMgrProxy::onStartDrag(int, int, int, int, int, const std::string&)
void WebAppMgrProxy::emitPaste(Window*)
void WebAppMgrProxy::emitCut(Window*)
void WebAppMgrProxy::emitCopy(Window*)
void WebAppMgrProxy::resizeEvent(Window*, int, int, bool)
void WebAppMgrProxy::inputQKeyEvent(Window*, QKeyEvent*)
void WebAppMgrProxy::inputEvent(Window*, Event*)
std::string WebAppMgrProxy::launchBootTimeApp(const char*)
std::string WebAppMgrProxy::launchNativeApp(const ApplicationDescription*, const std::string&, const std::string&, const std::string&, std::string&)
void WebAppMgrProxy::focusEvent(Window*, bool)
void WebAppMgrProxy::onPrepareAddWindowWithMetaData(int, int, int, int, int)
virtual void WebAppMgrProxy::onPrepareAddWindow(int, int, int, int)
std::string WebAppMgrProxy::appLaunch(const std::string&, const std::string&, const std::string&, const std::string&, std::string&)
static WebAppMgrProxy* WebAppMgrProxy::connectWebAppMgr(int, PIpcChannel*)
%s (%d): WebAppMgrProxy was Disconnected!!
WebAppMgrProxy
WebAppMgrProxy::appLaunch failed, %s.
%s (%d) WebAppMgrProxy instance is NULL
%s (%d): ERROR: WebAppMgr instance already Connected!!!
WebAppMgrProxyM.

Jake_M
06/03/2011, 01:18 PM
I'm not yet sure if this is real or not, but since I started JStops Auto GC, I'm finding my swap fills much more slowly.

Real? I think garbage collection is the only way 2.1 remains usable without restarting.

https://developer.palm.com/content/resources/develop/garbage_collection.html

JSTop has the Auto GC though...

rmausser
06/03/2011, 01:57 PM
I am really liking these settings guys.... try it out if you dare:

vm.swappiness = 100
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000
vm.dirty_ratio = 40
vm.dirty_background_ratio= 0

compcache= 262144 (256mb)

I think this is doing what carell said above. Its not so much that this ammount of swap is being used, its just that it is so big that linux doesnt feel the need to try and clear up room in the swap, which is when we get the hangs.

I would rather like to see 256mb or even more for the swap disk, and less for compcache, giving more room for actual RAM. Right now I am using up 64mb of ram for compcache that could be used for actual RAM, but even still I am not getting any big lags after a day of use. I even had 180mb of swap being used and it was totally fine with that.

I could see this being even better with say, 512mb of disk swap and 32mb of compcache.

PS: I am convinced that the Synergy messaging plugins leak memory like crazy. I turn that on and my swap goes up to 140mb within a matter of minutes.

Xanadu73
06/03/2011, 02:09 PM
I am really liking these settings guys.... try it out if you dare:

vm.swappiness = 100
vm.vfs_cache_pressure = 200
vm.dirty_expire_centisecs = 18000
vm.dirty_writeback_centisecs = 6000
vm.dirty_ratio = 40
vm.dirty_background_ratio= 0compcache= 262144 (256mb)

I think this is doing what carell said above. Its not so much that this ammount of swap is being used, its just that it is so big that linux doesnt feel the need to try and clear up room in the swap, which is when we get the hangs.

I would rather like to see 256mb or even more for the swap disk, and less for compcache, giving more room for actual RAM. Right now I am using up 64mb of ram for compcache that could be used for actual RAM, but even still I am not getting any big lags after a day of use. I even had 180mb of swap and it was totally fine with that.

I could see this being even better with say, 512mb of disk swap and 32mb of compcache.

PS: I am convinced that the Synergy messaging plugins leak memory like crazy. I turn that on and my swap goes up to 140mb within a matter of minutes.

I think something you're not taking into account is that compcache == Compressed Cache. *COMPRESSED*. This takes a lot of CPU power and RAM space to do that. We don't notice it too much on Desktops when zipping up a directory, but, on the wimpy Pre- that is already RAM limited, it's suicide.

More != Better.

It takes a lot of RAM and CPU time to calculate the compression algorithms necessary for all this. So when you're trying to get stuff into swap, you have to compress the stuff. This takes RAM. You're already out of RAM, so that has to get done in swap, which takes compression time... while you're already compressing something which requires it to be decompressed first in order to use/read it to decide what it should do with it... while at the same time trying to compress something...

Big ugly circle that brings our phones to its knees like we see over and over.

BIG compcache is really bad. But, it's your phone, so...


M.

rmausser
06/03/2011, 03:23 PM
I think something you're not taking into account is that compcache == Compressed Cache. *COMPRESSED*. This takes a lot of CPU power and RAM space to do that. We don't notice it too much on Desktops when zipping up a directory, but, on the wimpy Pre- that is already RAM limited, it's suicide.

More != Better.

It takes a lot of RAM and CPU time to calculate the compression algorithms necessary for all this. So when you're trying to get stuff into swap, you have to compress the stuff. This takes RAM. You're already out of RAM, so that has to get done in swap, which takes compression time... while you're already compressing something which requires it to be decompressed first in order to use/read it to decide what it should do with it... while at the same time trying to compress something...

Big ugly circle that brings our phones to its knees like we see over and over.

BIG compcache is really bad. But, it's your phone, so...


M.

As much as I appreciate your theories I am someone who has to do something and see it with my own eyes to be proven the results.

So far my Pre has not even hiccuped once with these settings. I will let you know if it does.

Also part of your theory is incorrect. Compression uses CPU power yes, but it does not require much RAM to do it. Overclocked at 1ghz, I have CPU power to spare, and will gladly afford some of it in exchange for more theoretical RAM space.