Page 1 of 2 12 LastLast
Results 1 to 20 of 30
Like Tree6Likes
  1.    #1  
    Sprint FrankenPre2, 2.2.4

    The short story is that this phone isn't showing any UI errors, but it will not send or receive email or messages.

    Gory details follow:

    This phone (my wife's) was franked without deleting Palm Backup in August 2012 (~6months ago).
    It suffered from the Database Full errors a couple of months later in Oct. 2012 and I performed the partition resizing fix MojoDB Partition Resize - WebOS Internals which solved that problem.

    The new problem is serious and no doubt related, but in this case the UI is NOT showing any errors.
    The phone will not send or receive email OR text messages.
    Hitting send in email shows a sending notification but the compose window never goes away and the same message can be "sent" over and over again.
    Send in messaging app does nothing.
    Neither app shows an error.

    Following everything in Lumberjack shows error -3962 thrown for just about anything that wants to use a db...
    For example I tried to rename an email account and got this:
    error -3962
    (palm://com.palm.db/merge {"objects":[{"_id":"yada,"alias":"NewAccountName"}]}: db: quota exceeded )

    This error is also thrown when Lumberjack is launched:

    User.Err: mojodb-luna[1214]
    [core.messageService] db: quota exceeded (-3962) - sender='com.palm.eventreporter.WebAppMgr' method='put' payload='{"objects": [{"_kind":"com.palm.contextupload:1","appid":"org.webosinternals.lumberjack","event":"launch"}]}

    The databases are big from looking at /var/db/main/ with internalz:
    indexes.db = 60MB
    objects.db = 101MB

    Whereas on my FrankenPre2 built with the same meta-doctor I've got:
    indexes.db = 12.7MB
    objects.db = 8.78MB

    I use my phone a lot more, have several times the number of apps and send and receive a little more email.
    My phone is set for 7 days of email and the problem one was at 7 and now dropped to 3 to try to get around this issue to no avail.

    Checking the partitions with novaterm df -h I see:
    > Total "Size" of /dev/mapper/store-media.: 13.0G
    > The "Used" size of /dev/mapper/store-media: 1.3G
    > The "Size" of /dev/mapper/store-cryptodb: 493.9M (expanded in 2012)
    > The "Use%" of /dev/mapper/store-cryptodb: 36%

    So, I couldn't find this particular problem detailed in the forums and want to know if anyone has a recommended course of action.
    Is there a way to "clean up" the databases?
    Should a doctoring fix this?

    -Any help appreciated!
    ---bbito
    Last edited by bbito; 03/16/2013 at 07:17 PM. Reason: Typo - MOD: please fix title typo: database
  2. #2  
    You may possibly want to nuke certain large but unimportant types, and when the system starts working again export old items and remove them. At this point, though, chances are you can't do anything else in the system, so try not stuff things up more through trying to use the device.
  3.    #3  
    Quote Originally Posted by GMMan View Post
    You may possibly want to nuke certain large but unimportant types, ...
    Thanks for the response, the only thought I've had thus far on pruning the db is a method I read of using Impostah to clear the 'media.' kinds one by one - are there any other ways to unclog the databases?
  4. #4  
    Well, these type of things usually don't get that bad until this system-autonukes everything. I'm not familiar with BDB (the database system that backs MojoDB). Just a thought, maybe you could try to invoke the autocompact and see if it can clean stuff up? There is the chance of it just messing with the DB more, but it's worth a try.
  5. #5  
    You could try

    luna-send -a com.palm.configurator -f -n 1 palm://com.palm.db/purge '{}'

    to clear out deleted records, but it's probably only a short-term fix.
  6.    #6  
    Quote Originally Posted by greenoyster View Post
    You could try

    luna-send -a com.palm.configurator -f -n 1 palm://com.palm.db/purge '{}'
    Thanks for the suggestion, but I'm resorting to the meta-doctor because both your suggestion and my attempt to prune with Impostah just throw errorCode: -3962...
    It seems the main system db is completely locked up!
  7.    #7  
    Quote Originally Posted by bbito View Post
    The databases are big from looking at /var/db/main/ with internalz:
    indexes.db = 60MB
    objects.db = 101MB
    A trip to the Franken meta-doctor fixed the problem for now and also reset the partition expansion from last Oct.

    The databases are looking much more normal in size now:
    indexes.db = 8.98MB
    objects.db = 4.75MB

    Hoping for smooth sailing for a while since I don't plan on hopping off this WebOS ride until really forced to.
    Thanks to the community for the suggestions.

    I'm not sure how to ask for a MOD to fix my title typo, but maybe this line will do it.
  8. #8  
    There is a database config in /etc/palm/mojodb.conf that may need to be edited. I had played with it, but I don't know if my changes had any effect on the system...

    NOTE: It does not look like my current mojodb.conf has a modified cacheSize value....

    I have also noticed that mojodb needs to bug fixin'. It hangs (and goes to anywhere form 40%-70% CPU utilization) and during that time, very little works (viewing and sending text and emails do NOT). generally stoping and starting works, or does reseting the device.... But, your problem sounds different.

    I've also been told by a few that the database has been known to get corrupted (or something to that extent) and fill up rather quickly. Not sure if I've seen this.... But it is likely.
    Did you know:

    webOS ran on a Treo 800 during initial development.
  9.    #9  
    Quote Originally Posted by dkirker View Post
    I have also noticed that mojodb needs to bug fixin'. It hangs (and goes to anywhere form 40%-70% CPU utilization) and during that time, very little works (viewing and sending text and emails do NOT). generally stoping and starting works, or does reseting the device.... But, your problem sounds different.
    Thanks for chiming-in.
    I had initially thought it was related to going through the FrankenPre2 process without nuking the 1.4.5 palm backups, but it sounds like more than a few Pre3s are showing similar symptoms. Perhaps it is a WebOS 2.x issue as it seems like it hasn't been a problem for 1.x phones. It would be nice if we knew that there was a 2.x version that was immune and then the change could be tracked down, but I suppose its more likely that it is a problem across 2.x...

    I started a new thread because I hadn't seen error -3962 specified from the folks with similar symptoms on the Pre3 so I'm curious if the same error is showing up in Pre3 logs.

    I'm not sure what type of corruption it may have been as I'm not anywhere near a database expert, but from my poking around I can say that they were still fully explorable with Impostah. I didn't go through everything, but I touched on almost all the dbs available through Impostah and I didn't see any rows full of garbage...

    One thing that I didn't have the time to fully explore was that it 'seemed' like the database was holding email much older than what the email app was actually displaying. Now I wish I had really confirmed that, but I was pulled away before I had the chance.

    I wanted to document the issue as well as time permitted and hopefully this may be useful to anyone who is looking to explore the various "database full" / error 3962 problems in 2.x and I'll post again if that phone starts blowing up its databases again. I'm not going to grow the partition again on that phone unless it starts acting-up again and just see how it fairs with the stock 135MB.
  10. #10  
    I initially had the "application database is full" warning message on my Pre3 (AT&T) on 4/11:

    > The application database is full.
    > You must restart your device. This
    > will clear the database. After restart, sign in to
    > your webOS Account to restore backed up data.


    I did not do the restart ("Restart Now" button) as the above warning message recommended, but instead followed the following forum posts and wiki page to adjust the /media/internal and /var/db partition sizes (thank you for these posts!!!):

    The famous post #171 by jrwolff:
    "The application database is almost full"?

    Post #202 by LewisR:
    "The application database is almost full"?

    MojoDB Partition Resize - WebOS Internals


    Everything went well for the first day after resizing the partitions, but then I ran into the even more DREADED "db: quota exceeded" error message, which I did not find a solution to in the forums yet, except to do a full device reset, which I wanted to avoid. The "db: quota exceeded" error message caused the following problems on my Pre3:

    - Cellular data stopped working completely without any kind of error message (even toggling Airplane Mode and Phone > Prefs > Network > "Data Usage" various ways did not help restore cellular data), but Wi-Fi still worked.

    - Sending and receiving Text messages stopped working without any error message, but I could still read existing Texts.

    - Sending and receiving Email stopped working, but I could still read existing email. When I tried to save a Draft, then I got the "db: quota exceeded" error message.

    - Photos app stopped updating so it did not show any of my new photos and screen captures, but Internalz Pro could see them.

    - Saw repeated 'Failed with error "db: quota exceeded", code -3962' and "db: quota exceeded (-3962)" messages in my /var/log/messages with Muffle System Logging patch not installed.



    After reading the following threads on "quota exceeded", I did not see a solution except for a trip to the Doctor or full device reset, which, again, I wanted to leave as a last resort, especially since I still had access to the command line and file system:

    Email account quota exceeded and can't delete accounts:
    http://forums.webosnation.com/hp-pre...-accounts.html

    Databsae [Database] error -3962 "db: quota exceeded" (this thread):
    http://forums.webosnation.com/webos-...-exceeded.html

    Post #221 by FatalException on 04/10/2014 with the same quota exceeded problem I was seeing, but again, no solution except for a trip to the Doctor:
    "The application database is almost full"?



    I finally came across the following BUG report that gave me some hope of something new to try:

    [BUG]db8 errorText in reached the maximum size
    https://developer.palm.com/distribut...p?f=11&t=17136

    with the key section saying:
    > Each kind owner has a database quota.
    > If you look in /etc/palm/mojodb.conf,
    > you'll see that (as of webOS 3.0.2)
    > we set the standard quota as
    > 62,914,560 (60MB [or 60*1024*1024]).
    >
    > System kinds have a 225MB
    > [or 225*1024*1024=235,929,600] quota.
    > DB8 data is stored in /var/db,
    > a separate ext3 partition.


    My Pre3's /etc/palm/mojodb.conf file has the following Quotas:

    > "quotas" : [
    > {"owner":"*","size":20971520},
    > {"owner":"com.palm.*","size":78643200} <<<

    which is 20MB (=20,971,520/1024/1024) and 75MB (=78,643,200/1024/1024), respectively.


    My largest /var/db/main/ database sizes, when viewed in Internalz, were:
    - objects.db = 74.6MB
    - indexes.db = 23.5MB

    so I can see that I am reaching that 75MB quota file size limit, which is probably causing those "db: quota exceeded" error messages.


    On 4/17, I increased that 78643200 (=75MB) quota value in /etc/palm/mojodb.conf to 235929600 (=225MB) using Internalz Pro to see if that would help (earlier I had increased my /var/db partition size from 135MB to 512MB, so there should be enough room to handle this larger quota limit) and then restarted by Pre3.


    Like magic, everything started working again.
    I first saw my Cellular Data signal come back, then I checked out the Photos app and all my new photos and screenshots showed up. I tested Email (set it to keep 3 days worth instead of my previous 1 month setting) and my new email showed up. Sending email and texts worked too. But the most important thing of all, no more "db: quota exceeded" messages in /var/log/messages.


    I hope my Pre3 that I really like and have only been using for 6 months keeps working.


    Now I'll have to keep an eye out on the file size of /var/db/main/objects.db and try to find out what is causing it to grow. It has already grown from 74.6MB to 76.0MB (increase of 1.4MB) in about 5 hours after doing the above and I don't even get that many texts/emails per day (maybe a dozen or so).

    I wonder if there is a way to see what is taking up the space in objects.db and trim it down somehow. Too bad Impostah does not show a database's size (or am I missing seeing this option?). Any ideas? Is there Mac app that can read the objects.db file? More stuff to research...


    Update 1 on 4/22:
    5 days after doing the above, everything is still working well on my Pre3. objects.db is down to 74.1MB in size, so that database can shrink in size, which is good.

    Update 2 on 4/26:
    Its almost been 10 days now and everything is still working well on my Pre3. objects.db is now down to 62.7MB in size and my indexes.db is down to 19.9 MB. I did not make any changes or run any purge commands myself, so it is interesting that these database can get smaller on their own, which, again, is good.
    Last edited by UI Designer; 07/23/2014 at 12:03 AM. Reason: Added "Update 2", Fixed some Links, Added some Dates, Added Highlighting
    gizmo21, ananimus and toto-w like this.
  11. #11  
    You can get stats on db usage with the following terminal commands:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/stats '{}'

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/quotaStats '{}'

    Another thing that may help in some circumstances is the purge command:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/purge '{}'

    Background: When database records are "deleted", they are not immediately removed from the database; instead it adds a _del: true property which makes the record invisible for normal queries. This allows sync services to sync the deletion to the server.

    However, this means that even data that has been "deleted" takes up space; adding and removing an account repeatedly can rapidly deplete the database disk space if the account has a lot of data. Email is the only service that properly purges data on account removal. Database purging will permanently remove these "deleted" records; normally it only gets run every 7 days (purgeWindow in mojodb.conf) but the purge command will force it to happen immediately. This number should not be set too low, or else deletions from the device may not be properly uploaded to the server (e.g. if network access is not available).

    It's also worth noting that increasing the database quota too closely to the /var/db partition size runs the risk of critically running out of disk space: even purging records requires some temporary disk space to carry out (for journaling). If the database gets into this state, it will automatically perform a hard reset (device wipe) rather than just returning database errors.

    While I'm at it, you can also dump a JSON backup copy of the full database contents using this command (may be slow):

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/dump '{"path": "/media/internal/db-backup.json"}'
    Last edited by jl85; 04/23/2014 at 06:09 AM. Reason: Typo -- thanks gizmo21
  12. #12  
    Quote Originally Posted by UI Designer View Post
    I initially had the "application database is full" warning message on my Pre3 (AT&T) on 4/11:

    > The application database is full.
    > You must restart your device. This
    > will clear the database. After restart, sign in to
    > your webOS Account to restore backed up data.




    I finally came across the following BUG report that gave me some hope of something new to try:

    [BUG]db8 errorText in reached the maximum size
    https://developer.palm.com/distribut...p?f=11&t=17136

    with the key section saying:
    > Each kind owner has a database quota.
    > If you look in /etc/palm/mojodb.conf,
    > you'll see that (as of webOS 3.0.2)
    > we set the standard quota as
    > 62,914,560 (60MB [or 60*1024*1024]).
    >
    > System kinds have a 225MB
    > [or 225*1024*1024=235,929,600] quota.
    > DB8 data is stored in /var/db,
    > a separate ext3 partition.


    My Pre3's /etc/palm/mojodb.conf file has the following Quotas:

    > "quotas" : [
    > {"owner":"*","size":20971520},
    > {"owner":"com.palm.*","size":78643200} <<<

    which is 20MB (=20,971,520/1024/1024) and 75MB (=78,643,200/1024/1024), respectively.


    My largest /var/db/main/ database sizes, when viewed in Internalz, were:
    - objects.db = 74.6MB
    - indexes.db = 23.5MB

    so I can see that I am reaching that 75MB quota file size limit, which is probably causing those "db: quota exceeded" error messages.


    I increased that 78643200 (=75MB) quota value in /etc/palm/mojodb.conf to 235929600 (=225MB) using InternalZ to see if that would help (earlier I had increased my /var/db partition size from 135MB to 512MB, so there should be enough room to handle this larger quota limit) and then restarted by Pre3.


    Like magic, everything started working again.
    I first saw my Cellular Data signal come back, then I checked out the Photos app and all my new photos and screenshots showed up. I tested Email (set it to keep 3 days worth instead of my previous 1 months setting) and my new email showed up. Sending email and texts worked too.


    I hope my Pre3 that I really like and have only been using for 6 months keeps working.


    Now I'll have to keep an eye out on the file size of /var/db/main/objects.db and try to find out what is causing it to grow. It has already grown from 74.6MB to 76.0MB (increase of 1.4MB) in about 5 hours after doing the above and I don't even get that many texts/emails per day (maybe a dozen or so).

    I wonder if there is a way to see what is taking up the space in objects.db and trim it down somehow. Too bad Impostah does not show a database's size (or am I missing seeing this option?). Any ideas? Is there Mac app that can read the objects.db file? More stuff to research...


    Update:
    5 days after doing the above, everything is still working well on my Pre3. objects.db is down to 74.1MB in size, so that database can shrink in size, which is good.
    That's a great find !!

    There are several posts in the german forum all having same or similar db full problems and all standard stuff like partition resizing/cleaning did not work.

    Will post it there an see if that will help them.

    Thx again for testing and posting!
  13. #13  
    Quote Originally Posted by jl85 View Post
    You can get stats on db usage with the following terminal commands:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/stats '{}'

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/quotaStats '{}'

    Another thing that may help in some circumstances is the purge command:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/purge '{}'

    Background: When database records are "deleted", they are not immediately removed from the database; instead it adds a _del: true property which makes the record invisible for normal queries. This allows sync services to sync the deletion to the server.

    However, this means that even data that has been "deleted" takes up space; adding and removing an account repeatedly can rapidly deplete the database disk space if the account has a lot of data. Email is the only service that properly purges data on account removal. Database purging will permanently remove these "deleted" records; normally it only gets run every 7 days (purgeWindow in mojodb.conf) but the purge command will force it to happen immediately. This number should not be set too low, or else deletions from the device may not be properly uploaded to the server (e.g. if network access is not available).

    It's also worth noting that increasing the database quota too closely to the /var/db partition size runs the risk of critically running out of disk space: even purging records requires some temporary disk space to carry out (for journaling). If the database gets into this state, it will automatically perform a hard reset (device wipe) rather than just returning database errors.

    While I'm at it, you can also dump a JSON backup copy of the full database contents using this command (may be slow):

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/dump '{"path": "/media/internal/db-backup.json"}'
    please correct typo in your commands to configurator

    perhaps it's also usefull to redirect output to a file:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/stats '{}' > /media/internal/palmdbstats.txt
  14. Barebuns's Avatar
    Posts
    86 Posts
    Global Posts
    87 Global Posts
    #14  
    Quote Originally Posted by UI Designer View Post
    I initially had the "application database is full" warning message on my Pre3 (AT&T) on 4/11:

    > The application database is full.
    > You must restart your device. This
    > will clear the database. After restart, sign in to
    > your webOS Account to restore backed up data.

    My Pre3's /etc/palm/mojodb.conf file has the following Quotas:

    > "quotas" : [
    > {"owner":"*","size":20971520},
    > {"owner":"com.palm.*","size":78643200} <<<

    which is 20MB (=20,971,520/1024/1024) and 75MB (=78,643,200/1024/1024), respectively.


    My largest /var/db/main/ database sizes, when viewed in Internalz, were:
    - objects.db = 74.6MB
    - indexes.db = 23.5MB

    so I can see that I am reaching that 75MB quota file size limit, which is probably causing those "db: quota exceeded" error messages.


    I increased that 78643200 (=75MB) quota value in /etc/palm/mojodb.conf to 235929600 (=225MB) using InternalZ to see if that would help (earlier I had increased my /var/db partition size from 135MB to 512MB, so there should be enough room to handle this larger quota limit) and then restarted by Pre3.
    Awesome! Thank You!
  15. #15  
    Quote Originally Posted by UI Designer View Post
    I initially had the "application database is full" warning message on my Pre3 (AT&T) on 4/11:

    > The application database is full.
    > You must restart your device. This
    > will clear the database. After restart, sign in to
    > your webOS Account to restore backed up data.


    I did not do the restart ("Restart Now" button) as the above warning message recommended, but instead followed the following forum posts and wiki page to adjust the /media/internal and /var/db partition sizes (thank you for these posts!!!):

    The famous post #171:
    "The application database is almost full"?

    Post #202:
    "The application database is almost full"?

    MojoDB Partition Resize - WebOS Internals


    Everything went well for the first day after resizing the partitions, but then I ran into the even more DREADED "db: quota exceeded" error message, which I did not find a solution to in the forums yet, except to do a full device reset, which I wanted to avoid. The "db: quota exceeded" error message caused the following problems on my Pre3:

    - Cellular data stopped working completely without any kind of error message (even toggling Airplane Mode and Phone > Prefs > Network > "Data Usage" various ways did not help restore cellular data), but Wi-Fi still worked.

    - Sending and receiving Text messages stopped working without any error message, but I could still read existing Texts.

    - Sending and receiving Email stopped working, but I could still read existing email. When I tried to save a Draft, then I got the "db: quota exceeded" error message.

    - Photos app stopped updating so it did not show any of my new photos and screen captures, but Internalz Pro could see them.

    - Saw repeated 'Failed with error "db: quota exceeded", code -3962' and "db: quota exceeded (-3962)" messages in my /var/log/messages with Muffle System Logging patch not installed.



    After reading the following threads on "quota exceeded", I did not see a solution except for a trip to the Doctor or full device reset, which, again, I wanted to leave as a last resort, especially since I still had access to the command line and file system:

    Email account quota exceeded and can't delete accounts:
    http://forums.webosnation.com/hp-pre...-accounts.html

    Databsae [Database] error -3962 "db: quota exceeded" (this thread):
    http://forums.webosnation.com/webos-...-exceeded.html

    Post #221 by FatalException on 04/10/2014 with the same quota exceeded problem I was seeing, but again, no solution except for a trip to the Doctor:
    "The application database is almost full"?



    I finally came across the following BUG report that gave me some hope of something new to try:

    [BUG]db8 errorText in reached the maximum size
    https://developer.palm.com/distribut...p?f=11&t=17136

    with the key section saying:
    > Each kind owner has a database quota.
    > If you look in /etc/palm/mojodb.conf,
    > you'll see that (as of webOS 3.0.2)
    > we set the standard quota as
    > 62,914,560 (60MB [or 60*1024*1024]).
    >
    > System kinds have a 225MB
    > [or 225*1024*1024=235,929,600] quota.
    > DB8 data is stored in /var/db,
    > a separate ext3 partition.


    My Pre3's /etc/palm/mojodb.conf file has the following Quotas:

    > "quotas" : [
    > {"owner":"*","size":20971520},
    > {"owner":"com.palm.*","size":78643200} <<<

    which is 20MB (=20,971,520/1024/1024) and 75MB (=78,643,200/1024/1024), respectively.


    My largest /var/db/main/ database sizes, when viewed in Internalz, were:
    - objects.db = 74.6MB
    - indexes.db = 23.5MB

    so I can see that I am reaching that 75MB quota file size limit, which is probably causing those "db: quota exceeded" error messages.


    I increased that 78643200 (=75MB) quota value in /etc/palm/mojodb.conf to 235929600 (=225MB) using InternalZ to see if that would help (earlier I had increased my /var/db partition size from 135MB to 512MB, so there should be enough room to handle this larger quota limit) and then restarted by Pre3.


    Like magic, everything started working again.
    I first saw my Cellular Data signal come back, then I checked out the Photos app and all my new photos and screenshots showed up. I tested Email (set it to keep 3 days worth instead of my previous 1 months setting) and my new email showed up. Sending email and texts worked too.


    I hope my Pre3 that I really like and have only been using for 6 months keeps working.


    Now I'll have to keep an eye out on the file size of /var/db/main/objects.db and try to find out what is causing it to grow. It has already grown from 74.6MB to 76.0MB (increase of 1.4MB) in about 5 hours after doing the above and I don't even get that many texts/emails per day (maybe a dozen or so).

    I wonder if there is a way to see what is taking up the space in objects.db and trim it down somehow. Too bad Impostah does not show a database's size (or am I missing seeing this option?). Any ideas? Is there Mac app that can read the objects.db file? More stuff to research...


    Update:
    5 days after doing the above, everything is still working well on my Pre3. objects.db is down to 74.1MB in size, so that database can shrink in size, which is good.
    Thank you for the details! Yeah, I suspect this is what I did when I was originally mucking with things. I will update the webOS-Internals article with this!

    EDIT: And it seems like I never actually made the changes that I had thought...... hmmmmm, I wonder what impact this has had?
    Last edited by dkirker; 04/24/2014 at 10:40 PM.
    Did you know:

    webOS ran on a Treo 800 during initial development.
  16. #16  
    Quote Originally Posted by jl85 View Post
    You can get stats on db usage with the following terminal commands:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/quotaStats '{}'

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/stats '{}'

    Quote Originally Posted by gizmo21 View Post
    ...perhaps it's also useful to redirect output to a file:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/stats '{}' > /media/internal/palmdbstats.txt


    Another thing that may help in some circumstances is the purge command:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/purge '{}'

    Background: When database records are "deleted", they are not immediately removed from the database; instead it adds a _del: true property which makes the record invisible for normal queries. This allows sync services to sync the deletion to the server.

    However, this means that even data that has been "deleted" takes up space; adding and removing an account repeatedly can rapidly deplete the database disk space if the account has a lot of data. Email is the only service that properly purges data on account removal.

    Database purging will permanently remove these "deleted" records; normally it only gets run every 7 days (purgeWindow in mojodb.conf) but the purge command will force it to happen immediately. This number should not be set too low, or else deletions from the device may not be properly uploaded to the server (e.g. if network access is not available).

    It's also worth noting that increasing the database quota too closely to the /var/db partition size runs the risk of critically running out of disk space: even purging records requires some temporary disk space to carry out (for journaling). If the database gets into this state, it will automatically perform a hard reset (device wipe) rather than just returning database errors.



    While I'm at it, you can also dump a JSON backup copy of the full database contents using this command (may be slow):

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/dump '{"path": "/media/internal/db-backup.json"}'


    Thank you, thank you, thank you, jl85!!! I have not seen these commands documented anywhere else before, especially the last one that dumps a JSON backup copy of my database contents -- I find this command to be the most useful of them all!!! I have been wanting to do something like this ever since upgrading from a Pre+ to a Pre3, especially since I cannot find an app for my Mac that can read the /var/db/main/objects.db db8 file like "SQLite Database Browser 2.0 b1" could do with my Pre+'s DB3 database files!!! Thank you, thank you, thank you, again!

    So after running all of jl85's recommended commands on 4/23 except for purge since I do not want to have anything purged quite yet until I figure out what is going on, I still do not fully understand why my objects.db is 74.1MB in size (except for not having been purged). I'll explain my findings below in more detail followed by my list of questions.



    Here are the results for each of the commands jl85 listed:



    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/quotaStats '{}'
    {
    "returnValue": true,
    "results": {
    "*": {
    "size": 20971520, < < < = 20.0MB
    "used": 1702486 < < < < = 1.4MB
    },
    "com.palm.*": {
    "size": 235929600, < < < = 225.0MB
    "used": 80601756 < < < < = 76.9MB
    }
    }
    }

    (Note: "< < < =" with MB values after them added by me to make the numbers easier to read)

    So it looks like I have enough space, which is good!



    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/stats '{}'

    This command took around 4 minutes to complete while the others only took a few seconds.

    There were 3,840 lines of output with 748 of them having a "size" value in them. I extracted the lines with size values along with their line number, imported them into a spreadsheet, sorted them by size, added them up, and looked more closely at the stats' with the largest "size" values.

    The "size" values add up to: 18.6 MB

    This is much smaller than the size I was expecting:
    - 74.1MB (size of object.db file),
    - 76.9MB ("used" valued returned from quotaStats), or
    - 98MB (file size of object.db plus indexes.db)

    How much overhead is there in the db8 database???
    I was not expecting an exact match, but something closer to the above values.

    Here are the sections from the above stats command that have the largest "size" values and account for most (14.6MB) of that 18.6MB:

    Near the top of the stats output (around line 40) is:

    "Object:1": {
    "indexes": {
    "_id": {
    "count": 177939,
    "size": 3774464 < < < < < 3.6MB
    },
    "_sync_revincdel": {
    "count": 177939,
    "size": 4840244 < < < < < 4.6MB
    }
    },
    "objects": {
    "count": 8226,
    "delCount": 169713,
    "delSize": 61815843, < < < < < 58.6MB <<<<<
    "size": 1925313 < < < < < < < < 1.8MB
    }
    }

    Sum of the "size" values for Object:1:
    3.6 + 4.6 + 1.8 = 10.0MB

    Sum with "delSize" included:
    10.0 + 58.6 = 68.6MB,
    which is getting pretty close to my object.db file size of 74.1MB.

    Does anyone know what "Object:1" is?
    I do not see an Object:1 db in Impostah's Databases list.
    Is it a summary of my entire database size?


    Further down around line 335 I see:

    "com.palm.activity:1": {
    "indexes": {
    "_id": {
    "count": 160578,
    "size": 3372138 < < < < 3.2MB
    }
    },
    "objects": {
    "count": 31,
    "delCount": 160547,
    "delSize": 60588959, < < < < 57.8MB <<<<
    "size": 9734 < < < < < < < 0.009MB (or 9.5KB)
    }
    }


    com.palm.smsmessage:1 is using 1.4 MB when I add up all of its "size" values.


    So now we are up to:
    10.0 + 3.2 + 1.4 = 14.6MB


    All the rest of the "size" values are less than 200KB with:
    - 428 out of the 748 size values having a size value of 0,
    - 152 having a size value of 1KB (1024B) or less,
    - 148 having a size value between 1KB to 100KB,
    - 15 having a size value between 100KB and 200KB,
    and these add up to around 4MB.


    Besides the "size" values, I also see 15 "delSize" values -- I did the same thing with these values as I did with the "size" values and they add up to: 118.1 MB

    (Note: this includes the 57.8MB delSize from Object:1 -- if Object:1 is a summary value, then if I subtract it out, I get 60.3MB, which is larger than the the Object:1 delSize value, so I am not sure if Object:1 is a summary value, unless it does not include everything -- do you know?)

    Here are the sections from the above stats command that have the largest "delSize" values and account for most (117.6MB) of that 118.1MB:
    - "Object:1", as seen above, at 58.6MB
    - com.palm.activity:1, as seen above, at 57.8MB
    - org.webosinternals.preware.justType:1 at 0.7MB
    - com.palm.email:1 at 0.2MB
    - com.palm.imap.email:1 at 0.2MB
    - com.palm.media.image.file:1 at 0.1MB
    - com.palm.browserhistory:1 at 0.04MB
    The rest (8 of them) are 10KB or less.


    The summed "delSize" value of 118.1 MB is larger than I was expecting and does not match the 58.3MB difference between:
    - quotaStats' 76.9MB "used" value and
    - summation of stats' "size" values calculated by me via Excel to be 18.6MB,
    I was looking for, but the "Object:1" delSize of 58.6MB is getting really close. So is that 60.3MB delSize sum when excluding Object:1 from the delSize grand total.



    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/dump '{"path": "/media/internal/db-backup.json"}'

    The size of the db-backup.json file is only 4.1MB,
    again, much smaller than the 74MB to 98MB range I was expecting and is even smaller than the 18.6 MB "size" values I added up from the stats command (I did not calculated the sum of the sizes for only the "objects" parts of stats -- do I need to do this for it to match?).

    I have not had a chance yet to make sure the db-backup.json dump file contains everything I am expecting, but it is over 8,000 lines long, I do not see any obvious problems or garbage in there yet while scanning through it, and it does contain my latest text messages, which is good. I plan to look at this file more when I have a chance.

    I do not see any "_del: true" properties in my dump file, like jl85 mentioned, so maybe there is nothing in there that needs purging after all or maybe it was not included in the dump. I do see some "_del":false values, though.



    Can anyone help me interpret the above results better so that the numbers add up more closely? I would like to understand why my objects.db file size is 74.1MB while stats' size values add up to much lower 18.6MB value, delSize values add up to a much higher 118.1 MB value, and the db-backup.json dump file is an even smaller 4.1MB in size.
    Am I doing something wrong or missing something?
    Are there any other commands I should try besides purge?

    Is there an option to the ...luna://com.palm.db/dump command or a similar command to dump EVERYTHING, even items marked for deletion (_del: true), since I am not sure if dump is really giving me everything.



    Quote Originally Posted by jl85 View Post
    Another thing that may help in some circumstances is the purge command:

    ...

    Database purging will permanently remove these "deleted" records; normally it only gets run every 7 days (purgeWindow in mojodb.conf) but the purge command will force it to happen immediately. This number should not be set too low, or else deletions from the device may not be properly uploaded to the server (e.g. if network access is not available).

    I do not see any "purgeWindow" or any similar purge related command in my /etc/palm/mojodb.conf file.
    Is purgeWindow in some other file or location on a Pre3?
    Since purgeWindow is missing in my mojodb.conf, could this cause my objects.db file to grow larger in size than it should be?
    Is purgeWindow in your Pre3's /etc/palm/mojodb.conf file?



    I will try running that ...luna://com.palm.db/purge command at some point to see if the databases shrink down to the values shown by stats, but I want to wait a little bit longer until I get some feedback on this post or need more space.



    So, in summary, here are my questions:

    1. Does anyone know what the "Object:1" in stats is?
      I do not see an Object:1 db in Impostah's Databases section.
      Is it a summary of my entire database size?

    2. Can anyone help me interpret the above results better so that the numbers add up more closely? I would like to understand why my objects.db file size is 74.1MB while stats' size values add up to much lower 18.6MB value, delSize values add up to a much higher 118.1 MB value, and the db-backup.json dump file is an even smaller 4.1MB in size.
      Am I doing something wrong or missing something?
      Are there any other commands I should try besides purge?

    3. Is there an option to the ...luna://com.palm.db/dump command or a similar command to dump EVERYTHING, even items marked for deletion (_del: true), since I am not sure if dump is really giving me everything.

    4. I do not see any "purgeWindow" or any similar purge related command in my /etc/palm/mojodb.conf file.
      Is purgeWindow in some other file or location on a Pre3?
      Did something change in one of the webOS releases (I am running 2.2.4), is the Doctor file for an AT&T Pre3 messed up, or could a patch have removed this?
      Since purgeWindow is missing in my mojodb.conf, could this cause my objects.db file to grow larger in size than it should be?
      Is purgeWindow in your Pre3's /etc/palm/mojodb.conf file?



    Thank you all again for your help, especially jl85! (and jrwolff, dkirker with the app database full warning message and gizmo21 for his helpful tips).
  17. #17  
    Quote Originally Posted by UI Designer View Post
    [B]

    So, in summary, here are my questions:

    1. Does anyone know what the "Object:1" in stats is?
      I do not see an Object:1 db in Impostah's Databases section.
      Is it a summary of my entire database size?

    2. Can anyone help me interpret the above results better so that the numbers add up more closely? I would like to understand why my objects.db file size is 74.1MB while stats' size values add up to much lower 18.6MB value, delSize values add up to a much higher 118.1 MB value, and the db-backup.json dump file is an even smaller 4.1MB in size.
      Am I doing something wrong or missing something?
      Are there any other commands I should try besides purge?

    3. Is there an option to the ...luna://com.palm.db/dump command or a similar command to dump EVERYTHING, even items marked for deletion (_del: true), since I am not sure if dump is really giving me everything.

    4. I do not see any "purgeWindow" or any similar purge related command in my /etc/palm/mojodb.conf file.
      Is purgeWindow in some other file or location on a Pre3?
      Did something change in one of the webOS releases (I am running 2.2.4), is the Doctor file for an AT&T Pre3 messed up, or could a patch have removed this?
      Since purgeWindow is missing in my mojodb.conf, could this cause my objects.db file to grow larger in size than it should be?
      Is purgeWindow in your Pre3's /etc/palm/mojodb.conf file?


    Thank you all again for your help, especially jl85! (and jrwolff, dkirker with the app database full warning message and gizmo21 for his helpful tips).

    no.4
    of my TP:
    Code:
    "db" : {
    		"permissions" : [
    			{"type":"db.role","object":"admin","caller":"com.palm.configurator","operations":{"*":"allow"}},
    			{"type":"db.role","object":"admin","caller":"com.palm.service.backup","operations":{"*":"allow"}},
    			{"type":"db.role","object":"admin","caller":"com.palm.odd.service","operations":{"*":"allow"}},
    			{"type":"db.role","object":"admin","caller":"com.palm.service.migrationscript","operations":{"*":"allow"}},
    			{"type":"db.role","object":"admin","caller":"com.palm.spacecadet","operations":{"*":"allow"}}
    		],
    		"quotas" : [
    			{"owner":"*","size":62914560}			{"owner":"com.palm.*","size":235929600}
    		],
    		"loadStepSize" : 173,
    		"purgeWindow": 7,
    	}
    so here it is 7 days on TP 3.0.5

    But that line is also not available in that conf 2.2.4 on my Pre2 is it elsewhere?

    Perhaps thats the reason so many phone devices have DB full and profile reloading right now?
  18. #18  
    Quote Originally Posted by UI Designer View Post
    I do not see any "purgeWindow" or any similar purge related command in my /etc/palm/mojodb.conf file.
    Is purgeWindow in some other file or location on a Pre3?
    Since purgeWindow is missing in my mojodb.conf, could this cause my objects.db file to grow larger in size than it should be?
    Is purgeWindow in your Pre3's /etc/palm/mojodb.conf file?

    Quote Originally Posted by gizmo21 View Post
    But that line is also not available in that conf 2.2.4 on my Pre2 is it elsewhere?

    Perhaps thats the reason so many phone devices have DB full and profile reloading right now?
    See /etc/palm/activities/com.palm.db/com.palm.db.purge.json

    {
    "activity" : {
    "name" : "mojodbpurge",
    "description" : "Scheduled mojodb purge activity",
    "schedule" : {
    "interval" : "24h"
    },
    "callback" : {
    "method" : "palm://com.palm.db/internal/scheduledPurge"
    },
    "type" : {
    "background" : true,
    "persist" : true,
    "power" : true
    }
    },
    "replace" : true,
    "start" : true
    }
    I set mine to 24h
    Did you know:

    webOS ran on a Treo 800 during initial development.
  19. #19  
    This is fascinating stuff! Going to have to try this stuff out.
  20. #20  
    hhm this one is also set to 24h on my TP and Pre2, but what about that 7 day window from the other file on phones?
Page 1 of 2 12 LastLast

Similar Threads

  1. Replies: 2
    Last Post: 02/04/2013, 09:26 PM
  2. Tweaks:Unknown Service Error & Preware database error
    By jblather in forum webOS Patches
    Replies: 23
    Last Post: 09/04/2012, 03:54 AM
  3. PreCentral Database Error?
    By Garrett92C in forum webOSNation.com - Site News, Feedback & Help
    Replies: 24
    Last Post: 04/09/2010, 09:24 PM
  4. Error:Database: can't find (0x0207)
    By Varangian in forum Palm OS Devices & Apps
    Replies: 4
    Last Post: 10/07/2004, 12:35 AM

Tags for this Thread

Posting Permissions