Page 2 of 2 FirstFirst 12
Results 21 to 30 of 30
Like Tree6Likes
  1. #21  
    Quote Originally Posted by UI Designer View Post
    Is there an option to the ...luna://com.palm.db/dump command or a similar command to dump EVERYTHING, even items marked for deletion (_del: true), since I am not sure if dump is really giving me everything.
    Add incDel: true to the dump parameters. This also works for regular db8 "find" queries.

    Quote Originally Posted by UI Designer View Post
    Does anyone know what the "Object:1" in stats is?
    I do not see an Object:1 db in Impostah's Databases section.
    Is it a summary of my entire database size?
    db8 uses a hierarchy of objects. For example com.palm.imap.email is a subkind of com.palm.email which is a subkind of Object:1 (all kinds inherit from Object:1). I don't know if the sizes are double-counted or not.
  2. #22  
    I have been noticing lately that my Pre3's object.db and indexes.db files in /var/db/main/ are climbing in size, but I do NOT know why.

    Could this possibly be caused by Google's constant login requests that I've been getting lately? [Updated 7/21: I think so -- see below!] Hopefully this is fixed by the Goggle sync https fix gsync-1.2.patch that I finally had a chance earlier today to update from v1.1 to 1.2 (I would have updated this sooner if it would have showed up in Preware).

    I have not been using my Pre3 any differently over the past few months since my last posts above -- just a few phone calls and texts per day, listen to 1 podcast per day via GuttenPodder, read a few web pages, and rarely use the Email app since I have other ways to access my email.


    Here is some history of my Pre3's object.db and indexes.db file sizes:

    On 4/21/2014 (resized partitions to fix db quota exceeded -- see post #10):
    - objects.db: 74.6MB
    - indexes.db: 23.5MB
    - (Combined = 98.1MB)

    On 4/26/2014 (sizes came down on their own):
    - object.db: 62.7MB
    - indexes.db: 19.9MB
    - (Combined = 82.6MB)

    On 5/20/2014:
    - object.db: 64.0MB
    - indexes.db: 20.8MB
    - (Combined = 84.6MB)

    On 6/24/2014 (almost a month ago):
    - object.db: 64.3MB
    - indexes.db: 22.4MB
    - (Combined = 86.6MB)

    On 7/14/2014 (4 days before this forum post):
    - object.db: 88.1MB
    - indexes.db: 28.8MB
    - (Combined = 116.9MB)

    On 7/18/2014 (date of this forum post at 4:28pm):
    - object.db: 104MB
    - indexes.db: 33.5MB
    - (Combined = 137.5MB)

    On 7/18/2014 (date of this forum post, but later at 5:50pm, 9pm, 10pm, 11pm):
    - object.db: 105MB
    - indexes.db: 33.7MB
    - (Combined = 138.7MB)

    On 7/21/2014 (3 days after this post):
    - object.db: 105MB
    - indexes.db: 33.8MB
    - (Combined = 138.8MB)

    On 7/22/2014 (4 days after this post and half a day after getting purge working in post #26):
    - object.db: 29.9MB
    - indexes.db: 14.8MB
    - (Combined = 44.7MB)



    You can see how these files have really increased in size in the last 4 days (7/18 back to 7/14) -- over 20MB! And in the last month (7/18 back to 6/24), they have increased by over 50MB!! I am concerned since in the months of May and June, these files stayed a fairly consistent size and only increased by about 2MB per month.

    So I decided today to finally run that purge command that jl85 pointed out in post #11 on 4/22/2014 to see if that would help shrink them back down in size. I ran the following command in WebOS Quick Install's Linux Commandline window:

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/purge '{}'

    After waiting about 3 minutes, I got the following error response (this is re-creatable):

    "errorCode": 12,
    "errorText": "bdb: db->get - Cannot allocate memory",
    "returnValue": false



    I think I have enough memory -- here are the results of quotaStats:

    Code:
    root@Pre3:/# luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/quotaStats '{}'
    {
        "returnValue": true,
        "results": {
            "*": {
                "size": 20971520, < < < = 20.0MB
                "used": 2789343 < < < < =  2.6MB
            },
            "com.palm.*": {
                "size": 235929600, < < < = 225.0MB
                "used": 92493143 < < < < = 88.21MB
            }
        }
    }
    (Note: "< < < =" with MB values after them added by me to make the numbers easier to read)


    Furthermore, df -h shows that none of the file systems are more than 75% full and most are less than 50% full.
    /var/db is using 156.3MB out of 493.9MB (32% full).


    When I do a dump of the database using:
    Code:
    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/dump '{"path": "/media/internal/db-backup.json"}'
    the size of the db-backup.json file is only 4.9MB. Back in April when I did this, it was 4.1MB, so that is about a 20% increase, which is small when compared to object.db's and indexes.db's increases of almost 70% each.


    Quote Originally Posted by jl85 View Post
    Add incDel: true to the dump parameters [to dump EVERYTHING, even items marked for deletion (_del: true),]. This also works for regular db8 "find" queries.
    How exactly do I add incDel: true to the dump parameters?
    What separator character is used between multiple parameters? [Update 7/21: it's a comma -- thank you jl85! (I should have guessed this)]
    Can you show me an example?

    (Also, as a side note, can anyone recommend any good Mac apps for viewing this dumped json file instead of just a text editor? I've tried VisualJSON, but it cannot parse this file correctly.)


    Anybody have any ideas what is going wrong with purge?
    What should I try instead to decrease the size of object.db and indexes.db, or at least stop their size from increasing even more?

    Thank you for your help!




    UPDATE on 7/18/2014 around 9pm:


    I started digging through /var/log/messages and came across the following entries that jumped out at me:


    2014-07-18T23:00:13.189758Z [1818] webos-device user.notice mojodb-luna[1167]: [db.mojodb] purging objects deleted more than 14 days ago...

    2014-07-18T23:01:46.259887Z [1911] webos-device user.err mojodb-luna[1167]: [db.bdb] bdb: Lock table is out of available lock entries

    2014-07-18T23:01:46.260864Z [1911] webos-device user.warning mojodb-luna[1167]: [db.bdb] bdb: transaction aborted

    2014-07-18T23:02:12.410888Z [1937] webos-device user.err mojodb-luna[1167]: [core.messageService] bdb: db->get - Cannot allocate memory (12) - sender='com.palm.configurator' method='purge' payload='{}'


    It kind of looks like to me that mojodb got purged, but maybe not bdb due to the "Lock table is out of available lock entries" error. Where is the lock table stored and is there any way to increase it?

    Does anyone know how the mojodb and bdb databases relate to the the object.db and indexes.db files, and what all this means? From GMMan's post #4 above, it sounds like BDB is a backup copy of MojoDB.

    Any other ideas of what I should look at or try in order to stop object.db and indexes.db from increasing in file size even further?

    Or does anyone think this will clean itself up in a week or so now that I have the updated (v1.2) Google Sync https fix patch installed that helped fix:
    - constant login requests
    - Memory critical, too many cards! (TMC) notifications
    - and who knows what else...




    UPDATE on 7/21/2014 around 3pm (3 days after my original post):


    Some good news -- my Pre3's object.db and indexes.db files in /var/db/main/ stopped increasing in size and are very close to where they where on 7/18 (3 days ago) after updating the Google Sync https fix patch from v1.1 to v1.2 on 7/18. Thank you Grabber5.0 and frantid for this patch!!!

    Too bad this updated gsync-1.2.patch did not show up in Preware where I would have been able to install it sooner and hopefully avoided this 50MB increase in my Pre3's object.db and indexes.db files. Here are their current sizes when viewed in Internalz Pro:

    - object.db: 105MB
    - indexes.db: 33.8MB
    - (Combined = 138.8MB)

    I have not had a chance to try jl85's great suggestion below on increasing maxLocks in /etc/palm/mojodb.conf yet in order to get purge working, but plan to later today. Thank you jl85 for all of your wonderful help and tips!




    UPDATE on 7/22/2014 around 3pm (4 days after my original post):


    Good news! Problem Solved!
    Increasing maxLocks from 6000 to 20000 in /etc/palm/mojodb.conf and rebooting fixed the purge command so that it now finishes (see post #26)!

    My Pre3's object.db and indexes.db in /var/db/main/ finally DECREASED in size (see post #28 below) about half of a day AFTER issuing the purge command in post #26! Their current size values are:

    - object.db: 29.9MB
    - indexes.db: 14.8MB
    - (Combined = 44.7MB)

    Thank you, jl85, for all of your help in keeping my Pre3 alive and working well!
    Last edited by UI Designer; 07/23/2014 at 03:26 PM. Reason: Add more details, make more clear, fix typos, add formating, add links, add updates, add highlighting
  3. #23  
    MojoDB uses BDB internally. All the MojoDB data is stored in the BDB files.

    Full dump command including deleted objects:

    Code:
    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/dump '{"path": "/media/internal/db-backup.json", "incDel": true}'
    Not sure what the BDB lock errors mean. Maybe there's too much stuff to purge and it doesn't think it has enough scratch space on the partition.
  4. #24  
    Quote Originally Posted by jl85 View Post
    MojoDB uses BDB internally. All the MojoDB data is stored in the BDB files.

    Full dump command including deleted objects:

    Code:
    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/dump '{"path": "/media/internal/db-backup.json", "incDel": true}'
    Thank you!


    Quote Originally Posted by jl85 View Post
    Not sure what the BDB lock errors mean. Maybe there's too much stuff to purge and it doesn't think it has enough scratch space on the partition.
    Do you know which partition the scratch space is in?
  5. #25  
    Quote Originally Posted by UI Designer View Post
    Do you know which partition the scratch space is in?
    It's part of the /var/db partition, along with the database. Since you said it's only 32% full maybe it's unrelated.

    In /etc/palm/mojodb.conf, there's a value for maxLocks and maxLockers. Maybe try bumping maxLocks to 20000 (the default value on the Touchpad) and reboot and try running purge again.
    bbito likes this.
  6. #26  
    Quote Originally Posted by jl85 View Post
    It's [the scratch space partition] part of the /var/db partition, along with the database. Since you said it's only 32% full maybe it's unrelated.

    In /etc/palm/mojodb.conf, there's a value for maxLocks and maxLockers. Maybe try bumping maxLocks to 20000 (the default value on the Touchpad) and reboot and try running purge again.

    Thank you, jl85, increasing maxLocks from 6000 to 20000 and rebooting WORKED! That was very smart of you to look at what value is being used on the TouchPad!

    Now the purge command finished and returned the following after about 5 minutes:

    "returnValue": true,
    "count": 142747


    Unfortunately, my Pre3's object.db and indexes.db files in /var/db/main/ did NOT shrink in size when viewed in Internalz Pro, even after another reboot. They are still at:

    - object.db: 105MB
    - indexes.db: 33.8MB
    - (Combined = 138.8MB)


    I was kind of guessing I would not see much change in size since when I ran the dump command earlier today with and without the "incDel": true parameter, I only saw a 1 line difference in the two files generated and the returned count values were 9921 and 9920. After the purge command succeeded, the returned count values are 9904 and 9903, so something did get deleted.



    When I look at /var/log/messages, the following messages caught my eye:

    2014-07-22T00:17:54.101165Z [619] webos-device user.notice mojodb-luna[1189]: [db.mojodb] purging objects deleted more than 14 days ago...

    2014-07-22T00:23:07.049407Z [932] webos-device user.notice mojodb-luna[1189]: [db.mojodb] purged 142747 objects


    So again, it looks like something got deleted (142,747 objects!), but I am not seeing a size decrease in object.db and indexes.db yet.



    When I look at the output of stats, which I ran as follows:
    Code:
    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/stats '{}' > /media/internal/palmdbstats14-07-21.txt
    I can see that most of the size, delSize, count, and delCount values decreased in size, so the purge command did do something. I ran stats before and after purge so that I could do a comparison, and I also compared it to the last time I ran stats, which was in post #16 above.

    Here are some of the stats changes I saw:

    Object:1 indexes count:
    - 4/27: 177,939
    - 7/21 before purge: 209,661
    - 7/21 AFTER purge: 66,910
    - (Purge Difference = 142,751)

    Object:1 indexes size:
    - 4/27: 3.6MB
    - 7/21 before purge: 4.2MB
    - 7/21 AFTER purge: 1.4MB
    - (Purge Difference = 2.8MB)

    Object:1 objects count:
    - 4/27: 8,226
    - 7/21 before purge: 9,921
    - 7/21 AFTER purge: 9,911
    - (Purge Difference = 10)

    Object:1 objects delCount:
    - 4/27: 169,713
    - 7/21 before purge: 199,740
    - 7/21 AFTER purge: 56,999
    - (Purge Difference = 142,741)

    Object:1 objects delSize:
    - 4/27: 58.6MB
    - 7/21 before purge: 68.0MB
    - 7/21 AFTER purge: 18.7MB
    - (Purge Difference = 49.3MB)

    Object:1 objects size:
    - 4/27: 1.8MB
    - 7/21 before purge: 2.2MB
    - 7/21 AFTER purge: 2.2MB
    - (Purge Difference = 0MB)

    com.palm.activity:1 indexes count:
    - 4/27: 160,578
    - 7/21 before purge: 183,283
    - 7/21 AFTER purge: 49,577
    - (Purge Difference = 133,706)

    com.palm.activity:1 indexes size:
    - 4/27: 3.2MB
    - 7/21 before purge: 3.7MB
    - 7/21 AFTER purge: 1.0MB
    - (Purge Difference = 2.7MB)

    com.palm.activity:1 objects count:
    - 4/27: 31
    - 7/21 before purge: 31
    - 7/21 AFTER purge: 31
    - (Purge Difference = 0)

    com.palm.activity:1 objects delCount:
    - 4/27: 160,547
    - 7/21 before purge: 183,252
    - 7/21 AFTER purge: 49,546
    - (Purge Difference = 133,706)

    com.palm.activity:1 objects delSize:
    - 4/27: 57.8MB
    - 7/21 before purge: 66.0MB
    - 7/21 AFTER purge: 17.9MB
    - (Purge Difference = 48.1MB)

    The rest of the differences I see so far are much smaller in size, so I am not listing them here.



    One more thing: while looking through /var/log/messages, I came across the following Database volume 32 full messages that I have not seen before today:

    Code:
    2014-07-22T00:07:55.565002Z [55] webos-device user.warning mojodb-luna[1189]: [db.serviceHandler] Database volume 32 full, space ok, no warning needed. 
    
    ...
    
    2014-07-22T00:09:15.595855Z [105] webos-device user.warning mojodb-luna[1189]: [db.serviceHandler] Database volume 32 full, space ok, no warning needed. 
    
    ...
    
    2014-07-22T00:09:23.408447Z [113] webos-device user.warning mojodb-luna[1189]: [db.serviceHandler] Database volume 32 full, space ok, no warning needed. 
    
    ...
    
    
    2014-07-22T00:25:26.149261Z [34] webos-device user.warning mojodb-luna[1181]: [db.serviceHandler] Database volume 32 full, space ok, no warning needed. 
    
    ...
    
    2014-07-22T00:26:45.059204Z [85] webos-device user.warning mojodb-luna[1181]: [db.serviceHandler] Database volume 32 full, space ok, no warning needed. 
    
    ...
    
    2014-07-22T00:26:50.500732Z [91] webos-device user.warning mojodb-luna[1181]: [db.serviceHandler] Database volume 32 full, space ok, no warning needed.
    I searched the forums for this message and could not find anything. Does anyone know what this "Database volume 32 full" message means? Which volume is volume 32?



    Any other ideas of what I should try in order to reduce the file size of object.db and indexes.db or at least get them back to the size they were a month ago or so without having to take a trip to the doctor or reset my Pre3? Is there a way to purge more recent objects than those deleted "14 days ago"? Is there a command to rebuild the database?
    Last edited by UI Designer; 07/23/2014 at 12:11 AM. Reason: Added more details, fix typos
  7. #27  
    From https://github.com/openwebos/db8/blo...SpaceAlert.cpp

    it looks like "Database volume 32 full, space ok, no warning needed." means that the partition is 32% full, which is OK. It also looks like mojodb only does a "compact" operation (to reduce the size of the .db files) if the disk space is low.

    This command should do a compact (haven't tested):

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/compact '{}'
    You should be able to purge more recent objects using (haven't tested):

    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/purge '{"window": 7}'
    which should purge deleted objects older than 7 days. I would advise against using a low purge window (less than 3), however -- if there are any "pending deletion" database records corresponding to emails, calendar events, etc, that haven't been sync'd to a server (e.g. no network), it could cause the sync engines to get confused.
  8. #28  
    Good news! My Pre3's object.db and indexes.db in /var/db/main/ finally DECREASED in size today, about half of a day AFTER issuing that purge command in post #26! Their current size values are:

    - object.db: 29.9MB
    - indexes.db: 14.8MB
    - (Combined = 44.7MB)


    Quote Originally Posted by jl85 View Post
    From https://github.com/openwebos/db8/blo...SpaceAlert.cpp

    It looks like "Database volume 32 full, space ok, no warning needed." means that the partition is 32% full, which is OK. It also looks like mojodb only does a "compact" operation (to reduce the size of the .db files) if the disk space is low.
    Thank you so much for clarifying that, jl85! That makes sense -- too bad that message was not more clear on its own, like including the word "percent"... I thought it was saying "volume 32" was full, but then was confused when it also said "space ok"...


    Quote Originally Posted by jl85 View Post
    This command should do a compact (haven't tested):

    Code:
    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/compact '{}'

    You should be able to purge more recent objects using (haven't tested):

    Code:
    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/purge '{"window": 7}'
    which should purge deleted objects older than 7 days. I would advise against using a low purge window (less than 3), however -- if there are any "pending deletion" database records corresponding to emails, calendar events, etc, that haven't been sync'd to a server (e.g. no network), it could cause the sync engines to get confused.

    Thank you, thank you again, jl85! I was wondering if there was a command like compact. I ended up NOT having time to try out the compact command yet, but when I checked the file size of my Pre3's object.db and indexes.db earlier today, they DROPPED in size!! So maybe the compact command ran on its own after the purge and took a long time to complete in the background or something else... Let me check /var/log/messages for any hints -- I found the following relevant log items:

    Code:
    2014-07-22T20:30:30.365814Z [31656] webos-device user.notice mojodb-luna[1181]: [db.mojodb] purging objects deleted more than 14 days ago...
    
    2014-07-22T20:30:30.423034Z [31656] webos-device user.notice mojodb-luna[1181]: [db.mojodb] purged 0 objects
    
    2014-07-22T20:30:30.423217Z [31656] webos-device user.notice mojodb-luna[1181]: [db.mojodb] compacting...
    
    2014-07-22T20:38:34.198699Z [32140] webos-device user.warning mojodb-luna[1181]: [db.bdb] During compact: 27466 db pages examined (max burst 8127), 6940 db pages freed (max burst 2258), 23986 db pages truncated (max burst 19131), -14725120 log bytes created by compacts (max burst 19206144), 98566144 bytes reclaimed by checkpoints (max burst 79384576), 9607470 bytes of keys stepped over (max burst 1243139), 21970012 bytes of values stepped over (max burst 9436462), 13828ms spent in stepping (max burst 2721ms), 451542ms spent in compact (max burst 2721ms) 
    
    2014-07-22T20:38:34.199218Z [32140] webos-device user.warning mojodb-luna[1181]: [db.bdb] Compact complete: took 483776ms, freed 98566144 bytes (including pre-checkpoint of 2101248 bytes). Volume /var/db has 354684928 bytes free out of 517939200 bytes (31.5 full) 
    
    2014-07-22T20:38:34.199584Z [32140] webos-device user.notice mojodb-luna[1181]: [db.mojodb] compaction complete

    So it looks like purge got called automatically on its own today around 1:30pm PST followed by compact. I wonder why it was not doing this automatically earlier? I searched through some of my older log files and I saw NO purge or compact commands in them.



    I am happy now that the file sizes of object.db and indexes.db finally decreased since now I do not have to worry as much about those dreaded "application database is full" and "db: quota exceeded" warning messages that I initially ran into in post #10.


    So, in summary, these are the steps I took to reduce my Pre3's object.db and indexes.db file sizes, which were the 2 biggest files taking up most of the space in /var/db and caused those above dreaded warning messages back in April on my Pre3:

    1. Install the latest Google sync https fix patch (gsync-n.n.patch) in order to stop object.db and indexes.db from increasing in size in the first place

    2. Increase maxLocks in /etc/palm/mojodb.conf from 6000 to 20000 (the default value on the Touchpad) using Internalz Pro to help the purge command complete without any errors, like "Cannot allocate memory" and "bdb: Lock table is out of available lock entries" (if you are not getting any error when using purge, then you can skip this step)

    3. Reboot

    4. Manually purge the database using the following command in Web OS Quick Install's Tools > Linux Commanline or your favorite Linux terminal app:
    Code:
    luna-send -a com.palm.configurator -f -n 1 luna://com.palm.db/purge '{}'
    5. WAIT a half a day or so for webOS to automatically purge the database again and compact it, which it should have been doing all along, but for some unknown reason, stopped doing (or give the above compact command a try and let us know how it works)


    This is in addition to adjusting the partition sizes and increasing the quota value described in post #10.



    Since I had to manually do the purge command myself, this makes me wonder if something is not setup or configured correctly on a Pre3, like was discussed in post #18 and in "The application database is almost full" post #259:

    - No "purgeWindow" command in /etc/palm/mojodb.conf

    - I noticed that the 24h value in /etc/palm/activities/com.palm.db/com.palm.db.purge.json did not seem to cause the purge command to run automatically, or it somehow stopped working for a while on my Pre3

    - Possible infrastructure issues (see post #259)


    For now, I'll be keeping an eye on the size of the files in /var/db/main/, especially objects.db and indexes.db, and am glad I have a way to reduce their size again when needed.

    Thank you again so much, jl85, for all of your help in keeping my Pre3 alive and working well! I was almost starting to think there for a little bit that it was almost the end of life for my trusty and wonderful Pre3, but not any more...


    Now if someone could get the Preware feeds working again so that it shows the latest patch updates and any other new items, that would be wonderful! I did not know this was broken until I read the following posts:

    Having a webOS SSL certificate updater app would be very nice too!




    Update 8/9/2014:


    My Pre3's object.db and indexes.db files in /var/db/main/ continue to drop in size on their own, reaching their lowest values I have ever seen:

    - object.db: 3.51MB
    - indexes.db: 6.80MB
    - (Combined = 10.31MB)

    Looks like the Purge command is working on its own.
    Last edited by UI Designer; 08/17/2014 at 04:19 AM. Reason: Add gsync patch step, add log items, add Preware Patch Portal Quotes, fix typos, fix links, add Update
    Grabber5.0 and bbito like this.
  9. #29  
    This is great info! It's nice having anther webOS insider on the forums.

    -- Sent from my Palm Pixi using Forums
  10. #30  
    I wish I would have seen this thread before I got the application size error and reset my device
Page 2 of 2 FirstFirst 12

Similar Threads

  1. Replies: 2
    Last Post: 02/04/2013, 09:26 PM
  2. Tweaks:Unknown Service Error & Preware database error
    By jblather in forum webOS Patches
    Replies: 23
    Last Post: 09/04/2012, 03:54 AM
  3. PreCentral Database Error?
    By Garrett92C in forum webOSNation.com - Site News, Feedback & Help
    Replies: 24
    Last Post: 04/09/2010, 09:24 PM
  4. Error:Database: can't find (0x0207)
    By Varangian in forum Palm OS Devices & Apps
    Replies: 4
    Last Post: 10/07/2004, 12:35 AM

Tags for this Thread

Posting Permissions